Collision avoidance systems PRORETA: situation analysis and intervention control

6th IFAC Symposium Advances in Automotive Control Munich, Germany, July 12-14, 2010

Collision avoidance systems PRORETA: situation analysis and intervention control Rolf Isermann ∗ Roman Mannale ∗ Ken Schmitt ∗ ∗

Institute of Automatic Control, Technische Universit¨ at Darmstadt, Landgraf-Georg-Str. 4, 64283 Darmstadt, Germany {risermann,rmannale,kschmitt}@iat.tu-darmstadt.de

Abstract: After a discussion of passive and active safety systems for automobiles and accident statistics it follows that a further progress in the reduction of accidents can be especially expected by next generation driver assistance systems with a sequence of warnings and active interventions. PRORETA is an Industry-University research project with the goal to develop steps towards accident free driving. The first project considers two vehicles moving in the same direction. The other vehicle is detected by a fusion of LIDAR and camera data providing the system with relative speeds, distance and locations. If the driver does not react to an obstacle on the own lane, the system automatically triggers an emergency braking and/or swerving to avoid a collision. This includes e.g. a fast and precise evasive trajectory control by automatic steering. The second project is dedicated to vehicles in opposite direction performing an overtaking maneuver on rural roads. The own vehicle detects the velocities and distances to the preceding and oncoming vehicle by RADAR and lane markings etc. with a camera. The measured data of the two sensors undergo a sensor fusion with Kalman filters. The overtaking maneuver is predicted by using the measured data of all three vehicles. If an accident free overtaking is in danger, warnings are given to the driver and if the driver does not react a full braking of the own vehicle is fired such that the driver can turn back behind the overtaken vehicle. The contribution describes the developed strategies and some basic calculated features and control systems. Measured data is shown and some videos give an impression of the driving experiments on the runway of an air field. Keywords: Collision avoidance, vehicles, road traffic, LIDAR, RADAR, video camera, automatic braking, automatic swerving, trajectory control, overtaking maneuver, driving experiments 1. INTRODUCTION Automobile safety is one of the high priority issues in the design of vehicles, construction and equipment to minimize the occurrence and consequence of automobile accidents. The improvements in automobile and roadway design have steadily reduced injury and death rates in developed countries. This positive development was reached by measures of passive and active safety systems. Passive safety systems provide vehicle components in order to protect occupants during a crash respectively to reduce the consequence of accidents. Some examples are: • Interior safety systems for passenger protection · Constructive crashworthy systems like crumple zones (front, rear, side), cell strength, sur⋆ The contribution results from the research cooperation PRORETA between Technische Universit¨ at Darmstadt and Continental AG. The research project is being carried out in cooperation of the Institutes of Automatic Control, Automotive Engineering Ergonomics (PRORETA1) and Multimodal Interactive Systems (PRORETA2). The participating Institutes thank the Continental AG for the generous support and good cooperation.

978-3-902661-72-2/10/$20.00 © 2010 IFAC

461

vival space, collapsible steering columns, cockpit padding. · Passenger restraint systems: seat belt with retractor and tightener, air bags (front, side, window) • Exterior safety systems for protection of external humans: · vehicle-related measures to minimize injuries to pedestrians, bicycle- and motorcycle-riders, e.g. hood measures and deformation behavior · exterior body shape and structure • Active passive safety systems · pre-crash-sensors for early actuating of occupant safety systems · pre-safe-systems for preparing passive safety systems for a possible accident (e.g. belt tightening) · pre-safe-structure: active chassis systems for passenger protection Active safety systems have the goal to prevent accidents by assisting the driver. Some examples are • Safe driving behavior of the vehicle by appropriate design of the chassis, suspensions, steering and braking systems

10.3182/20100712-3-DE-2013.00202

AAC 2010 Munich, Germany, July 12-14, 2010

• Human conditional safety: acceptable low physiological stress for drivers and occupants • Driver assistance systems: · ABS (anti-lock braking system) · TRC (traction control systems) · ESC (electronic stability control) · ACC (adaptive cruise control) · BAS (brake assist system) · AFS (active front steering) · LDW (lane departure warning) · LKS (lane keeping support) The passive safety systems have reached a well-developed status such that mainly the active passive systems will show further progress. Therefore larger steps in the reduction of accidents are expected by active safety systems. Accident statistics exist from several sources. The official German statistics DESTATIS distinguishes between the type of fatalities and kind of fatalities. The type of fatalities describe the traffic situation and therefore the origin of the accidents and the kind of fatalities describe the movement of the accident participants relative to each other. The major types of accidents are for Germany according to Statistisches Bundesamt (2007): • • • •

Warning&fullactiveactions Warning&carefulactiveactions Lateral&longitudinalcontrol,stabilisation

Comfort & Safetyfunctions

ESP

ABS ASR

Sensors Wheel speeds

Parking Assist (PA,A ) PA

Acceleration angular speed

Ultrasound

ACC

Radar Lidar

Collision Collision LDW - Avoidance ACC/FSR Mitigation (CAV) LKS (CMI) STA SensorMultinetwork Videocar-to-car Sensor camera commun. fusion

Mecha1979/1986 1995 1999 2005 20xx tronic actuators: ABS(79), ASR(86),ESP(95),EPS(96), EHB(01), AFS(03)

x-by-wire

Fig. 1. Roadmap for the development of driver assistance systems: 3 generations Accidentsituations: Vehiclesmove oppositedirection

Vehiclesmove samedirection Other vehicles • move ahead • stop

Other vehicles • movelaterally • cuttingin • crossing

CAV actions: • warning • braking

• swervingto

freelane

driving accident, 42 % accident between vehicles moving along, 21 % turning into a road or crossing it, 12 % crossing the road (pedestrian), 10 %

Overtaking • ruralroad • 2lanes oncoming vehicle

Overtaking Straightdriving • ownvehicle • autobahn • =2lanes turnstoleft oncoming • othervehicle vehicle, turnstoleft wrongway

• warning

• warning

• braking

• braking

AND swerving back

AND swervingto theright

• warning • braking AND

swerving totheright tofreelane

Fig. 2. Organisation chart for collision avoidance systems (longitudinal direction)

The kind of accidents with fatalities are distinguished as: • • • • •

DrivingAssistance

road departure, 34 % with oncoming vehicles, 21 % with vehicle turning or crossing, 15 % with pedestrians, 13 % with vehicles moving ahead or waiting stationary, or laterally moving, 11 %

A reduction of driving accidents and accidents with road departure can e.g. be supported by ABS, TRC, ESC, LDW and LKS. However, accidents between vehicles moving along, turning and crossing or with obstacles need newly developed driving assistance systems, so called anticollision systems or collision avoidance systems. Figure 1 shows a roadmap for driver assistance systems. Basic components are the required mechatronic actuators, ranging from ABS- to AFS-systems and sensors, from wheel speed through acceleration to RADAR and video camera. The development within the last 30 years shows three generations: 1st generation lateral and longitudinal stabilization and control 2nd generation warning and careful active actions 3rd generation warning and full active actions The last generation is still in the development phase, see also Zittlau and Hoppe [2008]. The scheme in Figure 2 distinguishes some accident situations between vehicles:

(b) move laterally, turn or cross (includes objects on road) • Collision avoidance (CAV) actions: warning, braking, swerving (2) Vehicles move in opposite directions • own vehicle: (a) overtaking maneuver - rural road - autobahn, freeway (b) straight driving - vehicles leave correct lane • CAV actions: warning, braking AND swerving The research project PRORETA is an Industry-University project on collision avoidance system between Continental AG and Technische Universit¨at Darmstadt. The project PRORETA 1 (2003-2006) was dedicated to cases 1 (a) and (b), and project PRORETA 2 (2006-2009) to case 2 (a) for rural roads. Both projects were performed by the cooperation of three research institutes: Automatic Control, Automotive Engineering and Ergonomics (1st project) and Multimodal Interactive Systems (2nd project). An important basis for these collision avoidance systems is the detection of the environment around the vehicle. Table 1 summarizes the present state of sensors like different types of RADAR, LIDAR and cameras. A comprehensive survey of the state of the art is given in the handbook on driver assistance systems, Winner et al. [2009]. In the following the developed collision avoidance systems, the assumed accident situations, the intervention planning and control functions will be summarized and experimental results with driving experiments will be shown.

(1) Vehicles move in the same direction • other vehicles: (a) move ahead or stop 462

AAC 2010 Munich, Germany, July 12-14, 2010

Table 1. Current state of environmental sensors Imagesensors

LIDAR

RADAR

Driver Driver

PRORETA prototype

Control Control Technology

Range Resolution

el.-magn.waves(24or 77GHz),measurement ofdistanceandspeed (Dopplereffect),pulseorfrequency-modulation,costreductionfor 77-GHz-RADAR: Silicon-Germanium technology 200m(77GHz) 50m(24GHz) goodmeasurement ofdistanceandspeed

Reliability dependson ofobject reflection, detection surface characteristics Environ- low mental impact

laserpulses(8001000nm),distance measurementbased ontimebetweensent andreceivedpulse (timeofflight)

200m verygoodfordistance,measurement, goodspatialresolution problemswith dirtyobjects high(atmospherical particles, e.g.dust,fog, rain,snow)

CMOScameras(mono orstereo)forinfrared orvisiblelight,nodirect measurement,examples forimageprocessing: objectdetection,lane detection

Intervention Interventionplanning planning

Ergonomic Ergonomic studies studies Experiments with test Experiments with test persons persons

Trajectory Trajectoryprediction prediction Analysis Analysis Fusionofenvironmental sensors&stateestim. fortheenvironment

50m moderatefordistance andspeed,verygoodfor objectdetection dependingon imageprocessing methods

Vehicle Vehicle state state estimation estimation

Environmental Environmental Sensors Sensors for for sensors vehicle states states sensors vehicle

Actuators Actuators

Vehicle Vehicle

Sensor type

Reactions of design notes, acceptance

Reactions , ofdrivers, design notes, acceptance

Environment Environment

lowforvisiblelight, mediumforinfrared (redusedthermal contrastduetoparticles, e.g.snow,rain,fog)

Fig. 3. Collision avoidance system overview for the development of PRORETA 1

2. COLLISION AVOIDANCE SYSTEM FOR VEHICLE MOVING IN THE SAME DIRECTION (PRORETA 1) An accident situation is considered where the own vehicle drives correctly on its lane and another vehicle or an object moving in the same direction appears in front. The driver has the chance to react appropriately to avoid an accident as long as possible. However, if the driver does not react the CAV system intervenes in the last possible moment. Figure 3 shows a system overview for the development of PRORETA 1, Isermann et al. [2008]. Based on information from the environment and the own car, predictions for the expected trajectory of the own car and of objects in the surroundings are being calculated. Using these predictions, a decision is made, whether an intervention is necessary or not and the intervention is planned. The intervention itself is then conducted fully automatically. An ergonomic study accompanied the development of the system. This study investigated how the driver reacts in critical situations and how he reacts on the interventions. In the following, the intervention decision, the planning of the intervention and the conduction of the intervention are described. The environment perception is described in detail in Darms and Winner [2006], Darms [2007]. Results from the ergonomic study can be found in Bender et al. [2007], Bender [2008]. The system was tested by simulations using a complex two track model followed by extensive driving tests with an experimental vehicle.

Fig. 4. Environmental Sensors of the test vehicle cycle. The distance to objects is determined by a time of flight measurement of emitted light impulses. The video sensor is based on a monochrome CMOS image sensor that provides data in a 40 ms cycle. The detection area is 44◦ , whereas the discretisation with approx. 0.07◦ is considerably finer than for the laser scanner. By means of image processing algorithms, ahead driving vehicle rear views and lane markings can be detected in the image. However, a direct distance measurement is not possible, for details see Darms and Winner [2006], Darms [2007].

2.1 The research vehicle

2.2 Evasive trajectory

A VW Golf IV, which was only equipped with additional sensors and actuators required for the developed functions, served as experimental vehicle, see Figure 4.

An evasive trajectory is required between intervention planning and control. For investigating several different intervention functions with different types of controllers, the type of intervention is selected using some flags. The flags used in this article are braking, emergency braking and evasion. If braking is chosen, the desired deceleration has to be transmitted. If an emergency braking is chosen, the maximum possible deceleration at every point in time is achieved using braking controllers. For an evasion, the desired position and heading are given for one time step TB , two time steps TB and ten time steps TB ahead in time, Figure 5. The coordinate system used is stationary for the duration of the evasion and is initialized at the

The driver assistance system uses an active front steering (AFS) and an electro hydraulic braking system (EHB) as actuators. For vehicle state estimation only ESC sensors and the sensors of the active front steering and braking system are necessary. For environment perception a laser scanner and a video sensor were used. The chosen design allows to scan the area in front of the vehicle. The detection area of the laser scanner covers an angular range of 22.5◦ with a resolution of 1.5◦ and is scanned in a 90 ms 463

AAC 2010 Munich, Germany, July 12-14, 2010

y p (t+10TB) p (t+2TB )

yM /2

p (t+TB )

yS

yM

yH 1 2

x c

beginning of the evasion to match the vehicle coordinate system at that point. The last position, which is supposed to be reached 10 time steps in the future, is used to allow the controller to react predictively after deviations of the first 2 time steps. Every point p(t) consists of the position (x, y) and the heading of the vehicle. All three points are put together in one matrix transmitted to the controller: Pevasion = [p (t + TB ) p (t + 2TB ) p (t + 10TB )] (1) Primary goal of the evasive trajectory is to reach a predefined lateral offset with the shortest possible traveled path. The vehicle dynamics and stability after the maneuver are taken into account. Vehicle dynamics of the trajectory are used to limit the maximum allowed lateral acceleration. This limit can be adapted to the actual traffic and driving situation, and especially whether conditions. The steering actuator also limits the maximum possible jerk. The designed evasive trajectory is described by a simple parametric model y = f (x) (2) The yaw angle ψ can be expressed as (assuming no side slip)   dy (3) ψ = arctan dx and its derivative with regard to time dψ 1 d2 y ψ˙ = =  2 2 vx dt dx dy 1 + dx

(4)

Based on this and using the Ackermann relations for the kinematics, the lateral acceleration is: d2 y 1 (5) ay = v ψ˙ =  2 2 vx v dx dy 1 + dx

Further simplification can be accomplished assuming vx = v.

The reference evasive trajectory is described by a sigmoide of the form yM (6) y(x) = 1 + e−a(x−c) yM is the maneuver width, describing the distance between minimum and maximum y-value. a defines the slope of the sigmoide, and c defines the position of the inflection point and therefore the length of the evasive maneuver, which is s = 2c, see Figure 4. The parameters of the sigmoide can be chosen according to the driving situation, such that the evasive path is 464

bveh

yA

bveh

ssteer

Fig. 5. Evasive trajectory for planning and control

yM 1 2

1 2

l veh

Fig. 6. Evasive Quantities for calculating the evasive trajectory (see text for details) minimal regarding the limitations for maximum lateral acceleration, maximal jerk and dynamics of the steering actuator, see St¨ahlin et al. [2006], St¨ahlin [2008]. 2.3 Intervention decision Based on the fused environment data it is decided if a collision is likely to occur and if so, which maneuver has to be carried out to avoid the collision. The strategy is to avoid the collision at the physically last possible moment by an intervention in order to give the driver the possibility to master the critical situation by his own actions as long as possible. In order to determine a threatening collision, predictions are first made for the own vehicle driving tube and the movement of the objects in the environment. By means of these predictions it can then be predicted whether a collision will occur. If this is the case, it is planned in a next step when and which intervention has to be carried out. Basically, there are three strategies to avoid a collision: Braking, steering or a combination of braking and steering. For the intervention decision it is calculated at what distance to the collision location the respective intervention has to be carried out, such that the collision can still be prevented. For a braking intervention the braking distance is calculated. In case of steering interventions the sigmoide is taken as the basis for the evasive trajectory. In Figure 6 the quantities necessary for the calculation of the evasive trajectory are presented. By means of the vehicle’s width bv and the obstacle’s width the necessary evasive width yA is determined together with a safety distance yS . Since the evasive width can be reached before the end of the maneuver, a maneuver width yM arises. However, the evasive trajectory is the trajectory until the evasive width yA is reached. The maneuver width is chosen according to the strategy used. If the maneuver width yM is chosen to be the same as the evasive width yA , the evasive trajectory length ssteer reaches its maximum for given maximal lateral acceleration and maximal lateral jerk. However, a smallest possible evasive length ssteer can be reached for the same acceleration and jerk limits for a larger maneuver width yM in front of the object. A further result is that for larger vehicle velocities steering has to be preferred compared to braking to avoid an accident, see St¨ahlin [2008]. 2.4 Lateral vehicle guidance If a collision with an obstacle is no longer avoidable by a reaction of the driver, then, according to the situation,

AAC 2010 Munich, Germany, July 12-14, 2010

k Pevasion

Calculation of command variables

Feedforwardcontrol

dFF dM

xR yR

[ ] -

Trans- e=Dy Feedback dFB formation control

x Sensors yEE and vehicle state estimation

[]

Actuating elements Vehicle

Fig. 7. Structure of linear feedback control combined with feedforward action the driver assistance system selects one of the intervention strategies described above. For the realization of the chosen intervention either the active steering and/or the electro hydraulic braking system are used according to the maneuver. If a braking maneuver should be carried out, the vehicle is decelerated, Schorn et al. [2005], , by utilization of the maximum force transmission available. The anti-lock braking system ABS supports in this case. In case a collision can only be prevented by an evasive maneuver or by a combined evasive and braking maneuver the control system receives from intervention planning a trajectory, see Figure 3. The vehicle is driven on this trajectory automatically around the obstacle. Different linear and nonlinear feedback controllers for an evasive maneuver were developed, see e.g. Schorn and Isermann [2006], Schorn et al. [2006], Schorn [2007]. Each lateral guidance feedback control transfers an additional steering angle to the interface of the steering system. Vehicle variables, which cannot be measured directly by sensors the vehicle is equipped with, are estimated, see also Schorn and Isermann [2006], Isermann [2006]. For combined steering and braking maneuvers different feedback controllers were developed as well. In the following only the lateral vehicle guidance is regarded. Exemplarily, one of the investigated approaches, a speed-dependent local linear feedback control approach with feedforward control is briefly described. To guide a vehicle on a desired trajectory, a speeddependent local linear feedback control approach with feedforward control was developed, Schorn and Isermann [2006]. A scheme of the control system is shown in Figure 7. Based on the self steer gradient SG a steer angle δF F is calculated for the feedforward control by means of vehicle velocity v, wheelbase l and curvature κ = R1 of the desired trajectory:
(7) δF F = l + SGv 2 κ

A feedback control is added to compensate disturbances and deviations. The parameters of a proportionalderivative (PD) controller are tuned by two parameters only and provide the required dynamics by means of the differential component. Using the vehicle orientation ψ, the control deviation is transformed from an earth-fixed coordinate system into a vehicle-fixed coordinate system as control deviation e = ∆y. The feedback of the vehicle’s longitudinal position xE is necessary for this purpose. The steering system is driven by the sum δM of the angles δF F and δF B of the feedforward and feedback control. As the velocity v influences the vehicle’s dynamics, the feedback controllers were designed for different operating points (velocities). Their outputs are weighted and super465

Fig. 8. Scenarios for practice system testing imposed based on Local Linear Models (LLM), Schorn and Isermann [2006], Nelles [2001]. 2.5 Experimental results from test drives The developed components environment recognition, intervention decision and feedback control were implemented as a whole system in a research vehicle and tested by means of numerous experiments. This happened using an obstacle that represents the rear view of a car and can be moved laterally on the lane. Two test scenarios can be seen in Figure 8. In the following sections some results from these tests are presented. It is required in each case that the lateral and back lane areas are monitored by additional sensors and thus permit driving maneuvers. a) Blocked Lane In the scenario “Suddenly appearing obstacle / blocked lane” from Figure 8a) a lane is blocked unexpectedly. An example for this would be an end of a traffic jam in the case of bad visibility or after a curve. The emergency evasion is then conducted as an automatic intervention. The position of the used obstacle is determined by the environmental sensors and the necessary evasive trajectory is calculated based on the information about the vehicle’s surroundings. The vehicle is then guided aside of the obstacle on the predefined evasive trajectory by the lateral guidance controller without the assistance of the driver. Figure 9 shows results of a test drive with the linear feedback control combined with feedforward control. A comparison of desired command variable and measured position shows that both values match very well. The evasive width yM is 3 m, the desired and the actual position correspond well, apart from a slight overshooting. The steering wheel angle indicates that the driver held the steering wheel in a straight position. The difference between total angle and steering wheel angle is provided only by the controller. The difference at the end of the intervention maneuver follows from the fact that the feedback control has been switched off at very low velocities. The experiments show that the maximal lateral accelerations are |ay | ≈ 7m/s2 . b) Cutting-in Vehicle As a second scenario a suddenly cutting-in vehicle is reproduced by moving the dummy obstacle just in front of the vehicle from the right to the left lane (Figure 8b). Evasion is not possible since further

150

3

100

2 1 0

testdrive desired

-1 0 velocity [m/s]

steeringwheelangle[°]

4

50 x-position[m]

15 10 5 0

6

8

10 12 14 time[s]

16

18

total driver

RadarSensor

VideoSensor

50 0 -50

Image processing

-100 -150

100 acceleration[m/s²]

y-position[m]

AAC 2010 Munich, Germany, July 12-14, 2010

10 5 0 -5 -10

6

8

10 12 14 time[s]

16

18

Sensorfusion

longitudinal lateral

6

8

10 12 14 time[s]

16

Situation analysis

18

Warnings/actions

Fig. 9. Experimental driving results for velocity-dependent linear feedback control combined with feedforward obstacles block the right lane. The necessary intervention is thus an emergency braking maneuver. By means of the environmental sensors it is recognized that both lanes of the road are blocked and it is calculated at which last possible moment the emergency braking maneuver must be started in order to come to a stop just before the obstacle. Assuming a maximum braking acceleration which is dependent on the road state (dry-wet), the required braking distance of the vehicle is calculated depending on the current speed. The driver assistance system triggers a braking intervention only if this minimal braking distance is reached in order to give the driver the chance to prevent the collision as long as possible by himself. The electro hydraulic braking system then decelerates the vehicle maximally with support by the anti-lock braking system ABS, on dry roads with a deceleration of as ≈ 7m/s2 . 3. COLLISION AVOIDANCE SYSTEM FOR OVERTAKING MANEUVERS AND ONCOMING TRAFFIC (PRORETA 2) Accident situations during overtaking maneuvers on rural roads are frequently the cause of severe injuries and fatalities. Accidents may originate in an erroneous situation assessment by the driver, e.g. misjudgement of distance and velocity of oncoming vehicles. Therefore, the 2nd PRORETA project developed a driver assistance system to avoid overtaking accidents. Figure 10 depicts a system overview for the development of PRORETA 2. A mono camera system and a farrange RADAR sensor, both series components, scan the environment in front of the own vehicle. The video data is processed via pixel-based segmentation and a combination of filterbank outputs and object detection. They allow a classification of the picture contents, like objects, road, land markings, free space, heaven etc. Detected vehicles from the camera and the RADAR sensor are fed into a sensor-fusion system applying an extended Kalmanfilter with yaw rate compensation, such providing an environment model, see Wojek et al. [2008], Hohm et al. [2008], Isermann et al. [2009]. The situation analysis has the task to detect dangerous overtaking maneuvers. Then, warnings and active braking commands are given to avoid an accident. 466

Fig. 10. Overview for the development of PRORETA 2 Overtaking Start

Passing

A A

Cutting-In A

B

A

Fig. 11. Phases of overtaking maneuvers In the following the situation analysis during overtaking maneuvers on two lane rural roads, the prediction of overtaking maneuvers, warnings and emergency braking is briefly described. The camera-based data processing algorithms are published in Wojek et al. [2008] and the multi-level sensor fusion of camera and RADAR data in Hohm et al. [2008]. An overtaking maneuver starts, when the own vehicle (A) enters the opposite lane with the intention to pass a leading vehicle (B), see Figure 11. When having catched up with vehicle B, the actual passing takes place. The subsequent cutting-in phase begins with entering the own lane again. The overtaking maneuver is completed when the vehicle has fully left the opposite lane. The PRORETA-2 system focuses on situations in which the forward road and potential oncoming traffic is viewable by the sensors. Thus, primarily overtaking maneuvers on straight roads or only slightly bended road geometries are addressed. The developed driver assistance system aims at warning the driver in overtaking maneuvers, which can not be performed without a conflict with an oncoming vehicle (C). The objective is to prompt the driver to abort the maneuver already during overtaking start or in an early stage of the passing phase. If there is too little time to react, or the driver doesn’t react for other reasons, the system initiates an automatic braking intervention, which allows the driver to drive back behind vehicle B before vehicle C is reached, see Figure 12. 3.1 The research vehicle The development of the driver assistance system for overtaking maneuvers was based on many experimental studies with the implemented camera and RADAR system on

AAC 2010 Munich, Germany, July 12-14, 2010

A

C B

A

C

B

Fig. 12. Objective of the PRORETA-System: Abortion of hazardous overtaking maneuvers normal rural roads and traffic situations and overtaking experiments with 3 vehicles on the runway of the University owned airfield. The used vehicle is a BMW 540i, see Figure 13, which is equipped with the market available camera system CSF200 and long range RADAR ARS300, both from Continental. The RADAR sensor operates at 77 GHz and provides distance, relative velocity and azimuth angle of target objects in a 66 ms cycle. The short range scan of the series sensor has a detection range of ±28◦ up to a distance of 60 m and the long range scan provides object detections in a range of ±8.5◦ up to 200 m. To meet the distance requirements imposed by overtaking situations, the detection range of the RADAR sensor used in the PRORETA project has been extended to 400 m by firmware adaption, Winner et al. [2009], Hohm et al. [2008]. Long Range Radar (77 GHz) ARS300

Electronic Brake System MK60E5

R=

YR

Camera Sensor CSF200

Accelerator Force Feedback Pedal

Fig. 13. Research vehicle and components The CSF200 camera is based on a CMOS color sensor with 752×480 pixels and provides image data every 60 ms. The horizontal field of view is ±18◦ and image processing algorithms can reliably detect vehicles up to a distance of 70 m. The camera image is also used to detect lane markings and determine the ego vehicles position within the lane. The RADAR sensor has strong longitudinal capabilities, while the camera sensor neither provides direct distance measurement nor velocity measurement, but has a good lateral resolution. Therefore, these sensors can be considered for complementary sensor data fusion. In the PRORETA-2 project, the sensor fusion is accomplished in an object tracking algorithm, based on an Extended Kalman Filter (EKF), see Hohm et al. [2008]. An accelerator force feedback pedal (AFFP) is installed to give haptic warnings to the driver, and electronically brake system allows automatic braking intervention. 3.2 Overtaking maneuver detection To assist the driver in dangerous overtaking situations, it is necessary to detect, that an overtaking maneuver is being conducted. Since driving maneuvers are primarily 467

1 c0

yR,M L bLane,L yR, M R

yR

θ

vX

vY

XR

ψɺ bLane,R xR

Camera based Lane Detection [bLane yLane θ c0 ] Vehicle Dynamics Sensors [ψɺ aY δ H ωFL ωFR ω RL ωRR ]

EKF xɺ = f (x, u) y = h(x, u)

Estimation Vehicle/Road vY ψɺ xR yR θ yR,M

[vX

R

yR,M L

c0 ]

Fig. 14. Vehicle/Road Model for State Estimation in Extended Kalman Filter defined by the vehicle movements along and lateral to the road, in a first step, the vehicle’s position and orientation are determined. Based on signals from vehicle dynamics sensors and a camera based lane detection system, the position, velocity and orientation with respect to the road are estimated in an odometry module. The state estimation is accomplished by coupling a vehicle model (see Isermann [2006]) and a road model in an Extended Kalman Filter (EKF), see Figure 14. While the vehicle dynamics sensors continuously provide measurements, the lane detection system switches between right and left lane during overtaking maneuvers and is temporarily unavailable due to dynamic movements of the vehicle. The observation model of the Kalman Filter is dynamically adapted, depending on whether the left lane or the right lane is detected by the lane detection system. This procedure allows a lane spanning ego localization and temporary breakdowns of the lane detection can be bridged. The output of the EKF in the odometry module is an estimation of the state vector
T (8) x = vX vY ψ˙ xR yR θ yR,M yR,M c0 R

L

Based on the state estimates from the odometry module and environment sensor data for the leading vehicle (B), longitudinal and lateral indicator variables are calculated. These indicator variables are used to detect the overtaking maneuver and comprise sub-maneuvers. Maneuvers like ‘Following”, “Overtaking Start”, “Passing”, “Cutting-In” or “Overtaking Abort” are modeled in a state diagram and the transitions between the maneuvers are modeled depending on the indicator variables. For details see Schmitt et al. [2009a] and Schmitt et al. [2009b]. In order to warn the driver in an early stage of critical overtaking maneuvers, an early detection of the overtaking start has been realized. To capture the initial lane change during overtaking start, the Time-to-Line-Crossing (TLC) is computed. The TLC is used in lane departure warning systems and predicts the time duration until the vehicle crosses the center line of the road, see Mammar et al. [2006]. By comparing the TLC with a threshold value, lane changes can be detected before the vehicle crosses the center line.

AAC 2010 Munich, Germany, July 12-14, 2010

xR, rel, C

t+TLC

vC

t

A

A

ETTC = f (xR,rel,B,vrel,B,arel,B)

B

B

B

vA

A

C

C

xR, rel, C TTCAC, pred = vA + vC

Fig. 16. Predicted Time-to-Collision at the end of overtaking maneuver as measure for the safety distance TLC

TLC
Acc. Pedal

I ETTC E

^

OTD [0,1]

I > Ith

Pedal [%]

Fig. 15. Overtaking Detection (OT D) based on Indicator Variables. I: Longitudinal Overtaking Indicator. For a reliable detection of overtaking maneuvers, also longitudinal characteristics of the lane change have to be taken into account. If the own vehicle is approaching vehicle B in a short distance, an overtaking maneuver is more likely than in cases where the own vehicle is falling back and/or the distance to vehicle B is high. Therefore, distance, relative velocity and relative acceleration to vehicle B are considered in the overtaking detection. These three variables are aggregated in a single, well interpretable measure by the Enhanced Time-to-Collision (ETTC). The ETTC is a variant of the ordinary Timeto-Collision, but includes the relative acceleration, see Winner et al. [2009]. It is given as 2xR,rel,AB q ET T CAB = (9) 2 −vrel,AB + vrel,AB − 2xR,rel,AB arel,AB The ET T CAB drops when starting accelerated overtaking maneuvers as well as in flying overtakings, in which the relative velocity is already high when approaching vehicle B.

To improve the reliability of overtaking detection ET T CAB is fused with the accelerator pedal position in a fuzzy logic system, see Figure 15. The resulting longitudinal overtaking indicator I covers the approach of the vehicles as well as the driver’s intention. The system detects an overtaking start (OT D = 1), if the Time-to-Line-Crossing as well as the longitudinal overtaking indicator reach corresponding threshold values, see Figure 15. 3.3 Overtaking prediction and threat analysis

Fig. 17. Relative distances during emergency braking and subsequent lane change. (a) xR,rel,AB < xR,rel,safe , (b) xR,rel,AB = xR,rel,safe . vehicle (C) while completing the overtaking maneuver. It is a continuous measure and can be interpreted independently from the speed range of the involved vehicles. Based on T T CAC,pred, the distance to the oncoming traffic can already be assessed when beginning the overtaking maneuver. If the predicted TTC is lower than a threshold, e.g. 2 s, vehicle C is too near and the overtaking maneuver should be refrained or aborted. 3.4 Warnings and emergency braking In case of a dangerous overtaking maneuver, a collision avoidance strategy is initialized, which aims at aborting the overtaking maneuver and driving back behind vehicle B. Besides different warnings, an automatic braking intervention is realized in order to compensate the driver’s reaction time in case of a suddenly occurring threat. The braking intervention is triggered, when the available time, given by the time gap between vehicle B and the oncoming vehicle C, is just sufficient for a collision avoiding braking and a subsequent lane change, compare Figure 17. To slow down the own vehicle (A) to the speed of vehicle B, the required braking duration is given by |vrel,AB (t)| τˆb,min(t) = . (10) ab A corresponding prediction for the distance at this instant of time, which will reach a minimum, is given by 1 2 (t). (11) x ˆR,rel,AB (t + τˆb,min (t)) = xR,rel,AB (t) − ab τˆb,min 2 If the braking intervention is triggered at a late instant of time, so that the minimal distance given by equation (11) violates the safety distance xrel,safe , an additional braking period, given by  0 , for xˆ ˆb,min (t)) ≥ xR,rel,safe R,rel,AB (t + τ  s
τˆb+ (t) = (12) 2 xR,rel,safe − x ˆR,rel,AB (t + τˆb,min (t))   , else,

If an overtaking situation is detected, it is continuously assessed, whether the maneuver can be conducted or completed without a conflict with oncoming traffic. If an overtaking maneuver has already been started, the actual acceleration of vehicle A is taken into account. For the instant of time, at which vehicle A has left the opposite lane after overtaking, the Time-to-Collision (T T CAC,pred) with respect to vehicle C is predicted, see Figure 16.

is necessary in order to gain the safety distance between A and B.

T T CAC,pred indicates the magnitude of the safety distance between the own vehicle (A) and the oncoming

The totally required time for the maneuver is modeled by τˆreq = τˆb,min (t) + τˆb+ (t) + τLC , (13)

468

ab

AAC 2010 Munich, Germany, July 12-14, 2010

30

Figure 18 illustrates a driving experiment, which simulates a dangerous overtaking maneuver. The system detects the beginning of an overtaking maneuver at t ≈ 4.3 s and the appearing oncoming vehicle at t ≈ 4.8 s. Consequently, the system starts continuously predicting the Time To Collision at the end of the overtaking maneuver T T CAC,pred. As T T CAC,pred is beyond the corresponding warning threshold, the system warns the driver and prefills the braking system in order to prepare a subsequent abort of the overtaking maneuver. Once the trigger condition for emergency braking is fulfilled at t ≈ 8 s, the system commands the full emergency braking pressure until the cut-in conditions are met at t ≈ 10 s, so that the driver can steer back behind the leading vehicle. At t ≈ 11.5 s, the system detects the completion of the abort maneuver and consequently stops warning.

Detection

Model

20 10

Maneuver (−)

0 1 Overtaking Detection (OTD) Overtaking Abort Detection 0

100

x

R,rel,C

(m)

200

0 Warning (−) TTCAC,pred (s)

3.5 Driving experiments

Vehicle B

0

3 2 1 0 1

0 10 τ

τ

The braking intervention is realized by feed forward control of the braking acceleration, while the set point of the feed forward control is adapted to the accelerator pedal position at the trigger point. For more detail see, Mannale et al. [2008], Schmitt et al. [2010].

Vehicle A

R

y (m)

5

(15)

As the trigger point depends on the relative kinematic situation of all three vehicles, trajectories will vary from case to case.

10 0

avail

5

τ

req

0 Brake pressure (bar)

Finally, the trigger condition is given by τˆavail ≤ τreq .

20

xR,rel,B (m)

The available time for the collision avoiding maneuver is calculated by considering the time gap between vehicle B and the oncoming vehicle (C), which leads to the TimeTo-Collision based estimate xR,rel,AC − xR,rel,AB τˆavail = . (14) |vrel,AC | − |vrel,AB |

v (m/s)

where the time reserve τLC for the subsequent lane change maneuver serves as a safety tuning parameter.

50

0 0

2

4

6

8 Time t (s)

10

t = 4.0 s

t = 9.5 s

t = 5.5 s

t = 10.5 s

t = 7.0 s

t = 11,5 s

t = 8.5 s

t = 13.0 s

4. CONCLUSIONS The described system for accident avoidance which was developed within the scope of the project PRORETA1 was presented to an audience selected by Continental Teves and TU Darmstadt. The guests had the possibility of experiencing the system within the scope of driving experiments. Every guest drove, amongst others, the scenarios presented in Figure 8. The system worked robustly and faultlessly. The second part presented the situation analysis and collision avoidance strategies within the PRORETA-2 driver assistance system for overtaking situations. Test drives with an experimental vehicle show, that based on data of environment sensors and vehicle dynamics sensors hazardous overtaking maneuvers can be detected so that the driver can be warned in an early stage of the overtaking maneuver. If an immediate reaction is required, an automatic braking intervention can initiate a timely abort of the overtaking maneuver, which allows the driver to return back behind the leading vehicle. 469

Fig. 18. Results of a driving experiment

12

14

16

AAC 2010 Munich, Germany, July 12-14, 2010

The results are promising, as the new developed methods can help to avoid severe traffic accidents, especially frontal collisions with oncoming traffic, by assisting the driver in timely abortion of hazardous overtaking maneuvers. Integrated into the PRORETA-2 prototype assistance system, the described methods were presented by numerous test drives without any faults to a selected audience of about 150 guests in October 2009. Future research should consider more complex traffic situations, e.g. with more vehicles and more driving lanes. Furthermore, a 360 degree environment perception is desirable, which allows to consider the sideward and backward traffic. This would also allow the integration of further functionalities like an active steering support for cutting in. The name PRORETA comes from the ancient world. In times of the roman galleys it used to be the boatswain standing at the bow of the ship warning of shallows. Following this idea, the driver assistance system PRORETA monitors the surroundings of the vehicle and intervenes in emergency situations to prevent an accident. REFERENCES Bender, E. (2008) Handlungen und Subjektivurteile von Kraftfahrzeugf¨ uhrern bei automatischen Brems- und Lenkeingriffen eines Unterst¨ utzungssystems zur Kollisionsvermeidung. Stuttgart, Ergonomia-Verlag. Bender, E., Darms, M., Schorn, M., St¨ahlin, U., Isermann, R., Winner, H. and Landau, K. (2007). Anti Collision System Proreta - On the Way to the Collision Avoiding Vehicle. Part 1: Basics of the System, Part 2: Results. ATZ, vol. 4, pp. 20-23 and vol. 5, pp. 32-35. Darms, M. (2007) Eine Basis-Systemarchitektur zur Sensordatenfusion von Umfeldsensoren f¨ ur Fahrerassistenzsysteme. D¨ usseldorf, VDI-Verlag Darms, M. and Winner, H. (2006) Umfelderfassung f¨ ur ein Fahrerassistenzsystem zur Unfallvermeidung D¨ usseldorf, VDI-Verlag Hohm, A., Wojek, C., Schiele, B. and Winner, H. (2008). Multi-Level Sensorfusion and Computer-Vision Algorithms within a Driver Assistance System for Avoiding Overtaking Accidents. FISITA World Automotive Congress, Munich. Isermann, R. (2006) (ed.). Fahrdynamik-Regelung: Modellbildung, Fahrerassistenzsysteme, Mechatronik. Wiesbaden, Vieweg Verlag. Isermann, R., Schiele, B., Winner, H., Hohm, A., Mannale, R., Schmitt, K., Wojek, C. and L¨ uke, S. (2009). Elektronische Fahrerassistenz zur Vermeidung ¨ von Uberholunf¨ allen - PRORETA 2. VDI-Berichte Nr. 2075, Elektronik im Kraftfahrzeug, D¨ usseldorf. Isermann, R., Schorn, M. and St¨ ahlin, U. (2008). Anticollision system PRORETA with automatic braking and steering. Vehicle System Dynamics, vol 46, supplement, pp. 683-694. Mammar, S., Glaser, S. and Netto, M. (2006). Time to Line Crossing for Lane Departure Avoidance: A Theoretical Study and an Experimental Setting. IEEE Transactions on Intelligent Transportation Systems, vol. 7, no. 2., pp. 226-241. Mannale, R., Hohm, A., Schmitt, K., Isermann, R. and Winner, H. (2008). Ansatzpunkte f¨ ur ein System zur 470

¨ Fahrerassistenz in Uberholsituationen. 3. Tagung Aktive Sicherheit durch Fahrerassistenz, Garching. Nelles, O. (2001) Nonliner Systems. Berlin, Springer Verlag. Schmitt, K., Habenicht, S. and Isermann, R. (2009a) Odometrie und Manvererkennung f¨ ur ein Fahrerassis¨ tenzsystem f¨ ur Uberholsituationen. VDI-Conference: 1. Automobiltechnische Kolloquium, Munich. Schmitt, K. and Isermann, R. (2009b). Vehicle State Estimation in Curved Road Coordinates for a Driver Assistance System for Overtaking Situations. 21st International Symposium on Dynamics of Vehicles on Roads and Tracks (IAVSD), Stockholm. Schmitt, K., Mannale, R. and Isermann, R. (2010) Collision Avoidance System PRORETA For Overtaking Maneuvers - automatic situation analysis, warnings and emergency braking. IFAC Symposium Advances in Automotive Control, July 12-14, 2010, Munich, Germany. Schorn, M. (2007) Quer- und L¨angsregelung eines Personenkraftwagens fr ein Fahrerassistenzsystem zur Unfallvermeidung. Fortschr.-Ber. VDI Reihe 12, 651. D¨ usseldorf, VDI-Verlag. Schorn, M. and Isermann, R. (2006) Automatic Steering and Braking for a Collision Avoiding Vehicle. 4th IFAC Symposium on Mechatronic Systems. September 12-14, Wiesloch/Heidelberg. Schorn, M., Schmitt, J., St¨ahlin, U. and Isermann, R. (2005). Model-based braking control with support by active steering. 16th IFAC World Congress, July 4-8, 2005, Prague, Czech Republic. Schorn, M., St¨ahlin, U., Khanafer, A. and Isermann, R. (2006). Nonlinear Trajectory Following Control for Automatic Steering of a Collision Avoiding Vehicle. American Control Conference, June 14-16, Minneapolis, Minnesota, USA. St¨ahlin, U. (2008) Eingriffsentscheidung f¨ ur ein Fahrerassistenzsystem zur Unfallvermeidung. D¨ usseldorf, VDIVerlag. St¨ahlin, U., Schorn, M. and Isermann, R. (2006) Notausweichen f¨ ur ein Fahrerassistenzsystem zur Unfallvermeidung. Fortschr.-Ber. VDI Reihe 12, 683. AUTOREG 2006, D¨ usseldorf, VDI-Verlag. Winner, H., Hakuli, S. and Wolf, G. (2009) Handbuch Fahrerassistenzsysteme. Wiesbaden, Vieweg+Teubner Verlag. Wojek, C., Dork, G., Schulz, A., Schiele and B. Wojek (2008). Sliding-Windows for Rapid Object-Class Localization: a Parallel Technique 30th DAGM Symposium 2008, Berlin, Springer Zittlau, D., Hoppe, J. (2008). L¨angs- und Querdynamikassistenzsysteme f¨ ur mehr Sicherheit. VDI/VDE-Tagung AUTOREG 2008, 12-13 Feb 2008. VDI Bericht 2009, 1-10.