- Email: [email protected]
Deciding the Behaviour of an Autonomous Mobile Road Vehicle
Copyright © IFAC Intelligent Autonomous Vehicles. Espoo. Finland. 1995
DECIDING THE BEHAVIOUR OF AN AUTONOMOUS MOBILE ROAD VEHICLE
Clemens Kujawski
Institut for Programmiersprachen und Programmentwicklung F akultiit fur I nformatik Universitiit der Bundeswehr Munchen 85577 Neubiberg Germany e-mail: [email protected]
Abstract: The seeing passenger car VaMoRs-P is able to perfonn basic manoeuvres. e.g. lane keeping, convoy driving, and lane changing. In order to make it capable of perfonning an appropriate behaviour even in complex traffic situations, an algorithm was designed which assesses the traffic situation, detennines a safety measure of different possible manoeuvres, and chooses one manoeuvre using a rule-based system. The transputer based implementation of this algorithm was demonstrated in real world traffic situations on French motorways. Keywords: autonomous vehicles, behaviour, numerical simulation. real-time AI, rule-based systems. trajectories
1. INTRODUcnON
which estimates horizontal and vertical curvature, lane width, and further state variables necessary to describe the road and the position of the AMY on the road. The estimated state variables are sent to the 'Dynamic Data Base (DDB)' (Dickmanns. 1994a) from where they can be accessed by any other module. The task of module 'Vehicle Control (VC)' (Maurer, et al.. 1994) is to control steering. accelerator, and brake of the AMY.
At the Department of Aeronautics and Astronautics of the 'Universitat der Bundeswehr Mtinchen ' the seeing passenger car VaMoRs-P has been developed in cooperation with Daimler Benz within the EUREKA project Prometheus (£rQgramme for a European Iraffic with Highest ~fficiency and Unprecedented Safety) (Dickmanns. et al .• 1994). This autonomous mobile vehicle (AMY) is equipped with four CCDcameras with different focal lengths viewing to the front and the rear. Among several software modules enabling VaMoRs-P to perfonn basic manoeuvres. e.g. lane keeping. convoy driving, and lane changing. the foUowing modules are essential: Digitized images are processed by the module KRONOS (Dickmanns, 1995). It provides edge extraction for the module 'Obstacle Detection and Tracking (ODT) ' (Thomanek. et al .• 1994) which estimates deviation. distance. width. height, velocity, and acceleration of surrounding vehicles and for the module 'Road Detection and Tracking (ROT) ' (Behringer. 1994)
The module 'Behaviour Decision (BD)' which is described below has been designed to make the AMY perfonn adequately in complex traffic situations.
2. MODULE BEHAVIOUR DECISION
2.1 Function
The function of module BD is to percept and to assess the actual traffic situation around the AMY. to make a
387
decision concerning further behaviour, and to give corresponding commands to module Vc. It has to take into account the data concerning the road including its markings, the surrounding vehicles as well as a planned velocity. The assessment of complex traffic situations has to be performed in real time, i. e. the reaction time of the AMV has to be comparable to the reaction time of a human driver. The following modes of behaviour are possible: lane keeping, changing to left/right lane, cancelling lane changing, and overtaking as a composed manoeuvre. Module BD makes appropriate decisions, if the AMV is driving on a motorway comparable to German motorways, the road and its markings are detected, all traffic participants surrounding the AMV are detected, and the motorway is well passable. At present there are further restrictions: There are no ghost-drivers. The only traffic signs taken into account are 'no overtaking' , speed limits, and their cancellations.
corresponding trajectories. The task of the third part of BD is to decide which actions are to be carried out and to issue appropriate driving commands.
Extrapolating and Assessing Trajectories. In the following a curvilinear coordinate system is used where the x-coordinate line points to the front of the AMV and runs along a curved line in the middle of the lane (fig. 1). The y-axis is straight and points to the right, the z-axis which is straight too points down. The origin of the x-coordinate line and the origin of the z-axis are placed in the geometric middle of the AMV. The origin of the y-axis is placed in the middle of the AMV's current lane. The z-coordinate denotes the height of the AMV and will not be taken into further considerations. The horizontal curvature of the x-coordinate line in front of the AMV is modelled as C
= cO+c1 xx.
(1)
The horizontal curvature cO at the place of the AMV and the average changing of horizontal curvature c J within a given look ahead range is calculated by module RDT and sent to module BD periodically. Collecting horizontal curvature over time and transforming it to a grid which is equidistant in space provides the curvature of the x-coordinate line behind theAMV.
2.2 Performance Requirements One goal of the Prometheus project is to achieve 'unprecedented safety'. This aspect of the performance of the AMV should compare favourably to an average human driver. Therefore the reaction time of the AMV must not be worse than the reaction time of man.
The horizontal curvature of a point on the road where the y-coordinate is not equal to zero results in 1 c".y = - 1 - - '
(2)
--y
According to Mitschke (1982) the average reaction time of a human driver is 0.6 s in case the danger is within the viewing direction of the driver, and it is 0.9s in case the driver must turn his eyes towards the danger. Since during the experiments the camera platform (Schiehlen and Dickmanns, 1994) was not activated and therefore the viewing direction of the AMV was fixed the first value will be used throughout.
c".o Defining the curvature like this makes the lanes running equidistantly to the x-coordinate line. Using the values deviation. distance. and vector velocity (as calculated by module ODT) the most probable trajectory of each surrounding vehicle detected by module ODT is extrapolated.
Assuming a mean delay time of about 300 ms for image processing, state estimation. controlling of the actuators, and for sending and receiving data through the DDB there will be about 300 ms left for module BD.
2.3 Realisation Module BD is divided into three partS. The task of the first part is to bridge over the temporary disappearing of a tracked vehicle in the blind spot between the front and the rear cameras. The task of the second part of module BD is to find out which possible manoeuvres of the AMV are safe by assessing
Fig. 1. The curvilinear coordinate system.
388
It is assumed that the considered vehicle will change lane, if its velocity in y-direction exceeds a given limit Furthennore it is assumed that the considered vehicle will finish its lane changing when a new lane is reached. Lane changing across more than one lane has not been modelled.
driving behind needs to cover this distance. Then dsafe at time tk is calculated as dsa:IS, '. Yi~.t I = max(minimum. Vy I x I ,f)' i" saJ~
(5)
if the regarded vehicle Vi is driving behind the AMY and it is calculated as
The extrapolation of the trajectories is done numerically according to
d sa/ t . Y"I.
= ma.x(minimum. v AMY. I. X I sa/ t )
(6)
otherwise. minimum is a quantity which is used if the velocity is very low, e.g. in stop & go traffic.
and
(4)
To find out whether a numerically extrapolated trajectory of the AMY Tj is safe with regard to vehicle Vi at time tk also the longitudinal distance d 10ng between AMY and vehicle Vi (defined as the difference between the x-coordinates minus both the half lengths of the AMY and the vehicle Vi) is computed and the quotient
where cx.y.k is the horizontal curvature of the lane at the place where the considered vehicle is located at time tk and cx.D}. is the horizontal curvature of the middle of the AMY's lane at the place (x. 0) at time tk' Currently three trajectories are extrapolated for the AMY. In case the AMY keeps the lane these three trajectories are ,.AMY changes to left lane", ,.AMY keeps lane", and ..AMY changes to right lane". Module RDT finds lane markings within search windows in the digitized video image. During a lane change these search windows are set to the new lane and the coordinate system is shifted by half the width of the new lane and half the width of the lane the AMY has come from. In case the AMY is changing the lane and the search windows have not yet been set to the new lane the three trajectories are ..AMY continues changing the lane", ,.Deviation is controlled to zero", and ,.AMY changes to the opposite lane". If the search windows have been set to the new lane, the trajectories ,.AMY changes to the next lane but one", ,.AMY continues lane changing", and ,.AMY drives back to the lane which it has come from" are extrapolated.
(7)
is calculated. If the lateral distance between the AMY and the regarded vehicle Vi does not fall below the given minimum, Qv., is set to 00. I'k
The assessment of the a trajectory Tj of the AMY is defined as AT
}
= fl'Jin(Qy l.k
I) . I"
A:
(8)
A value m =AT' can be interpreted as follows: While the manoeuvre 1s being executed the AMY will not come closer to any other vehicle than m times the safe distance. If m is greater or equal to one, there is no danger in executing the manoeuvre. (Remember that vehicles Vi are considered only while their lateral distance from the AMY is less than a given threshold.) Depending on the time needed for lane changing the extrapolation is done for a look ahead time of 7.5 s at least A time grid of !!J = 0.1 s has proved to be appropriate. It is neither too fine-grain to consume too much CPU time, nor is it too coarse so that a possibly dangerous situation will not be detected when the velocity is very high.
Of course a trajectory is only extrapolated, if the according lane does exist. To find out whether an extrapolated trajectory Tj of the AMY is safe with regard to vehicle Vi at time tk the lateral difference (defined as the difference of the y-coordinates minus both the half widths of the AMY and the vehicle Vi) is computed. If the lateral difference falls below a given threshold, the safe distance d safe to vehicle Vi is calculated according to a rule recommended to human drivers on Gennan motorways. This rule takes into account the distance covered by the second vehicle on the lane until the driver has reacted to an event. the brakes are activated, and the pressure of the braking system is built up. Let tsafe be the time the vehicle
In a situation where the AMY is surrounded by five other vehicles this algorithm will consume about 160 ms CPU time of two T800 transputers. This makes it the most costly part of module BD. Though the extrapolation time is 7.5 s, the extrapolation is executed about fom times a second. Therefore the assessment of possible trajectories of the AMY 389
trajectories but are also taking into account whether there is an according lane and whether the markings are dashed so that they are allowed to be crossed. The predicate rler simply indicates the existence of a right lane. The predicate cim becomes true. if the velocity of the AMY exceeds a threshold which has been gained from experience.
always refers to an up-to-date model of the actual traffic situation. Longitudinal control perfonned by module VC in VaMoRs-P is simulated by calculating the acceleration of the AMY in x-direction as
The predicate ller is calculated taking into account a lot of factors including the difference of the planned velocity and the actual velocity of the AMY. the difference of the planned velocity and the velocity of the vehicle driving in front of the AMY. the existence of a left lane. and a possibly existing restriction on overtaking.
where vr is the requested velocity. va is the actual velocity of the AMY. and kj are constants which were detennined in experiments. Experience evidence obtained in simulations triggered some further improvements of the algorithm. E. g .• a vehicle Vj which is catching up from behind in the AMY's lane has to be excluded from the assessment of lane changing trajectories of the AMY. Otherwise the AMY possibly would not perfonn lane changing to make way for this vehicle Vj.
Since two different conditions C jj and Cjk connected to two transitions S;Sj and SjSk can become true at the same time the STS has not been detenninistic per se. It has been made detenninistic by ordering the transitions in a list of priority. If the STS is in the state Sj and there are more transitions S;Sk] ... S;Sk n whose conditions become true at the same time, the STS is transferred to the state Sk I.' if the transition SjSkI. has the highest priority.
Knowledge Representation. Once the assessment of possible trajectories is given. it is known to module BD which manoeuvres could be done. To decide which manoeuvre should be done a small knowledge base is used. Different kinds of knowledge representation have been taken into consideration and production rules seemed to be most appropriate. In order to be independent of a commercial shell they were represented using a state transition system (STS) which in turn was implemented in C.
The interface protocol between module BD and module VC is also handled by the STS.
2.4 Implementation Module BD has been programmed using the ANSI-C programming language and integrated in the transputer system of VaMoRs-P. Two transputers 1'800 were necessary to achieve real time capability. Module BD receives data from module RDT and ODT via the DDB and exchanges data with module VC directly. Receiving data from module RDT. ODT. and VC module BD composes a set of data describing the traffic situation around the AMY. Temporary disappearing of a vehicle in a blind spot is bridged over. Using this data set possible trajectories of the AMY are assessed. predicates and conditions connected to the transitions are calculated, and should the situation arise the STS goes to a new state. Operations according to the actual state are executed and commands are sent to the module Vc. After possibly sending messages to the user module BD starts again producing a new set of data.
There are one or more than one operations (incl. NOP, no operation) assigned to each state. The operations are being executed on entering a state. The transition from a state Sj to a state Sj is perfonned, if the condition connected to this transition S;Sj becomes true. Operations and transitions are separated strictly: All operations are executed unconditionally. and no transition has a side effect. Conditions connected to the transitions are boolean expressions composed of boolean variables, conditions concerning time, and the predicates cim. "car is moving", ller. "left lane change request". rlcr. "right lane change request". cllp. "changing to left lane is possible", ccll. "cancel changing to left lane". erlp. "changing to right lane is possible", ccrl. "cancel changing to right lane".
Concurrency. To achieve real time capability even if the trajectories of more than four surrounding vehicles have to be computed. the algorithm of assessing extrapolated trajectories which consumes
The predicates ellp. cell. erlp and cerl are basically calculated by the method of assessing extrapolated
390
o'
most of the cycle time of module BD has been distributed on two transputers running concurrently. For this purpose the set of surrounding vehicles is split into two approximately equipotent subsets of vehicles. The assessment is done on the two transputers with each of the subsets. From the two arising assessments of trajectories the worst one is taken into consideration for calculating the predicates. It is true the trajectories of the AMV have to be extrapolated twice but since there is' little data exchange between the two transputers this kind of parallelization could be done very easily and the legibility of the program was kept.
0' 0' 0 -,
-, - ,
o o t -.
~
.........t
..I=-
Supporting Work, A PC based motorway simulator has been developed to test the module BD. Up to six user controlled vehicles slDTounding the AMV can be simulated. Employing either two joysticks or the keyboard as input device the user can easily compose complex traffic situations. The modules VC, ODT, and RDT are simulated on the PC. It is connected to a small transputer system on which module DDB and module BD are running using the same code as on the transputer system of the AMV. So once the code is tested and debugged with help of the simulator it can be run in the AMV without modification. Fig. 2 shows a screenshot of the simulator.
Fig. 2. Screenshot of the PC based motorway simulator. On the left side of the screen the graph of the STS is shown. It is possible to switch over to show the extrapolated trajectories. to show the messages sent from module BD. or to visualize the interface protocol between module BD and module VC. On the right side of the screen there are seven speedometers corresponding to seven vehicles including the AMV on the simulated motorway in the middle of the screen. Since it is zoomed in on the AMV here only the AMV and two of the slDTounding vehicles are visible. In the window on the lower right side of the screen the CPU utilization of the PC is shown. If it is less than 100%. the simulation is running in real time.
3. EXPERIMENTAL RESULTS
For safety reasons the user had to acknowledge actions triggered by module BD during the practical experiments.
Mter integrating module BD into VaMoRs (which was the predecessor of VaMoRs-P) overtakings triggered by decreasing the velocity of the vehicle driving in front of the AMV from 60 km/h down to 40 km/h were demonstrated. Experiments took place at the test area close to our university. Cancelling lane changing could also be demonstrated in practice: The vehicle which was driving in front of the AMV decreased its velocity so module BD triggered module VC to change lane. When the AMV was approaching the new lane, module ODT detected another vehicle driving on this lane. The second vehicle also decreased its velocity so module BD cancelled lane changing.
3.1 Criticism Though module BD showed its worth in different traffic situations. extensive simulation proved the method of assessing extrapolated trajectories to be too optimistic about lane changing on some rare occasions. A second algorithm for assessing different manoeuvres of the AMV was designed to overcome this difficulty. It can be used alternatively to or in combination with the method of assessing extrapolated trajectories.
When module BD was integrated in VaMoRs-P in autumn 1994, the whole system was able to detect the necessity of an overtaking as well as to perfonn it autonomously. During the Prometheus Board Member Meeting in October 1994 this was demonstrated in real traffic situations on the highway Al north of Paris achieving velocities up to 130 km/h.
Method of occupied areas. The second method of detennining the measure of safety of different possible manoeuvres of the AMV is quite simple. Three rectangular areas around the AMV arranged as shown in fig. 3 are considered. The left and right boundaries of the areas Aj correspond to the lane markings. The distance from the AMV to the front
391
4. CONCLUSION
8
e: ...
I---
~
n =. 0
I---
Ai
Experimental results with module BD, especially those gathered in Paris, were very satisfactory. But it has to be checked in what cases the method of occupied areas is sufficient. in what cases the method of extrapolating trajectories is more advantageous, and whether there are situations where a combination of both methods is required.
A2
A3
::l
0
...... 50
~
~
Furthermore the assessment of the behaviour of the surrounding vehicles will be refined. E. g. trafficators will be taken into account and more information about the whole traffic situation will be used.
~
Sln
All further improvements and refinements will have to show their usefulness in practice in the seeing passenger car VaMoRs-P.
Fig. 3. Areas around the AMV. and the back boundaries of the areas A ' is calculated for each of the surrounding vehicles separately. If the considered vehicle Vi is in front of the AMV like shown in fig. 3, the back boundaries of the areas k correspond to the back of the AMY, the distance fron{ the AMY to the front boundaries is calculated for each of the areas Aj separately as the maximum of - a given minimum , the safe distance depending on the velocity of the AMY,and the safe distance taking into account the difference of the AMY's and V;'s velocities as well as the time the AMY needs for changing to the corresponding lane. If the considered vehicle Vi is behind the AMV, the front boundaries of the areas A j correspond to the frontage of the AMV, the distance from the AMV to the back boundaries is calculated as the maximum of a given minimum, the safe distance depending on the velocity of the vehicle Vi, and the safe distance taking into account the difference of the AMV's and V;'s velocities as well as the time the AMV needs for changing to the corresponding lane. An area Aj is considered to be occupied, if there is a vehicle Vi partially or completely within the area A '. If the area Ai (A 3) is occupied, lane changing to the )eft (right) lane is not allowed or cancelling changing to the left (right) lane is necessary. Area A2 is used for calculating whether lane changing can be cancelled safely. Lane changing of the surrounding vehicles VI . IS modelled by extending their width to the lane they are changing to.
v:
This research has been supported by BMFT and DaimlerBenz AG under the EUREKA-Projekt Prometheus III (Grant ITW 9301A).
REFERENCES Behringer, R. (1994). Road Recognition from Multifocal Vision. In: Proceedings of the Intelligent Vehicles '94 Symposium, pp. 302-307. Paris, France. Dickmanns, D. (1995). Knowledge Based Real Time Vision. (this volume) Dickmanns, D. (J994a). The Dynamic Database User's Guide . Prometheus PRO-ART, InformatikForschungsgruppe, UniBw Miinchen. Germany. Dickmanns, E. D., R. Behringer, D. Dickmanns, T. Hildebrandt, M. Maurer, E Thomanek and J. Schiehlen (1994). The Seeing Passenger Car VaMoRs-P. In: Proceedings of the Intelligent Vehicles '94 Symposium, pp. 68-73. Paris, France. Maurer, M., R. Behringer, D. Dickmanns, T. Hildebrandt. E Thomanek, J. Schiehlen and E. D. Dickmanns (1994). VaMoRs-P. An advanced platform for visual autonomous road vehicle guidance. In: Proceedings of the SPlE Conference 'Mobile Robots lX' , pp. 239-248. Boston. USA. Mitschke, M. (1982). Dynamik der KraftJahrzeuge . Band A Antrieb und Bremsung. Springer. Berlin, Heidelberg. New York. Schiehlen, J. and E. D. Dickmanns (1994). A Camera Platform for Intelligent Vehicles. In: Proceedings of the Intelligent Vehicles '94 Symposium, pp. 393-398. Paris. France. Thomanek, E. E. D. Dickmanns and D. Dickmanns (1994). Multiple Object Recognition and Scene Interpretation for Autonomous Road Vehicle Guidance. In: Proceedings of the Intelligent Vehicles '94 Symposium. pp. 231-236. Paris, France.
The method of occupied areas is very fast and it can reduce the cycle time of module BD significantly, if it is used instead of the method of extrapolating trajectories.
392
DECIDING THE BEHAVIOUR OF AN AUTONOMOUS MOBILE ROAD VEHICLE
Clemens Kujawski
Institut for Programmiersprachen und Programmentwicklung F akultiit fur I nformatik Universitiit der Bundeswehr Munchen 85577 Neubiberg Germany e-mail: [email protected]
Abstract: The seeing passenger car VaMoRs-P is able to perfonn basic manoeuvres. e.g. lane keeping, convoy driving, and lane changing. In order to make it capable of perfonning an appropriate behaviour even in complex traffic situations, an algorithm was designed which assesses the traffic situation, detennines a safety measure of different possible manoeuvres, and chooses one manoeuvre using a rule-based system. The transputer based implementation of this algorithm was demonstrated in real world traffic situations on French motorways. Keywords: autonomous vehicles, behaviour, numerical simulation. real-time AI, rule-based systems. trajectories
1. INTRODUcnON
which estimates horizontal and vertical curvature, lane width, and further state variables necessary to describe the road and the position of the AMY on the road. The estimated state variables are sent to the 'Dynamic Data Base (DDB)' (Dickmanns. 1994a) from where they can be accessed by any other module. The task of module 'Vehicle Control (VC)' (Maurer, et al.. 1994) is to control steering. accelerator, and brake of the AMY.
At the Department of Aeronautics and Astronautics of the 'Universitat der Bundeswehr Mtinchen ' the seeing passenger car VaMoRs-P has been developed in cooperation with Daimler Benz within the EUREKA project Prometheus (£rQgramme for a European Iraffic with Highest ~fficiency and Unprecedented Safety) (Dickmanns. et al .• 1994). This autonomous mobile vehicle (AMY) is equipped with four CCDcameras with different focal lengths viewing to the front and the rear. Among several software modules enabling VaMoRs-P to perfonn basic manoeuvres. e.g. lane keeping. convoy driving, and lane changing. the foUowing modules are essential: Digitized images are processed by the module KRONOS (Dickmanns, 1995). It provides edge extraction for the module 'Obstacle Detection and Tracking (ODT) ' (Thomanek. et al .• 1994) which estimates deviation. distance. width. height, velocity, and acceleration of surrounding vehicles and for the module 'Road Detection and Tracking (ROT) ' (Behringer. 1994)
The module 'Behaviour Decision (BD)' which is described below has been designed to make the AMY perfonn adequately in complex traffic situations.
2. MODULE BEHAVIOUR DECISION
2.1 Function
The function of module BD is to percept and to assess the actual traffic situation around the AMY. to make a
387
decision concerning further behaviour, and to give corresponding commands to module Vc. It has to take into account the data concerning the road including its markings, the surrounding vehicles as well as a planned velocity. The assessment of complex traffic situations has to be performed in real time, i. e. the reaction time of the AMV has to be comparable to the reaction time of a human driver. The following modes of behaviour are possible: lane keeping, changing to left/right lane, cancelling lane changing, and overtaking as a composed manoeuvre. Module BD makes appropriate decisions, if the AMV is driving on a motorway comparable to German motorways, the road and its markings are detected, all traffic participants surrounding the AMV are detected, and the motorway is well passable. At present there are further restrictions: There are no ghost-drivers. The only traffic signs taken into account are 'no overtaking' , speed limits, and their cancellations.
corresponding trajectories. The task of the third part of BD is to decide which actions are to be carried out and to issue appropriate driving commands.
Extrapolating and Assessing Trajectories. In the following a curvilinear coordinate system is used where the x-coordinate line points to the front of the AMV and runs along a curved line in the middle of the lane (fig. 1). The y-axis is straight and points to the right, the z-axis which is straight too points down. The origin of the x-coordinate line and the origin of the z-axis are placed in the geometric middle of the AMV. The origin of the y-axis is placed in the middle of the AMV's current lane. The z-coordinate denotes the height of the AMV and will not be taken into further considerations. The horizontal curvature of the x-coordinate line in front of the AMV is modelled as C
= cO+c1 xx.
(1)
The horizontal curvature cO at the place of the AMV and the average changing of horizontal curvature c J within a given look ahead range is calculated by module RDT and sent to module BD periodically. Collecting horizontal curvature over time and transforming it to a grid which is equidistant in space provides the curvature of the x-coordinate line behind theAMV.
2.2 Performance Requirements One goal of the Prometheus project is to achieve 'unprecedented safety'. This aspect of the performance of the AMV should compare favourably to an average human driver. Therefore the reaction time of the AMV must not be worse than the reaction time of man.
The horizontal curvature of a point on the road where the y-coordinate is not equal to zero results in 1 c".y = - 1 - - '
(2)
--y
According to Mitschke (1982) the average reaction time of a human driver is 0.6 s in case the danger is within the viewing direction of the driver, and it is 0.9s in case the driver must turn his eyes towards the danger. Since during the experiments the camera platform (Schiehlen and Dickmanns, 1994) was not activated and therefore the viewing direction of the AMV was fixed the first value will be used throughout.
c".o Defining the curvature like this makes the lanes running equidistantly to the x-coordinate line. Using the values deviation. distance. and vector velocity (as calculated by module ODT) the most probable trajectory of each surrounding vehicle detected by module ODT is extrapolated.
Assuming a mean delay time of about 300 ms for image processing, state estimation. controlling of the actuators, and for sending and receiving data through the DDB there will be about 300 ms left for module BD.
2.3 Realisation Module BD is divided into three partS. The task of the first part is to bridge over the temporary disappearing of a tracked vehicle in the blind spot between the front and the rear cameras. The task of the second part of module BD is to find out which possible manoeuvres of the AMV are safe by assessing
Fig. 1. The curvilinear coordinate system.
388
It is assumed that the considered vehicle will change lane, if its velocity in y-direction exceeds a given limit Furthennore it is assumed that the considered vehicle will finish its lane changing when a new lane is reached. Lane changing across more than one lane has not been modelled.
driving behind needs to cover this distance. Then dsafe at time tk is calculated as dsa:IS, '. Yi~.t I = max(minimum. Vy I x I ,f)' i" saJ~
(5)
if the regarded vehicle Vi is driving behind the AMY and it is calculated as
The extrapolation of the trajectories is done numerically according to
d sa/ t . Y"I.
= ma.x(minimum. v AMY. I. X I sa/ t )
(6)
otherwise. minimum is a quantity which is used if the velocity is very low, e.g. in stop & go traffic.
and
(4)
To find out whether a numerically extrapolated trajectory of the AMY Tj is safe with regard to vehicle Vi at time tk also the longitudinal distance d 10ng between AMY and vehicle Vi (defined as the difference between the x-coordinates minus both the half lengths of the AMY and the vehicle Vi) is computed and the quotient
where cx.y.k is the horizontal curvature of the lane at the place where the considered vehicle is located at time tk and cx.D}. is the horizontal curvature of the middle of the AMY's lane at the place (x. 0) at time tk' Currently three trajectories are extrapolated for the AMY. In case the AMY keeps the lane these three trajectories are ,.AMY changes to left lane", ,.AMY keeps lane", and ..AMY changes to right lane". Module RDT finds lane markings within search windows in the digitized video image. During a lane change these search windows are set to the new lane and the coordinate system is shifted by half the width of the new lane and half the width of the lane the AMY has come from. In case the AMY is changing the lane and the search windows have not yet been set to the new lane the three trajectories are ..AMY continues changing the lane", ,.Deviation is controlled to zero", and ,.AMY changes to the opposite lane". If the search windows have been set to the new lane, the trajectories ,.AMY changes to the next lane but one", ,.AMY continues lane changing", and ,.AMY drives back to the lane which it has come from" are extrapolated.
(7)
is calculated. If the lateral distance between the AMY and the regarded vehicle Vi does not fall below the given minimum, Qv., is set to 00. I'k
The assessment of the a trajectory Tj of the AMY is defined as AT
}
= fl'Jin(Qy l.k
I) . I"
A:
(8)
A value m =AT' can be interpreted as follows: While the manoeuvre 1s being executed the AMY will not come closer to any other vehicle than m times the safe distance. If m is greater or equal to one, there is no danger in executing the manoeuvre. (Remember that vehicles Vi are considered only while their lateral distance from the AMY is less than a given threshold.) Depending on the time needed for lane changing the extrapolation is done for a look ahead time of 7.5 s at least A time grid of !!J = 0.1 s has proved to be appropriate. It is neither too fine-grain to consume too much CPU time, nor is it too coarse so that a possibly dangerous situation will not be detected when the velocity is very high.
Of course a trajectory is only extrapolated, if the according lane does exist. To find out whether an extrapolated trajectory Tj of the AMY is safe with regard to vehicle Vi at time tk the lateral difference (defined as the difference of the y-coordinates minus both the half widths of the AMY and the vehicle Vi) is computed. If the lateral difference falls below a given threshold, the safe distance d safe to vehicle Vi is calculated according to a rule recommended to human drivers on Gennan motorways. This rule takes into account the distance covered by the second vehicle on the lane until the driver has reacted to an event. the brakes are activated, and the pressure of the braking system is built up. Let tsafe be the time the vehicle
In a situation where the AMY is surrounded by five other vehicles this algorithm will consume about 160 ms CPU time of two T800 transputers. This makes it the most costly part of module BD. Though the extrapolation time is 7.5 s, the extrapolation is executed about fom times a second. Therefore the assessment of possible trajectories of the AMY 389
trajectories but are also taking into account whether there is an according lane and whether the markings are dashed so that they are allowed to be crossed. The predicate rler simply indicates the existence of a right lane. The predicate cim becomes true. if the velocity of the AMY exceeds a threshold which has been gained from experience.
always refers to an up-to-date model of the actual traffic situation. Longitudinal control perfonned by module VC in VaMoRs-P is simulated by calculating the acceleration of the AMY in x-direction as
The predicate ller is calculated taking into account a lot of factors including the difference of the planned velocity and the actual velocity of the AMY. the difference of the planned velocity and the velocity of the vehicle driving in front of the AMY. the existence of a left lane. and a possibly existing restriction on overtaking.
where vr is the requested velocity. va is the actual velocity of the AMY. and kj are constants which were detennined in experiments. Experience evidence obtained in simulations triggered some further improvements of the algorithm. E. g .• a vehicle Vj which is catching up from behind in the AMY's lane has to be excluded from the assessment of lane changing trajectories of the AMY. Otherwise the AMY possibly would not perfonn lane changing to make way for this vehicle Vj.
Since two different conditions C jj and Cjk connected to two transitions S;Sj and SjSk can become true at the same time the STS has not been detenninistic per se. It has been made detenninistic by ordering the transitions in a list of priority. If the STS is in the state Sj and there are more transitions S;Sk] ... S;Sk n whose conditions become true at the same time, the STS is transferred to the state Sk I.' if the transition SjSkI. has the highest priority.
Knowledge Representation. Once the assessment of possible trajectories is given. it is known to module BD which manoeuvres could be done. To decide which manoeuvre should be done a small knowledge base is used. Different kinds of knowledge representation have been taken into consideration and production rules seemed to be most appropriate. In order to be independent of a commercial shell they were represented using a state transition system (STS) which in turn was implemented in C.
The interface protocol between module BD and module VC is also handled by the STS.
2.4 Implementation Module BD has been programmed using the ANSI-C programming language and integrated in the transputer system of VaMoRs-P. Two transputers 1'800 were necessary to achieve real time capability. Module BD receives data from module RDT and ODT via the DDB and exchanges data with module VC directly. Receiving data from module RDT. ODT. and VC module BD composes a set of data describing the traffic situation around the AMY. Temporary disappearing of a vehicle in a blind spot is bridged over. Using this data set possible trajectories of the AMY are assessed. predicates and conditions connected to the transitions are calculated, and should the situation arise the STS goes to a new state. Operations according to the actual state are executed and commands are sent to the module Vc. After possibly sending messages to the user module BD starts again producing a new set of data.
There are one or more than one operations (incl. NOP, no operation) assigned to each state. The operations are being executed on entering a state. The transition from a state Sj to a state Sj is perfonned, if the condition connected to this transition S;Sj becomes true. Operations and transitions are separated strictly: All operations are executed unconditionally. and no transition has a side effect. Conditions connected to the transitions are boolean expressions composed of boolean variables, conditions concerning time, and the predicates cim. "car is moving", ller. "left lane change request". rlcr. "right lane change request". cllp. "changing to left lane is possible", ccll. "cancel changing to left lane". erlp. "changing to right lane is possible", ccrl. "cancel changing to right lane".
Concurrency. To achieve real time capability even if the trajectories of more than four surrounding vehicles have to be computed. the algorithm of assessing extrapolated trajectories which consumes
The predicates ellp. cell. erlp and cerl are basically calculated by the method of assessing extrapolated
390
o'
most of the cycle time of module BD has been distributed on two transputers running concurrently. For this purpose the set of surrounding vehicles is split into two approximately equipotent subsets of vehicles. The assessment is done on the two transputers with each of the subsets. From the two arising assessments of trajectories the worst one is taken into consideration for calculating the predicates. It is true the trajectories of the AMV have to be extrapolated twice but since there is' little data exchange between the two transputers this kind of parallelization could be done very easily and the legibility of the program was kept.
0' 0' 0 -,
-, - ,
o o t -.
~
.........t
..I=-
Supporting Work, A PC based motorway simulator has been developed to test the module BD. Up to six user controlled vehicles slDTounding the AMV can be simulated. Employing either two joysticks or the keyboard as input device the user can easily compose complex traffic situations. The modules VC, ODT, and RDT are simulated on the PC. It is connected to a small transputer system on which module DDB and module BD are running using the same code as on the transputer system of the AMV. So once the code is tested and debugged with help of the simulator it can be run in the AMV without modification. Fig. 2 shows a screenshot of the simulator.
Fig. 2. Screenshot of the PC based motorway simulator. On the left side of the screen the graph of the STS is shown. It is possible to switch over to show the extrapolated trajectories. to show the messages sent from module BD. or to visualize the interface protocol between module BD and module VC. On the right side of the screen there are seven speedometers corresponding to seven vehicles including the AMV on the simulated motorway in the middle of the screen. Since it is zoomed in on the AMV here only the AMV and two of the slDTounding vehicles are visible. In the window on the lower right side of the screen the CPU utilization of the PC is shown. If it is less than 100%. the simulation is running in real time.
3. EXPERIMENTAL RESULTS
For safety reasons the user had to acknowledge actions triggered by module BD during the practical experiments.
Mter integrating module BD into VaMoRs (which was the predecessor of VaMoRs-P) overtakings triggered by decreasing the velocity of the vehicle driving in front of the AMV from 60 km/h down to 40 km/h were demonstrated. Experiments took place at the test area close to our university. Cancelling lane changing could also be demonstrated in practice: The vehicle which was driving in front of the AMV decreased its velocity so module BD triggered module VC to change lane. When the AMV was approaching the new lane, module ODT detected another vehicle driving on this lane. The second vehicle also decreased its velocity so module BD cancelled lane changing.
3.1 Criticism Though module BD showed its worth in different traffic situations. extensive simulation proved the method of assessing extrapolated trajectories to be too optimistic about lane changing on some rare occasions. A second algorithm for assessing different manoeuvres of the AMV was designed to overcome this difficulty. It can be used alternatively to or in combination with the method of assessing extrapolated trajectories.
When module BD was integrated in VaMoRs-P in autumn 1994, the whole system was able to detect the necessity of an overtaking as well as to perfonn it autonomously. During the Prometheus Board Member Meeting in October 1994 this was demonstrated in real traffic situations on the highway Al north of Paris achieving velocities up to 130 km/h.
Method of occupied areas. The second method of detennining the measure of safety of different possible manoeuvres of the AMV is quite simple. Three rectangular areas around the AMV arranged as shown in fig. 3 are considered. The left and right boundaries of the areas Aj correspond to the lane markings. The distance from the AMV to the front
391
4. CONCLUSION
8
e: ...
I---
~
n =. 0
I---
Ai
Experimental results with module BD, especially those gathered in Paris, were very satisfactory. But it has to be checked in what cases the method of occupied areas is sufficient. in what cases the method of extrapolating trajectories is more advantageous, and whether there are situations where a combination of both methods is required.
A2
A3
::l
0
...... 50
~
~
Furthermore the assessment of the behaviour of the surrounding vehicles will be refined. E. g. trafficators will be taken into account and more information about the whole traffic situation will be used.
~
Sln
All further improvements and refinements will have to show their usefulness in practice in the seeing passenger car VaMoRs-P.
Fig. 3. Areas around the AMV. and the back boundaries of the areas A ' is calculated for each of the surrounding vehicles separately. If the considered vehicle Vi is in front of the AMV like shown in fig. 3, the back boundaries of the areas k correspond to the back of the AMY, the distance fron{ the AMY to the front boundaries is calculated for each of the areas Aj separately as the maximum of - a given minimum , the safe distance depending on the velocity of the AMY,and the safe distance taking into account the difference of the AMY's and V;'s velocities as well as the time the AMY needs for changing to the corresponding lane. If the considered vehicle Vi is behind the AMV, the front boundaries of the areas A j correspond to the frontage of the AMV, the distance from the AMV to the back boundaries is calculated as the maximum of a given minimum, the safe distance depending on the velocity of the vehicle Vi, and the safe distance taking into account the difference of the AMV's and V;'s velocities as well as the time the AMV needs for changing to the corresponding lane. An area Aj is considered to be occupied, if there is a vehicle Vi partially or completely within the area A '. If the area Ai (A 3) is occupied, lane changing to the )eft (right) lane is not allowed or cancelling changing to the left (right) lane is necessary. Area A2 is used for calculating whether lane changing can be cancelled safely. Lane changing of the surrounding vehicles VI . IS modelled by extending their width to the lane they are changing to.
v:
This research has been supported by BMFT and DaimlerBenz AG under the EUREKA-Projekt Prometheus III (Grant ITW 9301A).
REFERENCES Behringer, R. (1994). Road Recognition from Multifocal Vision. In: Proceedings of the Intelligent Vehicles '94 Symposium, pp. 302-307. Paris, France. Dickmanns, D. (1995). Knowledge Based Real Time Vision. (this volume) Dickmanns, D. (J994a). The Dynamic Database User's Guide . Prometheus PRO-ART, InformatikForschungsgruppe, UniBw Miinchen. Germany. Dickmanns, E. D., R. Behringer, D. Dickmanns, T. Hildebrandt, M. Maurer, E Thomanek and J. Schiehlen (1994). The Seeing Passenger Car VaMoRs-P. In: Proceedings of the Intelligent Vehicles '94 Symposium, pp. 68-73. Paris, France. Maurer, M., R. Behringer, D. Dickmanns, T. Hildebrandt. E Thomanek, J. Schiehlen and E. D. Dickmanns (1994). VaMoRs-P. An advanced platform for visual autonomous road vehicle guidance. In: Proceedings of the SPlE Conference 'Mobile Robots lX' , pp. 239-248. Boston. USA. Mitschke, M. (1982). Dynamik der KraftJahrzeuge . Band A Antrieb und Bremsung. Springer. Berlin, Heidelberg. New York. Schiehlen, J. and E. D. Dickmanns (1994). A Camera Platform for Intelligent Vehicles. In: Proceedings of the Intelligent Vehicles '94 Symposium, pp. 393-398. Paris. France. Thomanek, E. E. D. Dickmanns and D. Dickmanns (1994). Multiple Object Recognition and Scene Interpretation for Autonomous Road Vehicle Guidance. In: Proceedings of the Intelligent Vehicles '94 Symposium. pp. 231-236. Paris, France.
The method of occupied areas is very fast and it can reduce the cycle time of module BD significantly, if it is used instead of the method of extrapolating trajectories.
392