- Email: [email protected]
System Components for Driver Support by Evaluation of Image Sequences
Copyright © IFAC Advances in Automotive Control, Ascona, Switzerland, 1995
SYSTEM COMPONENTS FOR DRIVER SUPPORT BY EVALUATION OF IMAGE SEQUENCES
w.
Enkelmann, V. Gengenbach, W. Kriiger, H.-H. Nagel, G. Nirschl, M. Palmer, S. ROssle, W. Tolle Fraunhofer-Institut fiir Informations- und Datenverarbeitung (IITB) Fraunhoferstr. 1, D-76131 Karlsruhe, Germany Phone: +49 721 6091-274, Fax: +49 721 6091-413, e-mail: [email protected]
Abstract. Based on experiences by commissioning two experimental vehicles, a system architecture for investigating driver assistance approaches is described. H a road vehicle is equipped with a video camera, machine vision approaches can provide knowledge about the actual traffic situation around the vehicle, e.g. obstacles. Taking into consideration the complexity of traffic situations, a modelling approach for traffic scenes is necessary which is based on the instantiation of generic situation schemata within an interpretation cycle by utilizing sensor information about dri ver, vehicle and environment .
Key Words. Autonomous guided vehicles, driver support, real-time image sequence analysis, obstacle detection, modelling of traffic situations
1. INTRODUCTION
• • • •
If a road vehicle is equipped with a video camera, machine vision approaches can provide knowledge about the actual traffic environment around the vehicle. Experiences gathered with a van (MB 609D, see Figure 1) during driving experiments on public roads using image sequence analysis in combination with knowledge from a digital road map are reported in (Nagel et al., 1995).
route planning, situation modelling, image sequence evaluation functions, vehicle control and driver warning.
The most important aspects of these system components will be discussed in more detail.
2. DESCRIPTION OF A DRIVING MISSION
An overall model of all relevant aspects of driving a vehicle on public roads is tentatively decomp
Based on these experiences, a second vehicle for automatic driving - a sedan (BMW 735 iL, see Figure 1) - has been designed and commissioned to investigate approaches for driver assistance on public roads. A driver assistance system needs a representation of all aspects of a traffic situation, e.g. the state of the driver, the state of its own car, and the state of the environment. Suitable methods and tools for the representation, inspection and evaluation of system-internal situation models are being developed.
• • • •
a top MISSION MASTER level, a NAVIGATION level, a MANEUVER level and a CONTROL level, at which the variOUS loops for lateral and longitudinal control are executed in order to keep the vehicle on a collision-free course along the road.
The highest or MISSION MASTER level of an automatic FhG-C
Our current system architecture for the overall procedure (Struck et al., 1993; Enkelmann et al., 1993) consists of the following components (see Fig. 2): • man-machine-interaction, 215
notion "situation" includes
red by sub-missions, expected times and durations for breaks, preferences regarding, for example, the shortest , the fastest, or the most scenic route, etc .. These boundary conditions are processed by the mission and route planning component which has access to a complete road map for the area in question. A fraction of these services are already offered, e. g., by the Travelpilot® . Such a planning component submits details of the planned trip to the driver who then will either accept the resulting plan or suggest modifications in the form of boundary conditions or preferences not mentioned so far . The trip plan - after having been accepted by the driver - forms the basis for updating steps necessitated by changes in the goals or preferences of the driver during the trip or by external developments such as traffic jams or enforced detours which are either perceived via vehicle-based sensors or announced via messages transmitted by beacons or broadcast services to the FhG-Co-Driver.
• a state description of all situation relevant objects, • the admissible sequences of actions and • alternatives for the temporal development of the current situation. The state description refers to the relevant objects and their mutual relations at a given time instance. If the object relations at a given point in time are not only determined by the dynamics of object motion but also by considering additional degrees of freedom, we change from a static object description to a description of agents. In the context of modelling traffic situations, the notion 'agent ' expresses the fact that state changes of objects are not only determined by physical relations but also by the intentions of the agents. The action aspect takes into account the performed and the potential actions of the agents. The prediction aspect covers the temporal development of the current situation.
The actual version of the planned route provided by the Mission Master level determines the operations at the next lower system level, the NAVIGATION level. Based on the same road map used for route planning, a combination of odometry, verification of expected route characteristics such as sharp turns, and consistency with route characteristics extracted by machine vision such as location and type of intersections encountered, provides the means to decompose the planned route into subsections or 'legs' of the road travel. Driving the vehicle along a subsection will be considered as a task, for example following Route A from location B (say, the exit of an intersection) to location C (say, the next switch to a road of different type or number) . Driving a leg will consist in a series of maneuvers which are selected on the basis of a combination of the planned route and the actual traffic situation around the vehicle equipped with a FhG-Co-Driver.
To model sequences of situations (Kriiger, 1991) developed situation graphs on the basis of conceptual graphs (Sowa, 1984). A situation graph consists of situation nodes and prediction edges (see Figure 3). Each situation node represents a class of situations and contains a generic state schema which will be instantiated in the current situation . Moreover, a situation node contains an action schema which will be instantiated, too, in the current traffic situation . It determines the action corresponding to the current situation. Action schemata are parameterized commands to be executed by subsystems for human-machineinteraction or for sensor and vehicle control. A prediction edge from situation node sI to situation node s2 states that s2 is a possible successor of sI , i.e. the situations represented by node s2 are possible successors of the situations represented by node sI. Each prediction edge is marked with a priority where smaller numbers indicate higher priorities.
3. MODELLING OF TRAFFIC SITUATIONS
Situation recognition, i.e. the mapping of situation graphs to image sequences, is done by graph traversing and is based on mapping state schemata - contained in the graph 's situation nodes - to images. This mapping extends over several layers of abstraction, leading from the two-dimensional image and the three-dimensional scene represen-
Taking into consideration the complexity of traffic situations, (Nagel , 1988a) proposed a modelling approach for traffic scenes which is based on the instantiation of generic situation schemata within an interpretation cycle by utilizing sensor information about driver , vehicle and environment. The 216
1. Calculate optical flow vectors u from the image sequence. 2. Estimate model vectors UM that describe the expected image motion . 3. Evaluate the differences UD between calculated optical flow vectors U and estimated model vectors UM .
tation to the conceptual level (Figure 4) . The top layer is the traversing layer, having the task to traverse the given situation graph . To do this, the traversing layer needs services from the subordinated projection layer . The projection layer receives state schemata from the traversing layer. It's task is to find projections of these schemata onto projection graphs, which have to be built from sensorial data. To construct these graphs, the projection layer needs support from more sensororiented layers. In Figure 4, these layers are joined to a common layer, called evaluation layer. The task of this layer is to evaluate those types and relations which the projection layer regards as primitive. For primitive types and relations, the evaluation layer must have algorithms to evaluate these types and relations, i.e. to decide whether a scene domain individual is of a given type or whether a given relation holds between such individuals. Since the projection goal is given by the situation graph, only those evaluation functions have to be executed which are relevant to perform the actions corresponding to the current situation.
To detect objects the calculated optical flow vectors are compared with the corresponding model vectors which explicitly describe the expected image shift of a projected scene point of the road plane. The camera position is calibrated with respect to the road plane. The efficiency of this obstacle detection approach has been demonstrated (Enkelmann, 1991) using different algorithms for the calculation of optical flow vectors. The ego-motion of the camera has been obtained from sensors of our experimental vehicles (see Fig. 1) to determine the necessary model vectors. The optical flow vectors are calculated with a local analytical approach (Schnorr, 1991) . The necessary first and second order spatio-temporal derivatives of the gray value function are computed in video real-time with the hardware module OZA (german abbreviation of spatio-temporal derivatives) of our image sequence analysis system MiniVISTA (Gengenbach et al. , 1994) which is installed in the trunk of our experimental car BMW 735 iL (see Figure 5) . The spatio-temporal derivatives at each pixel have to pass a structure test in order to make sure that the underlying gray value distribution exhibits sufficient structure to locally estimate optical flow vectors. Only those sets of spatio-temporal derivatives which passed the structure test are transmitted to subsequent processes being executed - in the current version - on a transputer network . These processes calculate the optical flow vectors, compute the expected model vectors and perform the obstacle test .
An extension of these plain situation graphs to a hierarchical version is described in (Enkelmann, 1993). In the hierarchical situation graph the modelled situations are organized in a specialization hierarchy. This reduces the size of state schemata by inheritance from more general schemata and provides an automatic exception handling for situation graph traversing by falling back on more general situations if necessary.
4. OBSTACLE DETECTION In (Enkelmann et al., 1994) an approach for the detection of stationary obstacles and moving objects in front of a vehicle is presented . The detection of obstacles and other objects is based on the evaluation of optical flow because the relative motion between camera and the recorded scene can be calculated from reliably estimated optical flow vectors. Furthermore, optical flow vectors comprise information about the three-dimensional structure of the recorded scene.
Based on the realization of the hardware module for the computation of spatio-temporal derivatives within the image sequence analysis system MiniVISTA our approach for obstacle detection has been investigated with image sequences taken from cameras which were mounted in different heights in different vehicles (Enkelmann et al., 1994).
The basic procedure used to detect obstacles in image sequences recorded and evaluated in an experimental car consists of the following three steps (Enkelmann,1991) : 217
too fast approaches.
5. A DRIVER'S WARNING ASSISTANT
In order to correctly generate the necessary warning messages, the time requirements in the current traffic situation have to be analysed taking into account the available time as well as the minimum time needed by the driver and the DWA for their interaction . These time requirements are due to the tasks and properties of the driver, of the vehicle, and of the environment . The information is evaluated in a time-budget analysis (Geiser and Nirschl, 1991). With respect to warnings the following points are of special interest (see Figure 6) :
In contrast to approaches operating in a well structured environment such as motorways to drive automatically (Dickmanns et al., 1993; Ulmer, 1994) or to monitor the individual driver 's current behaviour (Kopf and Onken, 1992) , we started investigations of maneuvers (Nagel and Enkelmann, 1991) in a more complex scenario, i.e. inner city traffic situations. Furthermore, a driver's warning assistant was designed to support the driver in actual traffic situations. A driver's warning assistant (DWA) is an automatic co-pilot system that presents warning messages to a driver instead of acting directly upon the vehicle or the environment. (Nagel, 1994) presented a systematic structure for different types of sensor-based driver support systems in road vehicles.
earliest warning position: Point at which a driver normally starts braking. Drivers passing this point without braking will receive a pre-warning. latest warning position: Latest point at which a driver is able to stop in time when reacting to a generated warning message with maximum deceleration . Note that this has to take into account the driver 's reaction time. latest action onset position: Latest point at which an emergency stop will have been initiated still in time.
A crucial topic in the development of a DWA is the timing of warning messages for the driver. On the one hand the moment of warning has to be chosen early enough to enable the driver to react appropriately. This means that the inevitable reaction time which the driver needs to perceive and process the DWA 's warning message and to prepare a reaction has to be considered. On the other hand, the warning message should not be presented too early. A warning message given before the driver normally acts or reacts in a traffic situation risks to present information to the driver which he is already aware of. This would be annoying for the driver and would result in poor acceptance of the DWA.
The position of these points depends on the vehicle speed and the driver 's individual braking parameters, i.e. normal deceleration , maximum deceleration, and reaction time. Experimental results obtained with a DWA implemented in a passenger car (BMW 735 iL) which has been equipped with on board facilities to evaluate image sequences and vehicle data in real-time on a transputer network were discussed in (Enkelmann et al. , 1993) .
(Geiser and Nirschl, 1991) developed a system architecture of a DWA where driver modelling and warning message generation has been embedded within a comprehensive approach for the representation and recognition of traffic situations (Kriiger, 1991). The driver model component of their system architecture consist of a driver 's state model , a driver 's intent model, a normative driver model as well as a resource model.
6. CONCLUSIONS
One major difference between motorway and inner city traffic situations is the occurrence of intersections in inner city traffic situations. Therefore, we started investigations of approaching an intersection (Figure 6) , in order to study the feasibility of
Our system architecture presented so far provides an experimental platform for studying drivervehicle interactions with the option of automatically evaluating actual traffic situations around the vehicle in real-time. Our equipment thus offers an approach to record and disentangle the multitude of factors which influence the - often subconscious - reactions of a driver . It is our working hypothesis that only an automatic thorough
a DWA's functions and the acceptance of DWA 's warning messages by a human driver in case of
understanding of the actual traffic situation facilitates the design of a driver support system which 218
Kriiger , Wolfgang (1991) . Begriffsgraphen zur Situationsmodellierung in der Bildfolgenauswertung. Dissertation . Fakultiit fiir Informatik , Universitiit Karlsruhe (TH) . Erschienen in : Informatik-Fachberichte 311, Situationsmodellierung in der Bildfolgenauswertung, SpringerVerlag, Berlin Heidelberg, 1992. Nagel, H.-H . (1988a) . From Image Sequences towards Conceptual Descriptions. Image and Vision Computing 6(2), 59-74 . Nagel, H.-H. (1988b) . Wodurch zeichnet si ch ein "umsichtiges" Automobil aus? oder Zur Frage der impliziten und expliziten Reprasentation von Wissen in maschinellen Ko-Piloten fur den Stra6enverkehr. In: Elektronik im Kraftfahrzeug, Baden-Baden, 8.-9. September 1988. VDI-Berichte 687 . VDI Verlag Diisseldorf, 1988. pp. 25-40. Nagel, H.-H . (1994). AI Approaches towards Sensor-Based Driver Support in Road Vehicles. In: KI-94 : Advances in Artificial Intelligence, 18th German Annual Conference on Artificial Intelligence, Saarbriicken, Germany, September 18-29, 1994, B. Ne bel, L. Dreschler-Fischer (eds.), Lecture Notes in Artificial Intelligence 861, Springer- Verlag Berlin Heidelberg 1994. pp. 1-15. Nagel, H.-H . and W . Enkelmann (1991) . Generic Road Traffic Situations and Driver Support Systems. In: Proc. 5th PROMETHEUS Workshop, Munich/Germany, October 15-16, 1991. pp. 76-85. Nagel, H.-H ., W . Enkelmann and G . Struck (1995). FhG-Co-Driver: From Map-Guided Automatic Driving by Machine Vision to a Cooperative Driver Support. In: Special Issue of Computers and Mathematics with Applications, ID press. Schnorr, Christoph (1991). Funktionalanalytische Methoden zur Gewinnung von Bewegungsinformation aus TV-Bildfolgen. Dissertation . Fakultiit fiir Informatik, Universitiit Karlsruhe (TH). Sowa, J .F . (1984) . Conceptual Structures - Information Processing in Mind and Machine . Addison Wesley, Reading/MA. Struck, G ., J. Geisler, F. Laubenstein , H.-H. Nagel and G . Siegle (1993) . Interaction between Digital Road Map Systems and Trinocular Autonomous Driving. In : Intelligent Vehicles '93 Symposium, July 14-16, 1993, Tokyo, Japan. pp . 461-466 . Ulmer, B. (1994) . VITA 11 - Active Collision Avoidance in Real Traffic. In: Intelligent Vehicles '94 Symposium, October 24-26, 1994, Paris, France. pp . 1-6.
is competent and flexible enough to be accepted by a wide spectrum of users.
7. REFERENCES Dickmanns, E . D., R. Behringer, C. Briidigam, D. Dickmanns, F. Thomanek and V. v. Holt (1993). An AII-Transputer Visual AutobahnAutopilot/Copilot . In: Fourth International Conference on Computer Vision - ICCV '99, May 11-1,., 1999, Berlin, Germany, IEEE Computer Society Press, Los Alamitos/CA, 1999. pp. 608-615 . Enkelmann, W. (1991) . Obstacle Detection by Evaluation of Optical Flow Fields from Image Sequences. Image and Vision Computing 9(3) , 160-168. Enkelmann, W . (1993) . Konzeption und Nutzung von Wissensreprasentationen bei videogestiitzten Assistenzsystemen fiir die Stra6enfahrzeugfiihrung . In: 17. Fachtagung fur Kunstliche Intelligenz, Humboldt- Universitcit, Berlin, 19.-16. September 1999, O. Herzog, Th. Christaller, D. Schiitt (Hrsg.), Grundlagen und Anwendungen der Kiinstlichen Intelligenz, SpringerVerlag Berlin Heidelberg 1999. pp. 74-89. Enkelmann, W., G . Nirschl, V. Gengenbach, W . Kriiger , S. ROssle and W. Tolle (1993) . Realization of a Driver's Warning Assistant for Intersections. In: Intelligent Vehicles '99 Symposium, July 1,.-16, 1999, Tokyo, Japan. pp . 7277. Enkelmann, W ., V. Gengenbach, W . Kriiger, S. ROssle and W . Tolle (1994) . Obstacle Detection by Real-Time Optical Flow Evaluation. In: Intelligent Vehicles '94 Symposium, October 24-26, 1994, Paris, France . pp . 97-102 . Geiser, G . and G. Nirschl (1991). Towards a System Architecture of a Driver's Warning Assistant with Special Consideration of the TimeBudget of Driver-Vehicle Interactions. In: Proc. of the PROMETHEUS PRO-ART Workshop on Intelligent Co-Pilot, Grenoble, France , Dec. 12-19, 1991. pp. 259-27l. Gengenbach, V., M. Palmer and H. Ringshauser (1994) . MiniVISTA - Ein echtzeitf
..
~ .
: .ituatiOD
~phI
!ravening lay...
•
projectiOD of IIIIe ocbemata
projoction
lay ...
eVlh&atiOD at primitive typea ODd relatiODl
UuFODd
"""'" dcmain
Fig. 1. IITB's experimental vehicles of type MB 609D and BMW 735 iL.
evalualioo 1aya(1)
modell
Fig. 4. Matching of state schemata to sensor data.
Fig. 5. Rack mount ed in the trunk of our experimental car BMW 735 iL with MiniVISTA, transputer network, interfaces for sensors and actuators, and power supply.
Fig. 2. System architecture of an autopilot / co-pilot .
si tuition gnpb situltion node
/ action adlana
Iltest ICtion OGle! position
Iltest woming position
.tate schema
warning
rU~t
nonnalllCtioo onJCt positioo - c:arlicst warning position
Fig . 3. Structure of situation graph . Fig . 6. Approaching an intersection.
220
SYSTEM COMPONENTS FOR DRIVER SUPPORT BY EVALUATION OF IMAGE SEQUENCES
w.
Enkelmann, V. Gengenbach, W. Kriiger, H.-H. Nagel, G. Nirschl, M. Palmer, S. ROssle, W. Tolle Fraunhofer-Institut fiir Informations- und Datenverarbeitung (IITB) Fraunhoferstr. 1, D-76131 Karlsruhe, Germany Phone: +49 721 6091-274, Fax: +49 721 6091-413, e-mail: [email protected]
Abstract. Based on experiences by commissioning two experimental vehicles, a system architecture for investigating driver assistance approaches is described. H a road vehicle is equipped with a video camera, machine vision approaches can provide knowledge about the actual traffic situation around the vehicle, e.g. obstacles. Taking into consideration the complexity of traffic situations, a modelling approach for traffic scenes is necessary which is based on the instantiation of generic situation schemata within an interpretation cycle by utilizing sensor information about dri ver, vehicle and environment .
Key Words. Autonomous guided vehicles, driver support, real-time image sequence analysis, obstacle detection, modelling of traffic situations
1. INTRODUCTION
• • • •
If a road vehicle is equipped with a video camera, machine vision approaches can provide knowledge about the actual traffic environment around the vehicle. Experiences gathered with a van (MB 609D, see Figure 1) during driving experiments on public roads using image sequence analysis in combination with knowledge from a digital road map are reported in (Nagel et al., 1995).
route planning, situation modelling, image sequence evaluation functions, vehicle control and driver warning.
The most important aspects of these system components will be discussed in more detail.
2. DESCRIPTION OF A DRIVING MISSION
An overall model of all relevant aspects of driving a vehicle on public roads is tentatively decomp
Based on these experiences, a second vehicle for automatic driving - a sedan (BMW 735 iL, see Figure 1) - has been designed and commissioned to investigate approaches for driver assistance on public roads. A driver assistance system needs a representation of all aspects of a traffic situation, e.g. the state of the driver, the state of its own car, and the state of the environment. Suitable methods and tools for the representation, inspection and evaluation of system-internal situation models are being developed.
• • • •
a top MISSION MASTER level, a NAVIGATION level, a MANEUVER level and a CONTROL level, at which the variOUS loops for lateral and longitudinal control are executed in order to keep the vehicle on a collision-free course along the road.
The highest or MISSION MASTER level of an automatic FhG-C
Our current system architecture for the overall procedure (Struck et al., 1993; Enkelmann et al., 1993) consists of the following components (see Fig. 2): • man-machine-interaction, 215
notion "situation" includes
red by sub-missions, expected times and durations for breaks, preferences regarding, for example, the shortest , the fastest, or the most scenic route, etc .. These boundary conditions are processed by the mission and route planning component which has access to a complete road map for the area in question. A fraction of these services are already offered, e. g., by the Travelpilot® . Such a planning component submits details of the planned trip to the driver who then will either accept the resulting plan or suggest modifications in the form of boundary conditions or preferences not mentioned so far . The trip plan - after having been accepted by the driver - forms the basis for updating steps necessitated by changes in the goals or preferences of the driver during the trip or by external developments such as traffic jams or enforced detours which are either perceived via vehicle-based sensors or announced via messages transmitted by beacons or broadcast services to the FhG-Co-Driver.
• a state description of all situation relevant objects, • the admissible sequences of actions and • alternatives for the temporal development of the current situation. The state description refers to the relevant objects and their mutual relations at a given time instance. If the object relations at a given point in time are not only determined by the dynamics of object motion but also by considering additional degrees of freedom, we change from a static object description to a description of agents. In the context of modelling traffic situations, the notion 'agent ' expresses the fact that state changes of objects are not only determined by physical relations but also by the intentions of the agents. The action aspect takes into account the performed and the potential actions of the agents. The prediction aspect covers the temporal development of the current situation.
The actual version of the planned route provided by the Mission Master level determines the operations at the next lower system level, the NAVIGATION level. Based on the same road map used for route planning, a combination of odometry, verification of expected route characteristics such as sharp turns, and consistency with route characteristics extracted by machine vision such as location and type of intersections encountered, provides the means to decompose the planned route into subsections or 'legs' of the road travel. Driving the vehicle along a subsection will be considered as a task, for example following Route A from location B (say, the exit of an intersection) to location C (say, the next switch to a road of different type or number) . Driving a leg will consist in a series of maneuvers which are selected on the basis of a combination of the planned route and the actual traffic situation around the vehicle equipped with a FhG-Co-Driver.
To model sequences of situations (Kriiger, 1991) developed situation graphs on the basis of conceptual graphs (Sowa, 1984). A situation graph consists of situation nodes and prediction edges (see Figure 3). Each situation node represents a class of situations and contains a generic state schema which will be instantiated in the current situation . Moreover, a situation node contains an action schema which will be instantiated, too, in the current traffic situation . It determines the action corresponding to the current situation. Action schemata are parameterized commands to be executed by subsystems for human-machineinteraction or for sensor and vehicle control. A prediction edge from situation node sI to situation node s2 states that s2 is a possible successor of sI , i.e. the situations represented by node s2 are possible successors of the situations represented by node sI. Each prediction edge is marked with a priority where smaller numbers indicate higher priorities.
3. MODELLING OF TRAFFIC SITUATIONS
Situation recognition, i.e. the mapping of situation graphs to image sequences, is done by graph traversing and is based on mapping state schemata - contained in the graph 's situation nodes - to images. This mapping extends over several layers of abstraction, leading from the two-dimensional image and the three-dimensional scene represen-
Taking into consideration the complexity of traffic situations, (Nagel , 1988a) proposed a modelling approach for traffic scenes which is based on the instantiation of generic situation schemata within an interpretation cycle by utilizing sensor information about driver , vehicle and environment. The 216
1. Calculate optical flow vectors u from the image sequence. 2. Estimate model vectors UM that describe the expected image motion . 3. Evaluate the differences UD between calculated optical flow vectors U and estimated model vectors UM .
tation to the conceptual level (Figure 4) . The top layer is the traversing layer, having the task to traverse the given situation graph . To do this, the traversing layer needs services from the subordinated projection layer . The projection layer receives state schemata from the traversing layer. It's task is to find projections of these schemata onto projection graphs, which have to be built from sensorial data. To construct these graphs, the projection layer needs support from more sensororiented layers. In Figure 4, these layers are joined to a common layer, called evaluation layer. The task of this layer is to evaluate those types and relations which the projection layer regards as primitive. For primitive types and relations, the evaluation layer must have algorithms to evaluate these types and relations, i.e. to decide whether a scene domain individual is of a given type or whether a given relation holds between such individuals. Since the projection goal is given by the situation graph, only those evaluation functions have to be executed which are relevant to perform the actions corresponding to the current situation.
To detect objects the calculated optical flow vectors are compared with the corresponding model vectors which explicitly describe the expected image shift of a projected scene point of the road plane. The camera position is calibrated with respect to the road plane. The efficiency of this obstacle detection approach has been demonstrated (Enkelmann, 1991) using different algorithms for the calculation of optical flow vectors. The ego-motion of the camera has been obtained from sensors of our experimental vehicles (see Fig. 1) to determine the necessary model vectors. The optical flow vectors are calculated with a local analytical approach (Schnorr, 1991) . The necessary first and second order spatio-temporal derivatives of the gray value function are computed in video real-time with the hardware module OZA (german abbreviation of spatio-temporal derivatives) of our image sequence analysis system MiniVISTA (Gengenbach et al. , 1994) which is installed in the trunk of our experimental car BMW 735 iL (see Figure 5) . The spatio-temporal derivatives at each pixel have to pass a structure test in order to make sure that the underlying gray value distribution exhibits sufficient structure to locally estimate optical flow vectors. Only those sets of spatio-temporal derivatives which passed the structure test are transmitted to subsequent processes being executed - in the current version - on a transputer network . These processes calculate the optical flow vectors, compute the expected model vectors and perform the obstacle test .
An extension of these plain situation graphs to a hierarchical version is described in (Enkelmann, 1993). In the hierarchical situation graph the modelled situations are organized in a specialization hierarchy. This reduces the size of state schemata by inheritance from more general schemata and provides an automatic exception handling for situation graph traversing by falling back on more general situations if necessary.
4. OBSTACLE DETECTION In (Enkelmann et al., 1994) an approach for the detection of stationary obstacles and moving objects in front of a vehicle is presented . The detection of obstacles and other objects is based on the evaluation of optical flow because the relative motion between camera and the recorded scene can be calculated from reliably estimated optical flow vectors. Furthermore, optical flow vectors comprise information about the three-dimensional structure of the recorded scene.
Based on the realization of the hardware module for the computation of spatio-temporal derivatives within the image sequence analysis system MiniVISTA our approach for obstacle detection has been investigated with image sequences taken from cameras which were mounted in different heights in different vehicles (Enkelmann et al., 1994).
The basic procedure used to detect obstacles in image sequences recorded and evaluated in an experimental car consists of the following three steps (Enkelmann,1991) : 217
too fast approaches.
5. A DRIVER'S WARNING ASSISTANT
In order to correctly generate the necessary warning messages, the time requirements in the current traffic situation have to be analysed taking into account the available time as well as the minimum time needed by the driver and the DWA for their interaction . These time requirements are due to the tasks and properties of the driver, of the vehicle, and of the environment . The information is evaluated in a time-budget analysis (Geiser and Nirschl, 1991). With respect to warnings the following points are of special interest (see Figure 6) :
In contrast to approaches operating in a well structured environment such as motorways to drive automatically (Dickmanns et al., 1993; Ulmer, 1994) or to monitor the individual driver 's current behaviour (Kopf and Onken, 1992) , we started investigations of maneuvers (Nagel and Enkelmann, 1991) in a more complex scenario, i.e. inner city traffic situations. Furthermore, a driver's warning assistant was designed to support the driver in actual traffic situations. A driver's warning assistant (DWA) is an automatic co-pilot system that presents warning messages to a driver instead of acting directly upon the vehicle or the environment. (Nagel, 1994) presented a systematic structure for different types of sensor-based driver support systems in road vehicles.
earliest warning position: Point at which a driver normally starts braking. Drivers passing this point without braking will receive a pre-warning. latest warning position: Latest point at which a driver is able to stop in time when reacting to a generated warning message with maximum deceleration . Note that this has to take into account the driver 's reaction time. latest action onset position: Latest point at which an emergency stop will have been initiated still in time.
A crucial topic in the development of a DWA is the timing of warning messages for the driver. On the one hand the moment of warning has to be chosen early enough to enable the driver to react appropriately. This means that the inevitable reaction time which the driver needs to perceive and process the DWA 's warning message and to prepare a reaction has to be considered. On the other hand, the warning message should not be presented too early. A warning message given before the driver normally acts or reacts in a traffic situation risks to present information to the driver which he is already aware of. This would be annoying for the driver and would result in poor acceptance of the DWA.
The position of these points depends on the vehicle speed and the driver 's individual braking parameters, i.e. normal deceleration , maximum deceleration, and reaction time. Experimental results obtained with a DWA implemented in a passenger car (BMW 735 iL) which has been equipped with on board facilities to evaluate image sequences and vehicle data in real-time on a transputer network were discussed in (Enkelmann et al. , 1993) .
(Geiser and Nirschl, 1991) developed a system architecture of a DWA where driver modelling and warning message generation has been embedded within a comprehensive approach for the representation and recognition of traffic situations (Kriiger, 1991). The driver model component of their system architecture consist of a driver 's state model , a driver 's intent model, a normative driver model as well as a resource model.
6. CONCLUSIONS
One major difference between motorway and inner city traffic situations is the occurrence of intersections in inner city traffic situations. Therefore, we started investigations of approaching an intersection (Figure 6) , in order to study the feasibility of
Our system architecture presented so far provides an experimental platform for studying drivervehicle interactions with the option of automatically evaluating actual traffic situations around the vehicle in real-time. Our equipment thus offers an approach to record and disentangle the multitude of factors which influence the - often subconscious - reactions of a driver . It is our working hypothesis that only an automatic thorough
a DWA's functions and the acceptance of DWA 's warning messages by a human driver in case of
understanding of the actual traffic situation facilitates the design of a driver support system which 218
Kriiger , Wolfgang (1991) . Begriffsgraphen zur Situationsmodellierung in der Bildfolgenauswertung. Dissertation . Fakultiit fiir Informatik , Universitiit Karlsruhe (TH) . Erschienen in : Informatik-Fachberichte 311, Situationsmodellierung in der Bildfolgenauswertung, SpringerVerlag, Berlin Heidelberg, 1992. Nagel, H.-H . (1988a) . From Image Sequences towards Conceptual Descriptions. Image and Vision Computing 6(2), 59-74 . Nagel, H.-H. (1988b) . Wodurch zeichnet si ch ein "umsichtiges" Automobil aus? oder Zur Frage der impliziten und expliziten Reprasentation von Wissen in maschinellen Ko-Piloten fur den Stra6enverkehr. In: Elektronik im Kraftfahrzeug, Baden-Baden, 8.-9. September 1988. VDI-Berichte 687 . VDI Verlag Diisseldorf, 1988. pp. 25-40. Nagel, H.-H . (1994). AI Approaches towards Sensor-Based Driver Support in Road Vehicles. In: KI-94 : Advances in Artificial Intelligence, 18th German Annual Conference on Artificial Intelligence, Saarbriicken, Germany, September 18-29, 1994, B. Ne bel, L. Dreschler-Fischer (eds.), Lecture Notes in Artificial Intelligence 861, Springer- Verlag Berlin Heidelberg 1994. pp. 1-15. Nagel, H.-H . and W . Enkelmann (1991) . Generic Road Traffic Situations and Driver Support Systems. In: Proc. 5th PROMETHEUS Workshop, Munich/Germany, October 15-16, 1991. pp. 76-85. Nagel, H.-H ., W . Enkelmann and G . Struck (1995). FhG-Co-Driver: From Map-Guided Automatic Driving by Machine Vision to a Cooperative Driver Support. In: Special Issue of Computers and Mathematics with Applications, ID press. Schnorr, Christoph (1991). Funktionalanalytische Methoden zur Gewinnung von Bewegungsinformation aus TV-Bildfolgen. Dissertation . Fakultiit fiir Informatik, Universitiit Karlsruhe (TH). Sowa, J .F . (1984) . Conceptual Structures - Information Processing in Mind and Machine . Addison Wesley, Reading/MA. Struck, G ., J. Geisler, F. Laubenstein , H.-H. Nagel and G . Siegle (1993) . Interaction between Digital Road Map Systems and Trinocular Autonomous Driving. In : Intelligent Vehicles '93 Symposium, July 14-16, 1993, Tokyo, Japan. pp . 461-466 . Ulmer, B. (1994) . VITA 11 - Active Collision Avoidance in Real Traffic. In: Intelligent Vehicles '94 Symposium, October 24-26, 1994, Paris, France. pp . 1-6.
is competent and flexible enough to be accepted by a wide spectrum of users.
7. REFERENCES Dickmanns, E . D., R. Behringer, C. Briidigam, D. Dickmanns, F. Thomanek and V. v. Holt (1993). An AII-Transputer Visual AutobahnAutopilot/Copilot . In: Fourth International Conference on Computer Vision - ICCV '99, May 11-1,., 1999, Berlin, Germany, IEEE Computer Society Press, Los Alamitos/CA, 1999. pp. 608-615 . Enkelmann, W. (1991) . Obstacle Detection by Evaluation of Optical Flow Fields from Image Sequences. Image and Vision Computing 9(3) , 160-168. Enkelmann, W . (1993) . Konzeption und Nutzung von Wissensreprasentationen bei videogestiitzten Assistenzsystemen fiir die Stra6enfahrzeugfiihrung . In: 17. Fachtagung fur Kunstliche Intelligenz, Humboldt- Universitcit, Berlin, 19.-16. September 1999, O. Herzog, Th. Christaller, D. Schiitt (Hrsg.), Grundlagen und Anwendungen der Kiinstlichen Intelligenz, SpringerVerlag Berlin Heidelberg 1999. pp. 74-89. Enkelmann, W., G . Nirschl, V. Gengenbach, W . Kriiger , S. ROssle and W. Tolle (1993) . Realization of a Driver's Warning Assistant for Intersections. In: Intelligent Vehicles '99 Symposium, July 1,.-16, 1999, Tokyo, Japan. pp . 7277. Enkelmann, W ., V. Gengenbach, W . Kriiger, S. ROssle and W . Tolle (1994) . Obstacle Detection by Real-Time Optical Flow Evaluation. In: Intelligent Vehicles '94 Symposium, October 24-26, 1994, Paris, France . pp . 97-102 . Geiser, G . and G. Nirschl (1991). Towards a System Architecture of a Driver's Warning Assistant with Special Consideration of the TimeBudget of Driver-Vehicle Interactions. In: Proc. of the PROMETHEUS PRO-ART Workshop on Intelligent Co-Pilot, Grenoble, France , Dec. 12-19, 1991. pp. 259-27l. Gengenbach, V., M. Palmer and H. Ringshauser (1994) . MiniVISTA - Ein echtzeitf
..
~ .
: .ituatiOD
~phI
!ravening lay...
•
projectiOD of IIIIe ocbemata
projoction
lay ...
eVlh&atiOD at primitive typea ODd relatiODl
UuFODd
"""'" dcmain
Fig. 1. IITB's experimental vehicles of type MB 609D and BMW 735 iL.
evalualioo 1aya(1)
modell
Fig. 4. Matching of state schemata to sensor data.
Fig. 5. Rack mount ed in the trunk of our experimental car BMW 735 iL with MiniVISTA, transputer network, interfaces for sensors and actuators, and power supply.
Fig. 2. System architecture of an autopilot / co-pilot .
si tuition gnpb situltion node
/ action adlana
Iltest ICtion OGle! position
Iltest woming position
.tate schema
warning
rU~t
nonnalllCtioo onJCt positioo - c:arlicst warning position
Fig . 3. Structure of situation graph . Fig . 6. Approaching an intersection.
220