- Email: [email protected]
Interpretation of Traffic Scenes by Evaluation of Optical Flow Fields from Image Sequences
Copn'ig ht © I F:\C Co ntrol. COIl1» lil l'I'S. ( :Ollllllllllicatiol1s ill ' l 'r ;1I1 Sp0!"1 ;uioll,
SYSTH!S L:SI:--JC Ne W SENSO R
Pari s, Frall ce, 1 9X~1
INTERPRET A TION OF TRAFFIC SCENES BY EVALUATION OF OPTICAL FLOW FIELDS FROM IMAGE SEQUENCES W. Enkelmann Fmllll!w/iT- f ll.,lilul jiir f ll/imll(l li()IIS- III/{! [) 1I11'1I< 'I'I'(lrill'illlllg (ffTfi ) F mllll!w!I' nlmfJl' f. D-15 ()() }\(lr/.,nilll' f. FR(;
Abstract. TV cameras are very useful to record traffic scenes due to their high resolution compared to their size and costs. In comparison with single images , image sequ ences comprise information a bout the dynamic aspects of the recorded scene. Dynamic aspects play a crucial role in traffic scenarios. Traffic scenes with different complexity can be characterized by the structure of the scene. Depending on the structure of the sce ne different tas ks co uld be so lved by image seq uence evaluation . In this co ntribution two approaches for the evaluation of opti cal flow fields from image sequences are discussed. The first approach describes th e detection and tracking of moving objects on a complex street intersection. The second a pproac h presents an algorithm for the detection of obstacles on the road of an a utonomous vehicle . It will be shown how two-dimensiona l optical flow fi elds fi e lds can be interpreted to infer informati on a bout th e three-dimensional environment. Results obtained from investigations with real world image sequences will be presented . Keywords. Image sequence analysis, optical flow, object tracking, obstacle detection
If image sequences are taken from a moving camera,
INTRODUCTION
e.g. mounted on a robot's arm or on an a uton omous vehicle, the object detection task becomes much more difficult. But, furthermore, a new class of problems arise which could be solved by image sequence analysis due to the ego-motion of the camera. Some examples will be discussed in the following sections, e.g.
The evaluation of traffic scenes by image sequence analysis is a task with increasing attention by traffic experts as well as by robot a nd car ma nufacturers. One reason might be the progress in image sequence analysis during the last decade (N age l 86, Pa v lidis 86) which made this research field more attractive for potential applications. In compariso n with single images, image sequences comprise information about the dynamic aspects of the recorded scene. Dynamic aspects pl ay a crucial role in traffic scenarios. Traffic scenes with different complexity can be characterized by the
•
detection and tracking of objects mov ing relative to the recording camera as well as to the stationary environment,
•
the estimation a nd eva lu atio n of the relative displaceme nt betwee n an a utono mous ve hicle and
structure of the scene . Depending on the structure of the scene, different tasks could be solved by image sequence evaluation.
stationary parts of the sce ne as we ll as between other mov ing objects. •
estimation of motion par amete rs relative to the camera or the environment,
•
detection of stationary obstacles.
Let us assume the three-dimensional scene will be recorded with a stationary camera. In this case the movement of objects in the scene results in temporal variations in the image sequence. One task of image sequence analysis is the detection and evaluation of
Whenever dynamic aspects of image sequences have to be evaluated it is much easier if a reliable estimation of optical flow fields is available. An optical flow field u(x,t) maps the gray value g(x-utH,t-tlt) recorded at time t-tlt at the image plane location x-utlt into the gray value g(x,t) recorded at location x at time t. In
temporal variations in image sequences in order to provide information about the objects in the recorded scene. This information can be used for the surveillance of scenes, e.g., for the tracking of cars in order to provide information which could be evaluated by an automatic traffic flow analysis system.
43
44 most cases, this optical flow field is a good approximation to the temporal displacement of the image of a depicted surface element between time t-~t and time t . If we use optical flow field s for the evaluation of dynamic imagery we do not have to distinguish between image sequences which were recorded with stationary or non -sta ti onary came ras, since the evaluation of optica l flow fi e lds is applicable for both cases. The next section describes briefl y how optica l flow fi e lds ca n be calcu lated.
OPTICAL FLOW FIELDS Optical flow fi e lds desc ribe th e temporal shift of observable gray value stru ctures in image se quences (Nagel 85). Various approaches have been suggested to estima te optical flow fields from im age se quences - see, for example, N agel 83 + 84 + 85 or Hildreth 83 for a review of the literature. Optical fl ow fields comprise not only information about th e rel ative displacement of image points but also a bout the spatial structure of the recorded scene. Seve r al investigations in the literature show how these two-dimensi ona l fields can be interpreted to infer information a bout th e threedimensional environment. The algorithms deve loped for th e estimation of op tical flow fields or of si ngle optica l fl ow vectors can be characterized as feature based methods (Kories 85, Kori es & Zimme rm a nn 86 , Moravec 80) or as gradient based methods (Horn & Schunck 81 , Nagel & Enkelma nn 86, Enkelmann 88). One a pproac h of eac h class will be desc ribed briefly beca use the algorithms discussed in the followin g sections a r e based on these approaches. A very robust feature based method for the es timation of optical flow vectors has bee n developed by Kories and Zimmermann 86 . Th e digiti zed TV-frame is subjected to a bandpass filter . Blobs re presenting local max im a a nd minima of the gray value functi on are then determined as feature s by the so-ca ll ed monoton ici ty operator (Kori es & Zimme rm ann 84). The centroids of the detected blobs are track ed throug h subsequent fr ames, resulting in optica l fl ow vec tors. The method has been succesfull y app li ed to seve ra l ten thousands of images taken from various so urc es without a n y change of pa ramete rs . Its first steps a re now implemented in a VLSI-chip working a t video rate.
pixel position in a consecutive fram e. Smoothness requirements fac ilitate the estimation of optical flow fields even for a reas with constant or only linearly sloping gray value distributions. Horn a nd Schunck 81 formul a ted the estimation of optical fl ow fi elds as a minimi za tion problem. In order to avoid problems with the general sm oo thness constraint of Horn and Schunck 81, Nagel 83 deve loped the 'oriented sm ooth ness co nstr ain t'. Starting with an iterative solu tion a pproach fo r the system of non linear partial differe ntial equations resulting from the 'ori e nted smoothness constrai nt' , the iterative so lution app roac h co ul d be improved s ignifi ca ntl y usin g multi grid methods (Enkelma nn 85a + b, Enkelmann 88), It has bee n shown - experimentally as we ll as th eo reti ca lly th a t the multi g rid a pproac h of Enke lma nn 85a+85b+88 yields simil a r estim a tes for opti ca l flow vecto rs at image locatio ns wh ere the monoton icity ope rato r detected a blob (Enkelmann et a I. 88).
SURVEILLANCE OF SCENES TV-came ras a re ve ry use ful for the s urveillance of sce nes due to their high resolution co mp are d to their size a nd costs. Some applications requ ire only the eva lu ation of scenes recorded with a statio n a ry came ra, e.g. the su rveillance of a robots work space or a pl a tform of a n undergro und station where people e nter or leave a train (Haass 84). Many a lgorithms have bee n developed so fa r which explicitly make use of the a priori knowl edge that the recordin g camera is station a ry. Most of such a lgorithms a re si mple app roaches whi ch try to inte rprete differences betwee n co nsec uti ve fram es of a n image sequenc e. One app roac h (Hsu et a I. 84) which uses a n ex plicit math em a tical mode l of th e gr ay va lue image function to detect chan ges in stati ona ry image sequences, was succesfull y integrate d into the MORIO-Sys tem (Dresc hle r 8 1, Dresc hler & N age l 82, Dresc hl er-Fi sc he r et aI. 83). The MO RIO-System (MOving RIgid Objec ts ) ca n a utomatica lly infer a n a pproxim ate threedim e nsional polyhedra l model of a n objec t moving relative to a stati onary came ra. Nagel82 showed that a rel ation between cha nge detection a nd the estimation of op ti ca l fl ow fields exist. The foll owin g paragraphs desc ribe a system that can be used for the detection and trackin g of m ovi ng objects based on the evaluation of optical fl ow vectors.
In most situations, the mere gray value variations do
The
not provide sufficient information to completely determine both components of the optical flow field u(x) = (u(x), V(X))T which links th e pixel a t image
Trajecto ry estimation in Imagery of Objects in Natural
location x = (X,y)T in one fra me to the corresponding
abstr act propositional desc ription of the scene, the so-
ACTIONS
System
(Automatic
Cueing
and
Scenes, Sung & Zimmermann 86, Sung 88) is used to detect a nd track moving objects and construct a n
45
Illtl'rprl'tatioll of Tr;dlic SCl'Ill' S
called Geometrical Scene Description (Neumann 84).
a)
The geometrical scene description comprises for each time instance information about the three-dimensional location of all objects in the scene as well as addi tional object attributes. Such a description is used as an interface to the separately developed system VITRA (VIsual TRAnslator, Andre et a!. 86). The contents of the geometrical scene description are further interpreted with VITRA by recognizing events. Then, a natural language description is derived by verbalization of appropriate propositions selected in the step before (Schirra et a!. 87, Herzog et a!. 89). Thi s natural language system can be used as an interface to a human operator, who can, e.g., ask natural langu age questions about the recorded scene. Image sequences of thousands of frames have to be analysed in order to provide input for the generation of non-trivial natural language descriptions. In order to limit the required computations, a very robust method for the estimation of optical flow vectors (Kories and Zimmermann 86) has been applied in the ACTIONS system. The optical flow vectors are clustered in order to incrementally create candidate moving objects in the picture domain. Optical flow vectors for the image in Fig. 1 are shown in Fig. 2. Figure 1 shows moving objects marked by a frame parallel to the direc tion of motion indicated by an arrow inside the fr a me. The scene was recorded by a stationary camera from a building about 35 m high . The scene shows a tram moving from the left to the right of the field of view,
Calculate an optical flow field u(x,t) from the recorded image sequence. The multi grid approach described above has been used for this purpose (Enkelmann 88).
b)
Estimate a model vector field U~I(x,t) that describes the expected optical flow field without any obstacle. It is assumed In the current implementation that the road can be approximated by a plane without significant errors.
c)
Evaluate the differences U o between calculated optical flow field u and estimated model vector field
U , j'
Figure 3 shows the first frame of an image sequence used to investiga te an approach for obstacle detection. In Fig. 4 the optical flow field calculated with the multigrid approach of Enkelmann 88 is shown. A subsampling of eight was performed for better visualization.
Estimation of a model vector field The two-dimensional model vector field uM(x,t) assigns to each image location x = (X,y)T the maximal possible shift in the image plane if the projected threedimensional point X" = (X w ' Y" , Z)T does not belong to an obstacle. If we assume only a translational camera motion parallel to the road plane the components of the model vector u>! =(U,v)T are given by:
together with other vehicles moving in the opposite direction. In the upper left-hand corner, one car has already stopped while three others are slowing down in front of red traffic lights. Tracking such object candidates through extended image subsequences allows us to incrementally build up projected trajectories which - together with additional attributes like size, speed, orientation and internal blob structure of object candidates - provide the input data for the natural language generation steps.
OBSTACLE DETECTION In contrast to some approaches which detect obstacles in an industrial environment using stereo image pairs (Storjohann et a!. 88) or tracking only a few image features (Graefe et a!. 88), the approach for the detection of stationary as well as non-stationary obstacles described in the following paragraphs is based on the evaluation of optical flow.
u =
x'-
+-
. x· =(
f
x
(la)
y' v = ( _ C" _ + P ) _ y
11 bi
x
Zc
+ Px
)-
r
t..'
=
y' -
..
z~
y
Lower case character denote image pl a ne coordinates. Quoted characters correspond to the end of the time inte rv a l used to calculate the optical flow fi e ld shown in Fig. 4. The focal length f of the camera lens system and the intersection of the optical axis with the image plane (p" p) determine the camera parameters, as far as this contribution is concerned. The focus of expansion (FOE) is given by the intersection of the three-dimensional motion vector v F~ t = (v FX ,VFy,V FZ)T ~t with the image plane. In general, the direction of sensor motion is not parallel to the optical axis, so that the FOE is different from the projection point (p" Py) ofthe optical axis onto the image plane .
The basic procedure to detect obstacles in front of a moving vehicle consists of three steps (Enkelmann 87):
Subscripts W, C, F correspond to the world, camera, and vehicle coordinate system, respectively. Figure 5 shows the relation of denotations used.
'v\". Enkelm
46
Insertion of the coordinate transfonnation of three-
stop in front of the obstacle. The area we used in our
dimensional scene points on the road plane and the camera translation into Eq. (1) results in the following
current implementation has a distance of 8 to 11 m from the moving vehicle. If we resolve Eq. (2) to vcz~t for each image location in this area then the average of
equations.
the values for vcz~t in this area is used for the estimation of model vectors at the other image locations. The model vector field estimated with this
(X-p) -WOE, -p) ( 2a)
Ze / (ue?!'l)
procedure ist shown in Fig. 6.
- 1
(y-p) -(FOEy-p y )
(2b)
Ze l (ueztltl-l
The terms fVcx/vez and fvCY/vcz are equal to the image plane coordinates of the FOE. Equations (2) relate the components of the model vector uM(x) = (U,V)T to the image location x=(x,yJT, the FOE, the distance Zc of the projected three-dimensional scene point from the camera, and the component v c z~t of the sensor translation vector. The FOE was detennined from the calculated optical flow field using the approach of Bruss & Horn 83. In Figures 4 and 6-8 the FOE determined from the calculated optical flow field shown in Fig. 4 is marked with a little cross. The distance Zc of a three-dimensional scene point on the road plane can be calculated if we know the transformation matrix between the camera and the vehicle coordinate system. The transformation matrix was determined in a calibration procedure similar to that described by Lenz 87. In general, it is impossible to invert the imaging process. But, given the transfonnation matrix between vehicle and camera coordinates, the intersection of the three-dimensional line with the road plane can be calculated. If we insert the Zc component of this intersection point into Eq. (2) a model vector field can be estimated uniquely if the component v cz t1t of the camera translation vector is known. At those image locations where the Zc component of the intersection point is negative, i.e. it lies behind the
•
Detection of stationary obstacles To detect stationary obstacles, all differences between calculated optical flow vectors (Fig. 4) and estimated model vectors (Fig. 6) have to be evaluated. Image locations where the length of the model vector luMI is larger than the length of the corresponding optical flow vector lul are not considered as an image of an obstacle, since the distance of the projected scene point to the moving vehicle is larger than those points of the volume where obstacles have to be detected. If the length of the model vector is less than the length of the optical flow vector, the ratio of the absolute values of the difference vector IUnl and the model vector luMI are compared to a threshold 8. If the ratio lu DI / luMI is larger compared to the threshold 8 then the image location is considered to be a projection of a three-dimensional obstacle point. In areas around the FOE this ratio became larger than the threshold 8 even if the absolute differences were small. Therefore, another test is performed to make sure that the denominator of the ratio is significantly different from zero. The detection result for the optical flow field in Fig. 4 using the model vector field in Fig. 6 is shown in Fig. 7. For display purposes a subsampling with distance four has been done. Fig. 7 shows that obstacles beside the road are marked due to the fact that the corresponding scene points are not in the road plane. To concentrate the detection of
v cz ~t of the camera translation vector. If a dynamic model of the vehicle state is available then the value of
obstacles to the volume that will be passed through by the moving vehicle, additional limitations of the socalled 'motion tunnel' (Zimmermann et al. 86) have to be inserted. The volume. which is bounded by the road plane , the virtual plane parallel to the road plane and by the viewing angle of the camera, is reduced to a smaller volume that will be passed through by the moving vehicle by considering two additional virtual
the translation vector can directly be used to estimate the model vector field . In the current version of our implementation no vehicle state model is available. To cope with this problem, we selected interactively an
planes perpendicular to the road. The threedimensional coordinates of these additional virtual planes are determined by the width of the moving vehicle . This approach assumes that the velocity vector
image area where we a priori assume that the distance of the projected scene points is so small compared to the vehicle velocity (30 - 50 km/h) that is it not possible to
of the vehicle does not change. Using this bounded motion tunnel, a model vector field can be estimated
camera, the Zc component of an intersection point with a virtual plane parallel to the road plane is used instead. This virtual plane is given a priori by the height of the moving vehicle . The last unknown tenn in Eq . (2) is the component
with the procedure described above. This results in
Illll'l'prl'lalio ll or Tra lfi( SCl'Ill'S
larger model vectors beside the road and, therefore, the scene points outside this volume are not considered to be a projection of an obstacle (see Fig. 8). If we compare Fig. 7 to Fig. 8, the influence of the left
and right boundary of the motion tunnel is clearly visible. Image areas which cover the parking cars in the left hand corner, the trees in the right hand corner the rails of a bus track and the left box on the road are not considered as obstacles. The results presented so far show how obstacles which a re stationary relative to the environment can be detected by evaluation of calculated optical flow fields.
Detection of moving objects The detection of moving objects in image sequences taken from a moving camera becomes much more difficult due to the camera motion. If a camera is translating through a stationary e nvironment the optica l flow field which results from the ego-motion shows a typical structure shown , e.g. , in Fig. 6. The directions of each optical flow vector intersect at one point in the image plane , the focus of expansion. If another moving object becomes visible by the translating camera the optical flow field resulting from this additional motion will interfere with the optical fl ow field re su lting from pure ego-motion (Fig. 9). This interference can be detected if the approach for the detection of stationary obstacles in image sequences taken from a moving camera - as desc ribed above - will be extended by testing whether the calculated optica l fl ow vectors have the same direction as the estima ted mod e l vectors. If the calculated op tical flow vectors a re not compatible with the expected direction they are conside red to be an image of a moving object. The result of this evaluation of the optical flow field shown in Fig. 9 is presented in Fig 10. This encouraging results show how - stationary as well as non -stationary - obstacle ca ndidates can be detected in image sequences taken from a translating camera by the evaluation of optical flow fields.
CONCLUSION The approaches discussed in this contribution are exam ples for the interpretation of temporal variations in image sequences recorded by a stationary camera as well as by a translating camera. Further developments are neccessary to extend the app roaches to more general motion and more complex environments. e.g. scenes containing non-rigid objects, to derive con ce ptual descriptions for image sequences (NageI88).
47
ACKNOWLEDGEMENTS This work has been partially supported by the "Bundesministerium fur Forschung und Technologie" of the Federal Republic of Germany as well as by the Daimler-Benz AG. I thank Daimler-Benz AG for providing the facility to record image sequences from a moving bus. The kind help ofC.-K. Sung for providing Figures 1 and 2 is greatfully acknowledged. I thank H .- H. Nagel for his comments on a draft version of this paper.
REFERENCES Andre, E ., G. Bosch, G. Herzog, T. Rist (1986). Characterizing Trajectories of Moving Objects Using Natural Language Path Descriptions. Proc. of 7th European Conference on Artifcial Intelligence, Vo!. 2, Brighton/UK, 1986, pp. 1-8 Bruss, A.R. , B.K.P. Horn (1983). Passive Navigation. Computer Vision, Graphics, a nd Image Processing 21 (1983) 3-20 Dreschler, L. (1981). Ermittlung marka nter Punkte a uf den Bildern bewegter Objekte und Berechnung einer 3DBeschreibung auf dieser Grundlage. Dissertation (J uni 1981), Fachbereich Informa tik, Universitat Hamburg Dresc hler. L. ,H.-H. Nagel (1982). Volumetric Model and 3D-Trajectory of a Moving Car Derived from Monocular TV-Frame Seque nces of a Street Scene. Computer Graphics and Image Processing 20 (1982) 199-228 Dreschler-Fischer, L. . W. Enkelmann , H.-H. Nagel (1983 ). Lernen durch Beobachtung von Szenen mit bewegten Objekten: Phasen eine r Systementwick lun g. Proc. 5. DAGM-Symposium Mustererkennung 1983 , Karl s ruhe , 11 .- 13 . Oktober 1983, VDE -Fach · berichte 35 , VDE- Verlag GmbH Berlin -Offenbach. 1983, pp. 29-34 Enkelmann, W. (1985a). Mehrgitterverfahren zur Ermittlung von Verschiebungsvektorfeldern in Bildfolgen. Dissertation (Juli 1985), Fac hbereich Informatik , Universitat Hamburg Enkelmann, W. (1985b ) Ein Mehrgitterverfahren zur Ermittlung von Versc hiebungsvektorfeldern in Bildfolgen. 7. DAGMSymposium, Erlangen, 24.-26. September, 1985 , I:£. Niemann (ed.), Mustererkennung 1985, InformatlkFachberichte 107 , Springer- Verlag Berlin Heidel berg New York Tokyo, 1985, pp. 97-101 Enkelmann, W. (1987). Untersuchungen zur Hindernisdetektion durch Auswertung von Verschiebungsvektorfeldern. 9. DAGM-Symposium Mustererkennung, Braun sc hweig, 29.9.-l.10., 1987, E. Paul us (ed.). Informatik-Fachberichte 149 . Springer- Verlag Berltn Heidelberg New York Tokyo, 1987, pp. 282-286 Enkelmann, W . (1988). Investigations of Multigrid l\lgori~hms for the Estimation of Optical Flow FIelds In Image Sequences. Computer Vision, Graphics, and Image Processing (1988) 150-177
48
W. Ellkdlll
Enkelmann, W., R Kories, H.-H. Nagel, G. Zimmermann (1988). An Experimental Investigation of Estimation Approaches for Optical Flow Fields. W.N. Martin, J.K. Aggarwal (ed.), Motion Understanding: Robot and Human Vision, Kluwer Academic Publishers, Boston / Dordrecht / Lancaster, 1987, pp. 189-226 Graefe, V., U. Regensburger, U. Soldner (1988). Visuelle Entdeckung und Vermessung von Objekten in der Bahn eines autonom mobilen Systems. 10. DAGM Symposium, Zurich , 27.-29. September 1988, H. Bunke, O. Kubler, P. Stucki (Hrsg.), Mustererkennung 1988 Informatik-Fachberichte 180, Springer-Verlag Berlin Heidelberg New York London Paris Tokyo. 1988, pp. 312-318 Haass, U. (1984). Arbeitsraumuberwachung beim Industrieroboter durch automatische Bildverarbeitung. in: Sehr fortgeschrittene Handhabungssysteme. Fachberichte Messen Steuern , Regeln, Bd. 9, SpringerVerlag Berlin Heidelberg 1984, pp. 130-145
Nagel, H.-H. (1983). Constraints for the Estimation of Displacement Vector Fields from Image Sequences. Proc. Int. Joint Conference on Artificial Intelligence , Karlsruhe/FRG, August 8-12,1983, pp. 945-951 Nagel, H.-H. (1984). Recent Advances in Image Sequence Analysis. Proc. Premier Colloque Image - Traitement, Synthese, Technologie et Applications, Biarrit:zJFrance, May 21-25,1984,pp.545-558 Nagel, H.-H. (1985). Analyse und Interpretation von Bildfolgen. Informatik-Spektrum 8 (1985) 178-200,312-327 N agel, H.-H. (1986). Image Sequences - Ten (octal) Years - From Phenomenology towards a Theoretical Foundation. Proc. 8th International Conference on Pattern Recogni tion, Paris/France, October 27 -31, 1986, pp. 1174-1185
Herzog, G., C.-K. Sung, E. Andre, W. Enkelmann, H.-H. Nagel, T. Rist. W. Wahlster, G. Zimmermann (1989). Incremental Natural Language Description of Dynamic Imagery. Submitted for publication .
Nagel, H.-H., W. Enkelmann (1986). An Investigation of Smoothness Constraints for the Estimation of Displacement Vector Fields from Image Sequences. IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI-8 (1986) 565-593
Hildreth, E.C. (1983). The Measurement of Visual Motion. Ph.D. Thesis (August 1983), Dept. Electrical Engineering and Computer Science, Massachusetts Institute of Technology, Cambridge/MA
Nagel, H.-H. (1988.) From image sequences towards conceptual descriptions. Image and Vision Computing, Vol. 6, No.2 (may 1988) 59-74
Horn, B.K.P., B.G. Schunck (1981). Determining Optical Flow. Artificial Intelligence 17 (1981) 185-203
Neumann, B. (1984). Natural Language Description of Time- Varying Scenes. Report 105, Fachbereich Informatik, Universitat Hamburg, 1984
Hsu, Y.Z., H .-H . N agel, G. Rekers (1984) New Likelihood Test Methods for Change Detection in Image Sequences. Computer Vi sion, Graphics, and Image Processing 26 (1984) 73-106
Pavlidis, T. (1986). A Critical Survey ofImage Analysis Methods. Proc. 8th International Conference on Pattern Recognition, Paris/France, October 27-31,1986, pp. 505-512
Kories, R (1985). Maschinelles Bewegun gsse hen in naturlichen Szenen: Die Auswertun g von Bildfolgen gestutzt aufBildmerkmale. Di sse rtati on (J a nuar 1985), Fachbereich Biologie, Universitat Mainz
Schirra, J.RJ., G. Bosch, C. -K. Sung, G. Zimmermann (1987). From Image Sequences tow a rds Natural Language: A First Step toward Automatic Perce ption a nd Descri ption of Moti ons. Appl ied Artificial Intelligence 1 (1987 ) 287 -305 Storjohann, K., E. Schulze, W. v. Seelen (1988). Segmentierung dreidimensionaler Szenen mittels perspektivischer Kartierung. LO. DAGM Symposium , Zurich, 27.-29. September L988, H. Bunke , O. Kubler, P. Stucki (Hrsg.) , Mustererkennung 1988 Informatik-Fachberichte 180, Springer-Verlag Berlin Heidelberg New York London Paris Tokyo, 1988, pp. 248-254
Kories, R, G. Zimmermann (1984). Motion Detection in Image Sequences: an Evaluation of Feature Detectors . Proc. Int. Joint Conference on Pattern Recognition, Montreal / Canada, July 30 - August 2,1984, pp. 778-780 Kories, R., G. Zimmermann (1986). A Versatile Method for the Estima ti on of Displace ment Vector Fields from Image Sequences. Workshop on Motion: Representation and Analysis, Kiawah Island Resort, Charleston/SC, May 7-9 , 1986, IEEE Computer Society Press, 1986. 101- 106 Lenz, R. (1987). Linsenfehlerkorrigierte Eichung von Halbleiterkameras mit Standardobjektiven fur hochgenaue 3D-Messungen in Echtzeit. 9. DAGM Symposium, Braunschweig, 29.9-1.10.1987, E .Paulus (Hrsg.l , Mustererkennung 1987 Informatik-Fachberichte 149 Springer-Verlag Berlin Heidelberg New York, 1987, pp. 282-286 Moravec, H.P. (1980). Obstacle Avoidance a nd Navigation in the Real World by a Seeing Robot Rover. Ph.D. Thesis (September 1980), Department of Computer Science, Stanford University, available as CMU-RITR-3, Robotics Institute, Carnegie-Mellon University PittsburghlPA Nagel H.-H. (1982). On Change Detection and Displacemen t Vector Estimation in Image Sequences. Pa ttern Recognition Letters 1 (1982) 55-59
Sung, C.-K. (1988). Extraktion von typischen und komplexen Vorgangen aus einer langen Bildfolge einer Verkehrsszene. 10. DAGM Symposium, Zurich, 27.-29. September 1988, H. Bunke, O. Kubler, P. Stucki (Hrsg.), Mustererkennung 1988 Informatik Fachberichte 180, Springer- Verlag Berlin Heidelberg New York London Paris Tokyo, 1988, pp. 90-96 Sung, C.-K., G. Zimmermann (1986). Detektion und Verfolgung mehrerer Objekte in Bildfolgen. 8. DAGM Symposium Mustererkennung, Paderborn, 30.9. -2.10., 1986, G. Hartmann (ed.), Informatik-Fachberichte 125, SpringerVerlag Berlin Heidelberg New York Tokyo, 1986, pp. 181-184 Zimmermann, G., W. Enkelmann, G. Struck, R Niepold, R Kories (1986). Image Sequence Processing for the Derivation of Parameters for the Guidance of Mobile Robots. Proc. of the Int. Conf. on Intelligent Autonomous Systems, AmsterdamlThe Netherlands, December 8-10, 1986, Elsevier Science Publisher B.V., Amsterdam, 1986, pp. 654-658·
49
Illterpretatioll of Traffic SCl'Ill'S
;.,:
.•.....
.'
..
-.
'';'
".'
:...~... ,...:~ ;':,:: -: Fig. 1: First image of a sequence used to detect and track moving objects. The moving objects - detected from the optical flow vectors in Fig. 2 - are marked by a frame paralle l to the direction of motion indicated by an arrow inside the frame (S ung 88).
~
First frame of an image sequence used to investigate algorithms for obstacle detection .
Fig. 2: Optical flow vectors of th e image in Fig. 1. The optical flow vectors were calculated from features determined by the m onoton icity operator (Kories & Zimmermann 86).
Fig. 4: Optical flow fi e ld of the image in Fig. 3. The optical flow field was calcu la ted using a multi grid approach (E nkelman n 88). For better visualization a subsampling of eight was performed. The focus of expansion computed with the approach of Bruss & Horn 83 is marked by a little cross in the upper middle of the image.
0"9"1 of C.:Imera coordinate \ystem X
30 ve h,(l e
Origin o f Image
motion vector
coordinate
/
)C
_ _ +.
__
, ~
Z
"
~~~I~:'~:t~;'th ,mage p lane
y Focu~ of expanSion
, InterseCl l on of
~y~tem
X '
',
OPt l (7 1"~ I S c==:J
Y""
cv
c::===J
~ ' V,
O ri g i n of vehic le
coordinate system
X,
Fig. 5: Denotations of coordinates used for obstacle detection by evaluation of optical flow fields .
~ Model vector field u\(x ,t) that describes explicitly the expected opti cal flow field without any obstacles in the scene. This model vector field was calculated assuming that the camera moves in a three-
dimensional volume that is bounded by the road plane and a virtual plane pa rallel to the road plane with a distance from the road plane equal to the height of the moving vehicle.
\\'. Ellkcim;t1l1l
50
•
I Fig. 7: Image locations which are considered to be projections of three-dimensional obstacles. For better visualization they are marked only at every fourth pixeL The obstacles were detected by evaluating the differences between the calculated optical flow field (Fig. 4) and the estimated model vector field (Fig. 6).
Fig. 9: Optical flow field for a non-stationary image sequence with a moving car. The scene was taken from a moving camera mounted on a bus.
Fig. 8: In contrast to Fig. 7, only those obstacles are marked which are in the volume that will be passed through by the moving vehicle assuming constant egomotion parameters. Only the box in the middle of the image remains as an obstacle since the other objects are located outside the 'motion tunnel' .
Fig. 10: Image locations of the car moving relative to the stationary environment as well as to the nonstationary camera. The moving car was detected by evaluating the optical flow vector field in Fig. 9. The optical flow vectors at all image locations marked cannot be explained by the ego-motion of the camera.
SYSTH!S L:SI:--JC Ne W SENSO R
Pari s, Frall ce, 1 9X~1
INTERPRET A TION OF TRAFFIC SCENES BY EVALUATION OF OPTICAL FLOW FIELDS FROM IMAGE SEQUENCES W. Enkelmann Fmllll!w/iT- f ll.,lilul jiir f ll/imll(l li()IIS- III/{! [) 1I11'1I< 'I'I'(lrill'illlllg (ffTfi ) F mllll!w!I' nlmfJl' f. D-15 ()() }\(lr/.,nilll' f. FR(;
Abstract. TV cameras are very useful to record traffic scenes due to their high resolution compared to their size and costs. In comparison with single images , image sequ ences comprise information a bout the dynamic aspects of the recorded scene. Dynamic aspects play a crucial role in traffic scenarios. Traffic scenes with different complexity can be characterized by the structure of the scene. Depending on the structure of the sce ne different tas ks co uld be so lved by image seq uence evaluation . In this co ntribution two approaches for the evaluation of opti cal flow fields from image sequences are discussed. The first approach describes th e detection and tracking of moving objects on a complex street intersection. The second a pproac h presents an algorithm for the detection of obstacles on the road of an a utonomous vehicle . It will be shown how two-dimensiona l optical flow fi elds fi e lds can be interpreted to infer informati on a bout th e three-dimensional environment. Results obtained from investigations with real world image sequences will be presented . Keywords. Image sequence analysis, optical flow, object tracking, obstacle detection
If image sequences are taken from a moving camera,
INTRODUCTION
e.g. mounted on a robot's arm or on an a uton omous vehicle, the object detection task becomes much more difficult. But, furthermore, a new class of problems arise which could be solved by image sequence analysis due to the ego-motion of the camera. Some examples will be discussed in the following sections, e.g.
The evaluation of traffic scenes by image sequence analysis is a task with increasing attention by traffic experts as well as by robot a nd car ma nufacturers. One reason might be the progress in image sequence analysis during the last decade (N age l 86, Pa v lidis 86) which made this research field more attractive for potential applications. In compariso n with single images, image sequences comprise information about the dynamic aspects of the recorded scene. Dynamic aspects pl ay a crucial role in traffic scenarios. Traffic scenes with different complexity can be characterized by the
•
detection and tracking of objects mov ing relative to the recording camera as well as to the stationary environment,
•
the estimation a nd eva lu atio n of the relative displaceme nt betwee n an a utono mous ve hicle and
structure of the scene . Depending on the structure of the scene, different tasks could be solved by image sequence evaluation.
stationary parts of the sce ne as we ll as between other mov ing objects. •
estimation of motion par amete rs relative to the camera or the environment,
•
detection of stationary obstacles.
Let us assume the three-dimensional scene will be recorded with a stationary camera. In this case the movement of objects in the scene results in temporal variations in the image sequence. One task of image sequence analysis is the detection and evaluation of
Whenever dynamic aspects of image sequences have to be evaluated it is much easier if a reliable estimation of optical flow fields is available. An optical flow field u(x,t) maps the gray value g(x-utH,t-tlt) recorded at time t-tlt at the image plane location x-utlt into the gray value g(x,t) recorded at location x at time t. In
temporal variations in image sequences in order to provide information about the objects in the recorded scene. This information can be used for the surveillance of scenes, e.g., for the tracking of cars in order to provide information which could be evaluated by an automatic traffic flow analysis system.
43
44 most cases, this optical flow field is a good approximation to the temporal displacement of the image of a depicted surface element between time t-~t and time t . If we use optical flow field s for the evaluation of dynamic imagery we do not have to distinguish between image sequences which were recorded with stationary or non -sta ti onary came ras, since the evaluation of optica l flow fi e lds is applicable for both cases. The next section describes briefl y how optica l flow fi e lds ca n be calcu lated.
OPTICAL FLOW FIELDS Optical flow fi e lds desc ribe th e temporal shift of observable gray value stru ctures in image se quences (Nagel 85). Various approaches have been suggested to estima te optical flow fields from im age se quences - see, for example, N agel 83 + 84 + 85 or Hildreth 83 for a review of the literature. Optical fl ow fields comprise not only information about th e rel ative displacement of image points but also a bout the spatial structure of the recorded scene. Seve r al investigations in the literature show how these two-dimensi ona l fields can be interpreted to infer information a bout th e threedimensional environment. The algorithms deve loped for th e estimation of op tical flow fields or of si ngle optica l fl ow vectors can be characterized as feature based methods (Kories 85, Kori es & Zimme rm a nn 86 , Moravec 80) or as gradient based methods (Horn & Schunck 81 , Nagel & Enkelma nn 86, Enkelmann 88). One a pproac h of eac h class will be desc ribed briefly beca use the algorithms discussed in the followin g sections a r e based on these approaches. A very robust feature based method for the es timation of optical flow vectors has bee n developed by Kories and Zimmermann 86 . Th e digiti zed TV-frame is subjected to a bandpass filter . Blobs re presenting local max im a a nd minima of the gray value functi on are then determined as feature s by the so-ca ll ed monoton ici ty operator (Kori es & Zimme rm ann 84). The centroids of the detected blobs are track ed throug h subsequent fr ames, resulting in optica l fl ow vec tors. The method has been succesfull y app li ed to seve ra l ten thousands of images taken from various so urc es without a n y change of pa ramete rs . Its first steps a re now implemented in a VLSI-chip working a t video rate.
pixel position in a consecutive fram e. Smoothness requirements fac ilitate the estimation of optical flow fields even for a reas with constant or only linearly sloping gray value distributions. Horn a nd Schunck 81 formul a ted the estimation of optical fl ow fi elds as a minimi za tion problem. In order to avoid problems with the general sm oo thness constraint of Horn and Schunck 81, Nagel 83 deve loped the 'oriented sm ooth ness co nstr ain t'. Starting with an iterative solu tion a pproach fo r the system of non linear partial differe ntial equations resulting from the 'ori e nted smoothness constrai nt' , the iterative so lution app roac h co ul d be improved s ignifi ca ntl y usin g multi grid methods (Enkelma nn 85a + b, Enkelmann 88), It has bee n shown - experimentally as we ll as th eo reti ca lly th a t the multi g rid a pproac h of Enke lma nn 85a+85b+88 yields simil a r estim a tes for opti ca l flow vecto rs at image locatio ns wh ere the monoton icity ope rato r detected a blob (Enkelmann et a I. 88).
SURVEILLANCE OF SCENES TV-came ras a re ve ry use ful for the s urveillance of sce nes due to their high resolution co mp are d to their size a nd costs. Some applications requ ire only the eva lu ation of scenes recorded with a statio n a ry came ra, e.g. the su rveillance of a robots work space or a pl a tform of a n undergro und station where people e nter or leave a train (Haass 84). Many a lgorithms have bee n developed so fa r which explicitly make use of the a priori knowl edge that the recordin g camera is station a ry. Most of such a lgorithms a re si mple app roaches whi ch try to inte rprete differences betwee n co nsec uti ve fram es of a n image sequenc e. One app roac h (Hsu et a I. 84) which uses a n ex plicit math em a tical mode l of th e gr ay va lue image function to detect chan ges in stati ona ry image sequences, was succesfull y integrate d into the MORIO-Sys tem (Dresc hle r 8 1, Dresc hler & N age l 82, Dresc hl er-Fi sc he r et aI. 83). The MO RIO-System (MOving RIgid Objec ts ) ca n a utomatica lly infer a n a pproxim ate threedim e nsional polyhedra l model of a n objec t moving relative to a stati onary came ra. Nagel82 showed that a rel ation between cha nge detection a nd the estimation of op ti ca l fl ow fields exist. The foll owin g paragraphs desc ribe a system that can be used for the detection and trackin g of m ovi ng objects based on the evaluation of optical fl ow vectors.
In most situations, the mere gray value variations do
The
not provide sufficient information to completely determine both components of the optical flow field u(x) = (u(x), V(X))T which links th e pixel a t image
Trajecto ry estimation in Imagery of Objects in Natural
location x = (X,y)T in one fra me to the corresponding
abstr act propositional desc ription of the scene, the so-
ACTIONS
System
(Automatic
Cueing
and
Scenes, Sung & Zimmermann 86, Sung 88) is used to detect a nd track moving objects and construct a n
45
Illtl'rprl'tatioll of Tr;dlic SCl'Ill' S
called Geometrical Scene Description (Neumann 84).
a)
The geometrical scene description comprises for each time instance information about the three-dimensional location of all objects in the scene as well as addi tional object attributes. Such a description is used as an interface to the separately developed system VITRA (VIsual TRAnslator, Andre et a!. 86). The contents of the geometrical scene description are further interpreted with VITRA by recognizing events. Then, a natural language description is derived by verbalization of appropriate propositions selected in the step before (Schirra et a!. 87, Herzog et a!. 89). Thi s natural language system can be used as an interface to a human operator, who can, e.g., ask natural langu age questions about the recorded scene. Image sequences of thousands of frames have to be analysed in order to provide input for the generation of non-trivial natural language descriptions. In order to limit the required computations, a very robust method for the estimation of optical flow vectors (Kories and Zimmermann 86) has been applied in the ACTIONS system. The optical flow vectors are clustered in order to incrementally create candidate moving objects in the picture domain. Optical flow vectors for the image in Fig. 1 are shown in Fig. 2. Figure 1 shows moving objects marked by a frame parallel to the direc tion of motion indicated by an arrow inside the fr a me. The scene was recorded by a stationary camera from a building about 35 m high . The scene shows a tram moving from the left to the right of the field of view,
Calculate an optical flow field u(x,t) from the recorded image sequence. The multi grid approach described above has been used for this purpose (Enkelmann 88).
b)
Estimate a model vector field U~I(x,t) that describes the expected optical flow field without any obstacle. It is assumed In the current implementation that the road can be approximated by a plane without significant errors.
c)
Evaluate the differences U o between calculated optical flow field u and estimated model vector field
U , j'
Figure 3 shows the first frame of an image sequence used to investiga te an approach for obstacle detection. In Fig. 4 the optical flow field calculated with the multigrid approach of Enkelmann 88 is shown. A subsampling of eight was performed for better visualization.
Estimation of a model vector field The two-dimensional model vector field uM(x,t) assigns to each image location x = (X,y)T the maximal possible shift in the image plane if the projected threedimensional point X" = (X w ' Y" , Z)T does not belong to an obstacle. If we assume only a translational camera motion parallel to the road plane the components of the model vector u>! =(U,v)T are given by:
together with other vehicles moving in the opposite direction. In the upper left-hand corner, one car has already stopped while three others are slowing down in front of red traffic lights. Tracking such object candidates through extended image subsequences allows us to incrementally build up projected trajectories which - together with additional attributes like size, speed, orientation and internal blob structure of object candidates - provide the input data for the natural language generation steps.
OBSTACLE DETECTION In contrast to some approaches which detect obstacles in an industrial environment using stereo image pairs (Storjohann et a!. 88) or tracking only a few image features (Graefe et a!. 88), the approach for the detection of stationary as well as non-stationary obstacles described in the following paragraphs is based on the evaluation of optical flow.
u =
x'-
+-
. x· =(
f
x
(la)
y' v = ( _ C" _ + P ) _ y
11 bi
x
Zc
+ Px
)-
r
t..'
=
y' -
..
z~
y
Lower case character denote image pl a ne coordinates. Quoted characters correspond to the end of the time inte rv a l used to calculate the optical flow fi e ld shown in Fig. 4. The focal length f of the camera lens system and the intersection of the optical axis with the image plane (p" p) determine the camera parameters, as far as this contribution is concerned. The focus of expansion (FOE) is given by the intersection of the three-dimensional motion vector v F~ t = (v FX ,VFy,V FZ)T ~t with the image plane. In general, the direction of sensor motion is not parallel to the optical axis, so that the FOE is different from the projection point (p" Py) ofthe optical axis onto the image plane .
The basic procedure to detect obstacles in front of a moving vehicle consists of three steps (Enkelmann 87):
Subscripts W, C, F correspond to the world, camera, and vehicle coordinate system, respectively. Figure 5 shows the relation of denotations used.
'v\". Enkelm
46
Insertion of the coordinate transfonnation of three-
stop in front of the obstacle. The area we used in our
dimensional scene points on the road plane and the camera translation into Eq. (1) results in the following
current implementation has a distance of 8 to 11 m from the moving vehicle. If we resolve Eq. (2) to vcz~t for each image location in this area then the average of
equations.
the values for vcz~t in this area is used for the estimation of model vectors at the other image locations. The model vector field estimated with this
(X-p) -WOE, -p) ( 2a)
Ze / (ue?!'l)
procedure ist shown in Fig. 6.
- 1
(y-p) -(FOEy-p y )
(2b)
Ze l (ueztltl-l
The terms fVcx/vez and fvCY/vcz are equal to the image plane coordinates of the FOE. Equations (2) relate the components of the model vector uM(x) = (U,V)T to the image location x=(x,yJT, the FOE, the distance Zc of the projected three-dimensional scene point from the camera, and the component v c z~t of the sensor translation vector. The FOE was detennined from the calculated optical flow field using the approach of Bruss & Horn 83. In Figures 4 and 6-8 the FOE determined from the calculated optical flow field shown in Fig. 4 is marked with a little cross. The distance Zc of a three-dimensional scene point on the road plane can be calculated if we know the transformation matrix between the camera and the vehicle coordinate system. The transformation matrix was determined in a calibration procedure similar to that described by Lenz 87. In general, it is impossible to invert the imaging process. But, given the transfonnation matrix between vehicle and camera coordinates, the intersection of the three-dimensional line with the road plane can be calculated. If we insert the Zc component of this intersection point into Eq. (2) a model vector field can be estimated uniquely if the component v cz t1t of the camera translation vector is known. At those image locations where the Zc component of the intersection point is negative, i.e. it lies behind the
•
Detection of stationary obstacles To detect stationary obstacles, all differences between calculated optical flow vectors (Fig. 4) and estimated model vectors (Fig. 6) have to be evaluated. Image locations where the length of the model vector luMI is larger than the length of the corresponding optical flow vector lul are not considered as an image of an obstacle, since the distance of the projected scene point to the moving vehicle is larger than those points of the volume where obstacles have to be detected. If the length of the model vector is less than the length of the optical flow vector, the ratio of the absolute values of the difference vector IUnl and the model vector luMI are compared to a threshold 8. If the ratio lu DI / luMI is larger compared to the threshold 8 then the image location is considered to be a projection of a three-dimensional obstacle point. In areas around the FOE this ratio became larger than the threshold 8 even if the absolute differences were small. Therefore, another test is performed to make sure that the denominator of the ratio is significantly different from zero. The detection result for the optical flow field in Fig. 4 using the model vector field in Fig. 6 is shown in Fig. 7. For display purposes a subsampling with distance four has been done. Fig. 7 shows that obstacles beside the road are marked due to the fact that the corresponding scene points are not in the road plane. To concentrate the detection of
v cz ~t of the camera translation vector. If a dynamic model of the vehicle state is available then the value of
obstacles to the volume that will be passed through by the moving vehicle, additional limitations of the socalled 'motion tunnel' (Zimmermann et al. 86) have to be inserted. The volume. which is bounded by the road plane , the virtual plane parallel to the road plane and by the viewing angle of the camera, is reduced to a smaller volume that will be passed through by the moving vehicle by considering two additional virtual
the translation vector can directly be used to estimate the model vector field . In the current version of our implementation no vehicle state model is available. To cope with this problem, we selected interactively an
planes perpendicular to the road. The threedimensional coordinates of these additional virtual planes are determined by the width of the moving vehicle . This approach assumes that the velocity vector
image area where we a priori assume that the distance of the projected scene points is so small compared to the vehicle velocity (30 - 50 km/h) that is it not possible to
of the vehicle does not change. Using this bounded motion tunnel, a model vector field can be estimated
camera, the Zc component of an intersection point with a virtual plane parallel to the road plane is used instead. This virtual plane is given a priori by the height of the moving vehicle . The last unknown tenn in Eq . (2) is the component
with the procedure described above. This results in
Illll'l'prl'lalio ll or Tra lfi( SCl'Ill'S
larger model vectors beside the road and, therefore, the scene points outside this volume are not considered to be a projection of an obstacle (see Fig. 8). If we compare Fig. 7 to Fig. 8, the influence of the left
and right boundary of the motion tunnel is clearly visible. Image areas which cover the parking cars in the left hand corner, the trees in the right hand corner the rails of a bus track and the left box on the road are not considered as obstacles. The results presented so far show how obstacles which a re stationary relative to the environment can be detected by evaluation of calculated optical flow fields.
Detection of moving objects The detection of moving objects in image sequences taken from a moving camera becomes much more difficult due to the camera motion. If a camera is translating through a stationary e nvironment the optica l flow field which results from the ego-motion shows a typical structure shown , e.g. , in Fig. 6. The directions of each optical flow vector intersect at one point in the image plane , the focus of expansion. If another moving object becomes visible by the translating camera the optical flow field resulting from this additional motion will interfere with the optical fl ow field re su lting from pure ego-motion (Fig. 9). This interference can be detected if the approach for the detection of stationary obstacles in image sequences taken from a moving camera - as desc ribed above - will be extended by testing whether the calculated optica l fl ow vectors have the same direction as the estima ted mod e l vectors. If the calculated op tical flow vectors a re not compatible with the expected direction they are conside red to be an image of a moving object. The result of this evaluation of the optical flow field shown in Fig. 9 is presented in Fig 10. This encouraging results show how - stationary as well as non -stationary - obstacle ca ndidates can be detected in image sequences taken from a translating camera by the evaluation of optical flow fields.
CONCLUSION The approaches discussed in this contribution are exam ples for the interpretation of temporal variations in image sequences recorded by a stationary camera as well as by a translating camera. Further developments are neccessary to extend the app roaches to more general motion and more complex environments. e.g. scenes containing non-rigid objects, to derive con ce ptual descriptions for image sequences (NageI88).
47
ACKNOWLEDGEMENTS This work has been partially supported by the "Bundesministerium fur Forschung und Technologie" of the Federal Republic of Germany as well as by the Daimler-Benz AG. I thank Daimler-Benz AG for providing the facility to record image sequences from a moving bus. The kind help ofC.-K. Sung for providing Figures 1 and 2 is greatfully acknowledged. I thank H .- H. Nagel for his comments on a draft version of this paper.
REFERENCES Andre, E ., G. Bosch, G. Herzog, T. Rist (1986). Characterizing Trajectories of Moving Objects Using Natural Language Path Descriptions. Proc. of 7th European Conference on Artifcial Intelligence, Vo!. 2, Brighton/UK, 1986, pp. 1-8 Bruss, A.R. , B.K.P. Horn (1983). Passive Navigation. Computer Vision, Graphics, a nd Image Processing 21 (1983) 3-20 Dreschler, L. (1981). Ermittlung marka nter Punkte a uf den Bildern bewegter Objekte und Berechnung einer 3DBeschreibung auf dieser Grundlage. Dissertation (J uni 1981), Fachbereich Informa tik, Universitat Hamburg Dresc hler. L. ,H.-H. Nagel (1982). Volumetric Model and 3D-Trajectory of a Moving Car Derived from Monocular TV-Frame Seque nces of a Street Scene. Computer Graphics and Image Processing 20 (1982) 199-228 Dreschler-Fischer, L. . W. Enkelmann , H.-H. Nagel (1983 ). Lernen durch Beobachtung von Szenen mit bewegten Objekten: Phasen eine r Systementwick lun g. Proc. 5. DAGM-Symposium Mustererkennung 1983 , Karl s ruhe , 11 .- 13 . Oktober 1983, VDE -Fach · berichte 35 , VDE- Verlag GmbH Berlin -Offenbach. 1983, pp. 29-34 Enkelmann, W. (1985a). Mehrgitterverfahren zur Ermittlung von Verschiebungsvektorfeldern in Bildfolgen. Dissertation (Juli 1985), Fac hbereich Informatik , Universitat Hamburg Enkelmann, W. (1985b ) Ein Mehrgitterverfahren zur Ermittlung von Versc hiebungsvektorfeldern in Bildfolgen. 7. DAGMSymposium, Erlangen, 24.-26. September, 1985 , I:£. Niemann (ed.), Mustererkennung 1985, InformatlkFachberichte 107 , Springer- Verlag Berlin Heidel berg New York Tokyo, 1985, pp. 97-101 Enkelmann, W. (1987). Untersuchungen zur Hindernisdetektion durch Auswertung von Verschiebungsvektorfeldern. 9. DAGM-Symposium Mustererkennung, Braun sc hweig, 29.9.-l.10., 1987, E. Paul us (ed.). Informatik-Fachberichte 149 . Springer- Verlag Berltn Heidelberg New York Tokyo, 1987, pp. 282-286 Enkelmann, W . (1988). Investigations of Multigrid l\lgori~hms for the Estimation of Optical Flow FIelds In Image Sequences. Computer Vision, Graphics, and Image Processing (1988) 150-177
48
W. Ellkdlll
Enkelmann, W., R Kories, H.-H. Nagel, G. Zimmermann (1988). An Experimental Investigation of Estimation Approaches for Optical Flow Fields. W.N. Martin, J.K. Aggarwal (ed.), Motion Understanding: Robot and Human Vision, Kluwer Academic Publishers, Boston / Dordrecht / Lancaster, 1987, pp. 189-226 Graefe, V., U. Regensburger, U. Soldner (1988). Visuelle Entdeckung und Vermessung von Objekten in der Bahn eines autonom mobilen Systems. 10. DAGM Symposium, Zurich , 27.-29. September 1988, H. Bunke, O. Kubler, P. Stucki (Hrsg.), Mustererkennung 1988 Informatik-Fachberichte 180, Springer-Verlag Berlin Heidelberg New York London Paris Tokyo. 1988, pp. 312-318 Haass, U. (1984). Arbeitsraumuberwachung beim Industrieroboter durch automatische Bildverarbeitung. in: Sehr fortgeschrittene Handhabungssysteme. Fachberichte Messen Steuern , Regeln, Bd. 9, SpringerVerlag Berlin Heidelberg 1984, pp. 130-145
Nagel, H.-H. (1983). Constraints for the Estimation of Displacement Vector Fields from Image Sequences. Proc. Int. Joint Conference on Artificial Intelligence , Karlsruhe/FRG, August 8-12,1983, pp. 945-951 Nagel, H.-H. (1984). Recent Advances in Image Sequence Analysis. Proc. Premier Colloque Image - Traitement, Synthese, Technologie et Applications, Biarrit:zJFrance, May 21-25,1984,pp.545-558 Nagel, H.-H. (1985). Analyse und Interpretation von Bildfolgen. Informatik-Spektrum 8 (1985) 178-200,312-327 N agel, H.-H. (1986). Image Sequences - Ten (octal) Years - From Phenomenology towards a Theoretical Foundation. Proc. 8th International Conference on Pattern Recogni tion, Paris/France, October 27 -31, 1986, pp. 1174-1185
Herzog, G., C.-K. Sung, E. Andre, W. Enkelmann, H.-H. Nagel, T. Rist. W. Wahlster, G. Zimmermann (1989). Incremental Natural Language Description of Dynamic Imagery. Submitted for publication .
Nagel, H.-H., W. Enkelmann (1986). An Investigation of Smoothness Constraints for the Estimation of Displacement Vector Fields from Image Sequences. IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI-8 (1986) 565-593
Hildreth, E.C. (1983). The Measurement of Visual Motion. Ph.D. Thesis (August 1983), Dept. Electrical Engineering and Computer Science, Massachusetts Institute of Technology, Cambridge/MA
Nagel, H.-H. (1988.) From image sequences towards conceptual descriptions. Image and Vision Computing, Vol. 6, No.2 (may 1988) 59-74
Horn, B.K.P., B.G. Schunck (1981). Determining Optical Flow. Artificial Intelligence 17 (1981) 185-203
Neumann, B. (1984). Natural Language Description of Time- Varying Scenes. Report 105, Fachbereich Informatik, Universitat Hamburg, 1984
Hsu, Y.Z., H .-H . N agel, G. Rekers (1984) New Likelihood Test Methods for Change Detection in Image Sequences. Computer Vi sion, Graphics, and Image Processing 26 (1984) 73-106
Pavlidis, T. (1986). A Critical Survey ofImage Analysis Methods. Proc. 8th International Conference on Pattern Recognition, Paris/France, October 27-31,1986, pp. 505-512
Kories, R (1985). Maschinelles Bewegun gsse hen in naturlichen Szenen: Die Auswertun g von Bildfolgen gestutzt aufBildmerkmale. Di sse rtati on (J a nuar 1985), Fachbereich Biologie, Universitat Mainz
Schirra, J.RJ., G. Bosch, C. -K. Sung, G. Zimmermann (1987). From Image Sequences tow a rds Natural Language: A First Step toward Automatic Perce ption a nd Descri ption of Moti ons. Appl ied Artificial Intelligence 1 (1987 ) 287 -305 Storjohann, K., E. Schulze, W. v. Seelen (1988). Segmentierung dreidimensionaler Szenen mittels perspektivischer Kartierung. LO. DAGM Symposium , Zurich, 27.-29. September L988, H. Bunke , O. Kubler, P. Stucki (Hrsg.) , Mustererkennung 1988 Informatik-Fachberichte 180, Springer-Verlag Berlin Heidelberg New York London Paris Tokyo, 1988, pp. 248-254
Kories, R, G. Zimmermann (1984). Motion Detection in Image Sequences: an Evaluation of Feature Detectors . Proc. Int. Joint Conference on Pattern Recognition, Montreal / Canada, July 30 - August 2,1984, pp. 778-780 Kories, R., G. Zimmermann (1986). A Versatile Method for the Estima ti on of Displace ment Vector Fields from Image Sequences. Workshop on Motion: Representation and Analysis, Kiawah Island Resort, Charleston/SC, May 7-9 , 1986, IEEE Computer Society Press, 1986. 101- 106 Lenz, R. (1987). Linsenfehlerkorrigierte Eichung von Halbleiterkameras mit Standardobjektiven fur hochgenaue 3D-Messungen in Echtzeit. 9. DAGM Symposium, Braunschweig, 29.9-1.10.1987, E .Paulus (Hrsg.l , Mustererkennung 1987 Informatik-Fachberichte 149 Springer-Verlag Berlin Heidelberg New York, 1987, pp. 282-286 Moravec, H.P. (1980). Obstacle Avoidance a nd Navigation in the Real World by a Seeing Robot Rover. Ph.D. Thesis (September 1980), Department of Computer Science, Stanford University, available as CMU-RITR-3, Robotics Institute, Carnegie-Mellon University PittsburghlPA Nagel H.-H. (1982). On Change Detection and Displacemen t Vector Estimation in Image Sequences. Pa ttern Recognition Letters 1 (1982) 55-59
Sung, C.-K. (1988). Extraktion von typischen und komplexen Vorgangen aus einer langen Bildfolge einer Verkehrsszene. 10. DAGM Symposium, Zurich, 27.-29. September 1988, H. Bunke, O. Kubler, P. Stucki (Hrsg.), Mustererkennung 1988 Informatik Fachberichte 180, Springer- Verlag Berlin Heidelberg New York London Paris Tokyo, 1988, pp. 90-96 Sung, C.-K., G. Zimmermann (1986). Detektion und Verfolgung mehrerer Objekte in Bildfolgen. 8. DAGM Symposium Mustererkennung, Paderborn, 30.9. -2.10., 1986, G. Hartmann (ed.), Informatik-Fachberichte 125, SpringerVerlag Berlin Heidelberg New York Tokyo, 1986, pp. 181-184 Zimmermann, G., W. Enkelmann, G. Struck, R Niepold, R Kories (1986). Image Sequence Processing for the Derivation of Parameters for the Guidance of Mobile Robots. Proc. of the Int. Conf. on Intelligent Autonomous Systems, AmsterdamlThe Netherlands, December 8-10, 1986, Elsevier Science Publisher B.V., Amsterdam, 1986, pp. 654-658·
49
Illterpretatioll of Traffic SCl'Ill'S
;.,:
.•.....
.'
..
-.
'';'
".'
:...~... ,...:~ ;':,:: -: Fig. 1: First image of a sequence used to detect and track moving objects. The moving objects - detected from the optical flow vectors in Fig. 2 - are marked by a frame paralle l to the direction of motion indicated by an arrow inside the frame (S ung 88).
~
First frame of an image sequence used to investigate algorithms for obstacle detection .
Fig. 2: Optical flow vectors of th e image in Fig. 1. The optical flow vectors were calculated from features determined by the m onoton icity operator (Kories & Zimmermann 86).
Fig. 4: Optical flow fi e ld of the image in Fig. 3. The optical flow field was calcu la ted using a multi grid approach (E nkelman n 88). For better visualization a subsampling of eight was performed. The focus of expansion computed with the approach of Bruss & Horn 83 is marked by a little cross in the upper middle of the image.
0"9"1 of C.:Imera coordinate \ystem X
30 ve h,(l e
Origin o f Image
motion vector
coordinate
/
)C
_ _ +.
__
, ~
Z
"
~~~I~:'~:t~;'th ,mage p lane
y Focu~ of expanSion
, InterseCl l on of
~y~tem
X '
',
OPt l (7 1"~ I S c==:J
Y""
cv
c::===J
~ ' V,
O ri g i n of vehic le
coordinate system
X,
Fig. 5: Denotations of coordinates used for obstacle detection by evaluation of optical flow fields .
~ Model vector field u\(x ,t) that describes explicitly the expected opti cal flow field without any obstacles in the scene. This model vector field was calculated assuming that the camera moves in a three-
dimensional volume that is bounded by the road plane and a virtual plane pa rallel to the road plane with a distance from the road plane equal to the height of the moving vehicle.
\\'. Ellkcim;t1l1l
50
•
I Fig. 7: Image locations which are considered to be projections of three-dimensional obstacles. For better visualization they are marked only at every fourth pixeL The obstacles were detected by evaluating the differences between the calculated optical flow field (Fig. 4) and the estimated model vector field (Fig. 6).
Fig. 9: Optical flow field for a non-stationary image sequence with a moving car. The scene was taken from a moving camera mounted on a bus.
Fig. 8: In contrast to Fig. 7, only those obstacles are marked which are in the volume that will be passed through by the moving vehicle assuming constant egomotion parameters. Only the box in the middle of the image remains as an obstacle since the other objects are located outside the 'motion tunnel' .
Fig. 10: Image locations of the car moving relative to the stationary environment as well as to the nonstationary camera. The moving car was detected by evaluating the optical flow vector field in Fig. 9. The optical flow vectors at all image locations marked cannot be explained by the ego-motion of the camera.