- Email: [email protected]
RANGE ESTIMATION WITH AN OMNIDIRECTIONAL STEREO VISION SYSTEM
RANGE ESTIMATION WITH AN OMNIDIRECTIONAL STEREO VISION SYSTEM Eduardo L. L. Cabral(1) and José C. de Souza Jr.(2) (1)
Dept. of Mechatronic Engineering Escola Politécnica of University of São Paulo, Brazil [email protected] (2)
Escola de Engenharia Mauá Instituto Mauá de Tecnologia, SP, Brazil [email protected]
Abstract: This paper presents a compact panoramic stereo vision system based on a double lobed mirror with hyperbolic profile. This omnidirectional vision system is used to obtain range estimations in a form appropriated for mobile robot navigation. As hyperbolic mirrors ensure a single viewpoint the incident light rays are easily found from the points of the image. The geometry of the double lobed mirror naturally ensures matched epipolar lines in the two images of the scene. These two properties make the double lobed mirror especially suitable for panoramic stereo vision because range estimation becomes simple and fast. This is a great advantage for real-time applications. Copyright © 2006 IFAC Keywords: Machine vision, Omnidirectional vision, Stereo vision, Range estimation, Hyperbolic mirror.
1. INTRODUCTION Panoramic imaging is an important tool in machine vision and it is widely used in the area of mobile robots, see for instance (Koyasu et al., 2002; Shiguro et al., 1992; Bonarini et al., 1997; Delahoche et al., 2000). Among the many possible types of panoramic vision systems an attractive approach is to mount a single fixed camera under a mirror covering a hemisphere. These systems are normally referred as catadioptric. The advantages of a catadioptric system are the compactness, the wide angle of view, and the fast acquisition rate. However, a disadvantage is that the images acquired have a relatively low spatial resolution. The shape of the mirror determines the image formation model of a catadioptric panoramic system. Different mirror shapes have been used and tested (Gaspar et al., 2002; Yamazawa et al., 1999; Conroy and Moore, 1999). A catadioptric system based on a
hyperbolic mirror and a perspective (pinhole) camera forms an omnidirecional vision system with a single effective viewpoint. This characteristic is especially suitable for stereo imaging because the incident light rays are easily determined from the points of the image. Another important tool for mobile robots and machine vision is stereo vision. Stereo vision allows determining the range to objects within the field of view of the system. Panoramic stereo vision system is very convenient because the field of view is 360o. Omnidirectional systems based on hyperbolic mirrors have been used for stereo vision as reported in (Koyasu et al., 2002; Gluckman, et al., 1998). However, such systems were developed based on two mirrors and two cameras. Besides these systems are not compact they have the problem of requiring precise positioning of the cameras and the mirrors to guarantee perfect alignment of the images.
Panoramic stereo based on a double lobed mirror was first suggested by Southwell et al. (1996). A double lobed mirror is essentially a coaxial mirror pair, where the centres of both mirrors are collinear with the camera axis, and the mirrors have a symmetric profile around this axis. This arrangement has the great advantage to produce two panoramic views of the scene in a single image. Thus, it naturally ensures the alignment of the two images of the scene and removes the need for camera synchronization. A resolution invariant double lobed mirror was developed by Yamazawa et al. (1999), however, a catadioptric stereovision system based on a constant resolution mirror is not adequate for fast threedimensional stereo reconstruction since it requires solving a non-linear differential equation to obtain the mirror shape function and its slope at the reflection point. This paper presents a catadioptric system based on a hyperbolic double lobed mirror. Efficient equations for the stereo triangulation based on a single image acquired using the omnidirectional vision system are presented. The single viewpoint and the natural alignment of the images ensuring matched epipolar lines both in the omnidirectional and panoramic images make this system ideal for fast real-time stereo calculations. Accurate range estimations are obtained at distances far enough to allow this vision system to be used for mobile robot navigation. 2. THE OMNIDIRECTIONAL STEREO VISION SYSTEM The omnidirectional stereo vision system is composed by a conventional perspective (pinhole) camera aligned vertically (coaxially) with the double lobed mirror. Due to the rotational symmetry of the system only the radial cross section of the mirror need to be considered. Figure 1 presents a schematic of the hyperbolic double lobed mirror showing two incident and reflective light rays on the two mirror lobes. A perspective camera together with a hyperbolic mirror form a wide-angle imaging system with two centers of projections, located at the hyperbole focal points. In Fig. 1 points F0 and FI are the two focal points of the internal lobe hyperbole, and points F0 and FE are the focal points of the external lobe hyperbole. All incident light rays on the internal lobe pass through focal point FI and are reflected passing the other focal point F0. Similarly, all incident light rays on the external hyperbole pass through focal point FE and are reflected passing the other focal point F0. The internal and external hyperbole focus F0 must coincide with the camera optical centre. The optical axis of the internal lobe hyperbole is aligned with the camera axis, while the optical axis of the external hyperbole is not aligned with the camera axis. The profile of the external lobe is a hyperbole rotated around point F0 by an angle λ. The rotation of the hyperbole of the external lobe is
zE
z
External lobe
FE ME FI
MI
Internal lobe
φE θI
φI
λ θE F0 f
P
Camera optical center
λ Image plane
ρE ρI
r rE
Fig.1. Schematic of the hyperbolic double lobed mirror showing the incident and reflective light rays on the mirror emitted from a point P in space. necessary to ensure the same field of view with the same resolution of the images generated by both lobes. A hyperbolic mirror is defined in the mirror coordinate system centred at focal point F0 as z = c + a b2 + r 2 / b ,
(1)
where c, b and a are parameters of the hyperbole, with the restriction that c2 = a2 + b2. These parameters are different for the internal and external lobes. Equation (1) is valid for the internal hyperbole described in the camera coordinate system (F0-rz), and for the external hyperbole described in the external lobe coordinate system (F0-rEzE). 2.1 Image formation model Given a point P in space as shown in Fig. 1, two light rays emitted from this point reach the mirror surface, one on the internal lobe and other on the external lobe. The incident light ray with the elevation angle φI reaches the internal mirror lobe at point MI and projects onto the image at radius ρI. The incident light ray with the elevation angle φE reaches the external mirror lobe at point ME and projects onto the image at radius ρE. The equation for any incident light ray on the surface of the internal mirror lobe is z = − rcotanφ I + z FI ,
(2)
where zFI is the z coordinate of point FI. Similarly, for any incident light ray on the external lobe surface, z = −(r − rFE )cotanφ E + z FE ,
(3)
where (rFE, zFE) are the coordinates of the focal point FE. For any light ray reflected both from the inner and outer mirror lobes the following equation holds
tan θ = ρ f = rM z M ,
(4)
where f is the lens focal distance, θ is the camera vertical angle, (rM, zM) are the coordinates of a point on the mirror surface, and ρ is the image radius. The coordinates of any point lying on the internal mirror surface must obey the internal lobe profile given by (1). Also, the coordinates of any point lying on the external mirror surface must obey the external hyperbole equation given in the F0-rEzE coordinate system by (1). Equation (5) gives the coordinate transformation from the camera to the external lobe coordinate systems. r cos λ z = − sin λ
sin λ r , cos λ z ( E )
(5)
3. CALCULATION OF RANGE Given a point P in space, as shown in Fig. 1, the information available to calculate its coordinates in space (rP, zP) are the image radiuses in the internal and external images (ρI and ρE). Note that the radiuses of the points in the images are obtained through the search of corresponding points. The coordinates of point P are obtained by solving simultaneously the equations of the incident light rays from P to each mirror lobe reflection points MI and ME. tan φI [tan φE ( zFI − zFE ) − rFE ] rP 1 = zP (tan φE − tan φI ) zFE tan φE − zI tan φI + rFE (6)
The only unknowns in (6) are the elevation angles (φI and φE). Thus, to obtain the position of a point in space the tangent of the elevation angles are needed. From first optics and geometric considerations tanφI can be derived to give: 2k I ρ I f + k I2 ( f +
ρ E( E ) cos λ (E) = f sin λ
f 2 + ρ 2I
f 2 + ρ 2I ) 2 − ρ 2I
,
tan φ E( E )
where kE = (cE – aE)/(cE + aE), and f(E) and ρ (EE ) are respectively the lens focal distance and the image
(10)
To avoid the calculation of the angles, (10) can be rewritten in terms of the tangent of these angles, i.e.,
tan φ E =
tan φ (EE ) − tan λ 1 + tan φ (EE ) tan λ
.
(11)
In summary, given the image radiuses of the corresponding points in the internal and external images (ρI and ρE), using (7), (8), (9) and (11) the tangents of the elevation angles of the incident light rays are calculated and through (6) the position of the point in space is obtained. Note that in this process it is not necessary to calculate the elevation angles but only their tangents, which makes this calculation very efficient, fast and easy to be implemented in a digital computer. 4. CALCULATION OF THE MIRROR PROFILE The two lobes are designed to have the same field of view, which is described by the minimum and maximum elevation angles of the scene (φmin and φmax). Figure 2 presents a schematic of the double lobed mirror showing the incident light rays with the minimum and maximum elevation angles and the corresponding reflected rays. To achieve best resolution the projection of the mirror has to occupy the whole image. To achieve equal resolution in both mirror lobes the two images must have the same radius range, i.e., (ρ E , max − ρ E , min ) = (ρ I , max − ρ I , min )
(12)
where, ρI,max, ρI,min, ρE,maxn and ρE,min are the maximum and minimum image radius of the internal and external lobes. FE
φmax
∆zIE
φmax
FI
Similarly for the external mirror lobe, the same analysis results into: 2 2 2k E ρ E( E ) f ( E ) + f ( E ) + ρ E( E ) , (8) = 2 (E) (E) 2 (E) 2 2 (E) 2 kE ( f + f + ρE ) − ρE
(9)
φ E = φ (EE ) − λ .
(7)
where , kI = (cI – aI)/(cI + aI). For the derivation of this equation see (Souza Jr., 2004).
− sin λ ρ E . cos λ f
The elevation angle φE in the coordinate system F0rz, is obtained from φ (EE ) as:
(E)
where (r(E), z(E)) are the coordinates of a point described in the external lobe coordinate system.
tan φ I =
radius given in the external lobe coordinate system (F0-rEzE). f(E) and ρ (EE ) are calculated by inverting (5).
φmin
φmin
Fig. 2. Schematic of the double lobed mirror showing the incident light rays with the maximum and the minimum elevation angles.
The known parameters for the mirror design are the elevation angles φmin and φmax, the lens focal distance (f), the maximum image radius (ρmax), the axial distance between focal points (∆zIE), and the maximum external radius of the mirror. With the exception of ρmax, these parameters are chosen based on the application requirements. The axial distance between the focal points FI and FE, which is the distance between the two view points of the stereo system, is chosen to obtain the desired reconstruction accuracy for a given maximum distance. Geometric restrictions provide three relations between the internal and external hyperbole eccentricity, cI and cE, the angle λ, and the coordinates of focal points FI and FE. 2c I = z FI ; 2c E = rF2E + z F2E ; tan λ = rFE z FE .
(13) Equations (12) and (13) together with the expressions for the incident and reflected light rays with the maximum and the minimum elevation angles on the internal and external mirror lobes form a system of non-linear equations. The solution of this system results in the shapes (parameters) of the internal and external mirror lobes. 4.1 The mirror design
A double lobed mirror with hyperbolic profile was designed for a camera with a maximum image radius equals to 3.5945mm equipped with 25mm lens. The mirror has a field of view that encompasses elevation view angles of the scene between 22.6o and 85o. Adopting ∆zIE equals to 150mm and a maximum mirror radius equals to 60mm results into a rotation angle λ equals to 3.5067o. The main parameters of the mirror are shown in Table 1. Table 1. Main parameters of the mirror Parameter rFE rFI cE cI kE
Value 25.76mm 0.00mm 210.54mm 135.15mm 0.04190
Parameter zFE zFI aE aI kI
Value 420.30mm 270.30mm 193.61mm 123.79mm 0.04390
Fig. 3. Picture of the omnidirectional stereo system. The epipolar constraint for the vision system based on the double lobed mirror with hyperbolic sections vertically aligned are radial lines, and once the omnidirectional image is projected onto a cylinder (panoramic image) the epipolar curves become vertical lines. The arrangement of the double lobed mirror naturally ensures the vertical alignment of the two hyperboles, and since only one image generates the two views of the scene, the doubled lobed mirror also ensures matched epipolar lines both in the omnidirectional and panoramic images (see Fig. 4). Matched epipolar lines are desirable because efficient search methods can be used to perform stereo correspondence, thus, allowing fast and real-time three-dimensional reconstruction. The panoramic image is used for the correspondence problem. Corresponding points are search in the matched epipolar lines, which are the columns of the internal and external images. The horizontal edges are used as characteristics for correspondence. To quantify the similarity of corresponding pixels, the pixels intensity values at the horizontal edges positions in the original internal and external images are compared using a similarity function. The similarity function sum-of-absolute differences (SAD) is used. A seven-pixel length by
The camera sensor has 1280x1024 pixels, however the 62 central pixels are discharged from the omnidirectional image resulting into 225 pixels in height for each panoramic image of the two lobes. Figure 3 presents a picture of the omnidirectional vision system developed.
External image
5. CORRESPONDENCE A fundamental problem in stereo vision is establishing the matches between image elements in the two images (correspondence problem). The epipolar constraint reduces the problem of finding corresponding points in the two images to a onedimensional search.
Internal image (a)
(b)
Discharged pixels
Fig. 4. Epipolar geometry. (a) The epipolar lines are radial lines in the omnidirectional image. (b) When the image is projected onto a panorama the epipolar lines become vertical lines.
three-pixel wide window is used in the SAD. The SAD is preferred over other similarity functions for real-time applications due to its lower computational cost (Fielding and Kam, 2000). The correspondence method used is the dynamic programming (DP). Several studies have used this matching approach for stereo correspondence. The DP naturally accounts for the uniqueness of the corresponding pair. Central to the use of DP in stereo correspondence is the monotonic ordering constrain. The monotonic ordering constraint reduces the computational complexity of the DP from O(N2) to O(N), where N is the number of pixels in the epipolar lines (Fielding and Kam 2000). Maximum and minimum disparities constraints are added to the DP to minimize the computational cost. Although the dynamic programming presents a relatively high computational cost its performance justify its use.
Fig. 5. An example of an omnidirectional image.
6. ERROR ANALYSIS The finite number of pixels in the images (quantization error) and correspondence errors cause the errors present in the range estimation Souza Jr. (2004) presents an error analysis for the calculated position of a point in space. The error is a function of the position itself, i.e., the error depends on the radial distance (rP) and the height (zP) of the point. For a point located at 4000mm distance from the vision system the maximum error caused by an error in correspondence of one pixel is about 15% of the distance and for points located at a distance of 1000mm the error is about 3%. 7. RESULTS
The horizontal edges presented in the images are used as characteristic points for correspondence and range calculation. The horizontal edges are detected in the epipolar lines by checking the image intensity values in a seven-length pixel window. This window is passed through the epipolar lines of the internal and external panoramic images. If the relation between the minimum pixel intensity value and the mean intensity value of the pixels in the window is less than 0.75 an edge is consider existing in the central pixel of the window. This method detects only the high contrast edges. Figure 7 shows, as an example, the epipolar lines and the edges detected for the epilolar lines located at the 90o tangential angle.
Fig.6. Panoramic image obtained by projecting the omnidirectional image onto a cylinder. Figure 8 presents for each tangential angle the distances of the closest objects from the axis of the vision system. In Fig. 8 it is possible to observe the video monitor located at 500mm, and the four boxes 120
internal image external image edges of internal image edges of external image
100
Pixel intensity value
Figure 5 shows an omnidirectional image acquired with the system and Fig. 6 shows the panoramic image obtained by projecting the omnidirectional image onto a cylinder. The vision system is in the central position of a 4mx7m room. Four boxes are placed at about 1m around the vision system. Histogram equalization is applied in the panoramic image. The internal and external images in the panoramic image have 225 pixels in height and 360 pixels in length. Thus, the tangential resolution of the panoramic image is 1o.
80 60 40 20 0
0
50
100
Pixel
150
200
250
Fig. 7. Epipolar lines located at 90o tangential angle and the edges detected.
3500
Distance (mm)
3000
Point of the floor pattern
2500
The results show that the similarity function SAD associated with the dynamic programming are capable of estimating the range of the closest objects with very low computational cost. For mobile robot the closets horizontal edges (or the closest objects) are enough to provide information for safe robot navigation.
Points of the floor pattern
2000 1500 1000 500 0 0
Monitor 50
100
150 200 250 Tangential angle (degree)
300
350
Fig. 8. Distances of the closest objects from the vision system. located each one at a distance around 1000mm from the axis of the vision system. The distances of these objects detected by the system agree very well with the real positions. From Fig. 8 it is possible to note that the walls of the room are not detected very well by the system. The reason is the low contrast of the far edges and the difference in the shades presented in the internal and external images. Near objects are more important for collision avoidance and trajectory planning. For near objects edge detection and range estimation are performed by the system with very small errors. Thus, the errors presented in the range measurement do not hinder the application of the vision system for robot navigation. The edges belonging to the floor are almost all eliminated in Fig. 8 by the method used to detect edges which finds only the high contrast edges. However, few points of the floor pattern are still wrongly detected as belonging to objects. However, this is not a great problem for robot navigation since these points can be filter out by averaging the results obtained from several images. 8. CONCLUSIONS An omnidiectional stereo system based on a double lobed mirror with hyperbolic profile was designed and tested. The system can be used to give range measurements for the entirely panorama around the camera with the need for only one image. Using only a single camera eliminates the need for camera synchronization. As hyperbolic mirrors ensure a single viewpoint the incident light rays are easily found from the points of the image. This property and the perfect alignment of the images ensuring matched epipolar lines make this mirror especially suitable for real-time stereo calculations. Although there is an error associated with the distances measured, the results show that the sensor is capable of giving range measurements with the necessary accuracy for mobile robot navigation. Better accuracy could be achieved by smoothing the three-dimensional profiles obtained based on the errors.
The distances of the closest objects around the vision system can be used by a mobile robot navigation system, for example, to construct a map of the environment as the robot moves around, to find the robot position in the environment, and for collision avoidance. ACKNOWLEDGMENT The authors would like to tank Fundação de Amparo à Pesquisa do Estado de São Paulo, Brazil, (FAPESP) for the support of this project. REFERENCES Bonarini, A., P. Aliverti, and M. Lucioni (2000). An omnidirectional vision sensor for fast tracking for mobile robots, IEEE Trans. Instrumentation and Measurement, Vol. 49, No. 3. Conroy, T.L., and J.B. Moore (1999). Resolution invariant surfaces for panoramic vision systems, In IEEE ICCV'99, pp. 392-397. Delahoche, L., C. Pegard, B. Marhic, P. Vasseur (1997). A navigation system based on an ominidirectional vision sensor, In Intelligent Robots and Systems, Grenoble. Fielding, G. and M. Kam (2000). Weighted matchings for dense stereo correspondence, Pattern Recognition, Vol. 33, pp. 1511-1524. Gaspar, J.G., C. Deccó, J. Okamoto Jr., and J. S. Victor (2002). Constant resolution omnidirectional cameras, In OMNIVIS'02, Copenhagen. Gluckman, J., S.K. Nayar, and K.J. Thoresz (1998). Real-time omnidirectional and panoramic stereo, In Proc. of Image Understanding Workshop IUW’98. Koyasu, H., J. Miura, and Y. Shirai (2002). Recognizing moving obstacles for robot navigation using real-time omnidirectional stereo vision, J. of Robotics and Mechatronics, Vol. 14, No. 2, pp. 147-156. Shiguro, H., M. Yamamoto, and S. Tsuji (1992). Omni-directional stereo, IEEE Trans. Pattern Recognition and Machine Intell., Vol. 14, No. 2, pp. 225-262. Southwell, M.F.D., A. Basu, and J. Reyda (1996). Panoramic stereo, In ICPR; pp. 378-382, Vienna. Souza Jr., J.C. (2004). Omnidirectional stereo vision system base on a double lobed hyperbolic mirror, Ph.D. Thesis, Escola Politécnica of University of São Paulo, Brazil, (in Portuguese). Yamazawa, K., Y. Yagi, and M. Yachida (1999). Omnidirectional imaging with hyperboloidal projection, In IEEE/RSJ Int. Conf. Intell. Robots and Systems.
Dept. of Mechatronic Engineering Escola Politécnica of University of São Paulo, Brazil [email protected] (2)
Escola de Engenharia Mauá Instituto Mauá de Tecnologia, SP, Brazil [email protected]
Abstract: This paper presents a compact panoramic stereo vision system based on a double lobed mirror with hyperbolic profile. This omnidirectional vision system is used to obtain range estimations in a form appropriated for mobile robot navigation. As hyperbolic mirrors ensure a single viewpoint the incident light rays are easily found from the points of the image. The geometry of the double lobed mirror naturally ensures matched epipolar lines in the two images of the scene. These two properties make the double lobed mirror especially suitable for panoramic stereo vision because range estimation becomes simple and fast. This is a great advantage for real-time applications. Copyright © 2006 IFAC Keywords: Machine vision, Omnidirectional vision, Stereo vision, Range estimation, Hyperbolic mirror.
1. INTRODUCTION Panoramic imaging is an important tool in machine vision and it is widely used in the area of mobile robots, see for instance (Koyasu et al., 2002; Shiguro et al., 1992; Bonarini et al., 1997; Delahoche et al., 2000). Among the many possible types of panoramic vision systems an attractive approach is to mount a single fixed camera under a mirror covering a hemisphere. These systems are normally referred as catadioptric. The advantages of a catadioptric system are the compactness, the wide angle of view, and the fast acquisition rate. However, a disadvantage is that the images acquired have a relatively low spatial resolution. The shape of the mirror determines the image formation model of a catadioptric panoramic system. Different mirror shapes have been used and tested (Gaspar et al., 2002; Yamazawa et al., 1999; Conroy and Moore, 1999). A catadioptric system based on a
hyperbolic mirror and a perspective (pinhole) camera forms an omnidirecional vision system with a single effective viewpoint. This characteristic is especially suitable for stereo imaging because the incident light rays are easily determined from the points of the image. Another important tool for mobile robots and machine vision is stereo vision. Stereo vision allows determining the range to objects within the field of view of the system. Panoramic stereo vision system is very convenient because the field of view is 360o. Omnidirectional systems based on hyperbolic mirrors have been used for stereo vision as reported in (Koyasu et al., 2002; Gluckman, et al., 1998). However, such systems were developed based on two mirrors and two cameras. Besides these systems are not compact they have the problem of requiring precise positioning of the cameras and the mirrors to guarantee perfect alignment of the images.
Panoramic stereo based on a double lobed mirror was first suggested by Southwell et al. (1996). A double lobed mirror is essentially a coaxial mirror pair, where the centres of both mirrors are collinear with the camera axis, and the mirrors have a symmetric profile around this axis. This arrangement has the great advantage to produce two panoramic views of the scene in a single image. Thus, it naturally ensures the alignment of the two images of the scene and removes the need for camera synchronization. A resolution invariant double lobed mirror was developed by Yamazawa et al. (1999), however, a catadioptric stereovision system based on a constant resolution mirror is not adequate for fast threedimensional stereo reconstruction since it requires solving a non-linear differential equation to obtain the mirror shape function and its slope at the reflection point. This paper presents a catadioptric system based on a hyperbolic double lobed mirror. Efficient equations for the stereo triangulation based on a single image acquired using the omnidirectional vision system are presented. The single viewpoint and the natural alignment of the images ensuring matched epipolar lines both in the omnidirectional and panoramic images make this system ideal for fast real-time stereo calculations. Accurate range estimations are obtained at distances far enough to allow this vision system to be used for mobile robot navigation. 2. THE OMNIDIRECTIONAL STEREO VISION SYSTEM The omnidirectional stereo vision system is composed by a conventional perspective (pinhole) camera aligned vertically (coaxially) with the double lobed mirror. Due to the rotational symmetry of the system only the radial cross section of the mirror need to be considered. Figure 1 presents a schematic of the hyperbolic double lobed mirror showing two incident and reflective light rays on the two mirror lobes. A perspective camera together with a hyperbolic mirror form a wide-angle imaging system with two centers of projections, located at the hyperbole focal points. In Fig. 1 points F0 and FI are the two focal points of the internal lobe hyperbole, and points F0 and FE are the focal points of the external lobe hyperbole. All incident light rays on the internal lobe pass through focal point FI and are reflected passing the other focal point F0. Similarly, all incident light rays on the external hyperbole pass through focal point FE and are reflected passing the other focal point F0. The internal and external hyperbole focus F0 must coincide with the camera optical centre. The optical axis of the internal lobe hyperbole is aligned with the camera axis, while the optical axis of the external hyperbole is not aligned with the camera axis. The profile of the external lobe is a hyperbole rotated around point F0 by an angle λ. The rotation of the hyperbole of the external lobe is
zE
z
External lobe
FE ME FI
MI
Internal lobe
φE θI
φI
λ θE F0 f
P
Camera optical center
λ Image plane
ρE ρI
r rE
Fig.1. Schematic of the hyperbolic double lobed mirror showing the incident and reflective light rays on the mirror emitted from a point P in space. necessary to ensure the same field of view with the same resolution of the images generated by both lobes. A hyperbolic mirror is defined in the mirror coordinate system centred at focal point F0 as z = c + a b2 + r 2 / b ,
(1)
where c, b and a are parameters of the hyperbole, with the restriction that c2 = a2 + b2. These parameters are different for the internal and external lobes. Equation (1) is valid for the internal hyperbole described in the camera coordinate system (F0-rz), and for the external hyperbole described in the external lobe coordinate system (F0-rEzE). 2.1 Image formation model Given a point P in space as shown in Fig. 1, two light rays emitted from this point reach the mirror surface, one on the internal lobe and other on the external lobe. The incident light ray with the elevation angle φI reaches the internal mirror lobe at point MI and projects onto the image at radius ρI. The incident light ray with the elevation angle φE reaches the external mirror lobe at point ME and projects onto the image at radius ρE. The equation for any incident light ray on the surface of the internal mirror lobe is z = − rcotanφ I + z FI ,
(2)
where zFI is the z coordinate of point FI. Similarly, for any incident light ray on the external lobe surface, z = −(r − rFE )cotanφ E + z FE ,
(3)
where (rFE, zFE) are the coordinates of the focal point FE. For any light ray reflected both from the inner and outer mirror lobes the following equation holds
tan θ = ρ f = rM z M ,
(4)
where f is the lens focal distance, θ is the camera vertical angle, (rM, zM) are the coordinates of a point on the mirror surface, and ρ is the image radius. The coordinates of any point lying on the internal mirror surface must obey the internal lobe profile given by (1). Also, the coordinates of any point lying on the external mirror surface must obey the external hyperbole equation given in the F0-rEzE coordinate system by (1). Equation (5) gives the coordinate transformation from the camera to the external lobe coordinate systems. r cos λ z = − sin λ
sin λ r , cos λ z ( E )
(5)
3. CALCULATION OF RANGE Given a point P in space, as shown in Fig. 1, the information available to calculate its coordinates in space (rP, zP) are the image radiuses in the internal and external images (ρI and ρE). Note that the radiuses of the points in the images are obtained through the search of corresponding points. The coordinates of point P are obtained by solving simultaneously the equations of the incident light rays from P to each mirror lobe reflection points MI and ME. tan φI [tan φE ( zFI − zFE ) − rFE ] rP 1 = zP (tan φE − tan φI ) zFE tan φE − zI tan φI + rFE (6)
The only unknowns in (6) are the elevation angles (φI and φE). Thus, to obtain the position of a point in space the tangent of the elevation angles are needed. From first optics and geometric considerations tanφI can be derived to give: 2k I ρ I f + k I2 ( f +
ρ E( E ) cos λ (E) = f sin λ
f 2 + ρ 2I
f 2 + ρ 2I ) 2 − ρ 2I
,
tan φ E( E )
where kE = (cE – aE)/(cE + aE), and f(E) and ρ (EE ) are respectively the lens focal distance and the image
(10)
To avoid the calculation of the angles, (10) can be rewritten in terms of the tangent of these angles, i.e.,
tan φ E =
tan φ (EE ) − tan λ 1 + tan φ (EE ) tan λ
.
(11)
In summary, given the image radiuses of the corresponding points in the internal and external images (ρI and ρE), using (7), (8), (9) and (11) the tangents of the elevation angles of the incident light rays are calculated and through (6) the position of the point in space is obtained. Note that in this process it is not necessary to calculate the elevation angles but only their tangents, which makes this calculation very efficient, fast and easy to be implemented in a digital computer. 4. CALCULATION OF THE MIRROR PROFILE The two lobes are designed to have the same field of view, which is described by the minimum and maximum elevation angles of the scene (φmin and φmax). Figure 2 presents a schematic of the double lobed mirror showing the incident light rays with the minimum and maximum elevation angles and the corresponding reflected rays. To achieve best resolution the projection of the mirror has to occupy the whole image. To achieve equal resolution in both mirror lobes the two images must have the same radius range, i.e., (ρ E , max − ρ E , min ) = (ρ I , max − ρ I , min )
(12)
where, ρI,max, ρI,min, ρE,maxn and ρE,min are the maximum and minimum image radius of the internal and external lobes. FE
φmax
∆zIE
φmax
FI
Similarly for the external mirror lobe, the same analysis results into: 2 2 2k E ρ E( E ) f ( E ) + f ( E ) + ρ E( E ) , (8) = 2 (E) (E) 2 (E) 2 2 (E) 2 kE ( f + f + ρE ) − ρE
(9)
φ E = φ (EE ) − λ .
(7)
where , kI = (cI – aI)/(cI + aI). For the derivation of this equation see (Souza Jr., 2004).
− sin λ ρ E . cos λ f
The elevation angle φE in the coordinate system F0rz, is obtained from φ (EE ) as:
(E)
where (r(E), z(E)) are the coordinates of a point described in the external lobe coordinate system.
tan φ I =
radius given in the external lobe coordinate system (F0-rEzE). f(E) and ρ (EE ) are calculated by inverting (5).
φmin
φmin
Fig. 2. Schematic of the double lobed mirror showing the incident light rays with the maximum and the minimum elevation angles.
The known parameters for the mirror design are the elevation angles φmin and φmax, the lens focal distance (f), the maximum image radius (ρmax), the axial distance between focal points (∆zIE), and the maximum external radius of the mirror. With the exception of ρmax, these parameters are chosen based on the application requirements. The axial distance between the focal points FI and FE, which is the distance between the two view points of the stereo system, is chosen to obtain the desired reconstruction accuracy for a given maximum distance. Geometric restrictions provide three relations between the internal and external hyperbole eccentricity, cI and cE, the angle λ, and the coordinates of focal points FI and FE. 2c I = z FI ; 2c E = rF2E + z F2E ; tan λ = rFE z FE .
(13) Equations (12) and (13) together with the expressions for the incident and reflected light rays with the maximum and the minimum elevation angles on the internal and external mirror lobes form a system of non-linear equations. The solution of this system results in the shapes (parameters) of the internal and external mirror lobes. 4.1 The mirror design
A double lobed mirror with hyperbolic profile was designed for a camera with a maximum image radius equals to 3.5945mm equipped with 25mm lens. The mirror has a field of view that encompasses elevation view angles of the scene between 22.6o and 85o. Adopting ∆zIE equals to 150mm and a maximum mirror radius equals to 60mm results into a rotation angle λ equals to 3.5067o. The main parameters of the mirror are shown in Table 1. Table 1. Main parameters of the mirror Parameter rFE rFI cE cI kE
Value 25.76mm 0.00mm 210.54mm 135.15mm 0.04190
Parameter zFE zFI aE aI kI
Value 420.30mm 270.30mm 193.61mm 123.79mm 0.04390
Fig. 3. Picture of the omnidirectional stereo system. The epipolar constraint for the vision system based on the double lobed mirror with hyperbolic sections vertically aligned are radial lines, and once the omnidirectional image is projected onto a cylinder (panoramic image) the epipolar curves become vertical lines. The arrangement of the double lobed mirror naturally ensures the vertical alignment of the two hyperboles, and since only one image generates the two views of the scene, the doubled lobed mirror also ensures matched epipolar lines both in the omnidirectional and panoramic images (see Fig. 4). Matched epipolar lines are desirable because efficient search methods can be used to perform stereo correspondence, thus, allowing fast and real-time three-dimensional reconstruction. The panoramic image is used for the correspondence problem. Corresponding points are search in the matched epipolar lines, which are the columns of the internal and external images. The horizontal edges are used as characteristics for correspondence. To quantify the similarity of corresponding pixels, the pixels intensity values at the horizontal edges positions in the original internal and external images are compared using a similarity function. The similarity function sum-of-absolute differences (SAD) is used. A seven-pixel length by
The camera sensor has 1280x1024 pixels, however the 62 central pixels are discharged from the omnidirectional image resulting into 225 pixels in height for each panoramic image of the two lobes. Figure 3 presents a picture of the omnidirectional vision system developed.
External image
5. CORRESPONDENCE A fundamental problem in stereo vision is establishing the matches between image elements in the two images (correspondence problem). The epipolar constraint reduces the problem of finding corresponding points in the two images to a onedimensional search.
Internal image (a)
(b)
Discharged pixels
Fig. 4. Epipolar geometry. (a) The epipolar lines are radial lines in the omnidirectional image. (b) When the image is projected onto a panorama the epipolar lines become vertical lines.
three-pixel wide window is used in the SAD. The SAD is preferred over other similarity functions for real-time applications due to its lower computational cost (Fielding and Kam, 2000). The correspondence method used is the dynamic programming (DP). Several studies have used this matching approach for stereo correspondence. The DP naturally accounts for the uniqueness of the corresponding pair. Central to the use of DP in stereo correspondence is the monotonic ordering constrain. The monotonic ordering constraint reduces the computational complexity of the DP from O(N2) to O(N), where N is the number of pixels in the epipolar lines (Fielding and Kam 2000). Maximum and minimum disparities constraints are added to the DP to minimize the computational cost. Although the dynamic programming presents a relatively high computational cost its performance justify its use.
Fig. 5. An example of an omnidirectional image.
6. ERROR ANALYSIS The finite number of pixels in the images (quantization error) and correspondence errors cause the errors present in the range estimation Souza Jr. (2004) presents an error analysis for the calculated position of a point in space. The error is a function of the position itself, i.e., the error depends on the radial distance (rP) and the height (zP) of the point. For a point located at 4000mm distance from the vision system the maximum error caused by an error in correspondence of one pixel is about 15% of the distance and for points located at a distance of 1000mm the error is about 3%. 7. RESULTS
The horizontal edges presented in the images are used as characteristic points for correspondence and range calculation. The horizontal edges are detected in the epipolar lines by checking the image intensity values in a seven-length pixel window. This window is passed through the epipolar lines of the internal and external panoramic images. If the relation between the minimum pixel intensity value and the mean intensity value of the pixels in the window is less than 0.75 an edge is consider existing in the central pixel of the window. This method detects only the high contrast edges. Figure 7 shows, as an example, the epipolar lines and the edges detected for the epilolar lines located at the 90o tangential angle.
Fig.6. Panoramic image obtained by projecting the omnidirectional image onto a cylinder. Figure 8 presents for each tangential angle the distances of the closest objects from the axis of the vision system. In Fig. 8 it is possible to observe the video monitor located at 500mm, and the four boxes 120
internal image external image edges of internal image edges of external image
100
Pixel intensity value
Figure 5 shows an omnidirectional image acquired with the system and Fig. 6 shows the panoramic image obtained by projecting the omnidirectional image onto a cylinder. The vision system is in the central position of a 4mx7m room. Four boxes are placed at about 1m around the vision system. Histogram equalization is applied in the panoramic image. The internal and external images in the panoramic image have 225 pixels in height and 360 pixels in length. Thus, the tangential resolution of the panoramic image is 1o.
80 60 40 20 0
0
50
100
Pixel
150
200
250
Fig. 7. Epipolar lines located at 90o tangential angle and the edges detected.
3500
Distance (mm)
3000
Point of the floor pattern
2500
The results show that the similarity function SAD associated with the dynamic programming are capable of estimating the range of the closest objects with very low computational cost. For mobile robot the closets horizontal edges (or the closest objects) are enough to provide information for safe robot navigation.
Points of the floor pattern
2000 1500 1000 500 0 0
Monitor 50
100
150 200 250 Tangential angle (degree)
300
350
Fig. 8. Distances of the closest objects from the vision system. located each one at a distance around 1000mm from the axis of the vision system. The distances of these objects detected by the system agree very well with the real positions. From Fig. 8 it is possible to note that the walls of the room are not detected very well by the system. The reason is the low contrast of the far edges and the difference in the shades presented in the internal and external images. Near objects are more important for collision avoidance and trajectory planning. For near objects edge detection and range estimation are performed by the system with very small errors. Thus, the errors presented in the range measurement do not hinder the application of the vision system for robot navigation. The edges belonging to the floor are almost all eliminated in Fig. 8 by the method used to detect edges which finds only the high contrast edges. However, few points of the floor pattern are still wrongly detected as belonging to objects. However, this is not a great problem for robot navigation since these points can be filter out by averaging the results obtained from several images. 8. CONCLUSIONS An omnidiectional stereo system based on a double lobed mirror with hyperbolic profile was designed and tested. The system can be used to give range measurements for the entirely panorama around the camera with the need for only one image. Using only a single camera eliminates the need for camera synchronization. As hyperbolic mirrors ensure a single viewpoint the incident light rays are easily found from the points of the image. This property and the perfect alignment of the images ensuring matched epipolar lines make this mirror especially suitable for real-time stereo calculations. Although there is an error associated with the distances measured, the results show that the sensor is capable of giving range measurements with the necessary accuracy for mobile robot navigation. Better accuracy could be achieved by smoothing the three-dimensional profiles obtained based on the errors.
The distances of the closest objects around the vision system can be used by a mobile robot navigation system, for example, to construct a map of the environment as the robot moves around, to find the robot position in the environment, and for collision avoidance. ACKNOWLEDGMENT The authors would like to tank Fundação de Amparo à Pesquisa do Estado de São Paulo, Brazil, (FAPESP) for the support of this project. REFERENCES Bonarini, A., P. Aliverti, and M. Lucioni (2000). An omnidirectional vision sensor for fast tracking for mobile robots, IEEE Trans. Instrumentation and Measurement, Vol. 49, No. 3. Conroy, T.L., and J.B. Moore (1999). Resolution invariant surfaces for panoramic vision systems, In IEEE ICCV'99, pp. 392-397. Delahoche, L., C. Pegard, B. Marhic, P. Vasseur (1997). A navigation system based on an ominidirectional vision sensor, In Intelligent Robots and Systems, Grenoble. Fielding, G. and M. Kam (2000). Weighted matchings for dense stereo correspondence, Pattern Recognition, Vol. 33, pp. 1511-1524. Gaspar, J.G., C. Deccó, J. Okamoto Jr., and J. S. Victor (2002). Constant resolution omnidirectional cameras, In OMNIVIS'02, Copenhagen. Gluckman, J., S.K. Nayar, and K.J. Thoresz (1998). Real-time omnidirectional and panoramic stereo, In Proc. of Image Understanding Workshop IUW’98. Koyasu, H., J. Miura, and Y. Shirai (2002). Recognizing moving obstacles for robot navigation using real-time omnidirectional stereo vision, J. of Robotics and Mechatronics, Vol. 14, No. 2, pp. 147-156. Shiguro, H., M. Yamamoto, and S. Tsuji (1992). Omni-directional stereo, IEEE Trans. Pattern Recognition and Machine Intell., Vol. 14, No. 2, pp. 225-262. Southwell, M.F.D., A. Basu, and J. Reyda (1996). Panoramic stereo, In ICPR; pp. 378-382, Vienna. Souza Jr., J.C. (2004). Omnidirectional stereo vision system base on a double lobed hyperbolic mirror, Ph.D. Thesis, Escola Politécnica of University of São Paulo, Brazil, (in Portuguese). Yamazawa, K., Y. Yagi, and M. Yachida (1999). Omnidirectional imaging with hyperboloidal projection, In IEEE/RSJ Int. Conf. Intell. Robots and Systems.