- Email: [email protected]
Optical flow-based reactive navigation of a mobile robot
IAV2004 - PREPRINTS 5th IFAC/EURON Symposium on Intelligent Autonomous Vehicles Instituto Superior Técnico, Lisboa, Portugal July 5-7, 2004
OPTICAL FLOW-BASED REACTIVE NAVIGATION OF A MOBILE ROBOT
Mário Sarcinelli-Filho Hans Jörg Andreas Schneebeli Eliete Maria de Oliveira Caldeira Bruno Moreira Silva
Programa de Pós-Graduação em Engenharia Elétrica – UFES Av. Fernando Ferrari, s/n 29060-900 Vitória ES Brazil
Abstract: The use of optical flow to provide the sensorial information a mobile robot needs to navigate is addressed. An optical flow-based sensing subsystem is proposed to continuously detect obstacles, and a control subsystem is implemented to update the heading angle of the robot accordingly. All the computations are performed onboard the robot, due to the low memory and processing time demanded by the algorithms programmed, thus guaranteeing to the robot full autonomy. An experiment using such system is discussed, whose results validate the sensorial subsystem proposed. Copyright © 2002 IFAC
Keywords: Computer vision, Optical flow, Mobile robots, Motion parameters, Obstacle avoidance.
1. INTRODUCTION
(Dev et al., 1997; Sarcinelli-Filho et al., 2000) calculated from the optical flow field.
The problem addressed in this paper is the visualbased navigation of an autonomous wheeled mobile robot. The visual information is processed through the optical flow technique, which provides the information the robot needs to safely navigate in an unknown working-environment. The only one supposition about the robot working-environment is that it is an indoor environment with a plain ground, which guarantees that the robot will not slide.
In this paper, the optical flow technique is used to allow the robot to wander in its working-environment without colliding. It reacts to the environmental configuration itself or to the presence of people close to it. The objective is that the robot be able to avoid obstacles in real-time. Then, the main concerns of this work are: (1) to perform all the calculations onboard the robot, (2) to implement simpler algorithms to perform the optical flow and control action calculations, and (3) to account for the limited onboard computational setup. The objective of the first concern is to guarantee full autonomy to the robot, once it could navigate with no restriction on the distance to any external computational setup, like in (Dev et al., 1997) and (Carelli et al., 2002). The second concern is quite important to guarantee to the robot the capability of real-time response, mainly when considering the third concern previously mentioned. This last concern, by
The optical flow is the distribution of the apparent velocities of movement of the brightness pattern in an image, and arises from the relative movement of some objects and a viewer (Horn and Schunck, 1981). As the robot is moving around, even when the environment is static there will be a relative movement between the objects and the camera onboard the robot. From the optical flow thus generated the robot can get an inference on how deep an object present in a scene is. This information is embedded in the time to crash (or time to collision)
531
its turn, is a quite common feature of low cost mobile robots, like the one used in this work.
Many experiments run, however, have shown that such values are precise enough for the sake of wandering. An additional trick that allowed saving more computation time is the programming of the algorithm using the facilities of the Intel Open CV computer vision source library (Sarcinelli-Filho et al., 2002b).
One reason for choosing to implement the behaviour wander is that it demands to calculate the optical flow field over the entire image, and not exclusively over part of it. This last case is addressed in (Dev et al., 1997) and (Carelli et al., 2002), where it is implemented the behaviour corridor following, which demands to calculate the optical flow only over two small image regions. As it demands the calculation of the optical flow field over the entire image, thus being much more time consuming, the behaviour wander is more suitable for evaluating the possibility of using optical flow to provide the sensorial information the robot needs in real-time.
Based on the optical flow field thus obtained, the sensing subsystem delivers to the control subsystem information on the time to collision corresponding to distinct stripes in the visual field of the robot. The image, now represented by a matrix of 24x32 optical flow vectors, is segmented in different regions that execute similar movement (Sarcinelli-Filho et al., 2002a and 2002b). This allows characterizing distinct objects in the image frame considered and the distance from each of them to the robot (using the time to collision (Dev et al., 1997) corresponding to each object). The final result of such segmentation of movement is a vector of 32 values of time to collision. Each value of time to collision corresponds to the object closest to the robot in each angular interval of 1.525 degrees of the visual field of the robot (which is 48.8 degrees) (Sarcinelli-Filho et al., 2002a). Each angular interval of 1.525 degrees corresponds to a region whose width is 10 pixels, so that obtaining a vector of 32 times to collision is equivalent to using 32 CCD cameras, each one with a visual field of 1.525 degrees (Sarcinelli-Filho et al., 2002b).
The system here addressed for controlling the robot navigation is implemented onboard a Pioneer 2-DX mobile platform, for running the experiment presented in Section 4. Its onboard computer is based on the Intel Pentium MMX 266 MHz processor, having 128 Mbytes of RAM memory. A Sony D30 analogue camera and an Imagenation PXC200 frame grabber are also available onboard the robot, which deliver to the system images that are at most 480 pixelsx640 pixels bitmaps. In this work, however, 240 pixelsx320 pixels bitmaps are used, for the sake of reducing the image-acquisition time. Some important features of this work should be stressed, in comparison with other works in mobile robot navigation based on optical flow yet available in the literature. One is that all the calculations are performed onboard, using just the limited computational setup available, while other works use an external computer to process the images (Dev et al., 1997; Carelli et al., 2002).
Such vector of time to collision is the sensorial information used to control the heading angle of the robot, and is depicted in Figure 1. Such figure shows the first of the two image frames acquired, the optical flow vectors associated to it, and a bar diagram representing the vector of time to collision. Notice that the image frame is a 240 pixelsx320 pixels grey level bitmap and the values of time to collision corresponds to 32 regions of it, in the horizontal direction, as depicted in the bar diagram. The values
Another remarkable feature is that the heading angle of the robot, not its angular velocity, is here controlled, when avoiding obstacles. The main result of such strategy is that the robot performs smoother manoeuvres. The image resolution here adopted also deserves a remark: it is much greater than the resolution adopted in other works, like in (Lorigo et al., 1997), where images of 64 pixels by 64 pixels are processed. Also, neither a specialized hardware piece (Carelli et al., 2002) nor a special camera arrangement (Coombs et al., 1998; Camus et al., 1999) is adopted. This way, the experimental setup here used is the cheapest possible. 2. BACKGROUND The optical flow field used in this work is a rough least mean square estimate gotten from a small number of observations of the optical flow constraint (Sarcinelli-Filho et al., 2000; Sarcinelli-Filho et al., 2002a). The results of using such approach are a meaningful reduction in the computation time and a less precise inference of the time to collision values.
Fig. 1. The sensorial information delivered by the sensing subsystem.
532
in the vertical axis of the bar diagram are the times to collision, in seconds. From such figure, one can see that the information the control system of the robot receives is that the closest object is about the middle of its visual field, so that a deviation to the right or even to the left should be considered.
characterized. In such cases, the manoeuvre adopted is to rotate 15 degrees in the opposite direction and to resume moving ahead. Figure 4 illustrates a case of a side obstacle at the right side of the image frame. If two objects are detected, one in each side of the image frame, however, the system characterizes the situation like a situation of imminent collision ahead.
In addition to the vector of times to collision itself, the average value and the standard deviation corresponding to its elements are also calculated. However, in some situations an object can be either too close to the robot or moving towards it. In both cases the regions in the image frame considered that are detected as part of such object exhibit an optical flow vector of high magnitude. Thus, a simple comparison of this value with a threshold value allows detecting situations in which the robot is either about colliding to an object or close to an object moving towards it, both situations representing a risk. The value of such threshold is experimentally adjusted, and limits how close to an object the robot can get before starting an evasive manoeuvre.
The normal situation is characterized when no objects with high average optical flow magnitude are detected in the image frame. This means that all objects in the visual field of the robot are not close enough to deserve an abrupt manoeuvre. When this is the case, a new small heading angle is determined, the robot turns around its own axis according to such angle and after continues moving ahead. Such new heading angle is calculated as mentioned in (Sarcinelli-Filho et al., 2002b). Like there, each element of the vector of times to collision is considered as similar information coming from distinct sensors. Thus, such data are inputted to a data fusion algorithm, here a decentralized Kalman filter,
The time spent in calculating these three parameters (the average value and the standard deviation associated to the 32 values of time to collision used to build the graphic of Figure 1 and the average value of the magnitude of the optical flow vectors associated to each object detected in the scene) is very small, thus not decreasing the capability of the robot to react in real-time. 3. THE CONTROL SYSTEM IMPLEMENTED
(a)
Regarding the graphic shown in the bottom part of Figure 1 and the additional information associated to it, three distinct situations are here considered when implementing a system to control the heading angle of the robot. They are labelled imminent collision ahead, side obstacle and normal situation. Imminent collision ahead is characterized either when the average value and the standard deviation associated to the values of time to collision are both very small (meaning that a wide object is very close to the robot), or when an object for which the average magnitude of the optical flow vectors is greater than the threshold adopted is detected in the middle of an image frame. The middle of an image frame here means the elements 9 to 24 of the vector of times to collision. In such cases, the action the robot takes is to go back about 10 cm, to rotate 180 degrees and to move ahead again. Such situations are illustrated in Figure 2, were a wall is detected in front of the robot, and Figure 3, where the robot gets close to a waving hand, respectively.
(b)
However, if the object whose average magnitude of the optical flow vectors is greater than the threshold value is in the right side (columns 25 to 32) or in the left side (columns 1 to 8) of the visual field of the robot, the situation named side obstacle is
(c) Fig 2: Exemplifying the detection of a wall. Part (a) shows the wall, part (b) shows the optical flow vectors calculated, and part (c) shows the corresponding diagram of times to collision. 533
deviated a little to the right, firstly because of the chair leg and after because of the table leg, both appearing in the first image and not in the second one. The three manoeuvres above described represent the state changes in the state machine corresponding to the control system, when the robot is in the state Evaluation, as depicted in Figure 6. Thus, all the optical flow calculation and image segmentation, up to generate the vector of times to collision and to check for close objects, are performed when the robot gets into such state. The other circles in Figure 6 correspond to the three commands above mentioned (to go back 10 cm, to rotate and to move ahead), whose composition implements the above described composite manoeuvres. The state Idle in Figure 6 characterizes the robot state before being started. After the programmer starts the robot, it begins its navigation cycle by moving ahead, thus entering the state Evaluation. All the calculations associated to the generation of the vector of times to collision and the checking of objects close to the robot are then made. After having this vector available, the control system defines which situation (imminent collision ahead, side obstacle or normal situation) the robot is facing. According to the situation, the suitable manoeuvre is performed. In the normal situation case, the next state is to calculate the new heading angle (the current heading angle is always zero degrees). After finishing this calculation, the manoeuvre the robot performs is to rotate by the angle just calculated and after to move ahead again.
Fig. 3. Illustrating the detection of a waving human hand in the middle of the image. The real hand is in (a), while (b) shows just the part corresponding to the closest object detected. which generates the overall heading angle necessary to avoid the obstacles (in the range -15 degrees to +15 degrees). This situation is illustrated in Figure 5, which shows two images collected by the camera onboard the robot. A certain time has elapsed and some deviations have been performed, after getting the first image and before getting the second one. From the two images one can notice that the robot has
(a)
(b) Fig. 5. Illustrating the situation in which the robot performs smooth deviations (called normal situation). The images in parts (a) and (b) are two non-sequential images collected by the camera onboard the robot during its navigation.
Fig. 4. Detecting an object at the right side of the image. The object – a human leg – is in part (a), while part (b) shows the closest object detected. 534
End
Side obstacle
Evaluation
Imminent collision ahead
Go back
End
Rotate
End
End Normal situation
Calculate heading angle
Go ahead command
Start
Idle
Fig. 6. The state machine corresponding to the control system implemented. An important feature to be mentioned, in such context, is that the command execution is strictly sequential, as defined by the branches End in Figure 6. Only after the control system finishes the calculation of the new heading angle the rotation can start, as well as only after finishing the rotation the robot can start moving ahead again. In particular, the action to move ahead demands that the robot linear speed starts in zero and gets its final value (in this work 100 mm/s), which effectively demands a certain time to be performed. This way, the image frames used in the optical flow calculation are always acquired when the robot is performing pure translation.
Figure 7: The path followed by the robot while wandering. right and a rotation of a positive angle means a rotation to the left. Also, the two composite movements the robot executes are to go back 10 cm plus to rotate 180 plus to go ahead and to rotate much less than 180 plus to go ahead, as discussed in connection to Figure 6. By to go ahead, one should understand that the robot goes ahead, according to the new heading angle established, and its linear speed increases up to 100 mm/s, which consumes the time specified in Table I (about 1 s). As one can see, the time corresponding to the acquisition of two image frames, the calculation of the optical flow vectors plus the calculation of the new heading angle is compatible with the robot dynamics, thus showing that the use of optical flow for this kind of sensing is suitable, which is the main contribution of the paper.
4. EXPERIMENTAL RESULTS In order to check the performance of the robot when wandering in its working-environment using the sensing and control subsystems here discussed, an experiment is now presented. The robot is programmed to wander in the lab, avoiding all the chair legs, table legs, people legs, walls, etc., it detects.
Another feature to be mentioned is that the time corresponding to each cycle varies from about 1 second to about 4.5 seconds, thus characterizing an asynchronous system. Thus, it is not possible to define an image acquisition rate, when regarding the time between two consecutive instants the control system enters the state Evaluation. In addition, each image acquisition corresponds to the acquisition of two subsequent image frames, and the time to capture both frames is also not constant. It depends on the processes being executed in the onboard computer (a Windows ME computer).
Figure 7 shows the trajectory the robot followed when wandering. The layout of the lab has been superimposed to the figure in order to allow analyzing the robot actions. The robot, in this experiment, was programmed to get back to the state Idle after having its stop button pressed or via a special bottom in the interface menu. In such figure one can see that the robot stopped and rotated 180 when facing either a wall or a static person, and so on. An analysis of all the actions the robot has taken shows that it was effectively able to avoid the obstacles that appeared in its way, as expected, using only the optical flowbased sensorial information. Other examples have also shown that using optical flow it is possible to detect the movement of people inside the lab, which suggest the possibility of using such sensing system to follow a person, for example.
5. CONCLUSION A sensing subsystem based on optical flow is implemented to run in the limited computational setup available onboard a mobile robot. It is associated to a control subsystem designed to guide the robot to wander in an unknown indoor environment without colliding to any obstacle. The experimentation of such subsystems has shown that the robot is effectively able to avoid any obstacle, in real-time, based only on the information embedded in the optical flow.
In addition to Figure 7, Table I shows the times corresponding to the calculation of the vector of times to collision and to the movements the robot execute each time it gets into the state Evaluation. There, a rotation of a negative angle means a rotation to the
535
Table I - The time-table corresponding to the experimentation. Sequence Time to acquire two image frames and of to determine the new heading angle movements (ms) 1 141 2 115 3 137 4 121 5 138 6 135
Time to go back 10 cm (ms) 1003 1005
Time to rotate (ms) 504 1020 2584 1018 721 2474
Rotation angle (degrees)
Time to go ahead (ms)
6 12 180 -10 -4 180
1010 1005 1006 1011 1006 1007
Coombs, D., Herman, M., Hong, T. and Nashman, M. (1998). Real-time Obstacle Avoidance Using Central Flow Divergence and Peripheral Flow. IEEE Transactions on Robotics and Automation, 14, 49-59. Dev, A., Kröse, B. J. A. and Groen, F. C. A (1997). Navigation of a mobile robot on the temporal development of the optic flow. In: Proceedings of the 1997 IEEE/RSJ/GI International conference on Intelligent Robots and Systems (IROS'97), 558-563. Horn, B. K. P. and Schunck, B. G (1981). Determining optical flow. Artificial Intelligence, 17, 185-203. Lorigo, L. M., Brooks, R. A., Eric, W. and Grimson, L. (1997). Visually-Guided Obstacle Avoidance in Unstructured Environments. In: Proceedings of the 1997 IEEE/RSJ/GI International Conference on Intelligent Robots and Systems (IROS'97), 373-379. Sarcinelli-Filho, M., Schneebeli, H. J. A. and Caldeira, E. M. O (2000). On the use of optical flow in mobile robot navigation: The search for a suitable algorithm. In: Proceedings of the IEEE 43rd Midwest Symposium on Circuits and Systems, 922-925. Sarcinelli-Filho, M., Schneebeli, H. J. A. and Caldeira, E. M. O (2002a). Using optical flow to control mobile robot navigation. In: Proceedings of the XV IFAC World Congress on Automatic Control. Sarcinelli-Filho, M., Schneebeli, H. J. A., Caldeira, E. M. O. and Soria, C. M. (2002b). Optical flowbased obstacle detection and avoidance in mobile robot navigation. In: Proceedings of the 10th IEEE Mediterranean Conference on Control and Automation – MED2002.
ACKNOWLEDGEMENT The authors thank CNPq – National Council of Scientific and Technological Development, a Brazilian governmental institution promoting the scientific and technological development, for granting this research. They also thank CAPES (Brazil) and SETCIP (Argentina) for funding a partnership between Universidade Federal do Espírito Santo, Brazil, and Universidad Nacional de San Juan, Argentina, in which this work is included. As part of this partnership, Mrs. Eliete M. O. Caldeira got a scholarship to spend three months in San Juan, Argentina, working in this project. Dr. Mário Sarcinelli-Filho also thanks FACITEC, a Vitória city hall organism promoting scientific and technological development, for the additional financial support granted. REFERENCES Camus, T., Coombs, D., Herman., M. and Hong, T. H. (1999). Real-time Single-Workstation Obstacle Avoidance Using Only Wide-Field Flow Divergence. Videre: Journal of Computer Vision Research, MIT Press Journals, 1, 30-57. Carelli, R., Soria, C., Nasisi, O. and Freire, E. O. (2002). Stable AGV Corridor Navigation with Fused Vision-Based Control Signals. In: Proceedings of the 28th Annual Conference of the IEEE Industrial Electronics Society - IECON 02, 3, 2433-2438.
536
OPTICAL FLOW-BASED REACTIVE NAVIGATION OF A MOBILE ROBOT
Mário Sarcinelli-Filho Hans Jörg Andreas Schneebeli Eliete Maria de Oliveira Caldeira Bruno Moreira Silva
Programa de Pós-Graduação em Engenharia Elétrica – UFES Av. Fernando Ferrari, s/n 29060-900 Vitória ES Brazil
Abstract: The use of optical flow to provide the sensorial information a mobile robot needs to navigate is addressed. An optical flow-based sensing subsystem is proposed to continuously detect obstacles, and a control subsystem is implemented to update the heading angle of the robot accordingly. All the computations are performed onboard the robot, due to the low memory and processing time demanded by the algorithms programmed, thus guaranteeing to the robot full autonomy. An experiment using such system is discussed, whose results validate the sensorial subsystem proposed. Copyright © 2002 IFAC
Keywords: Computer vision, Optical flow, Mobile robots, Motion parameters, Obstacle avoidance.
1. INTRODUCTION
(Dev et al., 1997; Sarcinelli-Filho et al., 2000) calculated from the optical flow field.
The problem addressed in this paper is the visualbased navigation of an autonomous wheeled mobile robot. The visual information is processed through the optical flow technique, which provides the information the robot needs to safely navigate in an unknown working-environment. The only one supposition about the robot working-environment is that it is an indoor environment with a plain ground, which guarantees that the robot will not slide.
In this paper, the optical flow technique is used to allow the robot to wander in its working-environment without colliding. It reacts to the environmental configuration itself or to the presence of people close to it. The objective is that the robot be able to avoid obstacles in real-time. Then, the main concerns of this work are: (1) to perform all the calculations onboard the robot, (2) to implement simpler algorithms to perform the optical flow and control action calculations, and (3) to account for the limited onboard computational setup. The objective of the first concern is to guarantee full autonomy to the robot, once it could navigate with no restriction on the distance to any external computational setup, like in (Dev et al., 1997) and (Carelli et al., 2002). The second concern is quite important to guarantee to the robot the capability of real-time response, mainly when considering the third concern previously mentioned. This last concern, by
The optical flow is the distribution of the apparent velocities of movement of the brightness pattern in an image, and arises from the relative movement of some objects and a viewer (Horn and Schunck, 1981). As the robot is moving around, even when the environment is static there will be a relative movement between the objects and the camera onboard the robot. From the optical flow thus generated the robot can get an inference on how deep an object present in a scene is. This information is embedded in the time to crash (or time to collision)
531
its turn, is a quite common feature of low cost mobile robots, like the one used in this work.
Many experiments run, however, have shown that such values are precise enough for the sake of wandering. An additional trick that allowed saving more computation time is the programming of the algorithm using the facilities of the Intel Open CV computer vision source library (Sarcinelli-Filho et al., 2002b).
One reason for choosing to implement the behaviour wander is that it demands to calculate the optical flow field over the entire image, and not exclusively over part of it. This last case is addressed in (Dev et al., 1997) and (Carelli et al., 2002), where it is implemented the behaviour corridor following, which demands to calculate the optical flow only over two small image regions. As it demands the calculation of the optical flow field over the entire image, thus being much more time consuming, the behaviour wander is more suitable for evaluating the possibility of using optical flow to provide the sensorial information the robot needs in real-time.
Based on the optical flow field thus obtained, the sensing subsystem delivers to the control subsystem information on the time to collision corresponding to distinct stripes in the visual field of the robot. The image, now represented by a matrix of 24x32 optical flow vectors, is segmented in different regions that execute similar movement (Sarcinelli-Filho et al., 2002a and 2002b). This allows characterizing distinct objects in the image frame considered and the distance from each of them to the robot (using the time to collision (Dev et al., 1997) corresponding to each object). The final result of such segmentation of movement is a vector of 32 values of time to collision. Each value of time to collision corresponds to the object closest to the robot in each angular interval of 1.525 degrees of the visual field of the robot (which is 48.8 degrees) (Sarcinelli-Filho et al., 2002a). Each angular interval of 1.525 degrees corresponds to a region whose width is 10 pixels, so that obtaining a vector of 32 times to collision is equivalent to using 32 CCD cameras, each one with a visual field of 1.525 degrees (Sarcinelli-Filho et al., 2002b).
The system here addressed for controlling the robot navigation is implemented onboard a Pioneer 2-DX mobile platform, for running the experiment presented in Section 4. Its onboard computer is based on the Intel Pentium MMX 266 MHz processor, having 128 Mbytes of RAM memory. A Sony D30 analogue camera and an Imagenation PXC200 frame grabber are also available onboard the robot, which deliver to the system images that are at most 480 pixelsx640 pixels bitmaps. In this work, however, 240 pixelsx320 pixels bitmaps are used, for the sake of reducing the image-acquisition time. Some important features of this work should be stressed, in comparison with other works in mobile robot navigation based on optical flow yet available in the literature. One is that all the calculations are performed onboard, using just the limited computational setup available, while other works use an external computer to process the images (Dev et al., 1997; Carelli et al., 2002).
Such vector of time to collision is the sensorial information used to control the heading angle of the robot, and is depicted in Figure 1. Such figure shows the first of the two image frames acquired, the optical flow vectors associated to it, and a bar diagram representing the vector of time to collision. Notice that the image frame is a 240 pixelsx320 pixels grey level bitmap and the values of time to collision corresponds to 32 regions of it, in the horizontal direction, as depicted in the bar diagram. The values
Another remarkable feature is that the heading angle of the robot, not its angular velocity, is here controlled, when avoiding obstacles. The main result of such strategy is that the robot performs smoother manoeuvres. The image resolution here adopted also deserves a remark: it is much greater than the resolution adopted in other works, like in (Lorigo et al., 1997), where images of 64 pixels by 64 pixels are processed. Also, neither a specialized hardware piece (Carelli et al., 2002) nor a special camera arrangement (Coombs et al., 1998; Camus et al., 1999) is adopted. This way, the experimental setup here used is the cheapest possible. 2. BACKGROUND The optical flow field used in this work is a rough least mean square estimate gotten from a small number of observations of the optical flow constraint (Sarcinelli-Filho et al., 2000; Sarcinelli-Filho et al., 2002a). The results of using such approach are a meaningful reduction in the computation time and a less precise inference of the time to collision values.
Fig. 1. The sensorial information delivered by the sensing subsystem.
532
in the vertical axis of the bar diagram are the times to collision, in seconds. From such figure, one can see that the information the control system of the robot receives is that the closest object is about the middle of its visual field, so that a deviation to the right or even to the left should be considered.
characterized. In such cases, the manoeuvre adopted is to rotate 15 degrees in the opposite direction and to resume moving ahead. Figure 4 illustrates a case of a side obstacle at the right side of the image frame. If two objects are detected, one in each side of the image frame, however, the system characterizes the situation like a situation of imminent collision ahead.
In addition to the vector of times to collision itself, the average value and the standard deviation corresponding to its elements are also calculated. However, in some situations an object can be either too close to the robot or moving towards it. In both cases the regions in the image frame considered that are detected as part of such object exhibit an optical flow vector of high magnitude. Thus, a simple comparison of this value with a threshold value allows detecting situations in which the robot is either about colliding to an object or close to an object moving towards it, both situations representing a risk. The value of such threshold is experimentally adjusted, and limits how close to an object the robot can get before starting an evasive manoeuvre.
The normal situation is characterized when no objects with high average optical flow magnitude are detected in the image frame. This means that all objects in the visual field of the robot are not close enough to deserve an abrupt manoeuvre. When this is the case, a new small heading angle is determined, the robot turns around its own axis according to such angle and after continues moving ahead. Such new heading angle is calculated as mentioned in (Sarcinelli-Filho et al., 2002b). Like there, each element of the vector of times to collision is considered as similar information coming from distinct sensors. Thus, such data are inputted to a data fusion algorithm, here a decentralized Kalman filter,
The time spent in calculating these three parameters (the average value and the standard deviation associated to the 32 values of time to collision used to build the graphic of Figure 1 and the average value of the magnitude of the optical flow vectors associated to each object detected in the scene) is very small, thus not decreasing the capability of the robot to react in real-time. 3. THE CONTROL SYSTEM IMPLEMENTED
(a)
Regarding the graphic shown in the bottom part of Figure 1 and the additional information associated to it, three distinct situations are here considered when implementing a system to control the heading angle of the robot. They are labelled imminent collision ahead, side obstacle and normal situation. Imminent collision ahead is characterized either when the average value and the standard deviation associated to the values of time to collision are both very small (meaning that a wide object is very close to the robot), or when an object for which the average magnitude of the optical flow vectors is greater than the threshold adopted is detected in the middle of an image frame. The middle of an image frame here means the elements 9 to 24 of the vector of times to collision. In such cases, the action the robot takes is to go back about 10 cm, to rotate 180 degrees and to move ahead again. Such situations are illustrated in Figure 2, were a wall is detected in front of the robot, and Figure 3, where the robot gets close to a waving hand, respectively.
(b)
However, if the object whose average magnitude of the optical flow vectors is greater than the threshold value is in the right side (columns 25 to 32) or in the left side (columns 1 to 8) of the visual field of the robot, the situation named side obstacle is
(c) Fig 2: Exemplifying the detection of a wall. Part (a) shows the wall, part (b) shows the optical flow vectors calculated, and part (c) shows the corresponding diagram of times to collision. 533
deviated a little to the right, firstly because of the chair leg and after because of the table leg, both appearing in the first image and not in the second one. The three manoeuvres above described represent the state changes in the state machine corresponding to the control system, when the robot is in the state Evaluation, as depicted in Figure 6. Thus, all the optical flow calculation and image segmentation, up to generate the vector of times to collision and to check for close objects, are performed when the robot gets into such state. The other circles in Figure 6 correspond to the three commands above mentioned (to go back 10 cm, to rotate and to move ahead), whose composition implements the above described composite manoeuvres. The state Idle in Figure 6 characterizes the robot state before being started. After the programmer starts the robot, it begins its navigation cycle by moving ahead, thus entering the state Evaluation. All the calculations associated to the generation of the vector of times to collision and the checking of objects close to the robot are then made. After having this vector available, the control system defines which situation (imminent collision ahead, side obstacle or normal situation) the robot is facing. According to the situation, the suitable manoeuvre is performed. In the normal situation case, the next state is to calculate the new heading angle (the current heading angle is always zero degrees). After finishing this calculation, the manoeuvre the robot performs is to rotate by the angle just calculated and after to move ahead again.
Fig. 3. Illustrating the detection of a waving human hand in the middle of the image. The real hand is in (a), while (b) shows just the part corresponding to the closest object detected. which generates the overall heading angle necessary to avoid the obstacles (in the range -15 degrees to +15 degrees). This situation is illustrated in Figure 5, which shows two images collected by the camera onboard the robot. A certain time has elapsed and some deviations have been performed, after getting the first image and before getting the second one. From the two images one can notice that the robot has
(a)
(b) Fig. 5. Illustrating the situation in which the robot performs smooth deviations (called normal situation). The images in parts (a) and (b) are two non-sequential images collected by the camera onboard the robot during its navigation.
Fig. 4. Detecting an object at the right side of the image. The object – a human leg – is in part (a), while part (b) shows the closest object detected. 534
End
Side obstacle
Evaluation
Imminent collision ahead
Go back
End
Rotate
End
End Normal situation
Calculate heading angle
Go ahead command
Start
Idle
Fig. 6. The state machine corresponding to the control system implemented. An important feature to be mentioned, in such context, is that the command execution is strictly sequential, as defined by the branches End in Figure 6. Only after the control system finishes the calculation of the new heading angle the rotation can start, as well as only after finishing the rotation the robot can start moving ahead again. In particular, the action to move ahead demands that the robot linear speed starts in zero and gets its final value (in this work 100 mm/s), which effectively demands a certain time to be performed. This way, the image frames used in the optical flow calculation are always acquired when the robot is performing pure translation.
Figure 7: The path followed by the robot while wandering. right and a rotation of a positive angle means a rotation to the left. Also, the two composite movements the robot executes are to go back 10 cm plus to rotate 180 plus to go ahead and to rotate much less than 180 plus to go ahead, as discussed in connection to Figure 6. By to go ahead, one should understand that the robot goes ahead, according to the new heading angle established, and its linear speed increases up to 100 mm/s, which consumes the time specified in Table I (about 1 s). As one can see, the time corresponding to the acquisition of two image frames, the calculation of the optical flow vectors plus the calculation of the new heading angle is compatible with the robot dynamics, thus showing that the use of optical flow for this kind of sensing is suitable, which is the main contribution of the paper.
4. EXPERIMENTAL RESULTS In order to check the performance of the robot when wandering in its working-environment using the sensing and control subsystems here discussed, an experiment is now presented. The robot is programmed to wander in the lab, avoiding all the chair legs, table legs, people legs, walls, etc., it detects.
Another feature to be mentioned is that the time corresponding to each cycle varies from about 1 second to about 4.5 seconds, thus characterizing an asynchronous system. Thus, it is not possible to define an image acquisition rate, when regarding the time between two consecutive instants the control system enters the state Evaluation. In addition, each image acquisition corresponds to the acquisition of two subsequent image frames, and the time to capture both frames is also not constant. It depends on the processes being executed in the onboard computer (a Windows ME computer).
Figure 7 shows the trajectory the robot followed when wandering. The layout of the lab has been superimposed to the figure in order to allow analyzing the robot actions. The robot, in this experiment, was programmed to get back to the state Idle after having its stop button pressed or via a special bottom in the interface menu. In such figure one can see that the robot stopped and rotated 180 when facing either a wall or a static person, and so on. An analysis of all the actions the robot has taken shows that it was effectively able to avoid the obstacles that appeared in its way, as expected, using only the optical flowbased sensorial information. Other examples have also shown that using optical flow it is possible to detect the movement of people inside the lab, which suggest the possibility of using such sensing system to follow a person, for example.
5. CONCLUSION A sensing subsystem based on optical flow is implemented to run in the limited computational setup available onboard a mobile robot. It is associated to a control subsystem designed to guide the robot to wander in an unknown indoor environment without colliding to any obstacle. The experimentation of such subsystems has shown that the robot is effectively able to avoid any obstacle, in real-time, based only on the information embedded in the optical flow.
In addition to Figure 7, Table I shows the times corresponding to the calculation of the vector of times to collision and to the movements the robot execute each time it gets into the state Evaluation. There, a rotation of a negative angle means a rotation to the
535
Table I - The time-table corresponding to the experimentation. Sequence Time to acquire two image frames and of to determine the new heading angle movements (ms) 1 141 2 115 3 137 4 121 5 138 6 135
Time to go back 10 cm (ms) 1003 1005
Time to rotate (ms) 504 1020 2584 1018 721 2474
Rotation angle (degrees)
Time to go ahead (ms)
6 12 180 -10 -4 180
1010 1005 1006 1011 1006 1007
Coombs, D., Herman, M., Hong, T. and Nashman, M. (1998). Real-time Obstacle Avoidance Using Central Flow Divergence and Peripheral Flow. IEEE Transactions on Robotics and Automation, 14, 49-59. Dev, A., Kröse, B. J. A. and Groen, F. C. A (1997). Navigation of a mobile robot on the temporal development of the optic flow. In: Proceedings of the 1997 IEEE/RSJ/GI International conference on Intelligent Robots and Systems (IROS'97), 558-563. Horn, B. K. P. and Schunck, B. G (1981). Determining optical flow. Artificial Intelligence, 17, 185-203. Lorigo, L. M., Brooks, R. A., Eric, W. and Grimson, L. (1997). Visually-Guided Obstacle Avoidance in Unstructured Environments. In: Proceedings of the 1997 IEEE/RSJ/GI International Conference on Intelligent Robots and Systems (IROS'97), 373-379. Sarcinelli-Filho, M., Schneebeli, H. J. A. and Caldeira, E. M. O (2000). On the use of optical flow in mobile robot navigation: The search for a suitable algorithm. In: Proceedings of the IEEE 43rd Midwest Symposium on Circuits and Systems, 922-925. Sarcinelli-Filho, M., Schneebeli, H. J. A. and Caldeira, E. M. O (2002a). Using optical flow to control mobile robot navigation. In: Proceedings of the XV IFAC World Congress on Automatic Control. Sarcinelli-Filho, M., Schneebeli, H. J. A., Caldeira, E. M. O. and Soria, C. M. (2002b). Optical flowbased obstacle detection and avoidance in mobile robot navigation. In: Proceedings of the 10th IEEE Mediterranean Conference on Control and Automation – MED2002.
ACKNOWLEDGEMENT The authors thank CNPq – National Council of Scientific and Technological Development, a Brazilian governmental institution promoting the scientific and technological development, for granting this research. They also thank CAPES (Brazil) and SETCIP (Argentina) for funding a partnership between Universidade Federal do Espírito Santo, Brazil, and Universidad Nacional de San Juan, Argentina, in which this work is included. As part of this partnership, Mrs. Eliete M. O. Caldeira got a scholarship to spend three months in San Juan, Argentina, working in this project. Dr. Mário Sarcinelli-Filho also thanks FACITEC, a Vitória city hall organism promoting scientific and technological development, for the additional financial support granted. REFERENCES Camus, T., Coombs, D., Herman., M. and Hong, T. H. (1999). Real-time Single-Workstation Obstacle Avoidance Using Only Wide-Field Flow Divergence. Videre: Journal of Computer Vision Research, MIT Press Journals, 1, 30-57. Carelli, R., Soria, C., Nasisi, O. and Freire, E. O. (2002). Stable AGV Corridor Navigation with Fused Vision-Based Control Signals. In: Proceedings of the 28th Annual Conference of the IEEE Industrial Electronics Society - IECON 02, 3, 2433-2438.
536