- Email: [email protected]
Robust visual servoing for motion control of the ball on a plate
Mechatronics 13 (2003) 723–738
Robust visual servoing for motion control of the ball on a plate Jong Hyeon Park
a,*
, Young Jong Lee
b
a
b
School of Mechanical Engineering, Hanyang University, Seoul 133-791, Republic of Korea Department of Precision Mechanical Engineering, Hanyang University, Seoul 133-791, Republic of Korea
Abstract This paper presents visual servoing control for a ball on a flat plate to track its desired trajectory. An industrial robot manipulator is used as the actuators to change the angle of the plate so that the ball can track its given trajectory on the plate. The center position of the ball is measured with a machine vision system. Based on the Euler estimator, an algorithm to find the center of the ball with interlaced-scanned image data is proposed. To cope with the variations in the surface characteristics of the plate, and the size and mass of the ball, a slidingmode control is used for the visual servoing. Experimental results show that the proposed algorithm and the sliding-mode controller work well with robustness. Ó 2002 Elsevier Science Ltd. All rights reserved. Keywords: Machine vision; Sliding-mode control; Visual servoing; Interlaced scanning
1. Introduction When a human operator performs a task, he or she uses various sensing organs to deal with any change or uncertainty involved in the task and the environment. Among many sensing organs, eye vision is the prime information route to gather the data on the environment in many tasks. The operator’s hand-eye coordination is a key to success in many cases. For robots, which have the capability of positioning their end-effectors very precisely, without sensors such as vision systems, they are limited to performing simple tasks in a pre-determined environment. As in human counterparts, one of the prime sensors that robots need in performing tasks dexterously is vision. Visual information becomes more important especially when the *
Corresponding author. Tel.: +822-2290-0435; fax: +822-2298-4634. E-mail address: [email protected] (Jong H. Park).
0957-4158/03/$ - see front matter Ó 2002 Elsevier Science Ltd. All rights reserved. doi:10.1016/S0957-4158(02)00039-9
724
J.H. Park, Y.J. Lee / Mechatronics 13 (2003) 723–738
environment changes dynamically and no other non-contact means to measure the status of the changing environment is feasible. Many researchers have used machine vision systems in performing difficult tasks efficiently. Shirai and Inoue [1] described how a visual feedback loop can be used to correct the position of a robot to increase task accuracy. Hashimoto [2] proved the usefulness of the visual feedback using a robotic system. Hutchinson et al. [3] provides a good conceptual introduction to visual servoing and categorizes visual servoing into different sub-categories. Cameras can be fixed or moved around independent of the robot configuration, or can be attached to the robot so that its position and orientations are directly dependent on the robot configuration as in [4]. Control systems can be either position-based [5,6] or image-based [7]. In the former, the visual data is used to compute the relative position of the object(s) to the camera. In this paper, a machine vision system based on position control is proposed and used in order to control the trajectory of the ball on a flat plate. The vision system, fixed in a location, collects the information on the position of the ball on the plate. A fast algorithm to detect the position of the ball on the image plane combined with the Euler position estimator results in a fast control-loop at the regular frame rate. A 6DOF industrial robot is used to control the orientation of the plate to move the ball around on the plate. The visual servoing system is controlled by the sliding-mode controller, which has robustness to system uncertainty, and unmodelled disturbances. Section 2 describes an algorithm of searching the ball from the image data and the relationship between the world coordinates and the image coordinates. In Section 3, the dynamics of a ball on a plate and the sliding-mode controller is covered. Experiments and their results are covered in Section 4, followed by conclusions in Section 5.
2. Vision system 2.1. Image processing The vision system used in this paper takes binary data after digitizing the image data at the hardware level, the threshold levels for which are stored in a lookup table of the frame grabber. It is advantageous to use binary data in differentiating an object from its near-monotonic surroundings fast [8]. In the vision system used, one full-frame of image consists of 640 480 pixels. Searching the whole video data of a full-frame for a small moving object usually takes quite long time, and thus degrades the performance of the visual servoing system. Thus, based on the size of the ball in the image plane, only 100 100-pixelsized image data around the estimated position of the ball are searched to find the precise position of the ball (center), as shown in Fig. 1. Estimated position of the ball at ðk þ 1Þ-th frame, denoted by xb ðk þ 1Þ, is calculated using the Euler estimator [9], i.e.,
J.H. Park, Y.J. Lee / Mechatronics 13 (2003) 723–738
725
Fig. 1. Estimation of the ball center. i i
x^b ðk þ 1Þ ¼ i xb ðkÞ þ
xb ðkÞ i xb ðk 1Þ ; Dt
ð1Þ
where i xb ðkÞ is the measured position of the ball on the image plane at kth frame and Dt is the time gap between the kth and the ðk 1Þth frame samples. The center position of the ball in the image plane is computed by averaging the maximum position and the minimum positions of the image blob associated with the ball. Thus, i
xb ðkÞ ¼ 12½ max i x 2 BðkÞ þ min i x 2 BðkÞ ;
ð2Þ
where i x 2 R2 denotes the pixel index on the image plane and BðkÞ denotes the image blob at frame k that represents the ball. When an object moves fast, its image taken from video cameras with interlaced scanning may become blurred. Fig. 2 shows the images taken when a circular object moves fast horizontally and vertically. Thus, interlaced scanning, which scans even and odd fields of the video signal separately, introduces image distortions [10], and thus prevents an accurate measurement of the position of the object. To minimize the error in the position of the ball, only one field (even or odd) of the video data is used to measure the center of the ball. Fig. 3 shows the priority masks used for the algorithm. Mask Mx min in the figure is used to find the pixel that represents the lowerbound of the image blob of the ball in the x- direction. To find the bound, the following steps are taken: Step 1. Place the center of the mask, mc at the previously estimated center of the ball, i x^b ðk 1Þ.
726
J.H. Park, Y.J. Lee / Mechatronics 13 (2003) 723–738
Fig. 2. Image distortion due to the interlaced scanning: when a circular object moves: (a) horizontally, and (b) vertically.
Step 2. If the pixel at ‘‘first’’ is one of the pixels representing the ball, then the mask is moved such that its center is located at that pixel. Then, go back to Step 1. Step 3. If the pixel at ‘‘first’’ does not represent the ball, check the pixel at ‘‘second’’ of the mask. If it represents the ball, then the center of the mask is moved to the position of the pixel. Then, go back to Step 1. Step 4. If the pixel at ‘‘second’’ is not associated with the image of the ball, check if the pixel at ‘‘third’’ is associated with the image of the ball. If so, the center of the mask is shifted down to the position of the pixel, and then go back to Step 1. Otherwise, the lower-bound in the x- direction, i.e., min i x 2 BðkÞ, is set as the pixel at the center of the mask. To find the upper-bound pixel in the x- direction, Mx max , is used in a similar fashion to move the mask to the right of the object blob on the image plane. Sim-
J.H. Park, Y.J. Lee / Mechatronics 13 (2003) 723–738
727
Fig. 3. A mask used in computing the center of a ball.
ilarly, to find the upper- and lower-bounds in the y-direction, masks My max and My min are used, respectively. 2.2. Camera projection model In setting up the relationship between an object and its image on the image plane, the so-called scaled orthographic projection is often used especially when the motion range of the object in the direction perpendicular to the image plane is small compared with the distance between the object and the camera. Since that the depth variation of the ball on the plate is small due to small changes in the plane orientation as shown in Fig. 4, the scaled orthographic projection is used. The relationship between the reference coordinates and the image coordinates in the scaled orthographic projection [11] can be expressed by i x x ; ð3Þ ¼ S c i y y where ðx; yÞ is the position of the object about the camera reference coordinate and ði x; i yÞ is the position of the object in the image coordinates on the image plane. And,
728
J.H. Park, Y.J. Lee / Mechatronics 13 (2003) 723–738
Fig. 4. Configuration of the camera and the plate.
matrix Sc 2 R22 is a scale factor, which depends on many factors including the distance between the camera and the object, and the focal length of the camera lens. 3. Controller design 3.1. Equations of motion for the ball on a plate Suppose that the ball and the plate are aligned as in Fig. 5. The x–y plane is coplanar with the plane on which the ball is located. The z-direction is upward and normal to the plane. It will be assumed that the ball on the plane is perfectly a sphere. Then, from Newton’s second law, Ty mR€x J ay þ mgR sin hy ¼ 0;
ð4Þ
Tx þ mR€y J ax þ mgR sin hx ¼ 0;
ð5Þ
where R, m and J are the radius, the mass and the moment inertia of the ball; hx and hy are the inclination angle of the plate in the x- and y-direction; ax and ay are the angular acceleration of the ball in the x- and y-direction; Tx and Ty are the torque generated in the x- and y-direction respectively, at the contact area between the ball and the plate, associated with the deformation of the plate (and the ball) due to the contact force between them. Torque Tx and Ty are strongly depends on the type of the materials and the surface condition, such as roughness, of the ball and the plate. If the ball is uniform, its moment of inertia is J ¼ 25mR2 :
J.H. Park, Y.J. Lee / Mechatronics 13 (2003) 723–738
Fig. 5. The ball rolls down the inclined plate.
Fig. 6. Overall system block diagram.
729
730
J.H. Park, Y.J. Lee / Mechatronics 13 (2003) 723–738
In general, for a ball that is made of two different materials, J ¼ kmR2
2 Ml 2 Mh
ð6Þ
where Mh and Ml are the mass of the ball made of the heaviest material and the lightest material, respectively. Assuming that the plate inclination angle h is small, and thus sin hx hx and sin hy hy , and that the ball rolls without slip so that €x ¼ Ray and €y ¼ Rax , Eqs. (4) and (5) can be transformed into i
€xb i €yb
Sc Sc g hy Ty ¼ þ k þ 1 hx ðk þ 1ÞmR Tx
ð7aÞ
or €x ¼ f þ Qu;
ð7bÞ
where Sc Ty ; f ¼ ðk þ 1ÞmR Tx
Sc g ; Q¼ kþ1
hy u¼ : hx
Note that vector f cannot be identified and will be treated as a noise and that matrix Q can be assumed to be diagonal.
Fig. 7. Tracking an elliptical trajectory with a steel ball and a plastic plate.
J.H. Park, Y.J. Lee / Mechatronics 13 (2003) 723–738
731
3.2. Sliding-mode controller Since that sliding-mode controllers have robustness to uncertainty and unmodelled dynamics, they have been applied to robot manipulators [12–16]. In this article, a sliding-mode controller is designed to cope with the changes in the material of the plate and the material and the size of the ball as well as other model uncertainty. It is also assumed that the motion of the plate, and thus the ball is slow enough to neglect the dynamics of the industrial robot. In describing the controller, superscript i denoting the property in the image plane will be dropped for notational convenience. Let xd ðtÞ and xðtÞ be the desired trajectory of the ball (center), and the measured position of the ball at time t, with respect to the image coordinates. The sliding surface is then defined as sðtÞ ¼ x_ ðtÞ x_ d ðtÞ þ Kð xðtÞ xd ðtÞÞ ¼ ~x_ ðtÞ þ K~xðtÞ;
Fig. 8. Errors in tracking an elliptical trajectory with a steel ball and a plastic plate.
ð8Þ
732
J.H. Park, Y.J. Lee / Mechatronics 13 (2003) 723–738
where K is symmetric and positive definite. Then, the equivalent control that is asymptotically stable can be obtained using Eqs. (7b) and (8), and the condition s_ ðtÞ ¼ 0: h i b 1 €xd ðtÞ K~x_ ðtÞ f^ ; u^ðtÞ ¼ Q ð9Þ b are the estimate of f and Q, respectively. It is assumed where vector f^ and matrix Q that vector f has its upper- and lower-bound such that jf^ f j 6 Df
ð10Þ
and that diagonal matrix Q is also bounded by its lower-bound QL and upper-bound QU such that ð11Þ
0 < QL 6 Q 6 QU : If the estimate of matrix Q is selected as b ¼ ½ QL Q
QU 1=2 ;
ð12Þ
then 11
b 1 6 Q Q QL QU 2 6 Q Q U L 1=2
1=2
1=2
:
ð13Þ
In order to have robustness to uncertainty in the modelling, a discontinuous gain term is added to the equivalent control in Eq. (9) to result in the control input: h i b 1 €xd ðtÞ K~x_ ðtÞ f^ K sgnðsÞ ; ð14Þ uðtÞ ¼ Q where positive definite matrix K is diagonal such that K ¼ diagðk1 ; k2 Þ, and
Fig. 9. Tracking an elliptical trajectory with a compound ball and a plastic plate.
J.H. Park, Y.J. Lee / Mechatronics 13 (2003) 723–738
733
sgnðs1 Þ sgnðsÞ ¼ : sgnðs2 Þ From Eqs. (14) and (8), b 1 ð€xd K~x_ f^ K sgnðsÞÞ €xd þ K~x_ þ f T s s_ T s ¼ ½Q Q b 1 K sgnðsÞ T s: b 1 IÞð€xd K~x_ f^Þ þ f f^ Q Q ¼ ½ðQ Q To meet the stability condition of js j s_ T s 6 g 1 ; js2 j where g ¼ ½ g1
T
g2 ;
g1 ; g2 > 0;
Fig. 10. Errors in tracking an elliptical trajectory with a compound ball and a plastic plate.
ð15Þ
734
J.H. Park, Y.J. Lee / Mechatronics 13 (2003) 723–738
matrix K must be selected such that k1 b Q1 ðf f^Þ þ Q b Q1 g: b Q1 Þð€xd K~x_ f^Þ þ Q P ðI Q k2
ð16Þ
Using the bounds of f and Q in Eqs. (10) and (13), the constraint on K may be also expressed by
k1 1=2 1=2 1=2 €xd K~x_ f^ þ Q1=2 ðDf þ gÞ: ð17Þ P QU QL U QL k2 By using the saturation function with boundary layer thickness U instead of the sign function in order to remove chattering from the control law, the sliding-mode control becomes h i b 1 €xd ðtÞ Kð_xðtÞ x_ d ðtÞÞ f^ K sat s : uðtÞ ¼ Q ð18Þ U 4. Experiments and results A series of experiments were performed to measure the performance of the visual servoing system. The overall system components used in the experiments are shown in Fig. 6. The host computer, a Pentium II, generates the desired trajectory for the ball and also works as the sliding-mode controller. The vision board, a frame grabber, collects and stores video data in the binary form in the host computer memory. The MMC board in a PC slot generates the servo control input to the servo motor drives for an 6-DOF industrial robot, Samsung Fara-AS1. The servo motor drives are off-the-shelf products. The desired trajectory for the ball to track on the
Fig. 11. Tracking an elliptical trajectory with a steel ball and a rubber-coated plate.
J.H. Park, Y.J. Lee / Mechatronics 13 (2003) 723–738
735
plate are given in the form of an ellipsoid and a triangle. In order to verify the robustness of the system, different plates and different balls were tried out. In the first experiment, the ball on the plate is to follow an elliptical track at a constant speed. The ball is 17 mm in diameter, made of steel, and weighs about 28 g. The plate is made of plastic and has a smooth flat surface. The resulting track of the ball is shown in Fig. 7 in terms of the image coordinates. A dashed line is the reference trajectory and a solid line is the trajectory of the ball. Marks ‘þ’ and ‘ ’ denote the initial and the final position of the ball, respectively. Fig. 8 shows the tracking errors in the x- and y-axis, which are smaller than 7 mm, corresponding to 10 pixels. In the next experiment, the same plate was used but with a different ball, a compound ball made of hollow plastic and rubber. The tracking accuracy got worse than that in the first experiment, as shown in Fig. 9. It is because the inertia of ball was different from model. Fig. 10 indicates that the maximum tracking error in each direction is about 13 mm, corresponding to 20 pixels.
Fig. 12. Errors in tracking an elliptical trajectory with a steel ball and a rubber-coated plate.
736
J.H. Park, Y.J. Lee / Mechatronics 13 (2003) 723–738
Fig. 13. Tracking a triangular trajectory with a steel ball and a plastic plate.
In the third experiment, we used the same ball as the one used in the first experiment, but a different plate, which was coated with rubber. The ball tracked the desired trajectory well as shown in Fig. 11. However, the accuracy in tracking was worse than those in the previous two experiments. It is because there exist damping effects on the plate due to the deformation of the plated rubber. Fig. 12 shows that the maximum tracking error is about 16.7 mm, corresponding to 25 pixels. Next, a triangular trajectory was used instead of the elliptical one, but under the same condition as the one in the first experiment. The former resulted in a trajectory with sudden surges and drops in the trajectory while the latter resulted in a smooth sinusoidal trajectory in each direction of the coordinates. Fig. 13 shows that the ball follows the desired trajectory quite well. The error plot in each direction in Fig. 14 shows that the magnitude of the tracking error was less than 12 pixels.
5. Conclusions In this paper, we proposed a visual servoing system that consists of a machine vision system and an industrial robot, to move the flat plate held by the robot so that the ball on the plate can track its desired trajectory. In order to improve the tracking performance, we used the search area reduction method which guarantee high sampling frequency, and presented the center detection algorithm that overcomes the interlaced scanning effect. To cope with the uncertainty of the system model including the contact between the ball and the plate, materials of the ball and the plate, and the size and weight of the ball, a sliding-mode controller was designed. Through
J.H. Park, Y.J. Lee / Mechatronics 13 (2003) 723–738
737
Fig. 14. Errors in tracking a triangular trajectory with a steel ball and a plastic plate.
a series of experiments, it was shown that the system performed the ball tracking very well and that the performance was robust to the uncertainty.
References [1] Shirai Y, Inoue H. Guiding a robot by visual feedback in assembling tasks. Pattern Recognit 1973; 5:99–108. [2] Hashimoto K, Noritsugu T. Modeling and control of robotic yoyo with visual feedback. In: Proceedings of IEEE International Conference on Robotics and Automation, Minneapolis, USA. 1996. p. 2650–5. [3] Hutchinson S, Hager GD, Corke PI. A tutorial on visual servo control. IEEE Trans Robot Automat 1996;12(5):651–70. [4] Kelly R, Carelli R, Nasisi O, Kuchen B, Reyes F. Stable visual servoing of camera-in-hand robotic systems. IEEE/ASME Trans Mechatron 2000;5(1):39–48. [5] William J, Hulls CCW, Bell GS. Relative end-effector control using Cartesian position based visual servoing. IEEE Trans Robot Automat 1996;12(5):684–96.
738
J.H. Park, Y.J. Lee / Mechatronics 13 (2003) 723–738
[6] Allotta B, Colombo C. On the use of linear camera–object interaction models in visual servoing. IEEE Trans Robot Automat 1999;15(2):350–7. [7] Bors AG, Pitas I. Prediction and tracking of moving objects in image sequences. IEEE Trans Image Process 2000;9(8):1441–5. [8] Gonzalez RC, Woods RE. Digital image processing. Reading, MA: Addison-Wesley; 1992. [9] Piepmeier JA. Tracking a moving target with model independent visual servoing: a predictive estimation approach. In: Proceedings of IEEE International Conference on Robotics and Automation,Leuven, Belgium. 1998. p. 2652–7. [10] Yokokohji Y. Accurate image overlay on head-mounted displays using vision and accelerometers. In: Proceedingsof IEEE International Conference on Robotics and Automation, Detroit, USA. 1999. p. 3243–8. [11] Hager GD, Corke PI. A tutorial on visual servo control. IEEE Trans Robot Automat 1996;12:651– 70. [12] Slotine J-JE, Li W. Applied nonlinear control. Englewood Cliffs, NJ: Prentice-Hall; 1991. [13] Edwards C, Spurgeon SK. Sliding mode control: theory and applications. London: Taylor & Francis; 1998. [14] Kelly R. Robust asymptotically stable visual servoing of planar robots. IEEE Trans Robot Automat 1996:759–66. [15] Malis E, Chaumette F, Boudet S. 2 1/2 visual servoing. IEEE Trans Robot Automat 1999;15(2): 238–50. [16] Nguyen M-C, Graefe V. Self-learning vision-guided robots for searching and grasping objects. In: Proceedings of Robotics and Automation. 2000. p. 1633–8.
Robust visual servoing for motion control of the ball on a plate Jong Hyeon Park
a,*
, Young Jong Lee
b
a
b
School of Mechanical Engineering, Hanyang University, Seoul 133-791, Republic of Korea Department of Precision Mechanical Engineering, Hanyang University, Seoul 133-791, Republic of Korea
Abstract This paper presents visual servoing control for a ball on a flat plate to track its desired trajectory. An industrial robot manipulator is used as the actuators to change the angle of the plate so that the ball can track its given trajectory on the plate. The center position of the ball is measured with a machine vision system. Based on the Euler estimator, an algorithm to find the center of the ball with interlaced-scanned image data is proposed. To cope with the variations in the surface characteristics of the plate, and the size and mass of the ball, a slidingmode control is used for the visual servoing. Experimental results show that the proposed algorithm and the sliding-mode controller work well with robustness. Ó 2002 Elsevier Science Ltd. All rights reserved. Keywords: Machine vision; Sliding-mode control; Visual servoing; Interlaced scanning
1. Introduction When a human operator performs a task, he or she uses various sensing organs to deal with any change or uncertainty involved in the task and the environment. Among many sensing organs, eye vision is the prime information route to gather the data on the environment in many tasks. The operator’s hand-eye coordination is a key to success in many cases. For robots, which have the capability of positioning their end-effectors very precisely, without sensors such as vision systems, they are limited to performing simple tasks in a pre-determined environment. As in human counterparts, one of the prime sensors that robots need in performing tasks dexterously is vision. Visual information becomes more important especially when the *
Corresponding author. Tel.: +822-2290-0435; fax: +822-2298-4634. E-mail address: [email protected] (Jong H. Park).
0957-4158/03/$ - see front matter Ó 2002 Elsevier Science Ltd. All rights reserved. doi:10.1016/S0957-4158(02)00039-9
724
J.H. Park, Y.J. Lee / Mechatronics 13 (2003) 723–738
environment changes dynamically and no other non-contact means to measure the status of the changing environment is feasible. Many researchers have used machine vision systems in performing difficult tasks efficiently. Shirai and Inoue [1] described how a visual feedback loop can be used to correct the position of a robot to increase task accuracy. Hashimoto [2] proved the usefulness of the visual feedback using a robotic system. Hutchinson et al. [3] provides a good conceptual introduction to visual servoing and categorizes visual servoing into different sub-categories. Cameras can be fixed or moved around independent of the robot configuration, or can be attached to the robot so that its position and orientations are directly dependent on the robot configuration as in [4]. Control systems can be either position-based [5,6] or image-based [7]. In the former, the visual data is used to compute the relative position of the object(s) to the camera. In this paper, a machine vision system based on position control is proposed and used in order to control the trajectory of the ball on a flat plate. The vision system, fixed in a location, collects the information on the position of the ball on the plate. A fast algorithm to detect the position of the ball on the image plane combined with the Euler position estimator results in a fast control-loop at the regular frame rate. A 6DOF industrial robot is used to control the orientation of the plate to move the ball around on the plate. The visual servoing system is controlled by the sliding-mode controller, which has robustness to system uncertainty, and unmodelled disturbances. Section 2 describes an algorithm of searching the ball from the image data and the relationship between the world coordinates and the image coordinates. In Section 3, the dynamics of a ball on a plate and the sliding-mode controller is covered. Experiments and their results are covered in Section 4, followed by conclusions in Section 5.
2. Vision system 2.1. Image processing The vision system used in this paper takes binary data after digitizing the image data at the hardware level, the threshold levels for which are stored in a lookup table of the frame grabber. It is advantageous to use binary data in differentiating an object from its near-monotonic surroundings fast [8]. In the vision system used, one full-frame of image consists of 640 480 pixels. Searching the whole video data of a full-frame for a small moving object usually takes quite long time, and thus degrades the performance of the visual servoing system. Thus, based on the size of the ball in the image plane, only 100 100-pixelsized image data around the estimated position of the ball are searched to find the precise position of the ball (center), as shown in Fig. 1. Estimated position of the ball at ðk þ 1Þ-th frame, denoted by xb ðk þ 1Þ, is calculated using the Euler estimator [9], i.e.,
J.H. Park, Y.J. Lee / Mechatronics 13 (2003) 723–738
725
Fig. 1. Estimation of the ball center. i i
x^b ðk þ 1Þ ¼ i xb ðkÞ þ
xb ðkÞ i xb ðk 1Þ ; Dt
ð1Þ
where i xb ðkÞ is the measured position of the ball on the image plane at kth frame and Dt is the time gap between the kth and the ðk 1Þth frame samples. The center position of the ball in the image plane is computed by averaging the maximum position and the minimum positions of the image blob associated with the ball. Thus, i
xb ðkÞ ¼ 12½ max i x 2 BðkÞ þ min i x 2 BðkÞ ;
ð2Þ
where i x 2 R2 denotes the pixel index on the image plane and BðkÞ denotes the image blob at frame k that represents the ball. When an object moves fast, its image taken from video cameras with interlaced scanning may become blurred. Fig. 2 shows the images taken when a circular object moves fast horizontally and vertically. Thus, interlaced scanning, which scans even and odd fields of the video signal separately, introduces image distortions [10], and thus prevents an accurate measurement of the position of the object. To minimize the error in the position of the ball, only one field (even or odd) of the video data is used to measure the center of the ball. Fig. 3 shows the priority masks used for the algorithm. Mask Mx min in the figure is used to find the pixel that represents the lowerbound of the image blob of the ball in the x- direction. To find the bound, the following steps are taken: Step 1. Place the center of the mask, mc at the previously estimated center of the ball, i x^b ðk 1Þ.
726
J.H. Park, Y.J. Lee / Mechatronics 13 (2003) 723–738
Fig. 2. Image distortion due to the interlaced scanning: when a circular object moves: (a) horizontally, and (b) vertically.
Step 2. If the pixel at ‘‘first’’ is one of the pixels representing the ball, then the mask is moved such that its center is located at that pixel. Then, go back to Step 1. Step 3. If the pixel at ‘‘first’’ does not represent the ball, check the pixel at ‘‘second’’ of the mask. If it represents the ball, then the center of the mask is moved to the position of the pixel. Then, go back to Step 1. Step 4. If the pixel at ‘‘second’’ is not associated with the image of the ball, check if the pixel at ‘‘third’’ is associated with the image of the ball. If so, the center of the mask is shifted down to the position of the pixel, and then go back to Step 1. Otherwise, the lower-bound in the x- direction, i.e., min i x 2 BðkÞ, is set as the pixel at the center of the mask. To find the upper-bound pixel in the x- direction, Mx max , is used in a similar fashion to move the mask to the right of the object blob on the image plane. Sim-
J.H. Park, Y.J. Lee / Mechatronics 13 (2003) 723–738
727
Fig. 3. A mask used in computing the center of a ball.
ilarly, to find the upper- and lower-bounds in the y-direction, masks My max and My min are used, respectively. 2.2. Camera projection model In setting up the relationship between an object and its image on the image plane, the so-called scaled orthographic projection is often used especially when the motion range of the object in the direction perpendicular to the image plane is small compared with the distance between the object and the camera. Since that the depth variation of the ball on the plate is small due to small changes in the plane orientation as shown in Fig. 4, the scaled orthographic projection is used. The relationship between the reference coordinates and the image coordinates in the scaled orthographic projection [11] can be expressed by i x x ; ð3Þ ¼ S c i y y where ðx; yÞ is the position of the object about the camera reference coordinate and ði x; i yÞ is the position of the object in the image coordinates on the image plane. And,
728
J.H. Park, Y.J. Lee / Mechatronics 13 (2003) 723–738
Fig. 4. Configuration of the camera and the plate.
matrix Sc 2 R22 is a scale factor, which depends on many factors including the distance between the camera and the object, and the focal length of the camera lens. 3. Controller design 3.1. Equations of motion for the ball on a plate Suppose that the ball and the plate are aligned as in Fig. 5. The x–y plane is coplanar with the plane on which the ball is located. The z-direction is upward and normal to the plane. It will be assumed that the ball on the plane is perfectly a sphere. Then, from Newton’s second law, Ty mR€x J ay þ mgR sin hy ¼ 0;
ð4Þ
Tx þ mR€y J ax þ mgR sin hx ¼ 0;
ð5Þ
where R, m and J are the radius, the mass and the moment inertia of the ball; hx and hy are the inclination angle of the plate in the x- and y-direction; ax and ay are the angular acceleration of the ball in the x- and y-direction; Tx and Ty are the torque generated in the x- and y-direction respectively, at the contact area between the ball and the plate, associated with the deformation of the plate (and the ball) due to the contact force between them. Torque Tx and Ty are strongly depends on the type of the materials and the surface condition, such as roughness, of the ball and the plate. If the ball is uniform, its moment of inertia is J ¼ 25mR2 :
J.H. Park, Y.J. Lee / Mechatronics 13 (2003) 723–738
Fig. 5. The ball rolls down the inclined plate.
Fig. 6. Overall system block diagram.
729
730
J.H. Park, Y.J. Lee / Mechatronics 13 (2003) 723–738
In general, for a ball that is made of two different materials, J ¼ kmR2
2 Ml 2 Mh
ð6Þ
where Mh and Ml are the mass of the ball made of the heaviest material and the lightest material, respectively. Assuming that the plate inclination angle h is small, and thus sin hx hx and sin hy hy , and that the ball rolls without slip so that €x ¼ Ray and €y ¼ Rax , Eqs. (4) and (5) can be transformed into i
€xb i €yb
Sc Sc g hy Ty ¼ þ k þ 1 hx ðk þ 1ÞmR Tx
ð7aÞ
or €x ¼ f þ Qu;
ð7bÞ
where Sc Ty ; f ¼ ðk þ 1ÞmR Tx
Sc g ; Q¼ kþ1
hy u¼ : hx
Note that vector f cannot be identified and will be treated as a noise and that matrix Q can be assumed to be diagonal.
Fig. 7. Tracking an elliptical trajectory with a steel ball and a plastic plate.
J.H. Park, Y.J. Lee / Mechatronics 13 (2003) 723–738
731
3.2. Sliding-mode controller Since that sliding-mode controllers have robustness to uncertainty and unmodelled dynamics, they have been applied to robot manipulators [12–16]. In this article, a sliding-mode controller is designed to cope with the changes in the material of the plate and the material and the size of the ball as well as other model uncertainty. It is also assumed that the motion of the plate, and thus the ball is slow enough to neglect the dynamics of the industrial robot. In describing the controller, superscript i denoting the property in the image plane will be dropped for notational convenience. Let xd ðtÞ and xðtÞ be the desired trajectory of the ball (center), and the measured position of the ball at time t, with respect to the image coordinates. The sliding surface is then defined as sðtÞ ¼ x_ ðtÞ x_ d ðtÞ þ Kð xðtÞ xd ðtÞÞ ¼ ~x_ ðtÞ þ K~xðtÞ;
Fig. 8. Errors in tracking an elliptical trajectory with a steel ball and a plastic plate.
ð8Þ
732
J.H. Park, Y.J. Lee / Mechatronics 13 (2003) 723–738
where K is symmetric and positive definite. Then, the equivalent control that is asymptotically stable can be obtained using Eqs. (7b) and (8), and the condition s_ ðtÞ ¼ 0: h i b 1 €xd ðtÞ K~x_ ðtÞ f^ ; u^ðtÞ ¼ Q ð9Þ b are the estimate of f and Q, respectively. It is assumed where vector f^ and matrix Q that vector f has its upper- and lower-bound such that jf^ f j 6 Df
ð10Þ
and that diagonal matrix Q is also bounded by its lower-bound QL and upper-bound QU such that ð11Þ
0 < QL 6 Q 6 QU : If the estimate of matrix Q is selected as b ¼ ½ QL Q
QU 1=2 ;
ð12Þ
then 11
b 1 6 Q Q QL QU 2 6 Q Q U L 1=2
1=2
1=2
:
ð13Þ
In order to have robustness to uncertainty in the modelling, a discontinuous gain term is added to the equivalent control in Eq. (9) to result in the control input: h i b 1 €xd ðtÞ K~x_ ðtÞ f^ K sgnðsÞ ; ð14Þ uðtÞ ¼ Q where positive definite matrix K is diagonal such that K ¼ diagðk1 ; k2 Þ, and
Fig. 9. Tracking an elliptical trajectory with a compound ball and a plastic plate.
J.H. Park, Y.J. Lee / Mechatronics 13 (2003) 723–738
733
sgnðs1 Þ sgnðsÞ ¼ : sgnðs2 Þ From Eqs. (14) and (8), b 1 ð€xd K~x_ f^ K sgnðsÞÞ €xd þ K~x_ þ f T s s_ T s ¼ ½Q Q b 1 K sgnðsÞ T s: b 1 IÞð€xd K~x_ f^Þ þ f f^ Q Q ¼ ½ðQ Q To meet the stability condition of js j s_ T s 6 g 1 ; js2 j where g ¼ ½ g1
T
g2 ;
g1 ; g2 > 0;
Fig. 10. Errors in tracking an elliptical trajectory with a compound ball and a plastic plate.
ð15Þ
734
J.H. Park, Y.J. Lee / Mechatronics 13 (2003) 723–738
matrix K must be selected such that k1 b Q1 ðf f^Þ þ Q b Q1 g: b Q1 Þð€xd K~x_ f^Þ þ Q P ðI Q k2
ð16Þ
Using the bounds of f and Q in Eqs. (10) and (13), the constraint on K may be also expressed by
k1 1=2 1=2 1=2 €xd K~x_ f^ þ Q1=2 ðDf þ gÞ: ð17Þ P QU QL U QL k2 By using the saturation function with boundary layer thickness U instead of the sign function in order to remove chattering from the control law, the sliding-mode control becomes h i b 1 €xd ðtÞ Kð_xðtÞ x_ d ðtÞÞ f^ K sat s : uðtÞ ¼ Q ð18Þ U 4. Experiments and results A series of experiments were performed to measure the performance of the visual servoing system. The overall system components used in the experiments are shown in Fig. 6. The host computer, a Pentium II, generates the desired trajectory for the ball and also works as the sliding-mode controller. The vision board, a frame grabber, collects and stores video data in the binary form in the host computer memory. The MMC board in a PC slot generates the servo control input to the servo motor drives for an 6-DOF industrial robot, Samsung Fara-AS1. The servo motor drives are off-the-shelf products. The desired trajectory for the ball to track on the
Fig. 11. Tracking an elliptical trajectory with a steel ball and a rubber-coated plate.
J.H. Park, Y.J. Lee / Mechatronics 13 (2003) 723–738
735
plate are given in the form of an ellipsoid and a triangle. In order to verify the robustness of the system, different plates and different balls were tried out. In the first experiment, the ball on the plate is to follow an elliptical track at a constant speed. The ball is 17 mm in diameter, made of steel, and weighs about 28 g. The plate is made of plastic and has a smooth flat surface. The resulting track of the ball is shown in Fig. 7 in terms of the image coordinates. A dashed line is the reference trajectory and a solid line is the trajectory of the ball. Marks ‘þ’ and ‘ ’ denote the initial and the final position of the ball, respectively. Fig. 8 shows the tracking errors in the x- and y-axis, which are smaller than 7 mm, corresponding to 10 pixels. In the next experiment, the same plate was used but with a different ball, a compound ball made of hollow plastic and rubber. The tracking accuracy got worse than that in the first experiment, as shown in Fig. 9. It is because the inertia of ball was different from model. Fig. 10 indicates that the maximum tracking error in each direction is about 13 mm, corresponding to 20 pixels.
Fig. 12. Errors in tracking an elliptical trajectory with a steel ball and a rubber-coated plate.
736
J.H. Park, Y.J. Lee / Mechatronics 13 (2003) 723–738
Fig. 13. Tracking a triangular trajectory with a steel ball and a plastic plate.
In the third experiment, we used the same ball as the one used in the first experiment, but a different plate, which was coated with rubber. The ball tracked the desired trajectory well as shown in Fig. 11. However, the accuracy in tracking was worse than those in the previous two experiments. It is because there exist damping effects on the plate due to the deformation of the plated rubber. Fig. 12 shows that the maximum tracking error is about 16.7 mm, corresponding to 25 pixels. Next, a triangular trajectory was used instead of the elliptical one, but under the same condition as the one in the first experiment. The former resulted in a trajectory with sudden surges and drops in the trajectory while the latter resulted in a smooth sinusoidal trajectory in each direction of the coordinates. Fig. 13 shows that the ball follows the desired trajectory quite well. The error plot in each direction in Fig. 14 shows that the magnitude of the tracking error was less than 12 pixels.
5. Conclusions In this paper, we proposed a visual servoing system that consists of a machine vision system and an industrial robot, to move the flat plate held by the robot so that the ball on the plate can track its desired trajectory. In order to improve the tracking performance, we used the search area reduction method which guarantee high sampling frequency, and presented the center detection algorithm that overcomes the interlaced scanning effect. To cope with the uncertainty of the system model including the contact between the ball and the plate, materials of the ball and the plate, and the size and weight of the ball, a sliding-mode controller was designed. Through
J.H. Park, Y.J. Lee / Mechatronics 13 (2003) 723–738
737
Fig. 14. Errors in tracking a triangular trajectory with a steel ball and a plastic plate.
a series of experiments, it was shown that the system performed the ball tracking very well and that the performance was robust to the uncertainty.
References [1] Shirai Y, Inoue H. Guiding a robot by visual feedback in assembling tasks. Pattern Recognit 1973; 5:99–108. [2] Hashimoto K, Noritsugu T. Modeling and control of robotic yoyo with visual feedback. In: Proceedings of IEEE International Conference on Robotics and Automation, Minneapolis, USA. 1996. p. 2650–5. [3] Hutchinson S, Hager GD, Corke PI. A tutorial on visual servo control. IEEE Trans Robot Automat 1996;12(5):651–70. [4] Kelly R, Carelli R, Nasisi O, Kuchen B, Reyes F. Stable visual servoing of camera-in-hand robotic systems. IEEE/ASME Trans Mechatron 2000;5(1):39–48. [5] William J, Hulls CCW, Bell GS. Relative end-effector control using Cartesian position based visual servoing. IEEE Trans Robot Automat 1996;12(5):684–96.
738
J.H. Park, Y.J. Lee / Mechatronics 13 (2003) 723–738
[6] Allotta B, Colombo C. On the use of linear camera–object interaction models in visual servoing. IEEE Trans Robot Automat 1999;15(2):350–7. [7] Bors AG, Pitas I. Prediction and tracking of moving objects in image sequences. IEEE Trans Image Process 2000;9(8):1441–5. [8] Gonzalez RC, Woods RE. Digital image processing. Reading, MA: Addison-Wesley; 1992. [9] Piepmeier JA. Tracking a moving target with model independent visual servoing: a predictive estimation approach. In: Proceedings of IEEE International Conference on Robotics and Automation,Leuven, Belgium. 1998. p. 2652–7. [10] Yokokohji Y. Accurate image overlay on head-mounted displays using vision and accelerometers. In: Proceedingsof IEEE International Conference on Robotics and Automation, Detroit, USA. 1999. p. 3243–8. [11] Hager GD, Corke PI. A tutorial on visual servo control. IEEE Trans Robot Automat 1996;12:651– 70. [12] Slotine J-JE, Li W. Applied nonlinear control. Englewood Cliffs, NJ: Prentice-Hall; 1991. [13] Edwards C, Spurgeon SK. Sliding mode control: theory and applications. London: Taylor & Francis; 1998. [14] Kelly R. Robust asymptotically stable visual servoing of planar robots. IEEE Trans Robot Automat 1996:759–66. [15] Malis E, Chaumette F, Boudet S. 2 1/2 visual servoing. IEEE Trans Robot Automat 1999;15(2): 238–50. [16] Nguyen M-C, Graefe V. Self-learning vision-guided robots for searching and grasping objects. In: Proceedings of Robotics and Automation. 2000. p. 1633–8.