- Email: [email protected]
A Stable Tele-operation of a Mobile Robot with the Haptic Feedback
Budapest, Hungary, August 27-30, 2018 12th IFAC Symposium on Robot Control Budapest, Hungary, August 27-30, 2018 Available online at www.sciencedirect.com 12th IFAC Symposium on Robot Control 12th IFAC Symposium on Robot Control Budapest, Hungary, August 27-30, 2018 Budapest, Hungary, August 27-30, 2018
ScienceDirect
A Stable Tele-operation of a Mobile Robot IFAC PapersOnLine 51-22 (2018) 13–18 with the Haptic Feedback A Stable Tele-operation of a Mobile Robot with the Haptic Feedback Jangmyung Lee *, Hanuel Yoon and Donghyuk Lee Feedback A of a Mobile Robot with the Haptic A Stable Stable Tele-operation Tele-operation of a Mobile Robot with the Haptic Feedback Jangmyung Lee*,*, Hanuel Yoon and Donghyuk Lee
Jangmyung Lee*, Hanuel Yoon Pusan and Donghyuk Lee Department of Electronics Engineering, National University Jangmyung Lee*, Hanuel Yoon and Donghyuk Lee Department of Electronics Engineering, Pusan National University Department of Electronics Engineering, Pusan National University Department of Electronics Engineering, Pusan National University Abstract: — A hybrid joystick has been developed to share the six d.o.f. (degrees of freedom) for the three d.o.f.—inputs and for the three d.o.f.developed feedback signals. the d.o.f. remote(degrees controlofoffreedom) a mobilefor robot, Abstract: A hybrid joystick has been to share For the six the steering and acceleration commands can be generated by a three d.o.f. joystick motion, that is, x, y andthez three d.o.f.—inputs and for the three d.o.f.developed feedback signals. the d.o.f. remote(degrees controlofoffreedom) a mobilefor robot, Abstract: A hybrid joystick has been to share For the six Abstract: A hybrid joystick has beenbe developed tobyshare the changing six d.o.f. (degrees of that freedom) directional motions. Thefor road are dynamically are not clearly steering and— acceleration commands generated a three joystick motion, is, x, yfor andthez three d.o.f. inputs and the conditions three can d.o.f.which feedback signals. Ford.o.f. the remote control of a visible mobile robot, three d.o.f. inputsusually. and forroad the conditions three itd.o.f. feedback signals. Forthe theroll remote control of a visible mobilecan robot, remote operator Therefore is necessary to feedback and pitch values which bez directional motions. The which are dynamically changing are not clearly for steering and acceleration commands can be generated by a three d.o.f. joystick motion, that is, x, y andthe steering and acceleration commands can be generated by a three d.o.f. joystick motion, thatwhich is, x,The ycan and generated based upon the INS (Inertial Navigation System) to the operator through the joystick. yaw remote operator usually. Therefore it is necessary to feedback the roll and pitch values bez directional motions. The road conditions which are dynamically changing are not clearly visible for the directional motions. The road conditions which are dynamically changing are not clearly visible for the angle has been utilized to feedback the real direction of the mobile robot by the steering control. The generated based upon the Therefore INS (Inertial Navigation to thethe operator through joystick. remote operator usually. it is necessarySystem) to feedback roll and pitch the values whichThe canyaw be remote operator usually.toTherefore itthe ishybrid necessary to feedback the roll andand values which can be mechanism and utilized control of the joystick arethe newly designed thethe effectiveness of yaw this angle has been feedback real direction of mobile robot bypitch the steering control. The generated based upon thesystems INS (Inertial Navigation System) to the operator through joystick. The generated based upon the INS (Inertial Navigation System) to the operator through the joystick. The yaw hybrid joystick has been verified through the real tele-operation of a mobile robot driving on slanted and mechanism and utilized control systems of the joystick of arethe newly designed thesteering effectiveness of The this angle has been to feedback thehybrid real direction mobile robot and by the control. angle has been has utilized to feedback the real direction of the mobile robot by thedriving steeringoncontrol. The tiled roads. hybrid joystick beensystems verified through the real tele-operation ofdesigned a mobile robot slanted and mechanism and control of the hybrid joystick are newly and the effectiveness of this mechanism and control systems of the hybrid joystick are newly designed and the effectiveness of this tiled roads. hybrid joystick has been verified through the real tele-operation of a mobile robot driving on slanted and © 2018,joystick IFAC (International Federation of Automatic Control) Hosting Elsevier Ltd.driving All rights Keywords: Mobile Hybrid joystick, Impedance, Teleoperation. hybrid has robot, been verified through the real tele-operation of aby mobile robot on reserved. slanted and tiled roads. tiled roads. Mobile robot, Hybrid joystick, Impedance, Keywords: Teleoperation. Keywords: Mobile robot, Hybrid joystick, Impedance, Teleoperation. tilt and slope even though detection of the lanes and obstacles Keywords:1.Mobile robot, Hybrid joystick, Impedance, Teleoperation. INTRODUCTION can be implemented efficiently. For of thethe precise tilt and slope even though detection lanes tele-operation and obstacles 1. INTRODUCTION of the mobile robot, if the road conditions can be For the efficient tele-operations, a haptic joystick which can be implemented efficiently. For of thethe precise tele-operation tilt and slope even though detection lanes andtransferred obstacles 1. INTRODUCTION tilt and slope even though detection of the lanes andtransferred obstacles to the remote operator, it becomes a lot more convenient. canFor generate the operator’s commandsa and feedback thewhich force of robot, efficiently. if the road For conditions can be canthe be mobile implemented the precise tele-operation the efficient tele-operations, haptic joystick 1. INTRODUCTION can be implemented efficiently. For the precise tele-operation revealing the remote environment is very helpful. Also the a lotofmore convenient. of Therefore the remote mobile operator, robot, if it thebecomes road can be transferred canFor generate the operator’s commandsa and feedback thewhich force to the efficient tele-operations, haptic joystick this paper, newconditions type a hybrid has of the mobileinrobot, if the aroad conditions can bejoystick transferred For the the efficient tele-operations, aishaptic joystick which depending on the control system types, a different type of a to the remote operator, it becomes a lot more convenient. revealing remote environment very helpful. Also can generate the operator’s commands and feedback the force to been proposed for the efficient tele-operation of a mobile Therefore in this paper, a new type of a hybrid joystick the remote operator, it becomes a lot more convenient. has can generate operator’s commands and different feedback theMassie force haptic device is necessary to be designed. Therefore, depending onthe the control system types, type of a been revealing the remote environment is avery helpful. Also robot. The tele-operation ofa new the mobile canof be amodeled proposed for paper, the efficient tele-operation mobile Therefore in this type ofrobot a hybrid joystick has revealing the is[1] remote environment is very helpful. Also and Salisbury developed and commercialized a Massie master haptic device to be designed. Therefore, depending on thenecessary control system types, a different type of a as Therefore inofthis paper, a new type ofis, a hybrid joystick has a driving an automobile, that two d.o.f. can be robot. The tele-operation of the mobile robot can be modeled been proposed for the efficient tele-operation of a mobile depending on the control system types, a different type of a controller, PHANToM which has the force feedback and Salisbury developed and commercialized master been haptic device is[1] necessary to be designed. Therefore,a Massie proposed for the efficient tele-operation of a mobile assigned for the steering and one more d.o.f. can be assigned as a driving of an automobile, that is, two d.o.f. can be haptic device is necessary to be adesigned. Massie functions. Chen proposed 2T2R 4 Therefore, d.o.f. of robot. The tele-operation of the mobile robot can be modeled controller, PHANToM which the force (degree feedback and Salisbury [1][2]developed and has commercialized a master robot. The tele-operation ofchanges the mobile robot can be modeled for thedriving accelerator which theis, speed of theassigned mobile assigned for the steering and one more d.o.f. can be as a of an automobile, that two d.o.f. can be and Salisbury [1][2]developed and commercialized a al master freedom) parallel manipulator. etfeedback [3] functions. Chen proposed a has 2T2R 4 Inoue d.o.f. of robot. controller, hybrid PHANToM which the force (degree as athedriving of an which automobile, that twosensors d.o.f. can The conventional stick with twois, hall be for accelerator changes the speed of the mobile assigned for the steering and one more d.o.f. can be assigned controller, PHANToM which has the force feedback proposed parallel manipulator for a4 Inoue joystick. [4] freedom) manipulator. et Yi al [3] functions. ahybrid Chen [2]parallel proposed a 2T2R d.o.f. (degree of easily assigned forconventional thefor steering and one more can be assigned utilized the steering. To putd.o.f. the hand stick with two hall operator’s sensors can be for theThe accelerator which changes the speed of the mobile functions. Chen [2] proposed a mechanism 2T2R 4 joystick. d.o.f. (degree of robot. designed a spatial 3 d.o.f. parallel with RPS chain proposed a parallel manipulator for a Yi [4] freedom) hybrid parallel manipulator. Inoue et al [3] for the accelerator which changes the speed of the mobile which is holding the joystick, an upper plate has been easily utilized for the steering. To two put the hand robot. The conventional stick with hall operator’s sensors can be freedom) hybrid parallel manipulator. Inoue et al [3] structure. Li and Huang [5] verified a few structural designed 3 d.o.f. parallel mechanism with RPSYichain proposed a aspatial parallel manipulator for a joystick. [4] which robot. The conventional hall sensors be attached thefor bottom ofstick the with joystick. Between the upper is atholding the steering. joystick, antwo upper plate hascan been easily utilized the To put the operator’s hand proposed a parallel manipulator for a joystick. Yi [4] characteristics of 4 d.o.f. parallel manipulator with structure. and3 d.o.f. Huang [5] verified a with few RPS structural designed a Li spatial parallel mechanism chain plate easilyand utilized for the steering. To put the operator’s hand the lower plate, three linear actuators are connected attached at the bottom of the joystick. Between the upper which is holding the joystick, an upper plate has been designed a spatialof3and d.o.f. parallel mechanism with RPS chain constraints. designed a reflective characteristics 4 Collins d.o.f. parallel manipulator with6 which structure. LiLong and Huang [5][6] verified a afew structural is the holding theutilized joystick, an upper the plate has been in parallel, are changing velocity and plate and lower plate, linear actuators are connected attached at which the bottom ofthree the for joystick. Between the upper structure. Li and Huang [5] verified a few structural d.o.f. master controller with the pan-to-graph link structure. constraints. Longofand4 Collins designed a a reflective characteristics d.o.f. [6] parallel manipulator with6 feedback attached at the bottom of the joystick. Between the upper the road conditions by pitch and roll motions. in parallel, which are utilized for changing the velocity and characteristics of developed 4 with d.o.f.the parallel with plate and the lower plate, three linear actuators are connected The haptic devices so[6] far focusesmanipulator their d.o.f. master controller pan-to-graph structure. constraints. Long and Collins designed a link a applications reflective 6 Finally plate and lower plate, three linear actuators are connected thethe lower plate is controlled for theand yawing motion to feedback the road conditions by pitch roll motions. in parallel, which are utilized for changing the velocity and constraints. Long and Collinsso[6] designed a control a applications reflective to the control of the manipulators not to the of the6 The haptic devices developed focuses their d.o.f. master controller with the far pan-to-graph link structure. in parallel, which are utilized for changing the velocity and feedback to steering result. lower plate is controlled for theand yawing to feedbackthethe road conditions by pitch roll motion motions. d.o.f. master controller with the pan-to-graph link structure. mobile robot manipulators require more delicate to control ofsince the manipulators to the control of the Finally Thethe haptic devices developed so far not focuses their applications feedback the road result. conditions by pitch and roll motions. feedback to steering Finally the lower plate is controlled for the yawing motion to The haptic devices developed so far focuses their applications motions than the mobile robots. This the paper of six for sections including mobile robot ofsince manipulators not require to the control the manipulators to the more controldelicate of the Finally lowercomposes plate is controlled the yawing motionthis to feedback to steering result. to the control of the manipulators not to the control of the introduction. Thecomposes design the six proposed hybrid joystick this has motions than thesince mobilemanipulators robots. mobile robot require more robots delicate feedback to steering result.of of This paper sections including In many applications, the application of mobile is mobile robot since manipulators require more delicate been illustrated in detail in Section 2. The feedback motions than thesimple mobile robots. introduction. design of of the six proposed hybrid joystick this has This paperThecomposes sections including limited to the Therefore a simple motions than the mobilenavigations. robots. In many applications, the application of mobile robotstwo is algorithms Thisillustrated paper composes ofin sixSection sections including this of the road conditions to the operator have been been in detail 2. The feedback introduction. The design of the proposed hybrid joystick has d.o.f. joystick has beennavigations. utilized for Therefore the tele-operation of two the limited to the simple a simple In many applications, the application of mobile robots is introduction. The the proposed hybrid systems joystickbeen has described inofSection 3.detail InofSection 4, and algorithms the design road conditions to the the control operator been illustrated in in Section 2. The have feedback In many applications, the application of mobile robots is mobile robot. Recently Farrokh[7] and N. Diolaiti[8] utilized d.o.f. beennavigations. utilized for Therefore the tele-operation the experimental limitedjoystick to the has simple a simpleof two been illustrated in detail in Section 2. The feedback condition to verify the performance of the 3. conditions In Section 4, and algorithmsinofSection the road to the the control operatorsystems have been limited to theRecently simple Therefore a simple two described the force to navigations. the joystick to detect and avoid mobile robot. Farrokh[7] Diolaiti[8] utilized d.o.f.feedback joystick has been utilized forand theN. tele-operation of the the hybrid algorithms of the road conditions to the operator have been haptic joystick have been illustrated in detail. Section condition verify 4,the of and the described in Section 3. IntoSection the performance control systems d.o.f.feedback joystick hasdriving been utilized for N. theAndo[9], tele-operation of the the experimental obstacles in the path. And C[10], the force to the joystick to detect andRen. avoid mobile robot. Recently Farrokh[7] and N. Diolaiti[8] utilized described in Section 3.have In Section 4, of thethe control systems and 5 shows the experimental results teleoperation of hybrid haptic joystick been illustrated in detail. Section experimental condition to verify the performance of the mobile robot. Recently Farrokh[7] and N. Diolaiti[8] utilized and F. Arai[11] applied thejoystick force to the the obstacles in the driving path. And reflective N. C[10], the feedback force to the to Ando[9], detect joystick andRen. avoid experimental condition to verify the performance of the mobile robots with the designed hybrid haptic joystick and 5 shows the joystick experimental results of the in teleoperation of hybrid haptic have been illustrated detail. Section the feedback force to thevarious to detect and avoid tele-operation to perform tasks a limited and F. Arai[11] applied thejoystick force reflective joystick to the the the obstacles in the driving path. And N. with Ando[9], Ren.d.o.f. C[10], hybrid haptic joystick been illustrated in detail. Section conventional d.o.f. joystick demonstrate the mobile robots with 2thehave designed hybrid haptic joystick and 5 shows the experimental results of to the teleoperation of obstacles in the driving path. And N. Ando[9], Ren. C[10], tele-operation to perform various tasks a limited andFor F. Arai[11] applied the force reflective joystick to the the 5 shows the experimental results of the demonstrate teleoperation of superiority of the hybrid haptic joystick. Sectionand 6, general tele-operation thewith mobile robot,d.o.f. vision conventional 2thed.o.f. joystick the robots with designed hybridtoFinally haptic in joystick and F. the Arai[11] applied the forceof reflective joystick to the mobile tele-operation to perform various tasks with a limited d.o.f. mobile robots with the designed hybrid haptic in joystick and this research is concluded with suggestion of a future sensors are widely utilized [12-14]. In the image based telesuperiority of the hybrid haptic joystick. Finally Section 6, conventional 2 d.o.f. joystick to demonstrate the tele-operation to perform various tasks a limited For the general tele-operation of thewith mobile robot,d.o.f. vision the the conventional 2 d.o.f. joystick to demonstrate the research work. operation, it is not easy to obtain the read conditions such as this research is concluded with suggestion of a future of the hybrid haptic joystick. Finally in Section 6, sensors aregeneral widely tele-operation utilized [12-14]. In the imagerobot, basedvision tele- superiority For the of the mobile superiority of the hybrid haptic joystick. Finally in Section 6, For the general of mobile robot,such vision work. this research is concluded with suggestion of a future operation, is not tele-operation easy to obtain thethe as research sensors areitwidely utilized [12-14]. Inread the conditions image based telethis research sensors are widely utilized [12-14]. In the image based tele- research work. is concluded with suggestion of a future operation, it is not easy to obtain the read conditions such as Copyright IFAC operation,©it2018 is not easy to obtain the read conditions such as 13 research work. 2405-8963 © 2018, IFAC (International Federation of Automatic Control) Hosting by Elsevier Ltd. All rights reserved. Copyright 2018 responsibility IFAC 13 Control. Peer review©under of International Federation of Automatic 10.1016/j.ifacol.2018.11.511 Copyright © 2018 IFAC 13 Copyright © 2018 IFAC 13
IFAC SYROCO 2018 14 Budapest, Hungary, August 27-30, 2018
Jangmyung Lee et al. / IFAC PapersOnLine 51-22 (2018) 13–18
2. DESIGN OF A HYBRID HAPTIC JOYSTICK
between his command and the real driving directions by the yaw motion. 2.1 Pitch/Roll Motion of Upper Plate There are three frames: base frame {xB , yB , zB } , lower plate frame
{xL , yL , zL }
, and upper plate frame
{xU , yU , zU } . The motion components at the at the upper plate by the three linear actuator are pitch, roll, and z . Notice that there are three linear actuators vertically between the same size of upper and lower plates. Those are represented by P1 , P2 , and P3 . Therefore, x , y , and yaw motions cannot be generated by these three linear actuators. Notice that the x , y motion commands are generated by the joystick on the upper plate and sent to the mobile robot controller directly and the yaw motion is generated by a revolute joint below the lower plate. The pitch angle, , which is defined w.r.t. the center of
Fig. 1. Structure and functions of the hybrid joystick Figure 1 illustrates the structure of the hybrid haptic joystick proposed in this research, which can generate six d.o.f. motions. The part ① in Fig. 1 is the same as the conventional two d.o.f. joystick which generates the direction commands of the mobile robot. The two d.o.f. hall sensor detects the operator’s steering intention and converts to the driving direction of the mobile robot. The joystick has been attached on the top of the upper plate where the operator’s hand will be placed when he is using this joystick for the teleoperations. Changing the velocity can be implemented by pushing the upper plate downwards by the hand like accelerator for the automobile. To detect the pushing force which corresponds to the acceleration or deceleration command at the center of the upper plate, there are three force sensors at the connecting junctions of the linear
upper plate,
OU . Since OU is fixed at the center of the upper
plate, the pitch angle, theta, can be obtained as follows:
sin 1
P1 ( z )
P2 ( z ) P3 ( z ) 2 a
(1)
P2 and the center where a represents the length between P1 ,(z) point P of 1 , P2 (z) and
P3 (z).
The roll angle, , is defined w.r.t. the center of upper
actuators coming from the lower plate in part ②, respectively. To generate the deceleration command with the acceleration command by the pushing force at the upper plate, a threshold value has been set to differentiate the boundary between the acceleration and deceleration, that is, a threshold value can be used to represent the zero acceleration or the constant
OU . Since OU is fixed at the center of the upper plate, the roll angle, , can be obtained as follows: plate,
sin 1
velocity. At the part ② , there are three linear actuators connecting the upper plate to the lower plate. The motion of the three linear actuators has been utilized to generate the pitch and roll motions corresponding to the road conditions of the slope and tilt. When the mobile robot is driving a road with a slope and a tilt, the INS sensor attached at the mobile robot detects the slope and tilt values and feedback to the hybrid haptic joystick which reflects the values by the motion of the upper plate. The operator can feel the road slope and tilt by his hand on the upper plate and can change the velocity. For an example, when the mobile robot is driving on the upward slope, the speed of the mobile robot will be decreased, which can be recognized by the slope feedback and he can increase the dropped speed by pushing down the upper plate.
P2 ( z ) P3 ( z ) c
P1 , P2 (z) and where c represents the distance between
(2)
P3
(z)
points.
The yaw angle,
, is generated by a revolute joint between OL and OB which is limited within ± 60 to reveal the driving direction of the mobile robot. The revolute joint can be controlled without the angle limit. However it is bounded to cope with the wrist motion of a normal people, which is limited about ± 60 . The z directional motion can be generated by the same amount of motion by the three linear actuators, P1 , P2 and P3
At the part ③, there is a revolute joint which is used to reflect the result of the steering command by the yaw motion. When the steering commands are kept faithfully, the yaw motion becomes zero. The operator can recognize the difference
which can be represented as follows:
zd P1 P2 P3
14
(3)
IFAC SYROCO 2018 Budapest, Hungary, August 27-30, 2018
Jangmyung Lee et al. / IFAC PapersOnLine 51-22 (2018) 13–18
Notice that this z direction motion is independent from the pitch and roll motions.
User
Feedback Hall sensor Joystick (steering)
xd , yd
2-2 Pitch and Roll Motions of the Upper Plate
U Jq
T
q P1 P2
Vision Sensor data
OU , and
T
J13 J 23
J †12 J †22 J †32
Pitch Control
d
Roll Control
d
Yaw Control
d
f P1f , P2 , and f P3 are used for
3.1 Joystick and lower plate control The hall sensor joystick has been utilized for generating the steering command by the operator and for btransmitting
(5)
the commands directly to the mobile robot by
xd , yd . Not
any control actions are required for the two d.o.f. motion. The revolute joint at the lower plate has been utilized for revealing the yaw angle of the mobile robot corresponding to the steering commands, which has been controlled by PD algorithm to get the stable and fast performance with the simple algorithm.
To generate the roll and pitch motions by the three linear actuators, Eq. (4) needs to be solved for q . In solving the under-determined system of the Eq. (4), the constraint equation can be incorporated. However, the z directional motion is used as an input for the operator, which is not controlled to reveal the road conditions, it is separated in the inversion process. That is, the pseudo inverse of Eq. (4) has been utilized to obtain the linear actuator motions as follows:
P1 J †11 q J †U P2 J † 21 P3 J †31
changing the velocity of the mobile robot by converting to the command of dot{Z}.
linear actuators. The Jacobian matrix has 2X3 dimension and can be derived from the Eqs. (1) and (2) as follows:
J 22
vd
INS Sensor data
common components of
P3 represents the motions of the
J12
P3
Conversion to velocity
Figure 3 illustrates the block diagram of the whole system. The operator basically utilizes the vision information for monitoring and for controlling the mobile robot. The INS sensor attached at the mobile robot gathers the road conditions such as tilt and slope of the driving road and the steering information of the mobile robot, which are feedback back to the hybrid haptic joystick as roll, pitch and yaw, respectively. For the pitch and roll control, the force sensors installed between the end of the linear actuators and the upper plate , P2 , and f P3 as shown in Fig. 6. Notice that the providef P1f
represents the roll and pitch motions
J J 2 X 3 11 J 21
Upper Plate
f P1f , P2f ,
Fig. 2. Block diagram of the overall system
(4)
of the upper plate w.r.t. the origin of the upper plate
zd
Mobile Robot
The dynamic motion of pitch and roll motions of the upper plate are designed to transfer the road slope and tilt to the operator putting his hand on the upper plate. The operator may watch the monitoring screen to see how the mobile robot is moving as he operates. However the road conditions are very difficult to be watched by the screen. The roll and pitch motions of the upper plate are generated by the three linear actuators. Therefore, the motion equations can be represented as follows:
Where U
15
3.2 Control of the upper plate Three force sensors are installed at the end of the P1 , P2 , and P3 to sensor the pushing force by the operator to generate
(6)
zd command, that is, the velocity of the mobile robot, vd ,, and to generate the roll and pitch motions corresponding to the tilt and slope angles of the road based upon the impedance control that keeps the a desired relationship between the force and motion. The operating can push his hand on the upper plate with a certain force which can be used as a threshold value, f thre , to
3. CONTROL ALGORITHMS OF THE HYBRID HAPTIC JOYSTICK
differentiate between the acceleration and deceleration conditions. This threshold value can be adjusted for each operator before the real driving of the mobile robot. operation. The force at the center of the upper plate can be represented as, f P , and the value can be obtained by the three sensors as follows:
15
IFAC SYROCO 2018 16 Budapest, Hungary, August 27-30, 2018
Jangmyung Lee et al. / IFAC PapersOnLine 51-22 (2018) 13–18
f Pf P2f fP 1 3
P3
controller of the mobile robot. Two DC motors and encoders are used for driving and sensing the wheel motions. To sensor the roll, pitch and yaw angles, an INS sensor (EBIMU-9DOFV3) has been installed on the mobile robot with a CCD camera. The image and INS data are sent to the remote operator through the communication channel, UART. For the main controller, ARM Cortex STM32F4 has been used. The states of the mobile robot have been monitored by the PC screen. The INS data have been transmitted to the main controller to control the linear actuators to reflect the slope and the tilt conditions of the driving road. An extra INS has been attached at the bottom of the upper plate to measure the real roll, pitch, yaw angles of the upper plate, which has been utilized to control the yaw angle by PD control algorithm.
(7)
As summarized in Eqs. (8) and (9), the relation between
f pf 1 , pf 2,
p3
P1 , P2 , P3 needs to be determined
and
properly. Since the motions and forces of the upper plate are directly related to each other since the operator is putting his hand on the upper plate. Therefore the upper plate control by
P1 , P2P,1 ,Pand 3P2 , P3 can be properly controlled by a desired impedance Zd with the sensing of f pf 1 , p 2 , and f p 3 [15-20]. the three linear actuators,
The control of each link for the upper plate, which is represented as a desired length, ld , for one of the actuators of
4.2 Experimental environment
P1 , PP21,, P P and 3 2 , P3 .has been implemented by PID algorithm.
The length feedback, l , is implemented by the INS sensor attached under the upper plate. The relation between the force , f p ,and the length, l , can be defined by the
Figure 4 illustrates the experimental environments to show the effectiveness of the proposed hybrid haptic joystick in the tele-operation of a mobile robot on the various roads. A CCD camera is installed on the ceiling to monitor the motion of the mobile robot on the designed paths.
impedance as follows:
f
p
s Z sl s
where Z can be defined a desired impedance
(8)
Z d for
considering the operator’s manner of handling the joystick. The general form of desired impedance can be derived from the relation between the force and the length as follows:
f p md ld l bd ld l kd ld l Where md
(a) Straight line path with up/down slopes (9)
, bd , and kd represent inertia, damping, and
stiffness parameters, respectively, which can be obtained from the dynamic characteristics of the upper plate and the human hand.
4. SYSTEM COMPOSITION AND EXPERIMENTAL ENVIRONMENT (b) A straight up/down slopes & curved path
4.1 System composition
Fig. 3. Experimental environments
The operator controls the mobile robot at the remote site using the images sent from the CCD camera on the mobile robot. Based upon the images, he controls the mobile robot to avoid obstacles and to follow the desired path through the communication network using the two d.o.f. hall sensor joystick. To accelerate or decelerate the mobile robot, he may push or slight pull the upper plate, which is detected by the force sensors installed at the end of the linear actuators. At the same, time the road conditions of tilt and slope are measured by the INS sensor on the mobile robot and sent back to the operator hand on the upper plate by the roll and pitch motions, respectively. The DSP 28335 has been used for the main
Figure 4(a) shows the straight line path with
5 or
15 upward and downward slopes, Fig. (b) shows a curved path after the up/down slope paths. The slope 15 can cover most of roads with the slope limited by the laws. Notice that the slope affects a lot on the speed of the mobile robot with a given command, which is not easy to being recognized by the remote operator. In this research, the control capability of the individual person has not been considered. That is, one of theauthor solely conducted the remote operations of the mobile robot 16
IFAC SYROCO 2018 Budapest, Hungary, August 27-30, 2018
Jangmyung Lee et al. / IFAC PapersOnLine 51-22 (2018) 13–18
by using a conventional joystick (two d.o.f. hall sensor joystick) and the hybrid haptic joystick to demonstrate the superiority of the hybrid haptic joystick in the tele-operations.
17
controlled smoothly by the conventional joystick which cannot provide the down slope information properly to the operator. However, using the hybrid haptic joystick the curved motion after the down slope has been implemented successful with the force feedback at the upper plate as shown in Fig. 6. .The tele-driving using the hybrid joystick shows a lot better performance than the driving using the hall sensor joystick in terms of maximum errors as well as average errors. Not only the driving accuracy but also the driving stability has been improved when the hybrid haptic joystick has been utilized for the tele-operation of the mobile robot, which has been felt by the operator putting his hand on the upper plate which reflects the road conditions.
5. EXPERIMENTS AND DISCUSSIONS Figure 11 shows how the mobile robot motions are captured by the CCD camera on the ceiling and feedback to the remote operator who is going to control the mobile robot using the hybrid haptic joystick. Real motions of the mobile robot is recorded and analyzed through the image processing.
6. CONCLUSION In this research a hybrid haptic joystick has been designed and implemented for the tele-operation of a mobile robot by an operator watching the driving environment through the CCD camera images. The road conditions of slope and tilt have been efficiently transferred to the operator through the pitch and roll control of the upper plate where the operator is putting his hand on. With the feedback of the road conditions by the hybrid haptic joystick, the operator can control the mobile robot more precisely and stably that without the feedback by the conventional joystick. The efficiency of the hybrid haptic joystick in generating the commands as well as feedback the road conditions has been verified through the tele-operation of the mobile robot. However, in designing the impedance controller for the hybrid haptic joystick, the desired impedance has been determined empirically for the author. In the future research, the desired impedance needs to be adjusted for a certain operator using an artificial intelligence algorithm.
Fig. 4. Driving of the mobile robot on the straight line path with
15 slope
The maximum driving errors of the tele-operations by different joysticks for the 15 slope path the driving errors by the two d.o.f. hall sensor joystick becomes a lot bigger than those by the hybrid haptic joystick. The maximum error was 16.1 Cm by the hall sensor joystick, while it was only 6.8 Cm by the hybrid haptic joystick.
REFERENCES [1] T. Massi and K. Salisbury, “The PHANToM haptic interface: a device for probing virtual objects”, ASME Winter Annual Meeting, vol. 55, no. 1, pp. 295-300, 1994. [2] W.-J. Chen, “A novel 4-dof parallel manipulator and its kinematic modeling”, in: IEEE Int. Conf. on Robotics and Automation, Seoul, 23–25, pp. 3350–3355, May 2001. [3] H. Inoue et. Al, “Parallel manipulator”, Proc. ISRR, Robotics Research 3, 1985. [4] Yi Lu, Bo Hu, “Analyzing kinematics and solving active/constrained forces of a 3SPU + UPR parallel manipulator”, Mechanism and Machine Theory, vol. 42, pp. 1298-1313, 2007. [5] Q. Li, Z. Huang, “Type synthesis of 4-dof parallel manipulators”, in: IEEE Int. Conf. on Robotics and Automation, Taipei, 14–19, pp. 755–760, September 2003. [6] Gregory L. Long and Curtis L.Collins, “A pantograph linkage parallel platform master hand controller for force-reflection,” Proc. IEEE Int. Conf. Robotics and Automation, pp. 390-395, 1992. [7] Farrokh Janbi Sharifi, lraj Hassanzadeh, “Experimental Analysis of Mobile-Robot Teleoperation via Shared Impedance Control”, IEEE Transactions on Systems, vol. 41, no. 2, pp. 591-606, 2010. [8] N. Diolaiti, C. Melchiorri, “Teleoperation of a mobile robot through haptic feedback”, EEE International Workshop. Ottawa. Canada, Nov. 2002. [9] T. Massi and K. Salisbury, “The PHANToM haptic interface: a device for probing virtual objects”, ASME Winter Annual Meeting, vol. 55, no. 1, pp. 295-300, 1994. [10] W.-J. Chen, “A novel 4-dof parallel manipulator and its kinematic modeling”, in: IEEE Int. Conf. on Robotics and Automation, Seoul, 23–25, pp. 3350–3355, May 2001.
Fig. 5. Comparison of curved path motions of the mobile (Fig. 10 (b) path) Figure 6 illustrates the driving results of the mobile robot on the path shown in Fig. 5(b). In this comparison, the performance difference between the two joystick in the teleoperations is distinctive at the location around x= 5 m where the down slope is finished and the curved path is started. At the end of the down slope, the velocity of the mobile robot has been increased a lot and the turning motion cannot be 17
IFAC SYROCO 2018 18 Budapest, Hungary, August 27-30, 2018
Jangmyung Lee et al. / IFAC PapersOnLine 51-22 (2018) 13–18
[11] H. Inoue et. Al, “Parallel manipulator”, Proc. ISRR, Robotics Research 3, 1985. [12] Yi Lu, Bo Hu, “Analyzing kinematics and solving active/constrained forces of a 3SPU + UPR parallel manipulator”, Mechanism and Machine Theory, vol. 42, pp. 1298-1313, 2007. [13] Q. Li, Z. Huang, “Type synthesis of 4-dof parallel manipulators”, in: IEEE Int. Conf. on Robotics and Automation, Taipei, 14–19, pp. 755–760, September 2003. [14] Gregory L. Long and Curtis L.Collins, “A pantograph linkage parallel platform master hand controller for force-reflection,” Proc. IEEE Int. Conf. Robotics and Automation, pp. 390-395, 1992. [15]Selcuk KIZIR, and Zafer BINGUL, Fuzzy impedance and force control of a Stewart platform, Turkish Journal of Electrical Engineering & Computer Sciences, 22 (2014), pp. 924-939. [16]Cristian A. Vergara and Carlos F. Rodriguez, Hybrid Position-force Control for a Stewart Platform, Journal of Automation and Control Engineering, 5 (1) (2017), pp. 10-13. [17]Hannaneh Z. Arabshahi, Alireza and B. Novinzadeh, Impedance Control of the 3RPS Parallel Manipulator, in Proc. of the 2nd RSI/ISM International Conference on Robotics and Mechatronics, (2014) pp. 486-492.
18
ScienceDirect
A Stable Tele-operation of a Mobile Robot IFAC PapersOnLine 51-22 (2018) 13–18 with the Haptic Feedback A Stable Tele-operation of a Mobile Robot with the Haptic Feedback Jangmyung Lee *, Hanuel Yoon and Donghyuk Lee Feedback A of a Mobile Robot with the Haptic A Stable Stable Tele-operation Tele-operation of a Mobile Robot with the Haptic Feedback Jangmyung Lee*,*, Hanuel Yoon and Donghyuk Lee
Jangmyung Lee*, Hanuel Yoon Pusan and Donghyuk Lee Department of Electronics Engineering, National University Jangmyung Lee*, Hanuel Yoon and Donghyuk Lee Department of Electronics Engineering, Pusan National University Department of Electronics Engineering, Pusan National University Department of Electronics Engineering, Pusan National University Abstract: — A hybrid joystick has been developed to share the six d.o.f. (degrees of freedom) for the three d.o.f.—inputs and for the three d.o.f.developed feedback signals. the d.o.f. remote(degrees controlofoffreedom) a mobilefor robot, Abstract: A hybrid joystick has been to share For the six the steering and acceleration commands can be generated by a three d.o.f. joystick motion, that is, x, y andthez three d.o.f.—inputs and for the three d.o.f.developed feedback signals. the d.o.f. remote(degrees controlofoffreedom) a mobilefor robot, Abstract: A hybrid joystick has been to share For the six Abstract: A hybrid joystick has beenbe developed tobyshare the changing six d.o.f. (degrees of that freedom) directional motions. Thefor road are dynamically are not clearly steering and— acceleration commands generated a three joystick motion, is, x, yfor andthez three d.o.f. inputs and the conditions three can d.o.f.which feedback signals. Ford.o.f. the remote control of a visible mobile robot, three d.o.f. inputsusually. and forroad the conditions three itd.o.f. feedback signals. Forthe theroll remote control of a visible mobilecan robot, remote operator Therefore is necessary to feedback and pitch values which bez directional motions. The which are dynamically changing are not clearly for steering and acceleration commands can be generated by a three d.o.f. joystick motion, that is, x, y andthe steering and acceleration commands can be generated by a three d.o.f. joystick motion, thatwhich is, x,The ycan and generated based upon the INS (Inertial Navigation System) to the operator through the joystick. yaw remote operator usually. Therefore it is necessary to feedback the roll and pitch values bez directional motions. The road conditions which are dynamically changing are not clearly visible for the directional motions. The road conditions which are dynamically changing are not clearly visible for the angle has been utilized to feedback the real direction of the mobile robot by the steering control. The generated based upon the Therefore INS (Inertial Navigation to thethe operator through joystick. remote operator usually. it is necessarySystem) to feedback roll and pitch the values whichThe canyaw be remote operator usually.toTherefore itthe ishybrid necessary to feedback the roll andand values which can be mechanism and utilized control of the joystick arethe newly designed thethe effectiveness of yaw this angle has been feedback real direction of mobile robot bypitch the steering control. The generated based upon thesystems INS (Inertial Navigation System) to the operator through joystick. The generated based upon the INS (Inertial Navigation System) to the operator through the joystick. The yaw hybrid joystick has been verified through the real tele-operation of a mobile robot driving on slanted and mechanism and utilized control systems of the joystick of arethe newly designed thesteering effectiveness of The this angle has been to feedback thehybrid real direction mobile robot and by the control. angle has been has utilized to feedback the real direction of the mobile robot by thedriving steeringoncontrol. The tiled roads. hybrid joystick beensystems verified through the real tele-operation ofdesigned a mobile robot slanted and mechanism and control of the hybrid joystick are newly and the effectiveness of this mechanism and control systems of the hybrid joystick are newly designed and the effectiveness of this tiled roads. hybrid joystick has been verified through the real tele-operation of a mobile robot driving on slanted and © 2018,joystick IFAC (International Federation of Automatic Control) Hosting Elsevier Ltd.driving All rights Keywords: Mobile Hybrid joystick, Impedance, Teleoperation. hybrid has robot, been verified through the real tele-operation of aby mobile robot on reserved. slanted and tiled roads. tiled roads. Mobile robot, Hybrid joystick, Impedance, Keywords: Teleoperation. Keywords: Mobile robot, Hybrid joystick, Impedance, Teleoperation. tilt and slope even though detection of the lanes and obstacles Keywords:1.Mobile robot, Hybrid joystick, Impedance, Teleoperation. INTRODUCTION can be implemented efficiently. For of thethe precise tilt and slope even though detection lanes tele-operation and obstacles 1. INTRODUCTION of the mobile robot, if the road conditions can be For the efficient tele-operations, a haptic joystick which can be implemented efficiently. For of thethe precise tele-operation tilt and slope even though detection lanes andtransferred obstacles 1. INTRODUCTION tilt and slope even though detection of the lanes andtransferred obstacles to the remote operator, it becomes a lot more convenient. canFor generate the operator’s commandsa and feedback thewhich force of robot, efficiently. if the road For conditions can be canthe be mobile implemented the precise tele-operation the efficient tele-operations, haptic joystick 1. INTRODUCTION can be implemented efficiently. For the precise tele-operation revealing the remote environment is very helpful. Also the a lotofmore convenient. of Therefore the remote mobile operator, robot, if it thebecomes road can be transferred canFor generate the operator’s commandsa and feedback thewhich force to the efficient tele-operations, haptic joystick this paper, newconditions type a hybrid has of the mobileinrobot, if the aroad conditions can bejoystick transferred For the the efficient tele-operations, aishaptic joystick which depending on the control system types, a different type of a to the remote operator, it becomes a lot more convenient. revealing remote environment very helpful. Also can generate the operator’s commands and feedback the force to been proposed for the efficient tele-operation of a mobile Therefore in this paper, a new type of a hybrid joystick the remote operator, it becomes a lot more convenient. has can generate operator’s commands and different feedback theMassie force haptic device is necessary to be designed. Therefore, depending onthe the control system types, type of a been revealing the remote environment is avery helpful. Also robot. The tele-operation ofa new the mobile canof be amodeled proposed for paper, the efficient tele-operation mobile Therefore in this type ofrobot a hybrid joystick has revealing the is[1] remote environment is very helpful. Also and Salisbury developed and commercialized a Massie master haptic device to be designed. Therefore, depending on thenecessary control system types, a different type of a as Therefore inofthis paper, a new type ofis, a hybrid joystick has a driving an automobile, that two d.o.f. can be robot. The tele-operation of the mobile robot can be modeled been proposed for the efficient tele-operation of a mobile depending on the control system types, a different type of a controller, PHANToM which has the force feedback and Salisbury developed and commercialized master been haptic device is[1] necessary to be designed. Therefore,a Massie proposed for the efficient tele-operation of a mobile assigned for the steering and one more d.o.f. can be assigned as a driving of an automobile, that is, two d.o.f. can be haptic device is necessary to be adesigned. Massie functions. Chen proposed 2T2R 4 Therefore, d.o.f. of robot. The tele-operation of the mobile robot can be modeled controller, PHANToM which the force (degree feedback and Salisbury [1][2]developed and has commercialized a master robot. The tele-operation ofchanges the mobile robot can be modeled for thedriving accelerator which theis, speed of theassigned mobile assigned for the steering and one more d.o.f. can be as a of an automobile, that two d.o.f. can be and Salisbury [1][2]developed and commercialized a al master freedom) parallel manipulator. etfeedback [3] functions. Chen proposed a has 2T2R 4 Inoue d.o.f. of robot. controller, hybrid PHANToM which the force (degree as athedriving of an which automobile, that twosensors d.o.f. can The conventional stick with twois, hall be for accelerator changes the speed of the mobile assigned for the steering and one more d.o.f. can be assigned controller, PHANToM which has the force feedback proposed parallel manipulator for a4 Inoue joystick. [4] freedom) manipulator. et Yi al [3] functions. ahybrid Chen [2]parallel proposed a 2T2R d.o.f. (degree of easily assigned forconventional thefor steering and one more can be assigned utilized the steering. To putd.o.f. the hand stick with two hall operator’s sensors can be for theThe accelerator which changes the speed of the mobile functions. Chen [2] proposed a mechanism 2T2R 4 joystick. d.o.f. (degree of robot. designed a spatial 3 d.o.f. parallel with RPS chain proposed a parallel manipulator for a Yi [4] freedom) hybrid parallel manipulator. Inoue et al [3] for the accelerator which changes the speed of the mobile which is holding the joystick, an upper plate has been easily utilized for the steering. To two put the hand robot. The conventional stick with hall operator’s sensors can be freedom) hybrid parallel manipulator. Inoue et al [3] structure. Li and Huang [5] verified a few structural designed 3 d.o.f. parallel mechanism with RPSYichain proposed a aspatial parallel manipulator for a joystick. [4] which robot. The conventional hall sensors be attached thefor bottom ofstick the with joystick. Between the upper is atholding the steering. joystick, antwo upper plate hascan been easily utilized the To put the operator’s hand proposed a parallel manipulator for a joystick. Yi [4] characteristics of 4 d.o.f. parallel manipulator with structure. and3 d.o.f. Huang [5] verified a with few RPS structural designed a Li spatial parallel mechanism chain plate easilyand utilized for the steering. To put the operator’s hand the lower plate, three linear actuators are connected attached at the bottom of the joystick. Between the upper which is holding the joystick, an upper plate has been designed a spatialof3and d.o.f. parallel mechanism with RPS chain constraints. designed a reflective characteristics 4 Collins d.o.f. parallel manipulator with6 which structure. LiLong and Huang [5][6] verified a afew structural is the holding theutilized joystick, an upper the plate has been in parallel, are changing velocity and plate and lower plate, linear actuators are connected attached at which the bottom ofthree the for joystick. Between the upper structure. Li and Huang [5] verified a few structural d.o.f. master controller with the pan-to-graph link structure. constraints. Longofand4 Collins designed a a reflective characteristics d.o.f. [6] parallel manipulator with6 feedback attached at the bottom of the joystick. Between the upper the road conditions by pitch and roll motions. in parallel, which are utilized for changing the velocity and characteristics of developed 4 with d.o.f.the parallel with plate and the lower plate, three linear actuators are connected The haptic devices so[6] far focusesmanipulator their d.o.f. master controller pan-to-graph structure. constraints. Long and Collins designed a link a applications reflective 6 Finally plate and lower plate, three linear actuators are connected thethe lower plate is controlled for theand yawing motion to feedback the road conditions by pitch roll motions. in parallel, which are utilized for changing the velocity and constraints. Long and Collinsso[6] designed a control a applications reflective to the control of the manipulators not to the of the6 The haptic devices developed focuses their d.o.f. master controller with the far pan-to-graph link structure. in parallel, which are utilized for changing the velocity and feedback to steering result. lower plate is controlled for theand yawing to feedbackthethe road conditions by pitch roll motion motions. d.o.f. master controller with the pan-to-graph link structure. mobile robot manipulators require more delicate to control ofsince the manipulators to the control of the Finally Thethe haptic devices developed so far not focuses their applications feedback the road result. conditions by pitch and roll motions. feedback to steering Finally the lower plate is controlled for the yawing motion to The haptic devices developed so far focuses their applications motions than the mobile robots. This the paper of six for sections including mobile robot ofsince manipulators not require to the control the manipulators to the more controldelicate of the Finally lowercomposes plate is controlled the yawing motionthis to feedback to steering result. to the control of the manipulators not to the control of the introduction. Thecomposes design the six proposed hybrid joystick this has motions than thesince mobilemanipulators robots. mobile robot require more robots delicate feedback to steering result.of of This paper sections including In many applications, the application of mobile is mobile robot since manipulators require more delicate been illustrated in detail in Section 2. The feedback motions than thesimple mobile robots. introduction. design of of the six proposed hybrid joystick this has This paperThecomposes sections including limited to the Therefore a simple motions than the mobilenavigations. robots. In many applications, the application of mobile robotstwo is algorithms Thisillustrated paper composes ofin sixSection sections including this of the road conditions to the operator have been been in detail 2. The feedback introduction. The design of the proposed hybrid joystick has d.o.f. joystick has beennavigations. utilized for Therefore the tele-operation of two the limited to the simple a simple In many applications, the application of mobile robots is introduction. The the proposed hybrid systems joystickbeen has described inofSection 3.detail InofSection 4, and algorithms the design road conditions to the the control operator been illustrated in in Section 2. The have feedback In many applications, the application of mobile robots is mobile robot. Recently Farrokh[7] and N. Diolaiti[8] utilized d.o.f. beennavigations. utilized for Therefore the tele-operation the experimental limitedjoystick to the has simple a simpleof two been illustrated in detail in Section 2. The feedback condition to verify the performance of the 3. conditions In Section 4, and algorithmsinofSection the road to the the control operatorsystems have been limited to theRecently simple Therefore a simple two described the force to navigations. the joystick to detect and avoid mobile robot. Farrokh[7] Diolaiti[8] utilized d.o.f.feedback joystick has been utilized forand theN. tele-operation of the the hybrid algorithms of the road conditions to the operator have been haptic joystick have been illustrated in detail. Section condition verify 4,the of and the described in Section 3. IntoSection the performance control systems d.o.f.feedback joystick hasdriving been utilized for N. theAndo[9], tele-operation of the the experimental obstacles in the path. And C[10], the force to the joystick to detect andRen. avoid mobile robot. Recently Farrokh[7] and N. Diolaiti[8] utilized described in Section 3.have In Section 4, of thethe control systems and 5 shows the experimental results teleoperation of hybrid haptic joystick been illustrated in detail. Section experimental condition to verify the performance of the mobile robot. Recently Farrokh[7] and N. Diolaiti[8] utilized and F. Arai[11] applied thejoystick force to the the obstacles in the driving path. And reflective N. C[10], the feedback force to the to Ando[9], detect joystick andRen. avoid experimental condition to verify the performance of the mobile robots with the designed hybrid haptic joystick and 5 shows the joystick experimental results of the in teleoperation of hybrid haptic have been illustrated detail. Section the feedback force to thevarious to detect and avoid tele-operation to perform tasks a limited and F. Arai[11] applied thejoystick force reflective joystick to the the the obstacles in the driving path. And N. with Ando[9], Ren.d.o.f. C[10], hybrid haptic joystick been illustrated in detail. Section conventional d.o.f. joystick demonstrate the mobile robots with 2thehave designed hybrid haptic joystick and 5 shows the experimental results of to the teleoperation of obstacles in the driving path. And N. Ando[9], Ren. C[10], tele-operation to perform various tasks a limited andFor F. Arai[11] applied the force reflective joystick to the the 5 shows the experimental results of the demonstrate teleoperation of superiority of the hybrid haptic joystick. Sectionand 6, general tele-operation thewith mobile robot,d.o.f. vision conventional 2thed.o.f. joystick the robots with designed hybridtoFinally haptic in joystick and F. the Arai[11] applied the forceof reflective joystick to the mobile tele-operation to perform various tasks with a limited d.o.f. mobile robots with the designed hybrid haptic in joystick and this research is concluded with suggestion of a future sensors are widely utilized [12-14]. In the image based telesuperiority of the hybrid haptic joystick. Finally Section 6, conventional 2 d.o.f. joystick to demonstrate the tele-operation to perform various tasks a limited For the general tele-operation of thewith mobile robot,d.o.f. vision the the conventional 2 d.o.f. joystick to demonstrate the research work. operation, it is not easy to obtain the read conditions such as this research is concluded with suggestion of a future of the hybrid haptic joystick. Finally in Section 6, sensors aregeneral widely tele-operation utilized [12-14]. In the imagerobot, basedvision tele- superiority For the of the mobile superiority of the hybrid haptic joystick. Finally in Section 6, For the general of mobile robot,such vision work. this research is concluded with suggestion of a future operation, is not tele-operation easy to obtain thethe as research sensors areitwidely utilized [12-14]. Inread the conditions image based telethis research sensors are widely utilized [12-14]. In the image based tele- research work. is concluded with suggestion of a future operation, it is not easy to obtain the read conditions such as Copyright IFAC operation,©it2018 is not easy to obtain the read conditions such as 13 research work. 2405-8963 © 2018, IFAC (International Federation of Automatic Control) Hosting by Elsevier Ltd. All rights reserved. Copyright 2018 responsibility IFAC 13 Control. Peer review©under of International Federation of Automatic 10.1016/j.ifacol.2018.11.511 Copyright © 2018 IFAC 13 Copyright © 2018 IFAC 13
IFAC SYROCO 2018 14 Budapest, Hungary, August 27-30, 2018
Jangmyung Lee et al. / IFAC PapersOnLine 51-22 (2018) 13–18
2. DESIGN OF A HYBRID HAPTIC JOYSTICK
between his command and the real driving directions by the yaw motion. 2.1 Pitch/Roll Motion of Upper Plate There are three frames: base frame {xB , yB , zB } , lower plate frame
{xL , yL , zL }
, and upper plate frame
{xU , yU , zU } . The motion components at the at the upper plate by the three linear actuator are pitch, roll, and z . Notice that there are three linear actuators vertically between the same size of upper and lower plates. Those are represented by P1 , P2 , and P3 . Therefore, x , y , and yaw motions cannot be generated by these three linear actuators. Notice that the x , y motion commands are generated by the joystick on the upper plate and sent to the mobile robot controller directly and the yaw motion is generated by a revolute joint below the lower plate. The pitch angle, , which is defined w.r.t. the center of
Fig. 1. Structure and functions of the hybrid joystick Figure 1 illustrates the structure of the hybrid haptic joystick proposed in this research, which can generate six d.o.f. motions. The part ① in Fig. 1 is the same as the conventional two d.o.f. joystick which generates the direction commands of the mobile robot. The two d.o.f. hall sensor detects the operator’s steering intention and converts to the driving direction of the mobile robot. The joystick has been attached on the top of the upper plate where the operator’s hand will be placed when he is using this joystick for the teleoperations. Changing the velocity can be implemented by pushing the upper plate downwards by the hand like accelerator for the automobile. To detect the pushing force which corresponds to the acceleration or deceleration command at the center of the upper plate, there are three force sensors at the connecting junctions of the linear
upper plate,
OU . Since OU is fixed at the center of the upper
plate, the pitch angle, theta, can be obtained as follows:
sin 1
P1 ( z )
P2 ( z ) P3 ( z ) 2 a
(1)
P2 and the center where a represents the length between P1 ,(z) point P of 1 , P2 (z) and
P3 (z).
The roll angle, , is defined w.r.t. the center of upper
actuators coming from the lower plate in part ②, respectively. To generate the deceleration command with the acceleration command by the pushing force at the upper plate, a threshold value has been set to differentiate the boundary between the acceleration and deceleration, that is, a threshold value can be used to represent the zero acceleration or the constant
OU . Since OU is fixed at the center of the upper plate, the roll angle, , can be obtained as follows: plate,
sin 1
velocity. At the part ② , there are three linear actuators connecting the upper plate to the lower plate. The motion of the three linear actuators has been utilized to generate the pitch and roll motions corresponding to the road conditions of the slope and tilt. When the mobile robot is driving a road with a slope and a tilt, the INS sensor attached at the mobile robot detects the slope and tilt values and feedback to the hybrid haptic joystick which reflects the values by the motion of the upper plate. The operator can feel the road slope and tilt by his hand on the upper plate and can change the velocity. For an example, when the mobile robot is driving on the upward slope, the speed of the mobile robot will be decreased, which can be recognized by the slope feedback and he can increase the dropped speed by pushing down the upper plate.
P2 ( z ) P3 ( z ) c
P1 , P2 (z) and where c represents the distance between
(2)
P3
(z)
points.
The yaw angle,
, is generated by a revolute joint between OL and OB which is limited within ± 60 to reveal the driving direction of the mobile robot. The revolute joint can be controlled without the angle limit. However it is bounded to cope with the wrist motion of a normal people, which is limited about ± 60 . The z directional motion can be generated by the same amount of motion by the three linear actuators, P1 , P2 and P3
At the part ③, there is a revolute joint which is used to reflect the result of the steering command by the yaw motion. When the steering commands are kept faithfully, the yaw motion becomes zero. The operator can recognize the difference
which can be represented as follows:
zd P1 P2 P3
14
(3)
IFAC SYROCO 2018 Budapest, Hungary, August 27-30, 2018
Jangmyung Lee et al. / IFAC PapersOnLine 51-22 (2018) 13–18
Notice that this z direction motion is independent from the pitch and roll motions.
User
Feedback Hall sensor Joystick (steering)
xd , yd
2-2 Pitch and Roll Motions of the Upper Plate
U Jq
T
q P1 P2
Vision Sensor data
OU , and
T
J13 J 23
J †12 J †22 J †32
Pitch Control
d
Roll Control
d
Yaw Control
d
f P1f , P2 , and f P3 are used for
3.1 Joystick and lower plate control The hall sensor joystick has been utilized for generating the steering command by the operator and for btransmitting
(5)
the commands directly to the mobile robot by
xd , yd . Not
any control actions are required for the two d.o.f. motion. The revolute joint at the lower plate has been utilized for revealing the yaw angle of the mobile robot corresponding to the steering commands, which has been controlled by PD algorithm to get the stable and fast performance with the simple algorithm.
To generate the roll and pitch motions by the three linear actuators, Eq. (4) needs to be solved for q . In solving the under-determined system of the Eq. (4), the constraint equation can be incorporated. However, the z directional motion is used as an input for the operator, which is not controlled to reveal the road conditions, it is separated in the inversion process. That is, the pseudo inverse of Eq. (4) has been utilized to obtain the linear actuator motions as follows:
P1 J †11 q J †U P2 J † 21 P3 J †31
changing the velocity of the mobile robot by converting to the command of dot{Z}.
linear actuators. The Jacobian matrix has 2X3 dimension and can be derived from the Eqs. (1) and (2) as follows:
J 22
vd
INS Sensor data
common components of
P3 represents the motions of the
J12
P3
Conversion to velocity
Figure 3 illustrates the block diagram of the whole system. The operator basically utilizes the vision information for monitoring and for controlling the mobile robot. The INS sensor attached at the mobile robot gathers the road conditions such as tilt and slope of the driving road and the steering information of the mobile robot, which are feedback back to the hybrid haptic joystick as roll, pitch and yaw, respectively. For the pitch and roll control, the force sensors installed between the end of the linear actuators and the upper plate , P2 , and f P3 as shown in Fig. 6. Notice that the providef P1f
represents the roll and pitch motions
J J 2 X 3 11 J 21
Upper Plate
f P1f , P2f ,
Fig. 2. Block diagram of the overall system
(4)
of the upper plate w.r.t. the origin of the upper plate
zd
Mobile Robot
The dynamic motion of pitch and roll motions of the upper plate are designed to transfer the road slope and tilt to the operator putting his hand on the upper plate. The operator may watch the monitoring screen to see how the mobile robot is moving as he operates. However the road conditions are very difficult to be watched by the screen. The roll and pitch motions of the upper plate are generated by the three linear actuators. Therefore, the motion equations can be represented as follows:
Where U
15
3.2 Control of the upper plate Three force sensors are installed at the end of the P1 , P2 , and P3 to sensor the pushing force by the operator to generate
(6)
zd command, that is, the velocity of the mobile robot, vd ,, and to generate the roll and pitch motions corresponding to the tilt and slope angles of the road based upon the impedance control that keeps the a desired relationship between the force and motion. The operating can push his hand on the upper plate with a certain force which can be used as a threshold value, f thre , to
3. CONTROL ALGORITHMS OF THE HYBRID HAPTIC JOYSTICK
differentiate between the acceleration and deceleration conditions. This threshold value can be adjusted for each operator before the real driving of the mobile robot. operation. The force at the center of the upper plate can be represented as, f P , and the value can be obtained by the three sensors as follows:
15
IFAC SYROCO 2018 16 Budapest, Hungary, August 27-30, 2018
Jangmyung Lee et al. / IFAC PapersOnLine 51-22 (2018) 13–18
f Pf P2f fP 1 3
P3
controller of the mobile robot. Two DC motors and encoders are used for driving and sensing the wheel motions. To sensor the roll, pitch and yaw angles, an INS sensor (EBIMU-9DOFV3) has been installed on the mobile robot with a CCD camera. The image and INS data are sent to the remote operator through the communication channel, UART. For the main controller, ARM Cortex STM32F4 has been used. The states of the mobile robot have been monitored by the PC screen. The INS data have been transmitted to the main controller to control the linear actuators to reflect the slope and the tilt conditions of the driving road. An extra INS has been attached at the bottom of the upper plate to measure the real roll, pitch, yaw angles of the upper plate, which has been utilized to control the yaw angle by PD control algorithm.
(7)
As summarized in Eqs. (8) and (9), the relation between
f pf 1 , pf 2,
p3
P1 , P2 , P3 needs to be determined
and
properly. Since the motions and forces of the upper plate are directly related to each other since the operator is putting his hand on the upper plate. Therefore the upper plate control by
P1 , P2P,1 ,Pand 3P2 , P3 can be properly controlled by a desired impedance Zd with the sensing of f pf 1 , p 2 , and f p 3 [15-20]. the three linear actuators,
The control of each link for the upper plate, which is represented as a desired length, ld , for one of the actuators of
4.2 Experimental environment
P1 , PP21,, P P and 3 2 , P3 .has been implemented by PID algorithm.
The length feedback, l , is implemented by the INS sensor attached under the upper plate. The relation between the force , f p ,and the length, l , can be defined by the
Figure 4 illustrates the experimental environments to show the effectiveness of the proposed hybrid haptic joystick in the tele-operation of a mobile robot on the various roads. A CCD camera is installed on the ceiling to monitor the motion of the mobile robot on the designed paths.
impedance as follows:
f
p
s Z sl s
where Z can be defined a desired impedance
(8)
Z d for
considering the operator’s manner of handling the joystick. The general form of desired impedance can be derived from the relation between the force and the length as follows:
f p md ld l bd ld l kd ld l Where md
(a) Straight line path with up/down slopes (9)
, bd , and kd represent inertia, damping, and
stiffness parameters, respectively, which can be obtained from the dynamic characteristics of the upper plate and the human hand.
4. SYSTEM COMPOSITION AND EXPERIMENTAL ENVIRONMENT (b) A straight up/down slopes & curved path
4.1 System composition
Fig. 3. Experimental environments
The operator controls the mobile robot at the remote site using the images sent from the CCD camera on the mobile robot. Based upon the images, he controls the mobile robot to avoid obstacles and to follow the desired path through the communication network using the two d.o.f. hall sensor joystick. To accelerate or decelerate the mobile robot, he may push or slight pull the upper plate, which is detected by the force sensors installed at the end of the linear actuators. At the same, time the road conditions of tilt and slope are measured by the INS sensor on the mobile robot and sent back to the operator hand on the upper plate by the roll and pitch motions, respectively. The DSP 28335 has been used for the main
Figure 4(a) shows the straight line path with
5 or
15 upward and downward slopes, Fig. (b) shows a curved path after the up/down slope paths. The slope 15 can cover most of roads with the slope limited by the laws. Notice that the slope affects a lot on the speed of the mobile robot with a given command, which is not easy to being recognized by the remote operator. In this research, the control capability of the individual person has not been considered. That is, one of theauthor solely conducted the remote operations of the mobile robot 16
IFAC SYROCO 2018 Budapest, Hungary, August 27-30, 2018
Jangmyung Lee et al. / IFAC PapersOnLine 51-22 (2018) 13–18
by using a conventional joystick (two d.o.f. hall sensor joystick) and the hybrid haptic joystick to demonstrate the superiority of the hybrid haptic joystick in the tele-operations.
17
controlled smoothly by the conventional joystick which cannot provide the down slope information properly to the operator. However, using the hybrid haptic joystick the curved motion after the down slope has been implemented successful with the force feedback at the upper plate as shown in Fig. 6. .The tele-driving using the hybrid joystick shows a lot better performance than the driving using the hall sensor joystick in terms of maximum errors as well as average errors. Not only the driving accuracy but also the driving stability has been improved when the hybrid haptic joystick has been utilized for the tele-operation of the mobile robot, which has been felt by the operator putting his hand on the upper plate which reflects the road conditions.
5. EXPERIMENTS AND DISCUSSIONS Figure 11 shows how the mobile robot motions are captured by the CCD camera on the ceiling and feedback to the remote operator who is going to control the mobile robot using the hybrid haptic joystick. Real motions of the mobile robot is recorded and analyzed through the image processing.
6. CONCLUSION In this research a hybrid haptic joystick has been designed and implemented for the tele-operation of a mobile robot by an operator watching the driving environment through the CCD camera images. The road conditions of slope and tilt have been efficiently transferred to the operator through the pitch and roll control of the upper plate where the operator is putting his hand on. With the feedback of the road conditions by the hybrid haptic joystick, the operator can control the mobile robot more precisely and stably that without the feedback by the conventional joystick. The efficiency of the hybrid haptic joystick in generating the commands as well as feedback the road conditions has been verified through the tele-operation of the mobile robot. However, in designing the impedance controller for the hybrid haptic joystick, the desired impedance has been determined empirically for the author. In the future research, the desired impedance needs to be adjusted for a certain operator using an artificial intelligence algorithm.
Fig. 4. Driving of the mobile robot on the straight line path with
15 slope
The maximum driving errors of the tele-operations by different joysticks for the 15 slope path the driving errors by the two d.o.f. hall sensor joystick becomes a lot bigger than those by the hybrid haptic joystick. The maximum error was 16.1 Cm by the hall sensor joystick, while it was only 6.8 Cm by the hybrid haptic joystick.
REFERENCES [1] T. Massi and K. Salisbury, “The PHANToM haptic interface: a device for probing virtual objects”, ASME Winter Annual Meeting, vol. 55, no. 1, pp. 295-300, 1994. [2] W.-J. Chen, “A novel 4-dof parallel manipulator and its kinematic modeling”, in: IEEE Int. Conf. on Robotics and Automation, Seoul, 23–25, pp. 3350–3355, May 2001. [3] H. Inoue et. Al, “Parallel manipulator”, Proc. ISRR, Robotics Research 3, 1985. [4] Yi Lu, Bo Hu, “Analyzing kinematics and solving active/constrained forces of a 3SPU + UPR parallel manipulator”, Mechanism and Machine Theory, vol. 42, pp. 1298-1313, 2007. [5] Q. Li, Z. Huang, “Type synthesis of 4-dof parallel manipulators”, in: IEEE Int. Conf. on Robotics and Automation, Taipei, 14–19, pp. 755–760, September 2003. [6] Gregory L. Long and Curtis L.Collins, “A pantograph linkage parallel platform master hand controller for force-reflection,” Proc. IEEE Int. Conf. Robotics and Automation, pp. 390-395, 1992. [7] Farrokh Janbi Sharifi, lraj Hassanzadeh, “Experimental Analysis of Mobile-Robot Teleoperation via Shared Impedance Control”, IEEE Transactions on Systems, vol. 41, no. 2, pp. 591-606, 2010. [8] N. Diolaiti, C. Melchiorri, “Teleoperation of a mobile robot through haptic feedback”, EEE International Workshop. Ottawa. Canada, Nov. 2002. [9] T. Massi and K. Salisbury, “The PHANToM haptic interface: a device for probing virtual objects”, ASME Winter Annual Meeting, vol. 55, no. 1, pp. 295-300, 1994. [10] W.-J. Chen, “A novel 4-dof parallel manipulator and its kinematic modeling”, in: IEEE Int. Conf. on Robotics and Automation, Seoul, 23–25, pp. 3350–3355, May 2001.
Fig. 5. Comparison of curved path motions of the mobile (Fig. 10 (b) path) Figure 6 illustrates the driving results of the mobile robot on the path shown in Fig. 5(b). In this comparison, the performance difference between the two joystick in the teleoperations is distinctive at the location around x= 5 m where the down slope is finished and the curved path is started. At the end of the down slope, the velocity of the mobile robot has been increased a lot and the turning motion cannot be 17
IFAC SYROCO 2018 18 Budapest, Hungary, August 27-30, 2018
Jangmyung Lee et al. / IFAC PapersOnLine 51-22 (2018) 13–18
[11] H. Inoue et. Al, “Parallel manipulator”, Proc. ISRR, Robotics Research 3, 1985. [12] Yi Lu, Bo Hu, “Analyzing kinematics and solving active/constrained forces of a 3SPU + UPR parallel manipulator”, Mechanism and Machine Theory, vol. 42, pp. 1298-1313, 2007. [13] Q. Li, Z. Huang, “Type synthesis of 4-dof parallel manipulators”, in: IEEE Int. Conf. on Robotics and Automation, Taipei, 14–19, pp. 755–760, September 2003. [14] Gregory L. Long and Curtis L.Collins, “A pantograph linkage parallel platform master hand controller for force-reflection,” Proc. IEEE Int. Conf. Robotics and Automation, pp. 390-395, 1992. [15]Selcuk KIZIR, and Zafer BINGUL, Fuzzy impedance and force control of a Stewart platform, Turkish Journal of Electrical Engineering & Computer Sciences, 22 (2014), pp. 924-939. [16]Cristian A. Vergara and Carlos F. Rodriguez, Hybrid Position-force Control for a Stewart Platform, Journal of Automation and Control Engineering, 5 (1) (2017), pp. 10-13. [17]Hannaneh Z. Arabshahi, Alireza and B. Novinzadeh, Impedance Control of the 3RPS Parallel Manipulator, in Proc. of the 2nd RSI/ISM International Conference on Robotics and Mechatronics, (2014) pp. 486-492.
18