Telerobotic control using instructions from human hand motion

ELSEVIER

Copyright © IFAC Robot Control, Wroclaw, Poland, 2003

IFAC PUBLICATIONS www.elsevier.com/localelifac

TELEROBOTIC CONTROL USING INSTRUCTIONS FROM HUMAN HAND MOTION

IDepartment of Electrical and Electronic Eng., University of Newcastle upon Tyne, Newcastle upon Tyne, UK, [email protected], 2Departrnent of Cybernetic Eng., Wroclaw University of Technology, POLAND, 3 School ofInformation Systems, University of East Anglia, Norwich, UK

Abstract: The paper describes a novel method to control the position and orientation of robot manipulator moving in response to real time instructions from human hand motion. The system uses 3D electromagnetic technique to acquire accurate multi-phalange hand motion. The motion parameters are transmitted via Ethernet to a remote robot controller that updates the robot set point in real-time. The hand motion is measured using inductive sensors integrated into a data-glove at key anatomical locations. An RTlOO robot manipulator has been used to test the accuracy and repeatability of the telerobotic posture control obtainable by this method. A posture update rate of 10 updates/s was achieved. The experimental results show the positionaI accuracy achieved is +/Imm and orientation angles better than +/- 10 • This accuracy is a limitation imposed by the robot manipulator used. The measurement accuracy was found to be much higher than +/- Imm. The paper concludes with possible modes to integrate force feedback within the same posture data glove. Copyright©20031FAC Keywords: 3D electromagnetic sensing, posture data glove, Ethernet, telerobotics, distributed robot control.

1.

movements but at the same time system designers have considerable latitude to modify the user's task. This approach to human on-line control of the manipulator requires an accurate on-line measurement of hand motion including grasp and fingerlhand position. Previous work has used commercially available CyberGlove™ (Kramer, et al., 2000) and Isotrak n™ technologies (Isotrak Inc., 2002). However both systems suffer from a distinct lack of definition of the hand posture relying on analogue bending distortion in optic fibres and only have really one accurate point of measurement, the Polhemus sensor itself placed on the back of the hand (Sturman and Zeltzer, 1994). The work reported here applies inductive sensors at 11 locations on the data glove corresponding to key posture and position locations (ref. Fig. 1). The data glove sensors are interfaced to an electromagnetic 3D imaging system (Bladen, 1995). Position and orientation information of theses sensors are used to control a robot. Several previous publications in the fields of telerobotics and robot instruction from perception have been quoted, such as the application of Kang & Ikeuchi (1995), who considered modes of robot

INTRODUCTION

Teleoperators are robotic devices that synergistically combine human and machine. They are robotic devices but differ from autonomous robots in an important way. A robot is a programmable, automated materials handling machine, while a teleoperator is a remotely controlled materials handling machine. Robots are relatively inflexible: they are unable to respond to an environment in ways beyond the scope of their programming or to develop new behaviors on-line in real time. Because teleoperators take advantage of human creativity and intelligence, they are able to respond to an environment more flexibly and develop new behaviors as required. And, although robots are able to sense some things accurately, they lack the broad spectrum of senses available to people and the ability to integrate multisensor information swiftly and intelligently. Because teleoperators take advantage of human cognitive and perceptual abilities, they can perform more efficiently in unfamiliar and dynamic environments. The design of human-machine interfaces for teleoperators is a demanding task because of the requirement for real-time reproduction of hand and arm

527

instruction for grasp operations using such an input device and Bladen (1993) show a graphical visualization of colonoscopes during key-hole surgery.

Fig. 1. Sensor positions on data glove 2.

MULTI-POINT DATA GLOVE SENSING

The 3D imaging system (ref. Figure 2a) accurately captures hand posture. The hand posture data frame is then broadcast to a client PC via Ethemet. The client PC acts as the controller for a six-axis robot manipulator, RT 100 (ref. Figure 2b), which is programmed to generate continuant movement following instructions from human hand motion.

uninhibited control and interaction with the computer system. Thirdly, the sensor measurements of alternating electromagnetic filed are found to be stable, repeatable and accurate. These performance metrics are now presented. Stability is observed by placing the data-glove at a static position in the magnetic field and analyzing both position X, Y, Z and orientation, e and ~ data of each frame captured from one sensor in a time duration. Figure 3a shows the variation in meters, in form of standard deviation, between each frame. Repeating the stability experiment and choose different sensors, repeatability can be observed. By calculating the relative vibration percentage, conclusion could be reached that the stability of the system is higher than other systems. This system also achieved high accuracy, to quantitatively evaluate the accuracy, a single sensor has been located at two different positions along different directions. To avoid unexpected errors, the sensor has been kept static for about 10 seconds at each position while the position and orientation data is being recorded. The mean value of X, Y, Z, e and ~ is calculated and compared with the measurement taken with a ruler. High accuracy can be observed from Figure 3b displayed below. Average Standard Deviation of 147 frames is: X: -2.7707583026337E·7 Y: 9.94414862507267E-9 z: -7.21701178468776E·7 Theta: 9. 36560235985394E-7 Ph 2.46362843442193E·7

Fig. 3a. Stability statistics accuracy oIlhe position and OIientation measurement of the sensOl is: Actual Travelfng Distance: 0.0705 System Meas~ed TraveingDistance: 0.0697437935685407 AcluaI Rolatilg Angle: 0.6981 System Me~ed Rolatilg Angle: 0.68344469237852

Fig. 2a. The 3D imaging system, 2b. Robot RTlOO The objective of this paper is to present the theory of electromagnetic 3D imaging and its application in a new data glove used to control a manipulator remotely. This approach appears to offer a simple, convenient, low cost and robust human - computer interface for remote robot control. The 3D imaging Data Glove used is comfortable, light-weighted and do not restrict movement or affect the motion of the hand during use. Electromagnetic sensors provide a powerful technology for humancomputer interaction (HC!) and are being applied in a number of medical procedures such as endoscopy, (Rowland and Bell, 1998) and in (Bell, 2002) and quantitative analysis of Parkinson's Disease (Su et aI., 2002) and in (Warner et al.. 1991). One particular advantage is that the user may harness the technology easily and routinely by using ready-to-wear articles of clothing e.g. headsets or data-gloves (Pimental and Teixeira, 1991). Secondly the 3D sensors may simply be plugged into a computer system, allowing the user

Fig. 3b. Accuracy statistics To allow client PC to access the sensor data simultaneously, a data transmission program has been implemented to transfer the sensor data via network, further, live hand motion is used as on line instructions to robot RTlOO. Section 3 describes the principle of the electromagnetic 3D measurement. Section 4 introduces the algorithm to transmit data via Ethernet to allow a client PC to control the robot manipulator. Section 5 demonstrates the results of experiments carried on the 3D electromagnetic robot control system including a electromagnetic posture data-glove, an 3D imaging system and a 6-axis robot RT 100: (i) Position driven end point triggering movement (ii) Orientation driven rotating movement (iii) Position driven gripping movement

528

3.

PRINCIPLE OF 3D ELECTRO-MAGNETIC MEASUREMENT

(A), b is the side of the coil (in metres), unit vector in direction of R,

In order to measure the 3D position and orientation of a sensor an alternating current (AC) magnetic field must be applied. Each sensor coil then provides an induced voltage that is proportional to the magnetic field strength. To compute the full 3D position and orientation specification of the sensor coil requires several simultaneous measurements to be made. Nine generator magnetic sources, arranged in three orthogonal sets placed on a horizontal plane, have been used. The system is based on low strength magnetic field capable of capturing the absolute position of the magnetic sensor mounted on the data glove and avoiding the difficulties of calibration of conventional bending sensors. The technique is considered medically safe for use with patients undergoing medical treatment and unlike acoustic and microwave fields, the generators do not need to be in contact with the body or require matching media making the system convenient to use.

z -.

a

R

is the

is the unit vector in

direction of a Deriving the Cartesian form, resolve field element

a

and

into G,1"

a

ay

and

a

z'

a

R

re-eombine these

elements, for each generator coil with co-ordinates at (XGi, YGi, ZGi) , the magnetic field strength at the measurement sensor can be written as: B .

=~[3Z(OX(X-XGi)+OY(Y-YGi»)+

GI

R

5

]

(2)

0z(2z2_(Y-YGi)2_(X-XGi)2)

where i is the index reference number for the generator coil. The electromotive force (E.M.F.) induced in the sensor coil when placed in a magnetic field (ref. Figure 4b) is proportional to the resolved component of the field along its axis: (3) s = k s (B.a s ) where: Vs is the E.M.F induced in the sensor (V), ks is a coefficient defining the sensitivity of the sensor (Vis), is the unit vector in direction of the sensor, V

"z

...•

a

a

as

defined by the two Euler angles () and t/> (ref. Figure 4b) as:

.... _~- '". P(x.1,) ..-....

as =(Sin().a.xy +cosB.a) z

(4)

Expressing the E.M.F induced in the sensor coil related to each generator coil (and their magnetic field), the following e~uation can be obtained:

y

x

kGk S

"x

[3Z sin ()t~x - X Gi )cos t/> + lY - YGi )sin t/> J]

Si =-;5 +cos~2z2 -&-yGir -(x-xGY)

Fig. 4a. Diagram for the dipole field equation

V

(5) D.

where: Vs; represents the induced voltage inside sensor coil in respect to each generator coil, and i is the number of generator coil. Eqn. 5 represents a set of non-linear simultaneous equations which has to be solved to obtain x, y, z, e and $. Figure 5 displays typical position and orientation readings.

.....•

CIIn1el

Fig. 4b. Sensor orientation in a magnetic field defined bye, $ Eular angles

Sueceu

, 0

3

•

'] B = -kG" [2 a R cos a + a" a Sin a R3

5

6

7 8 9 10

(I)

)J

0

NIb 2 ,

and f.J 0

=

Z

lhel.

Pti

0.059 0.129 0.087 0.060 0..05 0.083

0292 0.:Jl1 0311 0.277 0311 0310 0273 0.291 0233 0327 OE

USl U21 1.718

·2.326 ·2.072 ·2.311

FoeItloll d

'anoe

-0.003

0.1E1 0.013 -0018 0.014

-0013 0.007 -0104

-0015

0.012·()072 0032 0010

1.279

-2.886

V7.

228C -2.464

1.599 1.500 1.C6S

1.63; 1.448 1.227

-2.735

·2.822 -2.:m -2.768 ·0932

Fig. 5. On-line position and orientation measurement for the 3D sensors

where: B is the magnetic flux density (T), R is the distance from the dipole to the point P (m), a is the angle from the coil axis to the point P (rad), kG is a constant equal to

y

-0124 -0.097

-oll54

2

The magnetic field, B at a point P, produced by a dipole at the origin (ref. Figure 4a), is given by:

x

4.

7 N' 4Jr 10 -, IS

DATA TRANSMISSION VIA ETHERNET

The internet provides a convenient and universal network interface and so was chosen as the data

411'

the number of turns, 1 is the current inside the coil

529

position data to the robot controller (ref. Figure 7a).

transrmsslon medium for this application (Gilbert, 1996). The IrnpriselBorland Delphi compiler was used to implement this application. It provides software support for both ends of the Ethernet interface viz; client and server sockets that allow the network application to form connections to other machines. In this application, Win Socket components are used to carry out data acquisition and transmission. The client robot control machine sends posture data request to the 3D motion data acquisition system, which acts as a server (ref. Figure 6a). After capturing a posture data frame, the 3D motion system sends the hand posture data to the client machine (ref. Figure 6b). The robot joint control angles are then calculated using inverse kinematic model. 5.

inIdaIIDl rubDt _ DD CIiont ...DtrD1 _>(robDt

.. IIITVI'

yes

oa:'IIITlld 7 ..

•

DO

dJwr YIrtoU _ _t ID world ..... nllData. Dbsl8t1es . . robDtat DriIl_1pasitlDn

.

ca_t ID Sener (ima&Ior _h1IIe).Y1acl_ _ t.

_

no

errar_DoI!Idt

tuCCBJd? ..

yes

elld paotmw d_ to s.ver _rIII-t _>Y1a

-_loot

~

•

ID ..... I)' tram

_

SenBr

~_h1IIe>.Y1ac_t

_bt

• •

EXPERIMENTS AND ANALYSIS

..... p - . . data tram WIn

Socbt recoMD& bulr..

..

Experiments have been made to investigate how a 6axis RT 100 robot manipulator behaves when driven from 3D position data derived from the data-glove, over the local area network. To allow the robot movements to be measured in its coordinate frame additional 3D sensors were attached to the robot limbs. Additionally software offsets were added to allow free movement of the hand (operator) and the robot as they both shared the same reference frames.

calc ..... robDtjalDt b_ Dn haIId pasltlDn (usb1c IllftrR K _ t b Mlldol)

Is _

pasltlana iIIIalalld •

no

WamI,. &

-~~:,- ~_ ..~

•

redraw rubDt mod"IDworld

coordinatab..... OD D8r pasltinn

inida1ize Sener (iJna&er macblne),ready to capture posture sensor data

~

error occurred? no

+

yes _

Fig. 6b. Client floating chart exit

.::5:::"J + +

Is data received the posture data nquest?

no

yes startsampliDa. capture posture data

+

Fig. 7a. Active hand sensor; 7b. Robot moving in concert with the hand sensor

send postun data to client machine

Once the RTlOO has arrived at the demand Zposition it remains at that position with no perceptible variation. This demonstrates two points: firstly that the hand demand signal is highly stable. Secondly the intrinsic delay in hand position measurement and transmission of that data via the Ethernet connection is very short indeed. Two tests have been carried out for this experiment, Figures 8 and 9 show 3D patterns of hand and robot movements. There is a time delay between hand and robot of 1s. This is not umeasonable bearing mind the joint speeds achievable by this manipulator. For example the Z- lift maximum speed is 15cm/s, gripper motion <5cmls.

Fig. 6a. Server floating chart The combined movements were measured by logging the robot borne sensor data after the RTlOO had completed its updated move before the next round of hand data is transmitted to the robot controller. Three experiments are now described which monitor specific world frame movements of the robot relative to the hand movements. Experiment 1: Robot moving to the position demanded by the human operator (re! Figure 7). The elaborated inverse kinematic model allows hand sensor to guide robot RT-100 in space. The sensor located on the tip of middle finger is used to provide

530

Experiment 2: Rotation Movement The hand sensing is now employed to provide 3D angular information for the control of the robot wrist. This movement can be driven in two different ways: I) using sensor orientation information directly obtained from the sensor located on the wrist (ref. Figure lOa) or 2) using orientation information formed by the position of two sensors located separately on two adjacent finger phalanges (ref. Fig. lOb). As described in section 11, the 3D electromagnetic posture sensor provides two angular measurements e and cP, together they provide the direction vector of the sensor. The wrist joint of the RTlOO robot may be controlled using sensor, S 10 on the data glove. Figure 11 is the diagram explaining the methodology to obtain wrist rotation instructions from the direction vector of sensor, SI O. The dot product of every two direction vectors obtained from different positions is used to calculate the hand rotating angle (Stephenson, 1965) which then instructs the robot RTl 00 as a wrist roll demand signal.

Fig. 11. Calculation of wrist rotation angle.

AI = cos

_I

[VtO

V

directionS! 0 . tl directionS! 0 ]

(6)

Ivto _ directionS! 0 1·lvt ] _ directionS! 0 I

-I [V

tO directionS!

A2 = cos

0.V

t2 directionS!

0]

(7)

Ivto _ directionS! 01'!Vt2 _ directionS! 01

where: A I, A2 are rotating angles from to to different time, tl and t2;

z

VtO_directionSIO

is the direction vector of

sensor SI 0 at time, to; V,I

.6

directionSIO

vector of sensor S10 at time, tl;

is the direction

V;2 _directionS! 0

is the

direction vector of sensor SI 0 at time, t2. The RTlOO robot has two geared motors that control the wrist movement of the robot. These motors are called WRIST I and WRIST2. When driven in combination, these motors provide the Pitch and Roll wrist movements for the robot wrist (ref. Figure 12). These joints require demand signals from the hand.

--

Fig. 8. Hand and Robot motion in 3D Space testl: Tracing around a box (Axes measured in metres) z

--

Fig. 12. Hand driven rotation movement

Fig. 9. Hand and Robot motion in 3D Space test2: Helix drawing (Axes measured in metres).

The pitch angle in degrees is given by the mean angular displacement of each motor i.e.

[(WRISTl + WRIST2). k]/ 2

(8)

Roll in degrees is given by the differential angular rotation of each motor i.e.

[(WRIsn - WRIST2). k]/ 2

Fig. lOa. Wrist sensor

(9)

where k is the constant RTlOO using to convert an angle in degrees to motor encoder counts. Hand was rotated (ROLL movement) repeatedly through an angle of approximately ±4S o and each ROLL movement captured via one sensor mounted on the gripper and from the data glove sensors.

lOb. Two sensors to provide rotation instructions

53\

Figures 13 and 14 show the hand and robot wrist motion results, taken from two tests carried out in this experiment, in 3D space.

z

X

,I

--

In comparison with current telerobotic practice the approach presented in this paper appears to have a number of advantages. Firstly, the "slave operator robot" has been replaced by a low cost glove and PCbased acquisition computer. This allows for simple interconnectivity on any desired network protocol. In this application, the Ethernet LAN has been employed. Secondly there is no requirement for a direct mechanical coupling between the master and slave manipulators as is the case commonly found in nuclear processing applications. Thirdly, the location of the human operator is selectable. Finally, the scope of tele-operation is extendable by the use of a data glove to capture movement

Fig. 13. Hand and robot wirst rotation in 3D Space, test 1. (Axes measured in Unit Vector)

REFERENCES

z

Kramer, D. (2000). Goniometer-based body-tracking device and method, US Patent 6,050,962, April 18, Immersion Corporation California USA. Isotrak n, (2002). 3D motion tracking system Polhemus Inc http://www.polhemus.com . Sturman, DJ. and D. Zeltzer (1994). A survey of glove-based input. IEEE Computer Graphics and Applications, 14(1):30-39. Bladen, J.S. (1995). Imaging medical endoscopes in three dimensions using magnetic fields. Ph.D Thesis, University of Sheffield, England Kang, S.B. and K. Ikeuchi (1995). Toward automatic robot instruction from perception - temporal segmentation of tasks from human hand motion. IEEE Trans. Robotics&Automation, 11: 670-681. Bladen, J.S., A.P. Anderson, G.D. Bell, B. Rameh, B. Evans and DJ. Heatley (1993). Non-radiological imaging of endoscopes. Lancet, 341, 719-722. Rowland, R. and G.D. Bell (1998). Non-radiological technique for 3D imaging of intestinal endoscopes. Med. BioI. Eng. Comput., 36; 285-290. Su, Y., C.R. Alien, D. Geng, G.D. Bell, D.Burn and U. Brechany (2002). Three-dimensional Motion System (Data-Gloves): Application for Parkinson's Disease. IEEE Trans Instrumentation & Measurement (in press). Bell, G.D. (2002). State of the Art in Gastroenterologic Endoscopy - A review of last year's most significant publications. Endoscopy, 34; 1-11. Warner, DJ., A.D. Will and G.W. Peterson (1991). Quantitative motion analysis for movementrelated potentia1s. Electroencephalography & Clinical Neurophysiology, 79; 29-30. Pimental, K. and K. Teixeira (1992). Virtual Reality Through the new looking glass. Windcrest McGraw Hill. Gilbert, H. (1996). Ethernet networks: design, implementation, operation, management. Gilbert Held, New York: J. Wiley. Stephenson, G. (1965). Mathematical Methods for Science Students. 3rd Edition Longmans. OxIM, (1997). RTlOO User Manual, Oxford, U.K.

--

Fig. 14. Hand and robot wirst rotation in 3D Space, Test 2. (Axes measured in Unit Vector) As the hand was held out in the horizontal plane angular changes occurred predominantly as y-and zvariation. There was found to be an accurate correlation between the two y-z- traces. However there is a constant angular misalignment This was found to be due to a slight angular misalignment of the wrist sensor placed on the robot. Repeating the experiment with all sensors aligned, more satisfactory result is obtained. 6.

CONCLUSIONS AND FURTHER WORK

In conclusion a number of tests have been carried out over an Ethernet local area network (LAN) to investigate the merits of controlling a manipulator using the movement of a human hand. It has been concluded from these experiments that a 3D motion capture computer can accurately and repeat ably control a robot manipulator by this method. Essentially there is a continuous, real-time and accurate segmentation of instructions from the human operator. The technology of hand posture acquisition has been tested for stability, repeatability and accuracy of measurement and has been found to be exceptionally robust. There is no detectable drift in measurement with ambient temperature, or when the data glove is removed from the operator and then re-used.

532