- Email: [email protected]
Recognition of human grasps by time-clustering and fuzzy modeling
Robotics and Autonomous Systems 57 (2009) 484–495
Contents lists available at ScienceDirect
Robotics and Autonomous Systems journal homepage: www.elsevier.com/locate/robot
Recognition of human grasps by time-clustering and fuzzy modeling Rainer Palm ∗ , Boyko Iliev, Bourhane Kadmiry AASS, Department of Technology, Orebro University, SE-70182 Orebro, Sweden
article
info
Article history: Received 2 August 2008 Received in revised form 17 October 2008 Accepted 23 October 2008 Available online 8 November 2008 Keywords: Grasp recognition Programming-by-demonstration Fuzzy clustering Fuzzy modeling
a b s t r a c t In this paper, we address the problem of recognition of human grasps for five-fingered robotic hands and industrial robots in the context of programming-by-demonstration. The robot is instructed by a human operator wearing a data glove capturing the hand poses. For a number of human grasps, the corresponding fingertip trajectories are modeled in time and space by fuzzy clustering and Takagi–Sugeno (TS) modeling. This so-called time-clustering leads to grasp models using time as an input parameter and fingertip positions as outputs. For a sequence of grasps, the control system of the robot hand identifies the grasp segments, classifies the grasps and generates the sequence of grasps shown before. For this purpose, each grasp is correlated with a training sequence. By means of a hybrid fuzzy model, the demonstrated grasp sequence can be reconstructed. © 2008 Elsevier B.V. All rights reserved.
1. Introduction Applications of multi-fingered robotic hands have so far been relatively few, although the field has attracted significant research efforts. One of the reasons is that they are fairly difficult to program, due to the inherently high dimensionality of the grasping and manipulation tasks. One approach to solve the problem is to program the robotic hand task directly by human demonstrations. That is, the operator performs a task while the robot ‘‘observes’’ the demonstrated actions with the help of motion capture devices or video cameras. The robot has to recognize the actions performed by the demonstrator and replicate them on its own. This approach is called Programming by Demonstration (PbD) and is used in complex robotics applications such as grasping and manipulation. Our interest also lies in developing methods for the control of anthropomorphic robotic (prosthetic) hands where the manipulation strategies are learned from human demonstrations. The idea is to exploit the structural similarity between the artificial and the human hand in order to achieve similar functionality. This will, on the one hand, allow for a more intuitive teaching of complex robotic hands. On the other hand, this approach can be applied for the control of prosthetic hands, where it is desired to mimic the motion of the human hand as closely as possible. Different techniques and motion capture setups have been applied in PbD. For example, Kang et al. [1] describe a system which observes, recognizes and maps human grasps to a robot
∗
Corresponding author. Tel.: +49 892605263; fax: +49 892605263. E-mail addresses: [email protected] (R. Palm), [email protected] (B. Iliev), [email protected] (B. Kadmiry). 0921-8890/$ – see front matter © 2008 Elsevier B.V. All rights reserved. doi:10.1016/j.robot.2008.10.012
manipulator. Human demonstration is captured by a stereo vision system and a data glove. Zoellner et al. [2] use a data glove with integrated tactile sensors, where the classification method is based on support vector machines (SVM). In a similar setup, Ikeuchi et al. [3] apply Hidden Markov Models (HMMs) to segment and recognize grasp sequences. Ekvall and Kragic [4] also address the PbD-problem, using the arm trajectory and hand pose as additional features for grasp classification. Martinez and Kragic [5] use combinations of SVM’s, conditional random fields (CRF’s) and HMM’s for modeling and recognition of object manipulation actions in the framework of robot learning. Li et al. [6] use the singular value decomposition (SVD) to generate feature vectors for human grasps and support vector machines (SVM) which are applied to the classification problem. Aleotti and Caselli [7] describe a virtual reality based PbD-system for grasp recognition, where final grasp postures are modeled based on the finger joint angles. A combination of HMM’s and SVM’s is presented by Vicente et al. [8] for learning manipulation actions. Calinon and Billard [9] use Gaussian Mixture Models (GMM) for robot learning of human gestures. A fuzzy logic approach for gesture recognition was published by Bimber [10]. The method is applied to 6 d.o.f. trajectories of a human arm, but cannot cope with our multi-dimensional problem. Ferguson et al. developed a pattern recognition system for the control of limb prostheses [11]. In this approach, a fuzzy system classifies myoelectric signals in the frequency domain. Because of this very specific application the method cannot be generalized to grasp recognition for PbD. To the knowledge of the authors, there are no further fuzzy logic approaches reported both for modeling and classification that fit the dynamical grasp recognition problem. Fleet et al. [12] uses Bayesian filters in the framework of visual motion analysis to identify image motions in the vicinity of occlusion boundaries.
R. Palm et al. / Robotics and Autonomous Systems 57 (2009) 484–495
An excellent overview about the ‘‘different approaches taken todate within the computer vision, robotics and artificial intelligence communities for the representation, recognition, synthesis and understanding of action’’ is given by Kruger et al. [13]. Most of the methods are of probabilistic nature in contrast to fuzzy modeling which is a possibilistic and rule-based approach. Despite the advanced state of the art, the cited methods on dynamic classification use discrete states or observations (hand poses) and do not consider the continuous evolution of a grasp in time and space. This appears to be a disadvantage, because neither the estimation of the occurrence of a specific grasp nor the problem of segmentation can be solved in a general way. The method presented in this paper tries to overcome some of these drawbacks by modeling fingertip trajectories in time and space. To solve a ‘fuzzy’ task like grasp recognition, it turns out that fuzzy modeling is a suitable approach. The main focus of this method is programming by demonstration and recognition of human grasps using fuzzy clustering and Takagi–Sugeno (TS) fuzzy modeling. Two types of models are considered:
• models of the fingertip trajectories. • models for segmentation and recognition. The first type of models incorporate the time variable of a grasp motion in the modeling process (time-clustering), i.e. it interprets the time as a model input, see Palm and Iliev [14]. These models can be used to replay, scale and recognize demonstrated grasp primitives in a most flexible way. The second type of model deals with the recognition of grasps from a grasp sequence [15]. The recognition model consists of a set of fuzzy expert rules to identify a grasp from a combination of grasps using membership functions of exponential type. This model is called a hybrid fuzzy model because it combines a fuzzy rule structure and a crisp decision making procedure. One of the main advantages of our approach is that grasp recognition, hand control and execution monitoring can be implemented within the same modeling framework. In our method, the grasps are modeled as continuous fingertip motions in time and space. The other methods like HMM use discrete observations and states. Therefore, with our method, the recognition task can be improved by refining the granulation of the time clusters. For the discrete methods like HMM, a refinement of the model would require the introduction of additional observations (hand poses) leading to a significant increase of possible states and a corresponding computing effort. In a direct comparison between the fuzzy approach and the HMM method, we could show that the fuzzy approach gives better results with respect to the recognition rates [16]. This paper is organized as follows: Section 2 describes the general principles of the PbD-methodology, and the experimental platform. Section 3 deals with the simulation of the grasp processes and the geometrical modeling and kinematical representation of the human hand. Section 4 discusses the learning of grasps by time-clustering and TS fuzzy modeling. Section 5 describes a new method for segmentation and recognition of grasp sequences using a hybrid fuzzy approach. Section 6 presents experimental results for time clustering and TS fuzzy modeling for the segmentation and recognition of selected grasps. Section 7 draws some conclusions and establishes directions for future work. 2. General (PbD)
principles
for
programming-by-demonstration
The long-term goal of the project ‘‘Dexterous Manipulation’’ at the Center for Applied Autonomous Sensor Systems (AASS) leads to a control methodology for grasping and manipulation in unstructured environments. Programming by demonstration (PbD) involves interpretation of human demonstrations and
485
their automatic conversion into program code that can be executed on a particular robot system. The PbD-methodology for robotic grasping requires three main tasks: segmentation of grasp sequences, grasp recognition, and generation of robot code [17]. The first task is to partition the data record into a sequence of episodes, where each one contains a single grasp. The second task is to recognize the grasp performed in each episode. Finally the demonstrated task is converted into a task level program which can be executed by an artificial robot hand or a prosthetic hand. This program activates grasp primitives corresponding to human grasps and adapted to the particular robot system. In the following, instead of going into detail with this high-level program, the problems of modeling, segmentation and recognition of human grasp sequences will be addressed. The experimental platform consists of a hand motion capturing device (data glove) and a hand simulation environment. The motions of a human demonstrator are recorded by a data glove (CyberGlove) which measures 18 joint angles in the hand and the wrist [18]. Most data gloves available on the market are of the same size (‘‘one fits many’’). Smaller or bigger sizes of hands have small influence on the fingertip modeling results, as long as they do not deviate too much from each other. Since humans mostly use a limited number of grasp types, the recognition process can be restricted to a certain grasp taxonomy, corresponding to those developed by Cutkosky [19] and Iberall [20], respectively. 3. Simulation of grasp primitives For the purpose of PbD we need a model of the human hand which allows the simulation of demonstrated grasps. In order to test the grasp primitives, a hand simulation was developed, with the help of which one can mimic the hand poses recorded by the data glove. This hand model allows one to compute the trajectories of both the finger angles and the fingertips. Since the size of the object to be grasped determines the starting and end points of the fingertips to a great extent, we used fingertip trajectories instead of finger angles for modeling. Moreover, we can simulate recorded demonstrations of grasps plus corresponding interactions with objects (see Fig. 2) and compare them with the training sequences of grasp primitives. Inspired by the grasp taxonomy of Iberall, 15 different grasps have been tested (see Fig. 2). These grasps are special cases of the following general classes [20]: 1. 2. 3. 4. 5. 6.
cylindrical grasp (grasps 1, 2, 3, 14) power grasp (grasps 4, 5) spherical grasp (grasps 6, 7, 12, 13) extension grasp (grasps 10, 11) precision grasp, nippers pinch (grasps 8, 9) penholder grasp (writing grip) (grasp 15).
The advantage of this selection is that the quality of a grasp recognition, both between classes and within a class, can be analyzed (see Section 6: Experiments and Simulations). 3.1. Geometrical modeling In order to study grasp primitives and to develop specific grasp models, a geometrical simulation of the hand is required (see Fig. 1). The hand model consists of 5 fingers which are linked to the wrist in such a way that the poses of a human operator can be displayed in a realistic way. The kinematic relations can be studied by means of the example of a single finger (see Fig. 3). Each finger is modeled with 3 links and 3 joints moving like a small planar robot. This turned out to be sufficient for the simulation of the grasp primitives in Fig. 2. The calculation of fingertip trajectories requires the formulation of transformations between the fingertips and the base frame of the hand. Translations and rotations between coordinate frames
486
R. Palm et al. / Robotics and Autonomous Systems 57 (2009) 484–495
Fig. 1. Simulation of the hand.
Fig. 2. Grasp primitives.
are calculated by homogeneous transformations, with the help of which a point PC 4 = (x4 , y4 , z4 , 1)T in local homogeneous fingertip coordinates can be transformed into the base frame C0 by PC 0 = T1 · T2 · T3 · T4 · PC 4 . The transformation matrix Ti defines the transformation between the coordinate systems Ci and Ci−1 .
An important modeling aspect is the inverse problem which is crucial both for the simulation of grasps and the control of robotic hands, as a feedforward component of the control law [14]. Given the fingertip position vector x(t ), compute the corresponding joint angle vector q(t ). Let q(t ) = f−1 (x)
x˙ (t ) = J (q)˙q;
q˙ (t ) = J + (q)˙x
(2)
∂q ∂x
3.2. Modeling of inverse kinematics
x(t ) = f(q);
for the existing finger kinematics. Therefore we deal with the differential kinematics which makes the computation of the inverse much easier. From (1) one obtains the differential transformations
(1)
be the nonlinear direct and inverse transformation for a single finger where the inverse transformation is not necessarily unique
where J (q) = is the Jacobian and J + (q) is the pseudo inverse Jacobian. Assuming x(t ) or x˙ (t ) to be given from the task, i.e. from captured human demonstrations, the inverse kinematics in (2) remains to be computed. In order to avoid the time-consuming calculation of the inverse Jacobian at every time step, the inverse differential kinematics is approximated by a TS fuzzy model q˙ (t ) =
cx X
wi (x)Jinv,i (xi ) · x˙
(3)
i=1
where wi (x) ∈ [0, 1] is the degree of membership of the vector x to a cluster Cxi with the cluster center xi . Jinv,i (xi ) are the inverse
R. Palm et al. / Robotics and Autonomous Systems 57 (2009) 484–495
487
we collected data sequences for 15 different grasps. Each demonstration is repeated several times to collect enough instances of every particular grasp. From those data, one model for each individual grasp is developed. In order to make a model of a grasp primitive, we focus on features like – Fingertip positions at selected time points – Angles between the finger links. The latter is even more important if one wants to recognize a grasp from others, only by using the kinematic data of the hand. Although these features are included or can be recovered from the data sequence (see [18]), their handling and generalization with respect to other grasps can hardly be managed without a grasp model. Therefore, there is a need for a model of the recorded grasp sequences that reflects the behavior of the hand in time. Such a model is generated by fuzzy clustering and TS fuzzy modeling [21]. The basic idea is to consider the 3 fingertip coordinates as model outputs and the time instants as model inputs. The incorporation of the time in the modeling process has the following advantages
Fig. 3. Configuration of a single finger.
• Approximation of complex hand motions by local linear time Jacobians in the cluster centers xi . cx is the number of clusters. Due to the errors 1x = x(t ) − xm (t ) between the desired position x(t ) and the position xm computed by the forward kinematics a correction of the angles is calculated via the analytical forward kinematics xm (t ) = f(q(t )) of the finger. This changes (3) into q˙ (t ) =
cx X
wi (x)Jinv,i (xi ) · (˙x + K · (x(t ) − xm (t ))).
(4)
i =1
It has to be emphasized that the correction or optimization loop using the forward kinematics f(q(t )) is started at every new time instant and stops until either a lower bound k1xk < is reached or a given number of optimization steps is executed. The gain K has to be determined so that the optimization loop is stable. This TS-modeling is based on a clustering algorithm whose steps are described in the next section in more detail. The degree of membership wi (x) of an input vector x belonging to a cluster Cxi is defined by a bell-shape-like function
wi (x) =
1
1 cx P (x−xi )T Mxi (x−xi ) m˜ x −1 j =1
.
(5)
(x−xj )T Mxj (x−xj )
Mxi define the induced matrices of the input clusters Cxi , (i = ˜ x > 1 determines the fuzziness of an individual 1 . . . cx ), m cluster. The complexity of the on-line calculation of (4) is much lower than the complexity of (2) because (4) avoids the online calculation of numerous trigonometric functions. The time consuming clustering algorithm leading to the inverse Jacobians Jinv,i is computed off-line. 4. Modeling of finger trajectories by time-clustering and TSmodeling 4.1. Principle of time-clustering In this section, we describe a new approach to the modeling of human grasps, using the so-called time clustering. A common problem encountered in the research on programming-bydemonstration, is the need for segmentation of the data sequences prior to grasp recognition [3]. However, for the modeling phase, we avoid the segmentation problem by recording only one grasp at a time. Each record starts with the pre-grasp posture and ends up with the final grasp posture of the hand. By means of the data glove,
models
• Modeling of dynamic grasp primitives by a small set of parameters
• Easy training of the grasp model by incorporating new samples. The TS fuzzy model is constructed from captured data, as follows. Let the trajectory of a fingertip be described by the nonlinear function x(t ) = f(t )
(6)
where x(t ) ∈ R , f ∈ R , and t ∈ R . Eq. (6) can be linearized at selected time points ti to be determined 3
x(t ) = x(ti ) +
3
+
1f(t ) |t · (t − ti ) 1t i
(7)
which is a linear equation in t. x(t ) = Ai · t + di 1f(t ) | 1 t ti
(8) 1f(t ) | 1 t ti
where Ai = ∈ R3 and di = x(ti ) − · t i ∈ R3 . Using (8) as a local linear model one can express (6) in terms of an interpolation between several local linear models by applying TS fuzzy modeling [22] (see Fig. 4) x(t ) =
ct X
wi (t ) · (Ai · t + di )
(9)
i=1
where wi (t ) ∈ [0, 1] is the degree of membership of the time point t to a cluster Pct with the cluster center ti , ct is the number of clusters, and i= 1 wi (t ) = 1. The ‘‘bubbles’’ in Fig. 4 covering the trajectories represent the local fuzzy models. The stripes on the time coordinate represent the projections of the local fuzzy models onto the time line. Let t be the time and x = [x, y, z ]T the fingertip coordinates. Then the general clustering and modeling steps are
• Choose an appropriate number ct of local linear models (data clusters)
• Find ct cluster centers (ti , xi , yi , zi ), i = 1 . . . ct , in the product space of the data quadruples (t , x, y, z ) by Fuzzy-c-elliptotype clustering
• Find the corresponding fuzzy regions in the space of input data (t ) by projection of the clusters of the product space, first into so-called Gustafson–Kessel clusters (GK) and then onto the input space [23] • Calculate ct local linear (affine) models (9) using the GK clusters from step 2.
488
R. Palm et al. / Robotics and Autonomous Systems 57 (2009) 484–495
5. Segmentation and recognition of grasps In Section 4 we have shown how fuzzy models can be used for modeling and imitation of human grasp motions. Now, we will show that they can also serve for recognition and classification of grasps from recorded human demonstrations. If we just observe captured motions of a human hand while executing several grasp actions, it is difficult to identify the exact moment when a grasp starts and terminates. This leads to the following two aspects of the PbD-task:
• Segmentation of recorded data and recognition of those time Fig. 4. Time-clustering principle for a fingertip and its motion in (x, y).
sequences that correspond to models of grasp primitives.
• Recognition and classification of the identified grasps according The degree of membership wi (t ) of an input data point t to an input cluster Ct i is determined by
wi ( t ) =
1
1 ct P (t −ti )T Mt i (t −ti ) m˜ t −1 j =1
.
(10)
(t −tj )T Mt j (t −tj )
The projected cluster centers ti and the induced matrices Mt i ˜t > define the input clusters Ct i (i = 1 . . . ct ). The parameter m 1 determines the fuzziness of an individual cluster. A detailed description of this very effective clustering method can be found in [21]. In this way, for each of the 15 grasp primitives in Fig. 2 a TS-fuzzy model is generated. These so-called model grasps are used to identify demonstrated grasps from a test sequence of a given combination of grasps. 4.2. Training of time cluster models using new data A grasp model can be built in several ways – A single user trains the grasp model by repeating the same grasp n times – m users train the grasp model by repeating the same grasp n times. The 1st model is generated by the time sequences [(t1 , t2 , . . . , tN )1 . . . (t1 , t2 , . . . , tM )n ] and the fingertip sequences [(x1 , x2 , . . . , xN )1 . . . (x1 , x2 , . . . , xM )n ]. The 2nd model is generated by the time sequences [((t1 , t2 , . . . , tN )11 . . . (t1 , t2 , . . . , tM )1n ) . . . ((t1 , t2 , . . . , tN )m 1 . . . (t1 , t2 , . . . , tM )m )] and the fingertip sequences [(( x1 , x2 , . . . , xN )11 . . . n m 1 (x1 , x2 , . . . , xM )n ) . . . ((x1 , x2 , . . . , xN )1 . . . (x1 , x2 , . . . , xM )m n )] where m is the number of users in the training process, N , M are lengths of time sequences where N ≈ M. Once a particular grasp model has been generated it might be necessary to take new data into account. These data may originate from different human operators to cover several ways of performing the same grasp type. Let, for simplicity, the old model to be built by a time sequence [t1 , t2 , . . . , tN ] and a respective fingertip sequence [x1 , x2 , . . . , xN ]. The old model is then represented by the input cluster centers ti and the output cluster centers xi (i = 1 . . . c). It is also described by the parameters Ai and di of the local linear models. Let [t˜1 , t˜2 , . . . , t˜M ], [˜x1 , x˜ 2 , . . . , x˜ M ] be the new training data. A new model can be built by ‘‘chaining’’ the old and the new training data leading, for the time sequences, to [t1 , t2 , . . . , tN , t˜1 , t˜2 , . . . , t˜M ], and for the fingertip sequences to [x1 , x2 , . . . , xN , x˜ 1 , x˜ 2 , . . . , x˜ M ]. The result is a model that involves properties of the old model and the new data. If the old sequence of data is not available, a corresponding sequence can be generated by running the old model with the time instants [t1 , t2 , . . . , tN ] as inputs and the fingertips [x1 , x2 , . . . , xN ] as outputs.
to these models. Segmentation of data in a grasping task deals with the recognition of the starting points of grasps. Ekvall et al. derived these time points by observing the velocity profile of the human arm motion [4]. However, in dexterous manipulation or in assembly tasks, an object can be re-grasped several times without significant arm motions. Therefore Ikeuchi et al. derived the start of a grasp by observing the state of tactile sensors of the data glove [3]. Our segmentation method, however, performs a convolution of a test sequence of grasps against a model signal, plus a subsequent time clustering. This more general approach can also be applied to many other segmentation and recognition problems. Recognition of a grasp from a grasp sequence is based on a recognition model. This model is represented by a set of recognition rules using the model grasps mentioned in the previous section. Segmentation and generation of the recognition model is done by the following steps: A. Computation of distance norms between a sequence of grasps and the individual model grasps involved. B. Computation of extrema (minima and maxima) of distance norms and subsequent segmentation by time clustering. C. Formulation of a set of fuzzy rules reflecting the relationship between extrema of the distance norms and model grasps involved. D. Computation of a vector of similarity degrees between the individual model grasps and the grasps in a grasp combination. E. Identification of the order of grasps in the grasp combination by searching for maximum similarity degrees. Steps A and B can be considered as the segmentation steps. Steps C–E belong to the actual identification and recognition process. In the following paragraphs, these steps will be discussed in more detail. It has to be emphasized that segmentation and recognition of grasps are executed for selected subsets (combinations) of the set of c = 15 grasps. Examples of applications where an a priori known combination of grasps could occur are assembly and handling tasks involving different tools. Our method can recognize which of these tasks was demonstrated. The number mc of grasps in a combination can be chosen to be between 2 and 15, and the possible number of combinations is nc =
c mc
.
We chose mc = 5 and nc = 12 because this choice is able to show the potential of the method and covers a lot of practical cases. Yet, it should be noticed that the method presented can be applied to any size of a combination. However, a change of the combination in size and content always requires a change of the corresponding set of recognition rules. In the framework of PbD by a human operator, the convolution procedure affects both the segmentation and the model generation. After having demonstrated a sequence of grasps, the robot analyzes which grasp in which order should be executed. In contrast to imitation, this is an off-line process which may take 2–5 s computing time in total, depending on the complexity of the task and the computing power available.
R. Palm et al. / Robotics and Autonomous Systems 57 (2009) 484–495
489
Fig. 5. Overlap principle.
5.1. Distance norms
Fig. 6. Time slices.
Without loss of generality, let grasp2 , grasp5 , grasp7 , grasp10 , grasp14 be an a priori known combination of grasps taken from the list shown in Fig. 2. Then a sequence of grasps is generated using the time series of the corresponding model grasps obtained from time clustering described in Section 4. This sequence containing grasps only from the above combination can be arbitrarily long. Grasps can occur in an arbitrary order, including repetitions of grasps. Each grasp is separated from the other by a ’pause’sequence (open hand) to obtain a definite distinction of the grasps. After that, each of the model grasps i = 2, 5, 7, 10, 14 is shifted along the time series of the grasp sequence and compared with it by taking the norm2 kXc − Xmi k between the difference of the fingertip positions Xc of the sequence of grasps and the fingertip positions Xmi of a model graspi . Shifting and comparison take place in steps of the time cluster centers t1 . . . tct of the individual grasp where ct is the number of time clusters. The fingertip positions Xmi and Xc are defined as
5.2. Extrema in the distance norms and segmentation
Xmi = (xm (t1 ), . . . xm (tct ))Ti
zmini = (zmin,1i , . . . , zmin,li )T ;
and
tmini = (tmin,1i , . . . , tmin,li )
Xc = (xc (t˜1 ), . . . , xc (t˜ct ))T .
and
The vectors xm and xc denote the 3 fingertip positions (x, y, z )j j = 1 . . . 5 in the hand coordinate system for each of the 5 fingers. t˜1 . . . t˜ct are the actual time points in the time series Xc of the grasp sequence. Because of scaling reasons the norm kXc − Xmi k is divided by the norm kXmi k > 0 of the model graspi ni =
kXc − Xmi k . kXmi k
(11)
With this result for each graspi i = 2, 5, 7, 10, 14 a time sequence ni (t˜l ), l = 1 . . . ct , is generated. During the pauses the norms ni do not change significantly because of the low correlation between the time series of the model grasp and the pause. But once the model grasp starts to overlap a grasp in the grasp combination, the norms ni change rapidly and reach an extremum (maximum or minimum) as the model grasp and the grasp in the grasp combination reach their highest overlap in time. Fig. 5 shows this ‘‘convolution principle’’ by the simple example of grasp 2 and 7 in a grasp sequence. Using only models, grasps in a sequence lead to an almost perfect overlap, together with a minimum norm ni for one of the model grasps i. However, this cannot be expected for the same sequence of test grasps, including the same type of grasps but being performed by another human operator at another time. As a conclusion, for a particular grasp sequence, one obtains 5 norm sequences ni , the processing of which are presented in the following paragraphs.
Using a model graspi for comparison with a sequence of M test grasps, the norm ni forms individual patterns at M distinct time intervals of the norm sequence. Within each of these time intervals the norm sequence ni reaches an extremum, i.e. either a local minimum or a maximum. In order to find the local extrema in ni the total time interval Tni of ni is partitioned into l time slices, within which the search takes place (see Fig. 6). To be able to identify all relevant extrema, the lengths Tslice of the time slices have to be bounded by Tgrasp,min < Tslice < Tdist,min /2
(12)
where Tgrasp,min is the minimum time length of a grasp. Tdist,min is the minimum time distance between the end of a grasp and the starting point of a new grasp which is equal to the length of the pause. Tgrasp,min and Tdist,min are supposed to be known. This search yields two pairs of vectors (13)
T
zmaxi = (zmax,1i , . . . , zmax,li )T ;
(14)
tmaxi = (tmax,1i , . . . , tmax,li )T where l = dTni /Tslice e. The elements of zmini and zmaxi contain l absolute values of local minima and maxima of ni , respectively. tmini and tmaxi contain the corresponding l time stamps of the local minima and maxima. Usually there are more elements (extrema) included in (13) and (14) than grasps exist in the grasp sequence l ≥ M. The segmentation task is to find the time slices that include the beginnings of grasps. To deal with an unknown number of grasps solutions, different strategies are possible. A ‘soft’ solution requires a variable number of time clusters and a repetitive search for the most likely starting points of grasps. A mixed hardware–software solution is to utilize sensor information about the established contact between the fingers and the object to be grasped. In the following, we assume the number M of grasps in a grasp sequence to be known. A segmentation procedure finds those extrema that indicate the starting points of the grasps. The segmentation is done by time clustering, where the time vectors tmaxi and tmini are the model inputs, zmini and zmaxi are the model outputs. We expect the elements of zmaxi and zmini to form M clusters tseg = (tseg ,1 . . . tseg ,M )T . The result of the clustering procedure is a vector of M time cluster centers, pointing to the starting points of the grasps. For each time point tseg ,r there is a pair (zmin,ij , zmax,ij ). Index j denotes a graspj in the grasp sequence, executed at time tseg ,r . Index i = 2, 5, 7, 10, 14 denote the model graspsi . This finalizes the segmentation procedure.
490
R. Palm et al. / Robotics and Autonomous Systems 57 (2009) 484–495
5.3. Set of fuzzy rules The two sets of vectors zmini and zmaxi build ‘fingerprint patterns’ for each of the grasps in a specific grasp combination. On the basis of these patterns, a set of rules decide whether a special combination of minima and maxima belong to a certain grasp. Obviously, for a selected grasp, these patterns change with the change of a grasp combination. For example, the pattern for grasp2 in the grasp combination 2, 5, 7, 10, 14 differs significantly from the pattern in grasp combination 1, 2, 3, 4, 5 etc. This is taken into account by the formulation of an individual set of rules for each grasp combination. In order to recognize a model graspi from a specific grasp sequence, a set of 5 rules is formulated, one rule for each grasp in the combination. A general recognition rule for graspi to be identified from the combination reads
grasp is graspi .
(n2 is max2,5 ) AND (n5 is min5,5 ) AND (n7 is max7,5 ) AND (n10 is min10,5 ) AND (n14 is max14,5 ) IF
grasp is grasp5
(17)
Analogue to that we obtain for the minimum
wmin,ij = zminij /zmin,tot j .
(18)
Then the local extrema zmaxij and zminij are expressed by the total extrema zmax,tot j and zmin,tot j and the weights wmax,ij and wmin,ij zmaxij = wmax,ij · zmax,tot j zminij = wmin,ij · zmin,tot j
(19)
i = 2, 5, 7, 10, 14.
(15)
To identify grasp5 , for example, rule (15) can be read. ‘‘IF (norm n2 of model grasp2 is max2,5 ) . . . AND (norm n14 of model grasp14 is max14,5 ) THEN grasp is grasp5 ’’. The full rule to identify grasp5 reads
THEN
wmax,ij = zmaxij /zmax,tot j .
A similarity degree aj ∈ [0, 1] is then defined showing the similarity between the model graspi and the actual grasp in the test grasp sequence at time tseg ,r .
(nj is exji ) . . . AND (nk is exki ) IF
THEN
The same is done for each local minimum zminij where zmin,tot j = max(zminij ), i = 2, 5, 7, 10, 14. The ratio between zmaxij and zmax,tot j is expressed by the weight
(16)
j = 2 . . . ; k = 14 are the indexes of the grasps in the grasp combination. i is the index of graspi to be identified. exji indicate fuzzy sets of local extrema which can be either minima minji or maxima maxji . Extrema appear at time points t˜ = tseg ,r at which model graspj meets graspj in the grasp sequence with a maximum overlap. 5.4. Similarity degrees and identification of grasps
aj =
Y
mij ,
(20)
mij = exp(−|wex,ij · zex,tot j − zexij |) zexij represents either a minimum zminij or maximum zmaxij of the norms ni at time tseg ,r , respectively (see Fig. 7). The weights wex,ij , once derived from the training grasp sequence, serve as model parameters for the membership function represented by (20). The product operation in (20) represents the AND-operation in the rules (15) and (16). The exponential function in (20) is a membership function indicating the distance of a norm ni to a local extremum wex,ij · zex,tot j . The exponential function reaches its maximum at exactly that time point tseg ,r when graspj in the grasp sequence has its local extremum (see, e.g., Fig. 8). Since the time points tseg ,r for the occurrence of grasps in the grasp sequence are known from the segmentation, we can compute the similarity degrees aj between a model graspk and graspj at time tseg ,r . The similarity degrees to identify a graspj at tseg ,r are computed in the following way: 1. Apply the recognition rules (15) to graspj by calculating (20), use the corresponding weights wex,ik , i, k = 2, 5, 7, 10, 14. 2. For a sequence of model grasps one obtains for ak
Without loss of generality we focus again on the grasp combination (2, 5, 7, 10, 14). Let a graspj occur in the grasp sequence at time tseg ,r . The possibility of this grasp to be classified as graspj is indicated by a so-called similarity degree aj ∈ [0, 1] that is closely related to the antecedent part of (15). The similarity degrees aj are computed with the help of the pair (zminij , zmaxij ) which has been computed as a result of the segmentation. Remember that index j denotes the grasp in the grasp sequence to be identified. Index i denotes the model grasps being shifted along the grasp sequence. Recall that for a grasp sequence containing only grasps (2, 5, 7, 10, 14) five norm curves ni , i = 1 . . . 5 are computed. Recall also that tseg ,r is the time point in tseg for the local extrema of the norms ni at which graspj occurs. At time point tseg ,r the norm curves ni reach either a local minimum or a maximum depending on which model graspi overlaps with graspj in the grasp sequence. Fig. 7 shows an example where a grasp sequence is built by all the grasps of the grasp combination (2, 5, 7, 10, 14) without any repetition. If the grasp sequence is built by model grasps, a perfect overlap occurs when model graspj meets graspj in the grasp sequence at time tseg ,r . To calculate aj each local maximum zmaxij is normalized to the total maximum zmax,tot j = max(zmaxij ), i = 2, 5, 7, 10, 14.
i = 2, 5, 7, 10, 14
i
ak =
1 if k = j < 1 if k 6= j.
3. For a sequence of test grasps one usually obtains aj < 1 and ak < aj if k 6= j. After having identified the types of grasps, one problem arises because of the confidence of the similarity degrees obtained. Obviously, a reliable grasp recognition requires a significant maximum similarity degree. If such a significance is too low, then the confidence of the grasp recognition is also low. In our experiments, we preferred, among others, Grubb’s test statistic to support a decision about a grasp appearing at a certain time point tseg ,r G=
max(ak ) − a¯ k s(ak )
;
k = 2, 5, 7, 10, 14;
(21)
ak are the similarity degrees for k = 2, 5, 7, 10, 14 at a fixed time tseg ,r , a¯ k is the mean value, and s(ak ) > 0 is the standard deviation. In a competition of two grasps, the grasp with the higher G-value wins. It has to be emphasized that the method described in this section also allows for the recognition of a single grasp from a set of possible grasps, which is a special case of the method discussed before. The idea is to identify a single grasp, knowing
R. Palm et al. / Robotics and Autonomous Systems 57 (2009) 484–495
491
Fig. 7. Norms of a grasp sequence.
6. Experiments and simulations
Fig. 8. Grasp membership functions.
that this grasp belongs to a well defined set of grasps or a grasp combination. In this case segmentation is an easy task, leading to a single starting point of the grasp to be identified. Several robotic tasks may require the identification of a single grasp rather than the identification of a full grasp sequence, which justifies the consideration of this special case. In our experiments, both cases have been considered. In the following section, however, we will mainly present the segmentation, identification, and recognition of a full grasp sequence.
In this section, the experimental evaluation of the proposed approaches is presented. At present, modeling, segmentation, and recognition of grasps are performed off-line. According to Section 3 we tested 15 different grasp primitives, and recorded 10 samples for each grasp, to perform a reliable modeling. First, the grasp primitives (grasp models) have been trained by an expert user who is strongly involved in the planning and execution of the experiments. The classification system has been tested by both the expert and by another two non-expert users. However because of the fixed test conditions, a significant difference in the classification results between expert and non-expert users could not be found. A grasp starts with an open hand and is completed when the fingers establish contact with the object. In the experiments, 12 grasp combinations of 5 grasps each have been tested (see Table 1). This selection intends to combine grasps from the classes in Section 3 in a unique way and to test the recognition quality both within a class and between classes. 6.1. Time clustering and modeling The choice of the numbers of clusters, both for the fingertip models and for the inverse kinematics, depend on the quality of the resulting TS fuzzy models. On the basis of a performance analysis for each grasp and finger, 10 fingertip position models with 10 cluster centers have been generated from collected data. Furthermore, 3 inverse Jacobian models for each grasp primitive and finger with 3 cluster centers have been built, which are
492
R. Palm et al. / Robotics and Autonomous Systems 57 (2009) 484–495
Table 1 Grasp combinations. Combination
Grasp
Grasp
Grasp
Grasp
Grasp
Comb 1 Comb 2 Comb 3 Comb 4 Comb 5 Comb 6 Comb 7 Comb 8 Comb 9 Comb 10 Comb 11 Comb 12
1 6 11 1 14 2 15 3 5 1 5 2
2 7 12 4 10 5 2 9 6 13 7 5
3 8 13 8 7 7 14 13 3 12 8 8
4 9 14 11 5 10 4 12 9 8 12 12
5 10 15 15 2 14 6 7 11 10 13 15
Fig. 10. Middle finger, original: solid, model: dashed.
Fig. 9. Index finger, original: solid, model: dashed.
15 Jacobians, to be computed off-line. Since there are 3 angles (q1 , q2 , q3 ) and 3 fingertip coordinates (x, y, z ) for a single finger, the Jacobians and their inverses are 3 × 3 square matrices. The 3rd link of each finger (next to the fingertip) does not have a sensor in the data glove. Therefore the angle of this link gets a fixed value greater than zero so that neither ill-conditioned Jacobians nor their inverses can computationally occur. For the identification of inverse Jacobians, small random noise excitation is added to the angles to prevent ill-conditioned Jacobians while modeling from data. The motion of a grasp lasts 3.3 s in the average which adds up to 33 timesteps 1t = 0.1 s. The time clustering procedure results in the cluster centers ti = 2.04, 5.43, 8.87, 12.30, 15.75, 19.19, 22.65, 26.09, 29.53, 32.94 where the time labels are measured in steps of 1t = 0.1 s. The time cluster centers are then complemented by the corresponding cluster centers for the x, y, z coordinates of the fingertips. This equidistant spacing can be found for every individual grasp primitive as a result of the time clustering. Figs. 9–13 shows modeling results for grasp 10 (plane (1 CD-ROM)) for the x, y, and z coordinates for the index, middle, ring, and pinkie finger plus the thumb. These results show a good or even excellent modeling quality.
Fig. 11. Ring finger, original: solid, model: dashed.
6.2. Grasp segmentation and recognition We tested 12 combinations of the 15 different grasps mentioned above. As an example, grasp combination 6 with the grasps 2, 5, 7, 10, 14 will be discussed. For the training phase, a sequence of the model grasps 2, 5, 7, 10, 14 is generated. Time clustering leads to a segmentation result with the starting points of the grasps at: grasp 2: t2 = 16.4, grasp 5: t5 = 39.9, grasp 7: t7 = 66.1, grasp 10: t10 = 88.6, grasp 14: t14 = 106.7.
Fig. 12. Pinkie finger, original: solid, model: dashed.
After the segmentation the norms ni = ni (t˜) are computed using the same model grasps according to (11). Based on the resulting curves in Fig. 7 the weights wex,ij for the rule model (16) and the similarity degrees (20) are determined.
R. Palm et al. / Robotics and Autonomous Systems 57 (2009) 484–495
493
Table 3 Grasp combination 6, test results. a/t
t2
t5
t7
t10
t14
a2 a5 a7 a10 a14
0.53 0.33 0.63 0.37 0.56
0.59 0.76 0.64 0.60 0.44
0.47 0.30 0.83 0.34 0.78
0.74 0.53 0.74 0.79 0.53
0.41 0.26 0.81 0.38 0.82
Table 4 Recognition rates.
Fig. 13. Thumb finger, original: solid, model: dashed.
Group
Grasp
Percentage (%)
≥75%
4. Hammer 5. Screwdriver 8.Precision. grasp sphere 9. Precision grasp cube 10. Small plane 11. Big plane 13. Fingertip big ball 15. Penholder grip
100 93 80 100 100 88 75 75
<75%, ≥ 50%
1. Cylinder 2. Big bottle 3. Small bottle 6. Small ball 7. Big ball 12. Fingertip small ball 14. Fingertip can
<50%
55 60 66 55 16 33 33
grasp. Although in the training situation, all grasps can be reliably recognized, some of the grasps may be more difficult to identify than others. Grasps 2 and 14, for example, show rather high numbers at other time points too, meaning that the recognition of these grasps may be less reliable. A test sample of the same grasp sequence confirms this assumption; there are good results for the grasps 5, 7, 10, and 14 but also a different (wrong) result for grasp 2 (see Fig. 15 and Table 3). Several tests with other grasp combinations have shown that some grasps can be more reliably recognized than others. Grasps with distinct maxima and minima in their ni patterns can be recognized better than those grasps which do not have this feature. Reliable grasps are also robust against variations in the time span of an unknown test grasp, compared to the time span of the respective model grasp. Our results show that the methods can handle up to 20% temporal difference. Table 4 shows the recognition rates of grasps from all grasp combinations taken into account. We divided the grasps into 3 groups:
Fig. 14. Grasp combination 6, training sequences.
1. grasps with a recognition rate ≥ 75% 2. grasps with a recognition rate < 75%& ≥ 50% 3. grasps with a recognition rate < 50%. Fig. 15. Grasp combination 6, test sequences. Table 2 Grasp combination 6, training results. a/t
t2
t5
t7
t10
t14
a2 a5 a7 a10 a14
0.99 0.70 0.78 0.63 0.64
0.70 1.00 0.66 0.67 0.49
0.53 0.39 1.00 0.44 0.95
0.63 0.67 0.67 1.00 0.52
0.53 0.39 0.90 0.44 0.99
The training results in Fig. 14 and Table 2 show that, at the starting points of the particular grasps, the type of the grasps 2, 5, 7, 10, and 14 can be reliably recognized. The rows a2, . . . , a14 denote the maximum similarity degrees at the starting points of a
Group 1 is the largest group, followed by groups 2 and 3. Here, we should point out the similarities between grasps 1 and 3, grasps 4, 5 and 15, grasps 7 and 11, and grasps 8 and 9 which gives space for misinterpretations and explains the low percentages for some of the grasps. The analysis confirms the obvious fact that distinct grasps can be discriminated quite well from each other, while the discrimination between similar grasps is difficult. Therefore, the merging of similar grasps and building of general classes can improve the recognition process significantly. For the general classes 1. 2. 3. 4.
cylindrical grasp (grasps 1, 2, 3, 14) power grasp (grasps 4, 5) spherical grasp (grasps 6, 7, 12, 13) extension grasp (grasps 10, 11)
494
R. Palm et al. / Robotics and Autonomous Systems 57 (2009) 484–495
Table 5 Recognition rates between general classes. Grasp general class
1
1. Cylinder 2. Big bottle 3. Small bottle 4. Hammer 5. Screwdriver 6. Small ball 7. Big ball 8. Precision. grasp sphere 9. Precision grasp cube 10. Small plane 11. Big plane 12. Fingertip small ball 13. Fingertip big ball 14. Fingertip can 15. Penholder grip
100% 71% 86% 14%
2
3
4
5
6
29% 14% 100% 58% 28%
14% 44% 100%
14% 28% 100% 100%
14% 100% 100%
100% 86%
100%
5. precision grasp, nippers pinch (grasps 8, 9) 6. penholder grasp (writing grip) (grasp 15) we obtain the recognition rates shown in Table 5. This table can be read as ‘‘grasp i can be identified with a recognition rate of x% to be in general class j’’. The bold numbers show the maxima in each row. The respective columns show that the bold numbers form exactly the general classes, which means that each general class can be reliably identified. 7. Summary and conclusions In this paper, a new fuzzy-logic based method to modeling, classification and recognition of human grasps is presented. The goal is to develop an easy way of programming-by-demonstration of grasps for artificial hands. Grasp primitives are captured by a data glove and modeled by TS-fuzzy models. Fuzzy clustering and modeling of time and space data is applied to the modeling of the fingertip trajectories of grasp primitives. Given a test sequence of grasps, the method allows us to identify the segments of the occurrence of grasps, and to recognize the grasp types. Each model grasp is correlated with the test grasp sequence. By means of a hybrid fuzzy model, the grasp sequence shown by the operator can be identified. With this method, both the segmentation of grasps and their recognition is possible. The method also allows the recognition of a single grasp from a set of possible grasps. The future work will concentrate on the improvement of the PbD process and the development of the recognition and classification of operator movements in a robotic environment, by using more sensor information about the robot workspace and objects to be handled, as well as their affordances. The data glove will be equipped by tactile sensors to improve and accelerate the segmentation process. The method will be simplified to make the recognition faster, leading, in turn, to a faster programming process for the robot. Furthermore the method will be extended and will be used for the programming of complex robot skills in the presence of external sensors. References [1] S.B. Kang, K. Ikeuchi, Towards automatic robot instruction form perception — mapping human grasps to manipulator grasps, IEEE Transactions on Robotics and Automation 13 (1) (1997). [2] R. Zoellner, O. Rogalla, R. Dillmann, J.M. Zoellner, Dynamic grasp recognition within the framework of programming by demonstration, in: Proceedings Robot and Human Interactive Communication, 10th IEEE International Workshop, Bordeaux, Paris, France, September 18–21, IEEE, 2001. [3] K. Bernardin, K. Ogawara, K. Ikeuchi, R. Dillman, A sensor fusion approach for recognizing continuous human grasp sequences using hidden Markov models, IEEE Transactions on Robotics 21 (1) (2005).
14%
86%
[4] S. Ekvall, D. Kragic, Grasp recognition for programming by demonstration, in: Proceedings of International Conference on Robotics and Automation, ICRA 2005, Barcelona, Spain, 2005. [5] D. Martinez, D. Kragic, Modeling and recognition of actions through motor primitives, in: Proceedings ICRA 2008, IEEE International Conference on Robots and Automation, IEEE, Pasadena, CA, USA, 2008. [6] Ch. Li, L. Khan, B. Prabhaharan, Real-time classification of variable length multi-attribute motions, Knowledge and Information Systems (2005) doi:10.1007/s10115-005-0223-8. [7] J. Aleotti, S. Caselli, Grasp recognition in virtual reality for robot pregrasp planning by demonstration, in: ICRA 2006—International Conference on Robotics and Automation, IEEE, Orlando, FL, USA, 2006. [8] I.S. Vicente, V. Kyrki, D. Kragic, M. Larson, Action recognition and understanding through motor primitives, Advanced Robotics 21 (15) (2007) 1687–1702. [9] S. Calinon, A. Billard, Incremental learning of gestures by imitation in a humanoid robot, in: Proceedings of the ACM/IEEE International Conference on Human–Robot Interaction (HRI), 2007, IEEE, ACM, Arlington, Virginia, USA, 2007. [10] O. Bimber, Continuous 6D gesture recognition: A fuzzy-logic approach, in: Proceedings of 7-th International Conference in Central Europe on Computer Graphics, Visualization and Interactive Digital Media, WSCG’99, 1999, pp. 24–30. [11] S. Ferguson, G. Dunlop, Grasp recognition from myoelectric signals, in: Proceedings Australasian Conference Robotics and Automation, 2002, pp. 78–83. [12] D.J. Fleet, M.J. Black, O. Nestares, Bayesian inference of visual motion boundaries, in: Exploring Artificial Intelligence in the New Millenium, Morgan, Kaufmann, 2003, pp. 139–173. [13] V. Kruger, D. Kragic, A. Ude, C. Geib, Action recognition and understanding through motor primitives, Advanced Robotics 21 (13) (2007) 1473–1501. [14] R. Palm, B. Iliev, Learning of grasp behaviors for an artificial hand by time clustering and Takagi-Sugeno modeling, in: Proceedings FUZZ-IEEE 2006 IEEE International Conference on Fuzzy Systems, Vancouver, BC, Canada, July 16–21, IEEE, 2006. [15] R. Palm, B. Iliev, Segmentation and recognition of human grasps for programming-by-demonstration using time clustering and Takagi-Sugeno modeling, in: Proceedings FUZZ-IEEE 2007 - IEEE International Conference on Fuzzy Systems, London, UK, July 23–26, IEEE, 2007. [16] R. Palm, B. Iliev, Grasp recognition by time clustering, fuzzy modeling, and Hidden Markov Models (HMM) — a comparative study, in: Proceedings FUZZIEEE 2008 - IEEE International Conference on Fuzzy Systems, Hong Kong, July 1–5, IEEE, 2008. [17] A. Skoglund, B. Iliev, B. Kadmiry, R. Palm, Programming by demonstration of pick-and-place tasks for industrial manipulators using task primitives, in: Proceedings Computational Intelligence in Robotics and Automation, CIRA 2007, Jacksonville, FL, USA, June 20–23, IEEE, 2007. [18] H.H. Asada, J.R. Fortier, Task recognition and human–machine coordination through the use of an instrument-glove. Progress report No. 2–5, March 2000, pp. 1–39. [19] M. Cutkosky, On grasp choice, grasp models, and the design of hands for manufacturing tasks, IEEE Transactions on Robotics and Automation 5 (3) (1989). [20] Thea Iberall, Human prehension and dexterous robot hands, The International Journal of Robotics Research 16 (1997) 285–299. [21] R. Palm, Ch. Stutz, Open loop dynamic trajectory generator for a fuzzy gain scheduler, Engineering Applications of Artificial Intelligence 16 (2003) 213–225.
R. Palm et al. / Robotics and Autonomous Systems 57 (2009) 484–495 [22] T. Takagi, M. Sugeno, Fuzzy identification of systems and its applications to modeling and control, IEEE Transactions on Systems, Man, and Cybernetics 15 (1) (1985) 116–132. [23] D.E. Gustafson, W.C. Kessel, Fuzzy clustering with a fuzzy covariance matrix, in: Proceedings of the 1979 IEEE CDC, 1979, pp. 761–766.
Rainer Palm has been Adjunct Professor at the Department of Technology (AASS), University Örebro since 2004. Until 2004 he was a Principal Research Scientist in the Neural Computation Department of the Siemens Corporate Research Technology in Munich, Germany, working both in research projects and industrial applications of Automatic Control, Fuzzy Control, Modeling of Dynamic Systems, and Distributed Systems. He received his Dipl. Ing. degree from the Technical University of Dresden, Germany, in 1975. Since 1982 he worked in robotic research, in particular in the field of sensor- guided robots. He received the Dr. Ing. degree from the Humboldt-University Berlin, Germany, in 1981, and the Dr. sc. techn. degree from the Academy of Sciences, Germany, in 1989. Since 1991 he held a position at the fuzzy logic task force group at Siemens AG. He is author /co-author, editor/co-editor of 15 books or chapters in books, author/ coauthor of 19 articles in journals, and author /co-author of 35 articles in Conference proceedings. He is also author / co-author of 14 patents. His research interest include automatic control, fuzzy control, and robotics. Spokesman of the fuzzy group 1994-2002. Associate editor of IEEE Transactions on Fuzzy Systems 1994–2004.
495
Boyko Iliev is a senior researcher at the Center for Applied Autonomous Sensors Systems (AASS), Dept. of Technology, Örebro University. He received his Ph.D. degree from Örebro University in Control Theory (robot control) in 2004 and, since then, his Postdoc at AASS. Before coming to Örebro, he received a diploma in Control Systems Engineering from the Technical University, Sofia, Bulgaria in 1997. His research interest includes robotic grasping and dexterous manipulation, biologically inspired systems, Control Theory and its applications in robotics. He is author /coauthor of numerous conference papers and Journal articles on robotics and control theory.
Bourhane Kadmiry is a Postdoc at the Dept. of Technology, Örebro University. He received his Ph.D. in Computer Sc. at Linköping University-Sweden and a Ph.D. in Robotics and AI at (LAAS) Toulouse University-France. He is also a holder of a Master of Sc. in Sensor Technology, Control and AI and a Master of Computer Engineering, both obtained at Rouen Technical University-France. His research interests include autonomous helicopter navigation control, visual servoing control for autonomous helicopter, fuzzy control, and control for an artificial (prosthetic) hand. He has authored 2 journal articles and 6 full refereed conference papers, and is a regular reviewer for international conferences. He is also project manager for Airbus aeronautic programs within Soerman Co.-Sweden.
Contents lists available at ScienceDirect
Robotics and Autonomous Systems journal homepage: www.elsevier.com/locate/robot
Recognition of human grasps by time-clustering and fuzzy modeling Rainer Palm ∗ , Boyko Iliev, Bourhane Kadmiry AASS, Department of Technology, Orebro University, SE-70182 Orebro, Sweden
article
info
Article history: Received 2 August 2008 Received in revised form 17 October 2008 Accepted 23 October 2008 Available online 8 November 2008 Keywords: Grasp recognition Programming-by-demonstration Fuzzy clustering Fuzzy modeling
a b s t r a c t In this paper, we address the problem of recognition of human grasps for five-fingered robotic hands and industrial robots in the context of programming-by-demonstration. The robot is instructed by a human operator wearing a data glove capturing the hand poses. For a number of human grasps, the corresponding fingertip trajectories are modeled in time and space by fuzzy clustering and Takagi–Sugeno (TS) modeling. This so-called time-clustering leads to grasp models using time as an input parameter and fingertip positions as outputs. For a sequence of grasps, the control system of the robot hand identifies the grasp segments, classifies the grasps and generates the sequence of grasps shown before. For this purpose, each grasp is correlated with a training sequence. By means of a hybrid fuzzy model, the demonstrated grasp sequence can be reconstructed. © 2008 Elsevier B.V. All rights reserved.
1. Introduction Applications of multi-fingered robotic hands have so far been relatively few, although the field has attracted significant research efforts. One of the reasons is that they are fairly difficult to program, due to the inherently high dimensionality of the grasping and manipulation tasks. One approach to solve the problem is to program the robotic hand task directly by human demonstrations. That is, the operator performs a task while the robot ‘‘observes’’ the demonstrated actions with the help of motion capture devices or video cameras. The robot has to recognize the actions performed by the demonstrator and replicate them on its own. This approach is called Programming by Demonstration (PbD) and is used in complex robotics applications such as grasping and manipulation. Our interest also lies in developing methods for the control of anthropomorphic robotic (prosthetic) hands where the manipulation strategies are learned from human demonstrations. The idea is to exploit the structural similarity between the artificial and the human hand in order to achieve similar functionality. This will, on the one hand, allow for a more intuitive teaching of complex robotic hands. On the other hand, this approach can be applied for the control of prosthetic hands, where it is desired to mimic the motion of the human hand as closely as possible. Different techniques and motion capture setups have been applied in PbD. For example, Kang et al. [1] describe a system which observes, recognizes and maps human grasps to a robot
∗
Corresponding author. Tel.: +49 892605263; fax: +49 892605263. E-mail addresses: [email protected] (R. Palm), [email protected] (B. Iliev), [email protected] (B. Kadmiry). 0921-8890/$ – see front matter © 2008 Elsevier B.V. All rights reserved. doi:10.1016/j.robot.2008.10.012
manipulator. Human demonstration is captured by a stereo vision system and a data glove. Zoellner et al. [2] use a data glove with integrated tactile sensors, where the classification method is based on support vector machines (SVM). In a similar setup, Ikeuchi et al. [3] apply Hidden Markov Models (HMMs) to segment and recognize grasp sequences. Ekvall and Kragic [4] also address the PbD-problem, using the arm trajectory and hand pose as additional features for grasp classification. Martinez and Kragic [5] use combinations of SVM’s, conditional random fields (CRF’s) and HMM’s for modeling and recognition of object manipulation actions in the framework of robot learning. Li et al. [6] use the singular value decomposition (SVD) to generate feature vectors for human grasps and support vector machines (SVM) which are applied to the classification problem. Aleotti and Caselli [7] describe a virtual reality based PbD-system for grasp recognition, where final grasp postures are modeled based on the finger joint angles. A combination of HMM’s and SVM’s is presented by Vicente et al. [8] for learning manipulation actions. Calinon and Billard [9] use Gaussian Mixture Models (GMM) for robot learning of human gestures. A fuzzy logic approach for gesture recognition was published by Bimber [10]. The method is applied to 6 d.o.f. trajectories of a human arm, but cannot cope with our multi-dimensional problem. Ferguson et al. developed a pattern recognition system for the control of limb prostheses [11]. In this approach, a fuzzy system classifies myoelectric signals in the frequency domain. Because of this very specific application the method cannot be generalized to grasp recognition for PbD. To the knowledge of the authors, there are no further fuzzy logic approaches reported both for modeling and classification that fit the dynamical grasp recognition problem. Fleet et al. [12] uses Bayesian filters in the framework of visual motion analysis to identify image motions in the vicinity of occlusion boundaries.
R. Palm et al. / Robotics and Autonomous Systems 57 (2009) 484–495
An excellent overview about the ‘‘different approaches taken todate within the computer vision, robotics and artificial intelligence communities for the representation, recognition, synthesis and understanding of action’’ is given by Kruger et al. [13]. Most of the methods are of probabilistic nature in contrast to fuzzy modeling which is a possibilistic and rule-based approach. Despite the advanced state of the art, the cited methods on dynamic classification use discrete states or observations (hand poses) and do not consider the continuous evolution of a grasp in time and space. This appears to be a disadvantage, because neither the estimation of the occurrence of a specific grasp nor the problem of segmentation can be solved in a general way. The method presented in this paper tries to overcome some of these drawbacks by modeling fingertip trajectories in time and space. To solve a ‘fuzzy’ task like grasp recognition, it turns out that fuzzy modeling is a suitable approach. The main focus of this method is programming by demonstration and recognition of human grasps using fuzzy clustering and Takagi–Sugeno (TS) fuzzy modeling. Two types of models are considered:
• models of the fingertip trajectories. • models for segmentation and recognition. The first type of models incorporate the time variable of a grasp motion in the modeling process (time-clustering), i.e. it interprets the time as a model input, see Palm and Iliev [14]. These models can be used to replay, scale and recognize demonstrated grasp primitives in a most flexible way. The second type of model deals with the recognition of grasps from a grasp sequence [15]. The recognition model consists of a set of fuzzy expert rules to identify a grasp from a combination of grasps using membership functions of exponential type. This model is called a hybrid fuzzy model because it combines a fuzzy rule structure and a crisp decision making procedure. One of the main advantages of our approach is that grasp recognition, hand control and execution monitoring can be implemented within the same modeling framework. In our method, the grasps are modeled as continuous fingertip motions in time and space. The other methods like HMM use discrete observations and states. Therefore, with our method, the recognition task can be improved by refining the granulation of the time clusters. For the discrete methods like HMM, a refinement of the model would require the introduction of additional observations (hand poses) leading to a significant increase of possible states and a corresponding computing effort. In a direct comparison between the fuzzy approach and the HMM method, we could show that the fuzzy approach gives better results with respect to the recognition rates [16]. This paper is organized as follows: Section 2 describes the general principles of the PbD-methodology, and the experimental platform. Section 3 deals with the simulation of the grasp processes and the geometrical modeling and kinematical representation of the human hand. Section 4 discusses the learning of grasps by time-clustering and TS fuzzy modeling. Section 5 describes a new method for segmentation and recognition of grasp sequences using a hybrid fuzzy approach. Section 6 presents experimental results for time clustering and TS fuzzy modeling for the segmentation and recognition of selected grasps. Section 7 draws some conclusions and establishes directions for future work. 2. General (PbD)
principles
for
programming-by-demonstration
The long-term goal of the project ‘‘Dexterous Manipulation’’ at the Center for Applied Autonomous Sensor Systems (AASS) leads to a control methodology for grasping and manipulation in unstructured environments. Programming by demonstration (PbD) involves interpretation of human demonstrations and
485
their automatic conversion into program code that can be executed on a particular robot system. The PbD-methodology for robotic grasping requires three main tasks: segmentation of grasp sequences, grasp recognition, and generation of robot code [17]. The first task is to partition the data record into a sequence of episodes, where each one contains a single grasp. The second task is to recognize the grasp performed in each episode. Finally the demonstrated task is converted into a task level program which can be executed by an artificial robot hand or a prosthetic hand. This program activates grasp primitives corresponding to human grasps and adapted to the particular robot system. In the following, instead of going into detail with this high-level program, the problems of modeling, segmentation and recognition of human grasp sequences will be addressed. The experimental platform consists of a hand motion capturing device (data glove) and a hand simulation environment. The motions of a human demonstrator are recorded by a data glove (CyberGlove) which measures 18 joint angles in the hand and the wrist [18]. Most data gloves available on the market are of the same size (‘‘one fits many’’). Smaller or bigger sizes of hands have small influence on the fingertip modeling results, as long as they do not deviate too much from each other. Since humans mostly use a limited number of grasp types, the recognition process can be restricted to a certain grasp taxonomy, corresponding to those developed by Cutkosky [19] and Iberall [20], respectively. 3. Simulation of grasp primitives For the purpose of PbD we need a model of the human hand which allows the simulation of demonstrated grasps. In order to test the grasp primitives, a hand simulation was developed, with the help of which one can mimic the hand poses recorded by the data glove. This hand model allows one to compute the trajectories of both the finger angles and the fingertips. Since the size of the object to be grasped determines the starting and end points of the fingertips to a great extent, we used fingertip trajectories instead of finger angles for modeling. Moreover, we can simulate recorded demonstrations of grasps plus corresponding interactions with objects (see Fig. 2) and compare them with the training sequences of grasp primitives. Inspired by the grasp taxonomy of Iberall, 15 different grasps have been tested (see Fig. 2). These grasps are special cases of the following general classes [20]: 1. 2. 3. 4. 5. 6.
cylindrical grasp (grasps 1, 2, 3, 14) power grasp (grasps 4, 5) spherical grasp (grasps 6, 7, 12, 13) extension grasp (grasps 10, 11) precision grasp, nippers pinch (grasps 8, 9) penholder grasp (writing grip) (grasp 15).
The advantage of this selection is that the quality of a grasp recognition, both between classes and within a class, can be analyzed (see Section 6: Experiments and Simulations). 3.1. Geometrical modeling In order to study grasp primitives and to develop specific grasp models, a geometrical simulation of the hand is required (see Fig. 1). The hand model consists of 5 fingers which are linked to the wrist in such a way that the poses of a human operator can be displayed in a realistic way. The kinematic relations can be studied by means of the example of a single finger (see Fig. 3). Each finger is modeled with 3 links and 3 joints moving like a small planar robot. This turned out to be sufficient for the simulation of the grasp primitives in Fig. 2. The calculation of fingertip trajectories requires the formulation of transformations between the fingertips and the base frame of the hand. Translations and rotations between coordinate frames
486
R. Palm et al. / Robotics and Autonomous Systems 57 (2009) 484–495
Fig. 1. Simulation of the hand.
Fig. 2. Grasp primitives.
are calculated by homogeneous transformations, with the help of which a point PC 4 = (x4 , y4 , z4 , 1)T in local homogeneous fingertip coordinates can be transformed into the base frame C0 by PC 0 = T1 · T2 · T3 · T4 · PC 4 . The transformation matrix Ti defines the transformation between the coordinate systems Ci and Ci−1 .
An important modeling aspect is the inverse problem which is crucial both for the simulation of grasps and the control of robotic hands, as a feedforward component of the control law [14]. Given the fingertip position vector x(t ), compute the corresponding joint angle vector q(t ). Let q(t ) = f−1 (x)
x˙ (t ) = J (q)˙q;
q˙ (t ) = J + (q)˙x
(2)
∂q ∂x
3.2. Modeling of inverse kinematics
x(t ) = f(q);
for the existing finger kinematics. Therefore we deal with the differential kinematics which makes the computation of the inverse much easier. From (1) one obtains the differential transformations
(1)
be the nonlinear direct and inverse transformation for a single finger where the inverse transformation is not necessarily unique
where J (q) = is the Jacobian and J + (q) is the pseudo inverse Jacobian. Assuming x(t ) or x˙ (t ) to be given from the task, i.e. from captured human demonstrations, the inverse kinematics in (2) remains to be computed. In order to avoid the time-consuming calculation of the inverse Jacobian at every time step, the inverse differential kinematics is approximated by a TS fuzzy model q˙ (t ) =
cx X
wi (x)Jinv,i (xi ) · x˙
(3)
i=1
where wi (x) ∈ [0, 1] is the degree of membership of the vector x to a cluster Cxi with the cluster center xi . Jinv,i (xi ) are the inverse
R. Palm et al. / Robotics and Autonomous Systems 57 (2009) 484–495
487
we collected data sequences for 15 different grasps. Each demonstration is repeated several times to collect enough instances of every particular grasp. From those data, one model for each individual grasp is developed. In order to make a model of a grasp primitive, we focus on features like – Fingertip positions at selected time points – Angles between the finger links. The latter is even more important if one wants to recognize a grasp from others, only by using the kinematic data of the hand. Although these features are included or can be recovered from the data sequence (see [18]), their handling and generalization with respect to other grasps can hardly be managed without a grasp model. Therefore, there is a need for a model of the recorded grasp sequences that reflects the behavior of the hand in time. Such a model is generated by fuzzy clustering and TS fuzzy modeling [21]. The basic idea is to consider the 3 fingertip coordinates as model outputs and the time instants as model inputs. The incorporation of the time in the modeling process has the following advantages
Fig. 3. Configuration of a single finger.
• Approximation of complex hand motions by local linear time Jacobians in the cluster centers xi . cx is the number of clusters. Due to the errors 1x = x(t ) − xm (t ) between the desired position x(t ) and the position xm computed by the forward kinematics a correction of the angles is calculated via the analytical forward kinematics xm (t ) = f(q(t )) of the finger. This changes (3) into q˙ (t ) =
cx X
wi (x)Jinv,i (xi ) · (˙x + K · (x(t ) − xm (t ))).
(4)
i =1
It has to be emphasized that the correction or optimization loop using the forward kinematics f(q(t )) is started at every new time instant and stops until either a lower bound k1xk < is reached or a given number of optimization steps is executed. The gain K has to be determined so that the optimization loop is stable. This TS-modeling is based on a clustering algorithm whose steps are described in the next section in more detail. The degree of membership wi (x) of an input vector x belonging to a cluster Cxi is defined by a bell-shape-like function
wi (x) =
1
1 cx P (x−xi )T Mxi (x−xi ) m˜ x −1 j =1
.
(5)
(x−xj )T Mxj (x−xj )
Mxi define the induced matrices of the input clusters Cxi , (i = ˜ x > 1 determines the fuzziness of an individual 1 . . . cx ), m cluster. The complexity of the on-line calculation of (4) is much lower than the complexity of (2) because (4) avoids the online calculation of numerous trigonometric functions. The time consuming clustering algorithm leading to the inverse Jacobians Jinv,i is computed off-line. 4. Modeling of finger trajectories by time-clustering and TSmodeling 4.1. Principle of time-clustering In this section, we describe a new approach to the modeling of human grasps, using the so-called time clustering. A common problem encountered in the research on programming-bydemonstration, is the need for segmentation of the data sequences prior to grasp recognition [3]. However, for the modeling phase, we avoid the segmentation problem by recording only one grasp at a time. Each record starts with the pre-grasp posture and ends up with the final grasp posture of the hand. By means of the data glove,
models
• Modeling of dynamic grasp primitives by a small set of parameters
• Easy training of the grasp model by incorporating new samples. The TS fuzzy model is constructed from captured data, as follows. Let the trajectory of a fingertip be described by the nonlinear function x(t ) = f(t )
(6)
where x(t ) ∈ R , f ∈ R , and t ∈ R . Eq. (6) can be linearized at selected time points ti to be determined 3
x(t ) = x(ti ) +
3
+
1f(t ) |t · (t − ti ) 1t i
(7)
which is a linear equation in t. x(t ) = Ai · t + di 1f(t ) | 1 t ti
(8) 1f(t ) | 1 t ti
where Ai = ∈ R3 and di = x(ti ) − · t i ∈ R3 . Using (8) as a local linear model one can express (6) in terms of an interpolation between several local linear models by applying TS fuzzy modeling [22] (see Fig. 4) x(t ) =
ct X
wi (t ) · (Ai · t + di )
(9)
i=1
where wi (t ) ∈ [0, 1] is the degree of membership of the time point t to a cluster Pct with the cluster center ti , ct is the number of clusters, and i= 1 wi (t ) = 1. The ‘‘bubbles’’ in Fig. 4 covering the trajectories represent the local fuzzy models. The stripes on the time coordinate represent the projections of the local fuzzy models onto the time line. Let t be the time and x = [x, y, z ]T the fingertip coordinates. Then the general clustering and modeling steps are
• Choose an appropriate number ct of local linear models (data clusters)
• Find ct cluster centers (ti , xi , yi , zi ), i = 1 . . . ct , in the product space of the data quadruples (t , x, y, z ) by Fuzzy-c-elliptotype clustering
• Find the corresponding fuzzy regions in the space of input data (t ) by projection of the clusters of the product space, first into so-called Gustafson–Kessel clusters (GK) and then onto the input space [23] • Calculate ct local linear (affine) models (9) using the GK clusters from step 2.
488
R. Palm et al. / Robotics and Autonomous Systems 57 (2009) 484–495
5. Segmentation and recognition of grasps In Section 4 we have shown how fuzzy models can be used for modeling and imitation of human grasp motions. Now, we will show that they can also serve for recognition and classification of grasps from recorded human demonstrations. If we just observe captured motions of a human hand while executing several grasp actions, it is difficult to identify the exact moment when a grasp starts and terminates. This leads to the following two aspects of the PbD-task:
• Segmentation of recorded data and recognition of those time Fig. 4. Time-clustering principle for a fingertip and its motion in (x, y).
sequences that correspond to models of grasp primitives.
• Recognition and classification of the identified grasps according The degree of membership wi (t ) of an input data point t to an input cluster Ct i is determined by
wi ( t ) =
1
1 ct P (t −ti )T Mt i (t −ti ) m˜ t −1 j =1
.
(10)
(t −tj )T Mt j (t −tj )
The projected cluster centers ti and the induced matrices Mt i ˜t > define the input clusters Ct i (i = 1 . . . ct ). The parameter m 1 determines the fuzziness of an individual cluster. A detailed description of this very effective clustering method can be found in [21]. In this way, for each of the 15 grasp primitives in Fig. 2 a TS-fuzzy model is generated. These so-called model grasps are used to identify demonstrated grasps from a test sequence of a given combination of grasps. 4.2. Training of time cluster models using new data A grasp model can be built in several ways – A single user trains the grasp model by repeating the same grasp n times – m users train the grasp model by repeating the same grasp n times. The 1st model is generated by the time sequences [(t1 , t2 , . . . , tN )1 . . . (t1 , t2 , . . . , tM )n ] and the fingertip sequences [(x1 , x2 , . . . , xN )1 . . . (x1 , x2 , . . . , xM )n ]. The 2nd model is generated by the time sequences [((t1 , t2 , . . . , tN )11 . . . (t1 , t2 , . . . , tM )1n ) . . . ((t1 , t2 , . . . , tN )m 1 . . . (t1 , t2 , . . . , tM )m )] and the fingertip sequences [(( x1 , x2 , . . . , xN )11 . . . n m 1 (x1 , x2 , . . . , xM )n ) . . . ((x1 , x2 , . . . , xN )1 . . . (x1 , x2 , . . . , xM )m n )] where m is the number of users in the training process, N , M are lengths of time sequences where N ≈ M. Once a particular grasp model has been generated it might be necessary to take new data into account. These data may originate from different human operators to cover several ways of performing the same grasp type. Let, for simplicity, the old model to be built by a time sequence [t1 , t2 , . . . , tN ] and a respective fingertip sequence [x1 , x2 , . . . , xN ]. The old model is then represented by the input cluster centers ti and the output cluster centers xi (i = 1 . . . c). It is also described by the parameters Ai and di of the local linear models. Let [t˜1 , t˜2 , . . . , t˜M ], [˜x1 , x˜ 2 , . . . , x˜ M ] be the new training data. A new model can be built by ‘‘chaining’’ the old and the new training data leading, for the time sequences, to [t1 , t2 , . . . , tN , t˜1 , t˜2 , . . . , t˜M ], and for the fingertip sequences to [x1 , x2 , . . . , xN , x˜ 1 , x˜ 2 , . . . , x˜ M ]. The result is a model that involves properties of the old model and the new data. If the old sequence of data is not available, a corresponding sequence can be generated by running the old model with the time instants [t1 , t2 , . . . , tN ] as inputs and the fingertips [x1 , x2 , . . . , xN ] as outputs.
to these models. Segmentation of data in a grasping task deals with the recognition of the starting points of grasps. Ekvall et al. derived these time points by observing the velocity profile of the human arm motion [4]. However, in dexterous manipulation or in assembly tasks, an object can be re-grasped several times without significant arm motions. Therefore Ikeuchi et al. derived the start of a grasp by observing the state of tactile sensors of the data glove [3]. Our segmentation method, however, performs a convolution of a test sequence of grasps against a model signal, plus a subsequent time clustering. This more general approach can also be applied to many other segmentation and recognition problems. Recognition of a grasp from a grasp sequence is based on a recognition model. This model is represented by a set of recognition rules using the model grasps mentioned in the previous section. Segmentation and generation of the recognition model is done by the following steps: A. Computation of distance norms between a sequence of grasps and the individual model grasps involved. B. Computation of extrema (minima and maxima) of distance norms and subsequent segmentation by time clustering. C. Formulation of a set of fuzzy rules reflecting the relationship between extrema of the distance norms and model grasps involved. D. Computation of a vector of similarity degrees between the individual model grasps and the grasps in a grasp combination. E. Identification of the order of grasps in the grasp combination by searching for maximum similarity degrees. Steps A and B can be considered as the segmentation steps. Steps C–E belong to the actual identification and recognition process. In the following paragraphs, these steps will be discussed in more detail. It has to be emphasized that segmentation and recognition of grasps are executed for selected subsets (combinations) of the set of c = 15 grasps. Examples of applications where an a priori known combination of grasps could occur are assembly and handling tasks involving different tools. Our method can recognize which of these tasks was demonstrated. The number mc of grasps in a combination can be chosen to be between 2 and 15, and the possible number of combinations is nc =
c mc
.
We chose mc = 5 and nc = 12 because this choice is able to show the potential of the method and covers a lot of practical cases. Yet, it should be noticed that the method presented can be applied to any size of a combination. However, a change of the combination in size and content always requires a change of the corresponding set of recognition rules. In the framework of PbD by a human operator, the convolution procedure affects both the segmentation and the model generation. After having demonstrated a sequence of grasps, the robot analyzes which grasp in which order should be executed. In contrast to imitation, this is an off-line process which may take 2–5 s computing time in total, depending on the complexity of the task and the computing power available.
R. Palm et al. / Robotics and Autonomous Systems 57 (2009) 484–495
489
Fig. 5. Overlap principle.
5.1. Distance norms
Fig. 6. Time slices.
Without loss of generality, let grasp2 , grasp5 , grasp7 , grasp10 , grasp14 be an a priori known combination of grasps taken from the list shown in Fig. 2. Then a sequence of grasps is generated using the time series of the corresponding model grasps obtained from time clustering described in Section 4. This sequence containing grasps only from the above combination can be arbitrarily long. Grasps can occur in an arbitrary order, including repetitions of grasps. Each grasp is separated from the other by a ’pause’sequence (open hand) to obtain a definite distinction of the grasps. After that, each of the model grasps i = 2, 5, 7, 10, 14 is shifted along the time series of the grasp sequence and compared with it by taking the norm2 kXc − Xmi k between the difference of the fingertip positions Xc of the sequence of grasps and the fingertip positions Xmi of a model graspi . Shifting and comparison take place in steps of the time cluster centers t1 . . . tct of the individual grasp where ct is the number of time clusters. The fingertip positions Xmi and Xc are defined as
5.2. Extrema in the distance norms and segmentation
Xmi = (xm (t1 ), . . . xm (tct ))Ti
zmini = (zmin,1i , . . . , zmin,li )T ;
and
tmini = (tmin,1i , . . . , tmin,li )
Xc = (xc (t˜1 ), . . . , xc (t˜ct ))T .
and
The vectors xm and xc denote the 3 fingertip positions (x, y, z )j j = 1 . . . 5 in the hand coordinate system for each of the 5 fingers. t˜1 . . . t˜ct are the actual time points in the time series Xc of the grasp sequence. Because of scaling reasons the norm kXc − Xmi k is divided by the norm kXmi k > 0 of the model graspi ni =
kXc − Xmi k . kXmi k
(11)
With this result for each graspi i = 2, 5, 7, 10, 14 a time sequence ni (t˜l ), l = 1 . . . ct , is generated. During the pauses the norms ni do not change significantly because of the low correlation between the time series of the model grasp and the pause. But once the model grasp starts to overlap a grasp in the grasp combination, the norms ni change rapidly and reach an extremum (maximum or minimum) as the model grasp and the grasp in the grasp combination reach their highest overlap in time. Fig. 5 shows this ‘‘convolution principle’’ by the simple example of grasp 2 and 7 in a grasp sequence. Using only models, grasps in a sequence lead to an almost perfect overlap, together with a minimum norm ni for one of the model grasps i. However, this cannot be expected for the same sequence of test grasps, including the same type of grasps but being performed by another human operator at another time. As a conclusion, for a particular grasp sequence, one obtains 5 norm sequences ni , the processing of which are presented in the following paragraphs.
Using a model graspi for comparison with a sequence of M test grasps, the norm ni forms individual patterns at M distinct time intervals of the norm sequence. Within each of these time intervals the norm sequence ni reaches an extremum, i.e. either a local minimum or a maximum. In order to find the local extrema in ni the total time interval Tni of ni is partitioned into l time slices, within which the search takes place (see Fig. 6). To be able to identify all relevant extrema, the lengths Tslice of the time slices have to be bounded by Tgrasp,min < Tslice < Tdist,min /2
(12)
where Tgrasp,min is the minimum time length of a grasp. Tdist,min is the minimum time distance between the end of a grasp and the starting point of a new grasp which is equal to the length of the pause. Tgrasp,min and Tdist,min are supposed to be known. This search yields two pairs of vectors (13)
T
zmaxi = (zmax,1i , . . . , zmax,li )T ;
(14)
tmaxi = (tmax,1i , . . . , tmax,li )T where l = dTni /Tslice e. The elements of zmini and zmaxi contain l absolute values of local minima and maxima of ni , respectively. tmini and tmaxi contain the corresponding l time stamps of the local minima and maxima. Usually there are more elements (extrema) included in (13) and (14) than grasps exist in the grasp sequence l ≥ M. The segmentation task is to find the time slices that include the beginnings of grasps. To deal with an unknown number of grasps solutions, different strategies are possible. A ‘soft’ solution requires a variable number of time clusters and a repetitive search for the most likely starting points of grasps. A mixed hardware–software solution is to utilize sensor information about the established contact between the fingers and the object to be grasped. In the following, we assume the number M of grasps in a grasp sequence to be known. A segmentation procedure finds those extrema that indicate the starting points of the grasps. The segmentation is done by time clustering, where the time vectors tmaxi and tmini are the model inputs, zmini and zmaxi are the model outputs. We expect the elements of zmaxi and zmini to form M clusters tseg = (tseg ,1 . . . tseg ,M )T . The result of the clustering procedure is a vector of M time cluster centers, pointing to the starting points of the grasps. For each time point tseg ,r there is a pair (zmin,ij , zmax,ij ). Index j denotes a graspj in the grasp sequence, executed at time tseg ,r . Index i = 2, 5, 7, 10, 14 denote the model graspsi . This finalizes the segmentation procedure.
490
R. Palm et al. / Robotics and Autonomous Systems 57 (2009) 484–495
5.3. Set of fuzzy rules The two sets of vectors zmini and zmaxi build ‘fingerprint patterns’ for each of the grasps in a specific grasp combination. On the basis of these patterns, a set of rules decide whether a special combination of minima and maxima belong to a certain grasp. Obviously, for a selected grasp, these patterns change with the change of a grasp combination. For example, the pattern for grasp2 in the grasp combination 2, 5, 7, 10, 14 differs significantly from the pattern in grasp combination 1, 2, 3, 4, 5 etc. This is taken into account by the formulation of an individual set of rules for each grasp combination. In order to recognize a model graspi from a specific grasp sequence, a set of 5 rules is formulated, one rule for each grasp in the combination. A general recognition rule for graspi to be identified from the combination reads
grasp is graspi .
(n2 is max2,5 ) AND (n5 is min5,5 ) AND (n7 is max7,5 ) AND (n10 is min10,5 ) AND (n14 is max14,5 ) IF
grasp is grasp5
(17)
Analogue to that we obtain for the minimum
wmin,ij = zminij /zmin,tot j .
(18)
Then the local extrema zmaxij and zminij are expressed by the total extrema zmax,tot j and zmin,tot j and the weights wmax,ij and wmin,ij zmaxij = wmax,ij · zmax,tot j zminij = wmin,ij · zmin,tot j
(19)
i = 2, 5, 7, 10, 14.
(15)
To identify grasp5 , for example, rule (15) can be read. ‘‘IF (norm n2 of model grasp2 is max2,5 ) . . . AND (norm n14 of model grasp14 is max14,5 ) THEN grasp is grasp5 ’’. The full rule to identify grasp5 reads
THEN
wmax,ij = zmaxij /zmax,tot j .
A similarity degree aj ∈ [0, 1] is then defined showing the similarity between the model graspi and the actual grasp in the test grasp sequence at time tseg ,r .
(nj is exji ) . . . AND (nk is exki ) IF
THEN
The same is done for each local minimum zminij where zmin,tot j = max(zminij ), i = 2, 5, 7, 10, 14. The ratio between zmaxij and zmax,tot j is expressed by the weight
(16)
j = 2 . . . ; k = 14 are the indexes of the grasps in the grasp combination. i is the index of graspi to be identified. exji indicate fuzzy sets of local extrema which can be either minima minji or maxima maxji . Extrema appear at time points t˜ = tseg ,r at which model graspj meets graspj in the grasp sequence with a maximum overlap. 5.4. Similarity degrees and identification of grasps
aj =
Y
mij ,
(20)
mij = exp(−|wex,ij · zex,tot j − zexij |) zexij represents either a minimum zminij or maximum zmaxij of the norms ni at time tseg ,r , respectively (see Fig. 7). The weights wex,ij , once derived from the training grasp sequence, serve as model parameters for the membership function represented by (20). The product operation in (20) represents the AND-operation in the rules (15) and (16). The exponential function in (20) is a membership function indicating the distance of a norm ni to a local extremum wex,ij · zex,tot j . The exponential function reaches its maximum at exactly that time point tseg ,r when graspj in the grasp sequence has its local extremum (see, e.g., Fig. 8). Since the time points tseg ,r for the occurrence of grasps in the grasp sequence are known from the segmentation, we can compute the similarity degrees aj between a model graspk and graspj at time tseg ,r . The similarity degrees to identify a graspj at tseg ,r are computed in the following way: 1. Apply the recognition rules (15) to graspj by calculating (20), use the corresponding weights wex,ik , i, k = 2, 5, 7, 10, 14. 2. For a sequence of model grasps one obtains for ak
Without loss of generality we focus again on the grasp combination (2, 5, 7, 10, 14). Let a graspj occur in the grasp sequence at time tseg ,r . The possibility of this grasp to be classified as graspj is indicated by a so-called similarity degree aj ∈ [0, 1] that is closely related to the antecedent part of (15). The similarity degrees aj are computed with the help of the pair (zminij , zmaxij ) which has been computed as a result of the segmentation. Remember that index j denotes the grasp in the grasp sequence to be identified. Index i denotes the model grasps being shifted along the grasp sequence. Recall that for a grasp sequence containing only grasps (2, 5, 7, 10, 14) five norm curves ni , i = 1 . . . 5 are computed. Recall also that tseg ,r is the time point in tseg for the local extrema of the norms ni at which graspj occurs. At time point tseg ,r the norm curves ni reach either a local minimum or a maximum depending on which model graspi overlaps with graspj in the grasp sequence. Fig. 7 shows an example where a grasp sequence is built by all the grasps of the grasp combination (2, 5, 7, 10, 14) without any repetition. If the grasp sequence is built by model grasps, a perfect overlap occurs when model graspj meets graspj in the grasp sequence at time tseg ,r . To calculate aj each local maximum zmaxij is normalized to the total maximum zmax,tot j = max(zmaxij ), i = 2, 5, 7, 10, 14.
i = 2, 5, 7, 10, 14
i
ak =
1 if k = j < 1 if k 6= j.
3. For a sequence of test grasps one usually obtains aj < 1 and ak < aj if k 6= j. After having identified the types of grasps, one problem arises because of the confidence of the similarity degrees obtained. Obviously, a reliable grasp recognition requires a significant maximum similarity degree. If such a significance is too low, then the confidence of the grasp recognition is also low. In our experiments, we preferred, among others, Grubb’s test statistic to support a decision about a grasp appearing at a certain time point tseg ,r G=
max(ak ) − a¯ k s(ak )
;
k = 2, 5, 7, 10, 14;
(21)
ak are the similarity degrees for k = 2, 5, 7, 10, 14 at a fixed time tseg ,r , a¯ k is the mean value, and s(ak ) > 0 is the standard deviation. In a competition of two grasps, the grasp with the higher G-value wins. It has to be emphasized that the method described in this section also allows for the recognition of a single grasp from a set of possible grasps, which is a special case of the method discussed before. The idea is to identify a single grasp, knowing
R. Palm et al. / Robotics and Autonomous Systems 57 (2009) 484–495
491
Fig. 7. Norms of a grasp sequence.
6. Experiments and simulations
Fig. 8. Grasp membership functions.
that this grasp belongs to a well defined set of grasps or a grasp combination. In this case segmentation is an easy task, leading to a single starting point of the grasp to be identified. Several robotic tasks may require the identification of a single grasp rather than the identification of a full grasp sequence, which justifies the consideration of this special case. In our experiments, both cases have been considered. In the following section, however, we will mainly present the segmentation, identification, and recognition of a full grasp sequence.
In this section, the experimental evaluation of the proposed approaches is presented. At present, modeling, segmentation, and recognition of grasps are performed off-line. According to Section 3 we tested 15 different grasp primitives, and recorded 10 samples for each grasp, to perform a reliable modeling. First, the grasp primitives (grasp models) have been trained by an expert user who is strongly involved in the planning and execution of the experiments. The classification system has been tested by both the expert and by another two non-expert users. However because of the fixed test conditions, a significant difference in the classification results between expert and non-expert users could not be found. A grasp starts with an open hand and is completed when the fingers establish contact with the object. In the experiments, 12 grasp combinations of 5 grasps each have been tested (see Table 1). This selection intends to combine grasps from the classes in Section 3 in a unique way and to test the recognition quality both within a class and between classes. 6.1. Time clustering and modeling The choice of the numbers of clusters, both for the fingertip models and for the inverse kinematics, depend on the quality of the resulting TS fuzzy models. On the basis of a performance analysis for each grasp and finger, 10 fingertip position models with 10 cluster centers have been generated from collected data. Furthermore, 3 inverse Jacobian models for each grasp primitive and finger with 3 cluster centers have been built, which are
492
R. Palm et al. / Robotics and Autonomous Systems 57 (2009) 484–495
Table 1 Grasp combinations. Combination
Grasp
Grasp
Grasp
Grasp
Grasp
Comb 1 Comb 2 Comb 3 Comb 4 Comb 5 Comb 6 Comb 7 Comb 8 Comb 9 Comb 10 Comb 11 Comb 12
1 6 11 1 14 2 15 3 5 1 5 2
2 7 12 4 10 5 2 9 6 13 7 5
3 8 13 8 7 7 14 13 3 12 8 8
4 9 14 11 5 10 4 12 9 8 12 12
5 10 15 15 2 14 6 7 11 10 13 15
Fig. 10. Middle finger, original: solid, model: dashed.
Fig. 9. Index finger, original: solid, model: dashed.
15 Jacobians, to be computed off-line. Since there are 3 angles (q1 , q2 , q3 ) and 3 fingertip coordinates (x, y, z ) for a single finger, the Jacobians and their inverses are 3 × 3 square matrices. The 3rd link of each finger (next to the fingertip) does not have a sensor in the data glove. Therefore the angle of this link gets a fixed value greater than zero so that neither ill-conditioned Jacobians nor their inverses can computationally occur. For the identification of inverse Jacobians, small random noise excitation is added to the angles to prevent ill-conditioned Jacobians while modeling from data. The motion of a grasp lasts 3.3 s in the average which adds up to 33 timesteps 1t = 0.1 s. The time clustering procedure results in the cluster centers ti = 2.04, 5.43, 8.87, 12.30, 15.75, 19.19, 22.65, 26.09, 29.53, 32.94 where the time labels are measured in steps of 1t = 0.1 s. The time cluster centers are then complemented by the corresponding cluster centers for the x, y, z coordinates of the fingertips. This equidistant spacing can be found for every individual grasp primitive as a result of the time clustering. Figs. 9–13 shows modeling results for grasp 10 (plane (1 CD-ROM)) for the x, y, and z coordinates for the index, middle, ring, and pinkie finger plus the thumb. These results show a good or even excellent modeling quality.
Fig. 11. Ring finger, original: solid, model: dashed.
6.2. Grasp segmentation and recognition We tested 12 combinations of the 15 different grasps mentioned above. As an example, grasp combination 6 with the grasps 2, 5, 7, 10, 14 will be discussed. For the training phase, a sequence of the model grasps 2, 5, 7, 10, 14 is generated. Time clustering leads to a segmentation result with the starting points of the grasps at: grasp 2: t2 = 16.4, grasp 5: t5 = 39.9, grasp 7: t7 = 66.1, grasp 10: t10 = 88.6, grasp 14: t14 = 106.7.
Fig. 12. Pinkie finger, original: solid, model: dashed.
After the segmentation the norms ni = ni (t˜) are computed using the same model grasps according to (11). Based on the resulting curves in Fig. 7 the weights wex,ij for the rule model (16) and the similarity degrees (20) are determined.
R. Palm et al. / Robotics and Autonomous Systems 57 (2009) 484–495
493
Table 3 Grasp combination 6, test results. a/t
t2
t5
t7
t10
t14
a2 a5 a7 a10 a14
0.53 0.33 0.63 0.37 0.56
0.59 0.76 0.64 0.60 0.44
0.47 0.30 0.83 0.34 0.78
0.74 0.53 0.74 0.79 0.53
0.41 0.26 0.81 0.38 0.82
Table 4 Recognition rates.
Fig. 13. Thumb finger, original: solid, model: dashed.
Group
Grasp
Percentage (%)
≥75%
4. Hammer 5. Screwdriver 8.Precision. grasp sphere 9. Precision grasp cube 10. Small plane 11. Big plane 13. Fingertip big ball 15. Penholder grip
100 93 80 100 100 88 75 75
<75%, ≥ 50%
1. Cylinder 2. Big bottle 3. Small bottle 6. Small ball 7. Big ball 12. Fingertip small ball 14. Fingertip can
<50%
55 60 66 55 16 33 33
grasp. Although in the training situation, all grasps can be reliably recognized, some of the grasps may be more difficult to identify than others. Grasps 2 and 14, for example, show rather high numbers at other time points too, meaning that the recognition of these grasps may be less reliable. A test sample of the same grasp sequence confirms this assumption; there are good results for the grasps 5, 7, 10, and 14 but also a different (wrong) result for grasp 2 (see Fig. 15 and Table 3). Several tests with other grasp combinations have shown that some grasps can be more reliably recognized than others. Grasps with distinct maxima and minima in their ni patterns can be recognized better than those grasps which do not have this feature. Reliable grasps are also robust against variations in the time span of an unknown test grasp, compared to the time span of the respective model grasp. Our results show that the methods can handle up to 20% temporal difference. Table 4 shows the recognition rates of grasps from all grasp combinations taken into account. We divided the grasps into 3 groups:
Fig. 14. Grasp combination 6, training sequences.
1. grasps with a recognition rate ≥ 75% 2. grasps with a recognition rate < 75%& ≥ 50% 3. grasps with a recognition rate < 50%. Fig. 15. Grasp combination 6, test sequences. Table 2 Grasp combination 6, training results. a/t
t2
t5
t7
t10
t14
a2 a5 a7 a10 a14
0.99 0.70 0.78 0.63 0.64
0.70 1.00 0.66 0.67 0.49
0.53 0.39 1.00 0.44 0.95
0.63 0.67 0.67 1.00 0.52
0.53 0.39 0.90 0.44 0.99
The training results in Fig. 14 and Table 2 show that, at the starting points of the particular grasps, the type of the grasps 2, 5, 7, 10, and 14 can be reliably recognized. The rows a2, . . . , a14 denote the maximum similarity degrees at the starting points of a
Group 1 is the largest group, followed by groups 2 and 3. Here, we should point out the similarities between grasps 1 and 3, grasps 4, 5 and 15, grasps 7 and 11, and grasps 8 and 9 which gives space for misinterpretations and explains the low percentages for some of the grasps. The analysis confirms the obvious fact that distinct grasps can be discriminated quite well from each other, while the discrimination between similar grasps is difficult. Therefore, the merging of similar grasps and building of general classes can improve the recognition process significantly. For the general classes 1. 2. 3. 4.
cylindrical grasp (grasps 1, 2, 3, 14) power grasp (grasps 4, 5) spherical grasp (grasps 6, 7, 12, 13) extension grasp (grasps 10, 11)
494
R. Palm et al. / Robotics and Autonomous Systems 57 (2009) 484–495
Table 5 Recognition rates between general classes. Grasp general class
1
1. Cylinder 2. Big bottle 3. Small bottle 4. Hammer 5. Screwdriver 6. Small ball 7. Big ball 8. Precision. grasp sphere 9. Precision grasp cube 10. Small plane 11. Big plane 12. Fingertip small ball 13. Fingertip big ball 14. Fingertip can 15. Penholder grip
100% 71% 86% 14%
2
3
4
5
6
29% 14% 100% 58% 28%
14% 44% 100%
14% 28% 100% 100%
14% 100% 100%
100% 86%
100%
5. precision grasp, nippers pinch (grasps 8, 9) 6. penholder grasp (writing grip) (grasp 15) we obtain the recognition rates shown in Table 5. This table can be read as ‘‘grasp i can be identified with a recognition rate of x% to be in general class j’’. The bold numbers show the maxima in each row. The respective columns show that the bold numbers form exactly the general classes, which means that each general class can be reliably identified. 7. Summary and conclusions In this paper, a new fuzzy-logic based method to modeling, classification and recognition of human grasps is presented. The goal is to develop an easy way of programming-by-demonstration of grasps for artificial hands. Grasp primitives are captured by a data glove and modeled by TS-fuzzy models. Fuzzy clustering and modeling of time and space data is applied to the modeling of the fingertip trajectories of grasp primitives. Given a test sequence of grasps, the method allows us to identify the segments of the occurrence of grasps, and to recognize the grasp types. Each model grasp is correlated with the test grasp sequence. By means of a hybrid fuzzy model, the grasp sequence shown by the operator can be identified. With this method, both the segmentation of grasps and their recognition is possible. The method also allows the recognition of a single grasp from a set of possible grasps. The future work will concentrate on the improvement of the PbD process and the development of the recognition and classification of operator movements in a robotic environment, by using more sensor information about the robot workspace and objects to be handled, as well as their affordances. The data glove will be equipped by tactile sensors to improve and accelerate the segmentation process. The method will be simplified to make the recognition faster, leading, in turn, to a faster programming process for the robot. Furthermore the method will be extended and will be used for the programming of complex robot skills in the presence of external sensors. References [1] S.B. Kang, K. Ikeuchi, Towards automatic robot instruction form perception — mapping human grasps to manipulator grasps, IEEE Transactions on Robotics and Automation 13 (1) (1997). [2] R. Zoellner, O. Rogalla, R. Dillmann, J.M. Zoellner, Dynamic grasp recognition within the framework of programming by demonstration, in: Proceedings Robot and Human Interactive Communication, 10th IEEE International Workshop, Bordeaux, Paris, France, September 18–21, IEEE, 2001. [3] K. Bernardin, K. Ogawara, K. Ikeuchi, R. Dillman, A sensor fusion approach for recognizing continuous human grasp sequences using hidden Markov models, IEEE Transactions on Robotics 21 (1) (2005).
14%
86%
[4] S. Ekvall, D. Kragic, Grasp recognition for programming by demonstration, in: Proceedings of International Conference on Robotics and Automation, ICRA 2005, Barcelona, Spain, 2005. [5] D. Martinez, D. Kragic, Modeling and recognition of actions through motor primitives, in: Proceedings ICRA 2008, IEEE International Conference on Robots and Automation, IEEE, Pasadena, CA, USA, 2008. [6] Ch. Li, L. Khan, B. Prabhaharan, Real-time classification of variable length multi-attribute motions, Knowledge and Information Systems (2005) doi:10.1007/s10115-005-0223-8. [7] J. Aleotti, S. Caselli, Grasp recognition in virtual reality for robot pregrasp planning by demonstration, in: ICRA 2006—International Conference on Robotics and Automation, IEEE, Orlando, FL, USA, 2006. [8] I.S. Vicente, V. Kyrki, D. Kragic, M. Larson, Action recognition and understanding through motor primitives, Advanced Robotics 21 (15) (2007) 1687–1702. [9] S. Calinon, A. Billard, Incremental learning of gestures by imitation in a humanoid robot, in: Proceedings of the ACM/IEEE International Conference on Human–Robot Interaction (HRI), 2007, IEEE, ACM, Arlington, Virginia, USA, 2007. [10] O. Bimber, Continuous 6D gesture recognition: A fuzzy-logic approach, in: Proceedings of 7-th International Conference in Central Europe on Computer Graphics, Visualization and Interactive Digital Media, WSCG’99, 1999, pp. 24–30. [11] S. Ferguson, G. Dunlop, Grasp recognition from myoelectric signals, in: Proceedings Australasian Conference Robotics and Automation, 2002, pp. 78–83. [12] D.J. Fleet, M.J. Black, O. Nestares, Bayesian inference of visual motion boundaries, in: Exploring Artificial Intelligence in the New Millenium, Morgan, Kaufmann, 2003, pp. 139–173. [13] V. Kruger, D. Kragic, A. Ude, C. Geib, Action recognition and understanding through motor primitives, Advanced Robotics 21 (13) (2007) 1473–1501. [14] R. Palm, B. Iliev, Learning of grasp behaviors for an artificial hand by time clustering and Takagi-Sugeno modeling, in: Proceedings FUZZ-IEEE 2006 IEEE International Conference on Fuzzy Systems, Vancouver, BC, Canada, July 16–21, IEEE, 2006. [15] R. Palm, B. Iliev, Segmentation and recognition of human grasps for programming-by-demonstration using time clustering and Takagi-Sugeno modeling, in: Proceedings FUZZ-IEEE 2007 - IEEE International Conference on Fuzzy Systems, London, UK, July 23–26, IEEE, 2007. [16] R. Palm, B. Iliev, Grasp recognition by time clustering, fuzzy modeling, and Hidden Markov Models (HMM) — a comparative study, in: Proceedings FUZZIEEE 2008 - IEEE International Conference on Fuzzy Systems, Hong Kong, July 1–5, IEEE, 2008. [17] A. Skoglund, B. Iliev, B. Kadmiry, R. Palm, Programming by demonstration of pick-and-place tasks for industrial manipulators using task primitives, in: Proceedings Computational Intelligence in Robotics and Automation, CIRA 2007, Jacksonville, FL, USA, June 20–23, IEEE, 2007. [18] H.H. Asada, J.R. Fortier, Task recognition and human–machine coordination through the use of an instrument-glove. Progress report No. 2–5, March 2000, pp. 1–39. [19] M. Cutkosky, On grasp choice, grasp models, and the design of hands for manufacturing tasks, IEEE Transactions on Robotics and Automation 5 (3) (1989). [20] Thea Iberall, Human prehension and dexterous robot hands, The International Journal of Robotics Research 16 (1997) 285–299. [21] R. Palm, Ch. Stutz, Open loop dynamic trajectory generator for a fuzzy gain scheduler, Engineering Applications of Artificial Intelligence 16 (2003) 213–225.
R. Palm et al. / Robotics and Autonomous Systems 57 (2009) 484–495 [22] T. Takagi, M. Sugeno, Fuzzy identification of systems and its applications to modeling and control, IEEE Transactions on Systems, Man, and Cybernetics 15 (1) (1985) 116–132. [23] D.E. Gustafson, W.C. Kessel, Fuzzy clustering with a fuzzy covariance matrix, in: Proceedings of the 1979 IEEE CDC, 1979, pp. 761–766.
Rainer Palm has been Adjunct Professor at the Department of Technology (AASS), University Örebro since 2004. Until 2004 he was a Principal Research Scientist in the Neural Computation Department of the Siemens Corporate Research Technology in Munich, Germany, working both in research projects and industrial applications of Automatic Control, Fuzzy Control, Modeling of Dynamic Systems, and Distributed Systems. He received his Dipl. Ing. degree from the Technical University of Dresden, Germany, in 1975. Since 1982 he worked in robotic research, in particular in the field of sensor- guided robots. He received the Dr. Ing. degree from the Humboldt-University Berlin, Germany, in 1981, and the Dr. sc. techn. degree from the Academy of Sciences, Germany, in 1989. Since 1991 he held a position at the fuzzy logic task force group at Siemens AG. He is author /co-author, editor/co-editor of 15 books or chapters in books, author/ coauthor of 19 articles in journals, and author /co-author of 35 articles in Conference proceedings. He is also author / co-author of 14 patents. His research interest include automatic control, fuzzy control, and robotics. Spokesman of the fuzzy group 1994-2002. Associate editor of IEEE Transactions on Fuzzy Systems 1994–2004.
495
Boyko Iliev is a senior researcher at the Center for Applied Autonomous Sensors Systems (AASS), Dept. of Technology, Örebro University. He received his Ph.D. degree from Örebro University in Control Theory (robot control) in 2004 and, since then, his Postdoc at AASS. Before coming to Örebro, he received a diploma in Control Systems Engineering from the Technical University, Sofia, Bulgaria in 1997. His research interest includes robotic grasping and dexterous manipulation, biologically inspired systems, Control Theory and its applications in robotics. He is author /coauthor of numerous conference papers and Journal articles on robotics and control theory.
Bourhane Kadmiry is a Postdoc at the Dept. of Technology, Örebro University. He received his Ph.D. in Computer Sc. at Linköping University-Sweden and a Ph.D. in Robotics and AI at (LAAS) Toulouse University-France. He is also a holder of a Master of Sc. in Sensor Technology, Control and AI and a Master of Computer Engineering, both obtained at Rouen Technical University-France. His research interests include autonomous helicopter navigation control, visual servoing control for autonomous helicopter, fuzzy control, and control for an artificial (prosthetic) hand. He has authored 2 journal articles and 6 full refereed conference papers, and is a regular reviewer for international conferences. He is also project manager for Airbus aeronautic programs within Soerman Co.-Sweden.