- Email: [email protected]
Planning grasps for industrial robotized applications using neural networks
Robotics and Computer Integrated Manufacturing 16 (2000) 451}463
Planning grasps for industrial robotized applications using neural networks Gino Dini*, Franco Failli Department of Mechanical, Nuclear and Production Engineering, University of Pisa, Via Bonanno Pisano, 25/b, 56126 Pisa, Italy
Abstract The present paper proposes a computer-aided module for the detection of grasps of 3D objects by using arti"cial neural networks. This module considers the grasps performed by the most common classes of grippers used in industrial applications: two-jaw grippers, three-jaw grippers, magnetic grippers, vacuum grippers and expandable grippers. The neural networks are properly tailored and trained in order to evaluate and compare the di!erent grasping alternatives according to geometrical and technological aspects of object surfaces. A case study is also discussed in order to point out the performances of this methodology. 2000 Elsevier Science Ltd. All rights reserved. Keywords: Gripper; Grasp planning; Neural networks
1. Introduction to grasp planning The need of computer-aided systems for planning of robotized operations (e.g. assembly, welding, machine loading and unloading, palletization, etc.) has been stressed in several occasions both by the academic and the industrial world. The reasons of this interest are various and mainly arise from the following considerations: E modern products require highly #exible production systems able to be recon"gured in short times according to market changes and requirements. In the same way, the structure used for planning, scheduling and o!-line programming the plant must be #exible as well, by means of software tools capable of giving fast responses for every new operative condition; E the recent interest toward concurrent engineering has also involved aspects related to the planning of operations, to be considered simultaneously in the design of a product; E the problem of planning is usually characterized by a very huge number of potential solutions. For this reason, the use of computer-aided systems able to process and evaluate in short times a high number of alternatives is strongly recommended. * Corresponding author. Fax: #39-050-913040. E-mail address: [email protected] (G. Dini).
Generally speaking, three main sub-problems have to be approached and solved in computer-aided planning of robotized operations: sequencing, path planning and grasp planning. The "rst one consists of determining, under certain constraints, the feasible sequences of elementary operations needed to successfully perform a speci"c task. The second one concerns the generation of end-e!ector trajectories needed to complete each operation, avoiding collisions and reducing the total time requested to cover the path. The third one is obviously present only in planning of operations requiring manipulation of objects, such as assembly or pick-and-place tasks; it consists of selecting the best way of grasping an object by a robotic hand according to the geometrical and technological features of its surfaces as well as the speci"c task to be accomplished. The three previous sub-problems, although they are separately analyzed in most literature contributions, are strictly interconnected in a computer-aided planning system. This fact is more evident in planning of assembly operations: the feasibility of an assembly sequence can be proved only if the grasp and the trajectories of the ende!ector are completely known; on the other hand, the selection of grasps can be performed only if the assembly sequence is completely planned and the trajectories are de"ned in order to verify the accessibility of the grasping surfaces. The automated planning of grasps perhaps represents, among the previous sub-problems, the most complex
0736-5845/00/$ - see front matter 2000 Elsevier Science Ltd. All rights reserved. PII: S 0 7 3 6 - 5 8 4 5 ( 0 0 ) 0 0 0 2 1 - 1
452
G. Dini, F. Failli / Robotics and Computer Integrated Manufacturing 16 (2000) 451}463
procedure owing to the high number of involved constraints concerning both the geometrical features of the object, the gripper, the robot and the kinematic and dynamic aspects that characterized this operation. Furthermore, it has to be stressed that grasp planning appears as a multi-solution problem: the combinations of the surfaces that guarantee a correct gripping are often more than one. At the same time, several possible gripper con"gurations can be adopted for a stable and correct grasp. The problem is therefore to detect the preferred solution among a set of feasible ones. The detection of grasping surfaces has to be performed assuring the ful"llment of the following three objectives (Fig. 1): E stability: the selected surfaces and grippers must guarantee a stable grasp under di!erent static and dynamic conditions evaluated in function of the morphology of the object and the task to be performed; E accessibility: the selected surfaces must be easily reached by the gripper, avoiding interferences and collisions with the remaining part of the object; E accuracy: the selected surfaces and grippers must guarantee the grasping accuracy requested by the
Fig. 1. Concepts of stability, accessibility and accuracy in grasping of objects.
robotized operation. This aspect is crucial in assembly where, for example, high precision insertions obviously need high accurate grasps achievable through a proper selection of gripping surfaces. Unfortunately, in most cases, these three aspects cannot be optimized simultaneously, but a compromise among the di!erent requirements has to be reached. An example of this situation is depicted in Fig. 1, where a surface which simultaneously represents the preferred choice in terms of stability, accessibility and accuracy does not exist. A good compromise could be the surface no. 1 which represents the best solution in terms of accessibility and accuracy during the insertion into the hole, but it is not the optimal choice in terms of stability.
2. Related works In these last years, a very high number of contributions has been given to the problem of grasp planning, demonstrating the great interest toward this topic among scientists. This fact can be explained, as mentioned before, by the ever growing interest of industry and market to computer-aided planning systems. Grasp planning is a very stimulating subject also for the researchers working in computer science "eld due to its peculiar aspects of representing a good benchmark for testing new algorithms or optimization procedures. A "rst classi"cation of contributions can be made taking into account the two research "elds of non-industrial and industrial applications. The former concerns aspects of advanced and dexterous robotics and mainly deals with the problem of planning grasps for multi"ngered robot hands, including very complicated procedures concerning the selection of grasp postures based on attempts to mimic human grasping behavior [1,2]. The latter includes the present work and considers the grasp planning problem restricted to industrial cases, i.e. use of industrial grippers, grasps on object of industrial interest, grasps to perform industrial tasks, etc. This `restrictiona leads to a deep simpli"cation of computation algorithms with respect to the non-industrial application "eld: the structures of grippers, the limited number of "ngers (usually two and sometimes three), the simple shape of "nger-tips (usually #at or V-shaped) and the simple motion needed to approach the grasping surfaces allow a strong reduction of the number of feasible grasps. The problem of selecting grasping con"gurations has been studied by many researchers under di!erent points of view. Some of them [3,4] concentrate their e!orts to the theoretical analysis of grasp stability by identifying the combination of forces and moments exerted on the object to keep it in a stable equilibrium ( force-closure condition); two problems are usually dealt with: a forward problem, consisting of analyzing whether a grasp, de"ned
G. Dini, F. Failli / Robotics and Computer Integrated Manufacturing 16 (2000) 451}463
by a set of contacts, is force closure or not; a reverse problem, regarding the selection of places to put the "nger tips so that the grasp is force closure. Another important stability criterion concerns the resistance of a grip to slippage and twisting due to the weight of the object, to the inertial forces or to the loads exerted during the contact of the part with the external environment (typically in peg-in-hole operations). This last aspect is discussed in [5], where the resistance to slippage is expressed in terms of friction between surfaces, shape of contact areas and distance of the grasp from the centroid of the object, and in [6] where the distance of the grasp from the resultant of the external forces is also taken into account. The problem of accessibility is discussed in many contributions which emphasize the high complexity of this aspect. In [7,8] this problem is approached in connection with the planning of collision-free trajectories and kinematically feasible motions of the robot arm. In particular, in [7] the approach is based on a kinematic analysis of the con"guration-space representation of the motion constraints; a simple and e$cient algorithm calculating the possible collision-free paths in pick-andplace operations on polyhedrical objects is proposed. Meaningful examples of grasp planning contributions speci"cally focused to industrial cases are reported in [9,10]. The former addresses the task of automatic grasp planning for assembly operations, presenting a system for selecting grasps and parallel two-jaw grippers according to the object geometry and assembly task speci"cations (e.g. placing an object on another, inserting a peg into a hole, grasping a screwdriver to turn a screw, turning a nut onto a bolt, "tting a part into a slot, etc.). The latter deals with the same speci"c problem (i.e. assembly operations by means of parallel two-jaw grippers), concentrating its e!orts in grasping rotational components and using three di!erent heuristics to determine feasible grasps, gripper functional attributes and areas available for grasping a component. Another e$cient way to deal with the problem of grasp planning is the application of Artixcial Intelligence software tools [6,11}13]. The use of these methodologies, in fact, allows an e!ective management of the knowledge needed to perform the optimal choice among the feasible solutions. Moreover, they allow a representation of a knowledge di$cult to be formally de"ned and resulting from experiences of many years of pick-and-place activities. This knowledge can be also modi"ed and adapted to the actual requirements of the user. Last but not least, these methods usually give the outputs in short computational times with respect to other more traditional techniques. From this point of view, Artixcial Neural Networks (ANNs) can be surely considered a very attractive programming tool. The particular procedure of knowledge acquisition and the capability to face unknown
453
situations, without having the explicit rule for solution, make ANNs an e!ective tool for some typical problems of computer-aided planning in manufacturing engineering [14]. Using this approach, in fact, the technological knowledge can be easily expressed through a set of representative examples. In this way, a knowledge based on theories partially resulted from empirical rules can be easily implemented in a computer; this is the case of problems such as the sequencing of machining operations, the selection of cutting parameters, the selection of cutting tools and, also, the selection of grasps. A further advantage related to the use of ANNs in grasp planning is the high computing speed allowed by this programming tool, especially if the network is implemented in hardware platforms based on parallel architectures. This important feature can be used in applications where a fast response on some aspects of the production process is requested, for example during a CAD session for a real implementation of the concepts of concurrent engineering. The innovative contribution of this work is therefore represented by testing the feasibility of using the neural network software tool in the research area of grasp planning. The networks will be properly tailored and trained in order to solve the problems related to the choice of grasping surfaces using di!erent types of industrial grippers.
3. Overall description of the proposed grasp planning system The previous considerations have led to the development of a computer-aided system for the detection of grasps of 3D objects, using ANNs, in order to evaluate and compare the di!erent alternatives. This evaluation is performed considering the grasps obtained by the most common classes of grippers used in industrial applications: E two-jaw grippers, with pivoting or parallel motion of "ngers; E three-jaw grippers, with self-centering motion of "ngers; E magnetic grippers, with a single circular magnetic plate; E vacuum grippers, with a single circular vacuum cup; E expandable grippers, such as collet or rubber gripper used for internal grasps of parts with holes. Fig. 2 gives a schematical overview of the system, emphasizing the following steps needed to generate the "nal response: (a) inputs: the inputs of the system are divided in two groups. The former includes morphological and technological aspects of the object, without any reference to the speci"c operation to be performed; these data can be introduced manually, through a proper
454
G. Dini, F. Failli / Robotics and Computer Integrated Manufacturing 16 (2000) 451}463
Fig. 2. Schematic overview of the proposed grasp planning system.
interactive session, otherwise they can be automatically extracted from the 3D CAD model of the object. The latter includes all the aspects characterizing the robotized operation such as object orientation, accessible surfaces, direction of insertion, etc.; (b) pre-processing of inputs: the inputs are pre-processed in order to select feasible and potential grasps for each class of grippers. This "rst selection is made by rules and conditions described in detail in Section 4. Successively, the geometrical and technological aspects of each potential grasp are extracted and translated in proper inputs for the ANNs; (c) evaluation of grasps: each grasps previously selected is evaluated by ANNs, one for each class of grippers.
This step generates an output which represents a `judgementa on the quality of the grasp given on the basis of the knowledge acquired in training; (d) post-processing of ANN outputs: this part of the system has the objective of re-evaluating the grasps according to additional criteria concerning, for instance, the preference of using a certain type of gripper to perform a speci"c operation. This task is obtained by using proper parameters (ranging between 0 and 1) which weigh the results generated by the system. For instance, in high-precision grasping, jaw or expandable grippers are preferred to magnetic or vacuum ones due to their intrinsic accuracy in grasping an object (the following weights have been adopted: 1.0
G. Dini, F. Failli / Robotics and Computer Integrated Manufacturing 16 (2000) 451}463
455
evaluating grasps by ANNs. In this way, each ANN does not process all the combinations of surfaces, but only those ones which guarantee a grasp from a purely geometrical point of view. This selection is performed through a two-step procedure. First of all, only certain types of surface combinations are taken into account for each class of gripper (#at, curved and cylindrical surfaces). Table 1 shows the combinations considered in this system (at the moment, other combinations have not been taken into account being quite unusual in real mechanical parts). Moreover, the following restrictions have been introduced:
for grasps with two- and three-jaw grippers, 0.8 for grasps with expandable grippers, 0.5 for grasps with magnetic and vacuum grippers); (e) xnal results: at the end of the process, the system produces a list of preferred grasps, each of them represented by the type of gripper to be used and the set of gripping surfaces of the object. The various grasps are sorted according to the judgements assigned by the ANNs: the grasp with the best judgement represents the best solution generated by the system; the others can be valid alternatives if other constraints prevent the adoption of the best solution (e.g. minimization of gripper changes in successive operations).
E the term curved surface identi"es single-curvature surfaces having constant radius, subdivided in concave and convex ones; E the term cylindrical surface identi"es complete rotational surfaces, subdivided in external (e.g. a peg) and internal ones (e.g. a hole).
The system does not perform any collision check among the gripper, the object and the external environment. This control, in fact, can be accomplished only if the geometrical aspects of the gripper, the "xtures and the devices used in the surroundings and the trajectory of the robot are completely known. For this reason, it has been decided to postpone this control to a successive simulation module (not described in this paper), where the alternatives proposed by the grasp planning system can be analyzed, under this point of view, and "nally selected.
The second step of this selection consists of applying and verifying geometrical conditions speci"c for the different types of grippers. In the next paragraphs these conditions are shown and discussed. 4.1. Two-jaw grippers Using two-jaw grippers, a correct and stable grasping cannot leave out of consideration two geometrical requirements of surfaces: mutual orientation and mutual visibility. A pair of #at surfaces S and S satis"es the require ment of mutual orientation when the angle 0 between the normal n and n agrees with the following condition (Fig. 3a): 2
(1) "1803!0 ") , k Q
4. Selection of potential grasps In this context, a potential grasp is identi"ed by one or more object surfaces which must satisfy, at least, the minimum geometrical requirements for a correct grasping by a given type of gripper. The selection of all the potential grasps is therefore an important step which has to be accomplished before Table 1 Surface combinations considered for each type of gripper Gripper
Surface combinations Single #at Single surface curved surface
Two-jaw Three-jaw Magnetic Vacuum Expandable
Single cylindrical surface (external)
Single cylindrical surface (internal)
Two #at surfaces
Two curved surfaces
One #at surface and one curved surface
One #at surface and one cylindrical surface (external or internal)
One curved surface and one cylindrical surface (external or internal)
Two or more co-axial surfaces.
456
G. Dini, F. Failli / Robotics and Computer Integrated Manufacturing 16 (2000) 451}463
Fig. 3. Concepts of mutual orientation (a) and mutual visibility (b) for a pair of #at surfaces.
being the friction angle between the surface of the object and the "nger-tip and k a safety factor. Obviously, Q the optimal condition is obtained when the surfaces are parallel and opposed (i.e. 0"1803), but small variations from this value are allowed if the "nger-tips are properly shaped or self-aligning jaws are used. It has to be noticed that the analysis of n and n directions, together with their application points, allows a distinction between external and internal grasps. A pair of #at surfaces S and S satis"es the requirement of mutual visibility where exists a surface S obtained as S "SH5SH, (2) where SH and SH are, respectively, the projections of the surfaces S and S on the bisector plane (Fig. 3b). As far as curved surfaces are concerned, the previous conditions (1) and (2) can be still veri"ed considering as normal of the surface that vector, whose direction passes through the center of the surface curvature, and by which the angle 0 assumes a value as near as possible to 1803. An example of this concept is given in Fig. 4a, where d represents the minimum distance from the edge of KGL the surface at which the grasp acts. The directions of n and n also lead to the identi"cation of the bisector plane and therefore to the veri"cation of mutual visibility condition. Cylindrical surfaces present a much higher #exibility in terms of grasping, since they can be approached by the "ngers from any direction (in the plane of grasping). For this reason, a grasp performed on one cylindrical surface (external or internal) intrinsically satis"es both mutual orientation and mutual visibility conditions. If two cylindrical surfaces are used to perform a grasp, or a cylindrical surface is used together a #at or a curved surface, the veri"cation of the orientation and visibility conditions presents similarities with respect to the previous cases and an example is depicted in Fig. 4b. 4.2. Three-jaw grippers If the grasp is performed on a single cylindrical surface (external or internal), no speci"c condition has to be
Fig. 4. Examples of di!erent situations in grasping curved and cylindrical surfaces.
veri"ed. In this situation, in fact, an in"nite number of grasps assuring three points of contact spaced at 1203 exists. If the grasp is performed on a single curved surface, in order to assure the presence of the aforesaid three points of contact, the following condition, referred to the arc obtained by sectioning the curved surface with a plane perpendicular to its axis, has to be satis"ed: l!2d * pR,
where l is the length of the arc and R is its radius (Fig. 4c). If the grasp is performed on two or more curved surfaces, the following conditions have to be satis"ed: E co-axiality: the surfaces must have the same axis; E homogeneity: the surfaces must have the same radius R and must belong to the same category (external or internal); E mutual visibility: this condition takes into account, for each surface, the segment s of the axis obtained G through the radial projection of the surface (Fig. 4d); the condition is satis"ed if exists a segment s given by L s "s 5s 525s , L L E mutual orientation: the set of surfaces must include three points disposed at 1203 from each other having a distance from the edges of surface at least of d .
G. Dini, F. Failli / Robotics and Computer Integrated Manufacturing 16 (2000) 451}463
457
4.3. Magnetic grippers The potential grasps are selected verifying the material and the orientation of the surface. First of all, the surface must be made of ferromagnetic material; secondly, being 0 the angle formed between its normal and the gravity vector, the following condition has to be satis"ed: d "1803!0 ") , k Q where k is a safety factor and d represents the maximum Q angle of surface inclination, with respect to a horizontal plane, which allows a correct grasp with a magnetic gripper. 4.4. Vacuum grippers The conditions are quite similar to those presented for magnetic grippers. A "rst control is made on the surface quality features: threaded or porous surfaces are discarded due to the impossibility of a correct gripping through a vacuum cup. A second control is made on the surface orientation: being 0 the angle formed between its normal and the gravity vector, the following condition has to be satis"ed: f "1803!0 ") , k Q where k is a safety factor and f represents the maximum Q angle of surface inclination, with respect to a horizontal plane, which allows a correct grasp with a vacuum gripper. 4.5. Expandable grippers No particular condition has to be satis"ed in this case, being internal cylindrical surfaces the unique possibility of grasping for this kind of gripper.
5. Neural network architectures The neural networks are used to evaluate the feasible grasps previously selected, implementing a concept schematically shown in Fig. 5. The inputs of the ANNs are represented by the di!erent grasp parameters p . These parameters are G summarized in Table 2 for each kind of gripper and, considering for brevity only the grasps performed by a two-jaw gripper, are evaluated as follows: E surface combinations (p ): it indicates the kind of surfa ces used in the grasp (#at, curved or cylindrical) and it is represented by a numerical value ranging between 0 and 1 in order to distinguish the di!erent combinations listed in Table 1;
Fig. 5. General structure of a neural network and its use in the grasp planning system.
E available area (p ): it is proportional to the area avail able for grasping and calculated through the expression p "A/A , being A the area of the surface S (Fig. 6)
and A the maximum area among all the surfaces
S obtained from each pair of object surfaces; E slenderness of available area (p ): it is calculated as p "4(A/P, where P is the perimeter of the surface S ; E distance between surfaces (p ): it is proportional to the distance u existing between the centroids G and G of the portions of the surfaces S and S obtained pro jecting to themselves the surface S (Fig. 6). It is calculated as p "u/u , being u the maximum
distance evaluated among all the possible pairs of surfaces; E mutual orientation (p ): it is calculated as p " "n ) n ", where n and n are the normal unit vectors of the surfaces S and S (Fig. 6); E external/internal grasp (p ): it distinguishes between external (p "1) and internal grasps ( p "0); E distance from gravity axis (p ): it is evaluated in func tion of the distance v between the center point of the segment G G (indicated in Fig. 6 as GCP: grasp center point) and the line of action of the gravity vector (gravity axis). The parameter is calculated as p "0.1(A/v if v*0.1(A, otherwise p "1; E orientation with respect to the gravity axis (p ): it is calculated as p ""q ) r", where q is the unit vector having the same direction of the segment G G and r is the unit vector perpendicular to the gravity axis and having the direction joining the GCP and the gravity axis; E distance from assembly axis (p ): it is calculated in the same way of p , but the distance v is between the GCP and the line, parallel to the assembly direction of the object, that coincides with the axis of the surfaces in contact during insertion with the object positioned on the workplace (assembly axis in Fig. 6);
Orientation Surface respect to roughness assembly (p ) axis (p ) Orientation Distance respect to from gravity (p ) assembly axis (p )
Two-jaw gripper Three-jaw gripper Magnetic gripper Vacuum gripper Expandable gripper
Distance from gravity axis (p ) External/ internal grasp (p ) Distance Mutual between orientation surfaces (p ) (p ) Available area (p )
Slenderness of available area (p )
Fig. 6. Nomenclature adopted for the de"nition of grasp parameters p . G
Surface combinations (p )
Inputs (grasp parameters p ) ANN
Table 2 Inputs of the neural networks for each type of gripper
G. Dini, F. Failli / Robotics and Computer Integrated Manufacturing 16 (2000) 451}463 Surface regularity (p )
458
E orientation with respect to the assembly axis (p ): it is calculated in the same way of p , but considering the assembly axis instead of the gravity axis; E surface roughness (p ): it indicates the status of surfa ces ("nished, roughed, etc.) and it is calculated as a numerical value ranging between 0 and 1 in order to distinguish the di!erent possibilities; E surface regularity (p ): it takes into account the pres ence and the distribution of holes and/or protruding parts on the surfaces. The parameter is calculated as p "p ) p . The single terms p and p are FMJC NPMR FMJC NPMR evaluated in function of four variables: the ratio r"A /A, being A the area of the available surface GPP GPP covered by the irregularities; the kind of gripper; the number of irregularities; the distribution of irregularities on the surface. As far as this last aspect is concerned, three types of distributions have been considered: centered distribution, when the irregularities are mainly concentrated in the center of the surface; peripheral distribution, when the irregularities are mainly concentrated near the border of the surface; intermediate distribution, when the distribution cannot be classi"ed in one of the previous types. Table 3 shows the di!erent expressions adopted to calculate the parameters p and p . For example, considerFMJC NPMR ing a grasp performed by a vacuum gripper on a surface having a centered/intermediate distribution of holes (very critical situation for this kind of gripper), the parameter p is calculated by the function f . The FMJC concave shape of this function, in fact, allows a strong distinction between surfaces having very low values of r (grasping is still possible) and surfaces having medium or high values of r (grasping is very di$cult).
G. Dini, F. Failli / Robotics and Computer Integrated Manufacturing 16 (2000) 451}463
459
Table 3 Determination of the parameters related to the irregularities of a grasped surface
The output of each ANN is a linguistic variable which can assume one of the following terms referred to the `qualitya of grasp: optimum, good, quite good, acceptable, bad. The ANN architecture adopted in this work has been the radial basis function (RBF) [15,16], formed by the two layers of neurons: a hidden layer, containing neurons with Gaussian transfer functions, and an output layer, containing neurons with linear transfer functions (Fig. 5). The output layer, in particular, includes "ve neurons, one for each linguistic variable previously mentioned: the activation of an output neuron therefore means the assignment of the corresponding linguistic variable to the examined grasp. The activation of two contiguous outputs can also be accepted (e.g. good}quite good). This means that the network has not given a result univocally set to one output, but it has generated an intermediate response between two outputs. Accepting this kind of results, the possible levels of evaluation increase from 5 to 9.
The adoption of RBF architecture is justi"ed by the following two reasons: E it presents a high training speed, especially if it is compared with ANNs based on back propagation training methods; E it allows an easier optimization of performances, since the only parameter which can be used to modify its structure is the number of neurons in the hidden layer. Furthermore, this number is automatically set in the training procedure. The key aspect in training and validation is represented by the creation of examples. The basic idea is to create examples from real cases adopting the following procedure: E one or more objects are proposed to the system which extracts from them the potential grasps using the same selection procedure described in Section 4; E for each potential grasp, the parameters p are then G evaluated, using the same criteria previously explained;
460
G. Dini, F. Failli / Robotics and Computer Integrated Manufacturing 16 (2000) 451}463
6. Example of application and results The behavior of the system has been tested in di!erent situations. In the next sections, an example of application is given both for the training and validation phase and for the detection of grasps in a case study. 6.1. Training and validation of ANNs
Fig. 7. Training and validation procedure.
E the system proposes the grasps to a human expert (the program user), together with the corresponding parameters and the localization of surfaces on the object. The user gives a judgement taken from the list of linguistic expressions previously mentioned; E the system stores the examples formed by the parameters p and the corresponding judgements, in order G to use them in training and validation. The training and validation are performed through an iterative procedure which intrinsically allows a satisfactory optimization of the ANN structure. This procedure can be described as follows (Fig. 7): (a) a "rst training set (TS ) formed by a limited number of examples is used to train the ANN; (b) a "rst validation set (VS ) formed by examples con cerning grasps, di!erent from TS , are used to vali date the ANN; (c) the examples of VS which give unexpected outputs are extracted and added to TS in order to create a new training set TS ; (d) the examples of VS which give correct outputs are extracted and added to other di!erent examples in order to create a new validation set VS ; (e) the training and validation sets TS and VS are used for a new procedure, repeating the steps from (a) until no error (or a limited number of errors) occurs in step (b).
Training and validation of ANNs have been performed by using a set of examples extracted from di!erent families of very simple objects, such as boxes, cylinders, etc. The di!erent objects belonging to each family are obtained starting from a representative object and varying di!erent geometrical and technological aspects within a predetermined range. It is obvious that the choice of these objects strongly a!ects the training and therefore the behavior of the networks. This critical operation has to be performed in function of the range of situations to be covered or, in other words, in function of the technological knowledge domain which has to be implemented in the system. Table 4 reports some examples used in training and validation for grasps performed by two-jaw grippers. The examples are created, by the procedure described in Section 5, from a family of hollow cylinders obtained starting from a representative object and varying the following aspects: E E E E E
y"e\ G N \UG
outer diameter D (from 10 to 40 mm); inner diameter d (from 8 to 30 mm); height h (from 10 to 80 mm); orientation of simmetry axis (horizontal or vertical); position of center of gravity (centered or shifted of 0.5D); E position of assembly axis (centered or shifted of 0.5D); E regularity of surfaces (without or with a presence of protruding parts so that A "0.1A). GPP On the contrary, the surface roughness has been considered unchanged. In this way, 128 examples of grasp evaluations have been obtained. Other examples of twojaw gripper grasps have been created considering families of boxes (228 examples) and prisms (48 examples), for a total of 404 examples. Considering, for brevity, the training and validation of three ANNs for two-jaw, magnetic and vacuum grippers, Table 5 shows the data concerning:
where p (with i"1, 2,2, n) are the ANN inputs and G w (with i"1, 2,2, n) are the parameters belonging to G the example which gives the maximum error. This procedure allows, for each iteration, the introduction of an entity which gives the best result in situations where the previous structure failed, increasing rapidly the performance of the network. Neurons are added until the sum-squared errors falls beneath an error goal or a maximum number of neurons has been reached.
E the number of examples used in training: it can be noticed that the ANN for two-jaw grippers has required the highest number of examples being this kind of grasp characterized by the highest number of di!erent alternatives; E the maximum sum-squared error: this parameter has been set, for each network, to a compromise value in order to obtain acceptable results and, at the same time, to keep low the total computational time;
In training, a further iterative procedure is used to optimize the design of the RBF network, creating successive structures by adding one neuron at a time in the hidden layer. This neuron has a transfer function given by L
G. Dini, F. Failli / Robotics and Computer Integrated Manufacturing 16 (2000) 451}463
461
Table 4 Set of 128 examples extracted from a family of hollow cylinders and used in training and validation phases of the ANN for two-jaw grippers
E the maximum number of neurons accepted in the hidden layer: this parameter has been set su$ciently high in order to obtain the termination of training procedure due to the reaching of the error goal; E the number of neurons in the hidden layer obtained at the end of the process: the obtained values demonstrate the `di$cultya encountered in training the ANN for twojaws grippers with respect to the others.
6.2. Case study The object illustrated in Fig. 8 is proposed to the system. It can be noticed that the gravity axis coincides with the assembly axis; furthermore, the surface 6 is not accessible, being the object placed on a workbench. No particular constraint concerning the accuracy of grasping has been given.
462
G. Dini, F. Failli / Robotics and Computer Integrated Manufacturing 16 (2000) 451}463
Table 5 Data concerning the training phase of ANNs used for the evaluation of grasps performed by two-jaw, magnetic and vacuum grippers ANN
Size of training set (no. of examples)
Maximum sum-squared error
Maximum no. of neurons in hidden layer
No. of neurons in hidden layer
Two-jaw gripper Magnetic gripper Vacuum gripper
326 80 100
4.00 1.00 1.00
250 150 150
191 64 85
The results, obtained considering ANNs trained by the procedure described in Section 6.1, are summarized in Table 6. In particular, the following considerations can be made: E the best grasp is represented by the cylindrical surface 1 performed by a two-jaw gripper. The reason of this choice can be explained considering the wide area of grasping, the low value of slenderness, the external approaching of gripper and the central position of grasp with respect to the gravity and assembly axis; E just below the best one, three grasps can be found: an internal grasp performed by a two-jaw gripper on cylindrical surface 5; an external grasp performed by a two-jaw gripper on curved surfaces 3 and 15; a grasp on the #at surface 2 performed by a magnetic gripper. The di!erent evaluation given to these grasps with respect to the "rst choice can be explained considering: the internal approach to surface 5, which can introduce accessibility problems in grasping the object; a lower grasping area and a higher slenderness of surfaces 3 and 15; the presence of the hole in the center of surface 2; E a lower value has been given to other grasps with two-jaw grippers characterized by a GCP not positioned on the axis of the object. As examples taken among them, the grasp performed on #at surface 8 and cylindrical surface 5 is considered better than the grasp performed on #at surfaces 7 and 9. This fact can be explained by the di!erent orientation of these grasps with respect to the object axis: the former has p "1 and p "1 and therefore presents a more stable gras ping, the latter has p "0 and p "0 which can increase the possibility of slippage during grasping due to the weight of the object; E grasps performed by magnetic devices, except that one acting on surface 2, have not received a good evaluation due to their small and slender area as well as their distance from the axis of the object; E the same reasoning can be done for grasps performed by vacuum grippers. All these grasps present acceptable or bad evaluation due to their small and slender area as well as their distance from the axis of the object. In addition, surface 2, although it is centered with respect to the axis, has a big hole which prevents a good evaluation by the system.
Fig. 8. Case study: a valve housing.
Table 6 Grasps detected by the system for the object illustrated in Fig. 7 and sorted in order of importance No. of grasp
Grasping surfaces
Type of gripper
Evaluation by ANNs
1 2 3 4 5 6 7 8 10 11 12 13 14 15 16 17 18 19 20 21 22
1}1 5}5 3}15 2 8}5 12}5 5}3 5}15 8}12 1}5 10 14 10 14 7}9 11}13 4 16 4 16 2
Two-jaw Two-jaw Two-jaw Magnetic Two-jaw Two-jaw Two-jaw Two-jaw Two-jaw Two-jaw Magnetic Magnetic Vacuum Vacuum Two-jaw Two-jaw Magnetic Magnetic Vacuum Vacuum Vacuum
Optimum Good-optimum Good Good Quite good-good Quite good-good Quite good Quite good Quite good Acceptable-quite good Bad-acceptable Bad-acceptable Bad-acceptable Bad-acceptable Bad Bad Bad Bad Bad Bad Bad
G. Dini, F. Failli / Robotics and Computer Integrated Manufacturing 16 (2000) 451}463
The results shown in Table 6 can be post-processed according to di!erent instances concerning task requirements. If, for example, a high-precision positioning of the object is requested, the system re-arranged the outputs penalizing those solutions which do not guarantee a high accuracy such as the grasps performed by magnetic or vacuum grippers.
463
Acknowledgements The authors would like to express sincere thanks to Prof. B. Lazzerini and Dr. F. Marcelloni of the Department of Information Engineering of the University of Pisa for their technical support, and to Dr. M. Montanari for the active contribution to this work. References
7. Concluding remarks The feasibility of implementing a neural algorithm in a computer-aided grasp planning system has been successfully demonstrated. This new approach, in particular, gives the chance to acquire the technological knowledge without using complicated heuristic rules, usually di$cult to be formulated and managed, but presenting to the system a set of grasping examples together with their evaluation given by a human expert. This aspect, although representing the most evident advantage of the proposed technique, at the same time is the most critical point: it is important to pay attention on the selection of proper examples which have to be focused to a given technological and geometrical domain and equally distributed within it. The procedure proposed to train and validate the ANNs works in this direction allowing, through its iterative process, a self-calibrating detection of the training set size. Future developments of this research project will follow two main directions: improvement of system performances and integration of the system in a more complex assembly planning package. The former could include improvements concerning some aspects such as capability to deal with more complex object surfaces than those ones considered in the system (at present, only #at, cylindrical and curved surfaces are taken into account); extension of surface combinations considered for each type of gripper; capability of grasping on edges or vertices of the object. The latter could involve aspects related to the selection of gripper features (dimensions, "nger strokes, grasping force, "nger-tip shape, etc.) and integration problems related to sequencing and path planning activities, such as selection of grasps according to assembly sequence of components in order to minimize gripper changes; evaluation of the exact "nger-tip positions on selected surfaces; veri"cation of surface accessibility by means of collision check and path planning procedures.
[1] Kawarazaki N, Hasegawa T, Nishihara K. Grasp planning algorithm for a multi-"ngered hand-arm robot. Proceedings of IEEE International Conference of Robotics and Automation, 1998. p. 933}9. [2] Bekey GA, Liu H, Tomovic R, Karplus WJ. Knowledge-based control of grasping in robot hands using heuristics from human motor skills. IEEE Trans Robotics Automat 1993;9(6):709}22. [3] Xiong YL, Sanger DJ, Kerr DR. Geometric modelling of bounded and frictional grasps. Robotics 1993;11:185}92. [4] Montana DJ. The conditions for contact grasp stability. Proceedings of IEEE International Conference on Robotics and Automation, 1991. p. 412}7. [5] Wolter JD, Volz RA, Woo AC. Automatic generation of gripping positions. IEEE Trans Systems Man Cybernet 1985;SMC15(2):204}13. [6] Dini G. A module for the automated planning of grasps in robotized assembly operations. Ann CIRP 1993;42(1):1}4. [7] Jones JL, Lozano-Perez T. Planning two-"ngered grasps for pickand-place operations on polyhedra. Proceeding of IEEE International Conference, 1990. p. 683}8. [8] Terasaki H, Hasegawa T, Takahashi H, Arakawa T. Automatic grasping and regrasping by space characterization for pick-andplace operations. Proceeding of IEEE/RSJ International Workshop on Intelligent Robots and Systems, 1991. p. 58}63. [9] Dubey A, Nnaji BO. Task-oriented and feature-based grasp planning. Robotics Comput Integrated Manuf 1992;9(6):471}84. [10] Boubekri N, Ramanujam G. Design of grasping methodologies for rotational assembly components. Int J Flexible Manuf Systems 1995;7:373}88. [11] Pham DT, Yeo SH. A knowledge-based system for robot gripper selection: criteria for choosing grippers and surfaces for gripping. Int J Mach Tools Manuf 1988;28(4):301}13. [12] Canzi U, Cugini U, Rigillo F. Kali: a knowledge-based system for grasp planning. Proceedings of 5th International Conference on Advanced Robotics, 1991. [13] Pham DT, Tacgin E. Grippex: a hybrid expert system for selecting robot gripper types. Int J Mach Tools Manuf 1992;32(3):349}60. [14] Dagli CH, editor. Arti"cial neural networks for intelligent manufacturing. London: Chapman & Hall, 1994. [15] Wasserman PD. Advanced methods in neural computing. New York: Van Nostrand Reinhold, 1993. [16] Chen S, Cowan CFN, Grant PM. Orthogonal least squares learning algorithm for radial basis function networks. IEEE Trans Neural Networks 1991;2(2):302}9.
Planning grasps for industrial robotized applications using neural networks Gino Dini*, Franco Failli Department of Mechanical, Nuclear and Production Engineering, University of Pisa, Via Bonanno Pisano, 25/b, 56126 Pisa, Italy
Abstract The present paper proposes a computer-aided module for the detection of grasps of 3D objects by using arti"cial neural networks. This module considers the grasps performed by the most common classes of grippers used in industrial applications: two-jaw grippers, three-jaw grippers, magnetic grippers, vacuum grippers and expandable grippers. The neural networks are properly tailored and trained in order to evaluate and compare the di!erent grasping alternatives according to geometrical and technological aspects of object surfaces. A case study is also discussed in order to point out the performances of this methodology. 2000 Elsevier Science Ltd. All rights reserved. Keywords: Gripper; Grasp planning; Neural networks
1. Introduction to grasp planning The need of computer-aided systems for planning of robotized operations (e.g. assembly, welding, machine loading and unloading, palletization, etc.) has been stressed in several occasions both by the academic and the industrial world. The reasons of this interest are various and mainly arise from the following considerations: E modern products require highly #exible production systems able to be recon"gured in short times according to market changes and requirements. In the same way, the structure used for planning, scheduling and o!-line programming the plant must be #exible as well, by means of software tools capable of giving fast responses for every new operative condition; E the recent interest toward concurrent engineering has also involved aspects related to the planning of operations, to be considered simultaneously in the design of a product; E the problem of planning is usually characterized by a very huge number of potential solutions. For this reason, the use of computer-aided systems able to process and evaluate in short times a high number of alternatives is strongly recommended. * Corresponding author. Fax: #39-050-913040. E-mail address: [email protected] (G. Dini).
Generally speaking, three main sub-problems have to be approached and solved in computer-aided planning of robotized operations: sequencing, path planning and grasp planning. The "rst one consists of determining, under certain constraints, the feasible sequences of elementary operations needed to successfully perform a speci"c task. The second one concerns the generation of end-e!ector trajectories needed to complete each operation, avoiding collisions and reducing the total time requested to cover the path. The third one is obviously present only in planning of operations requiring manipulation of objects, such as assembly or pick-and-place tasks; it consists of selecting the best way of grasping an object by a robotic hand according to the geometrical and technological features of its surfaces as well as the speci"c task to be accomplished. The three previous sub-problems, although they are separately analyzed in most literature contributions, are strictly interconnected in a computer-aided planning system. This fact is more evident in planning of assembly operations: the feasibility of an assembly sequence can be proved only if the grasp and the trajectories of the ende!ector are completely known; on the other hand, the selection of grasps can be performed only if the assembly sequence is completely planned and the trajectories are de"ned in order to verify the accessibility of the grasping surfaces. The automated planning of grasps perhaps represents, among the previous sub-problems, the most complex
0736-5845/00/$ - see front matter 2000 Elsevier Science Ltd. All rights reserved. PII: S 0 7 3 6 - 5 8 4 5 ( 0 0 ) 0 0 0 2 1 - 1
452
G. Dini, F. Failli / Robotics and Computer Integrated Manufacturing 16 (2000) 451}463
procedure owing to the high number of involved constraints concerning both the geometrical features of the object, the gripper, the robot and the kinematic and dynamic aspects that characterized this operation. Furthermore, it has to be stressed that grasp planning appears as a multi-solution problem: the combinations of the surfaces that guarantee a correct gripping are often more than one. At the same time, several possible gripper con"gurations can be adopted for a stable and correct grasp. The problem is therefore to detect the preferred solution among a set of feasible ones. The detection of grasping surfaces has to be performed assuring the ful"llment of the following three objectives (Fig. 1): E stability: the selected surfaces and grippers must guarantee a stable grasp under di!erent static and dynamic conditions evaluated in function of the morphology of the object and the task to be performed; E accessibility: the selected surfaces must be easily reached by the gripper, avoiding interferences and collisions with the remaining part of the object; E accuracy: the selected surfaces and grippers must guarantee the grasping accuracy requested by the
Fig. 1. Concepts of stability, accessibility and accuracy in grasping of objects.
robotized operation. This aspect is crucial in assembly where, for example, high precision insertions obviously need high accurate grasps achievable through a proper selection of gripping surfaces. Unfortunately, in most cases, these three aspects cannot be optimized simultaneously, but a compromise among the di!erent requirements has to be reached. An example of this situation is depicted in Fig. 1, where a surface which simultaneously represents the preferred choice in terms of stability, accessibility and accuracy does not exist. A good compromise could be the surface no. 1 which represents the best solution in terms of accessibility and accuracy during the insertion into the hole, but it is not the optimal choice in terms of stability.
2. Related works In these last years, a very high number of contributions has been given to the problem of grasp planning, demonstrating the great interest toward this topic among scientists. This fact can be explained, as mentioned before, by the ever growing interest of industry and market to computer-aided planning systems. Grasp planning is a very stimulating subject also for the researchers working in computer science "eld due to its peculiar aspects of representing a good benchmark for testing new algorithms or optimization procedures. A "rst classi"cation of contributions can be made taking into account the two research "elds of non-industrial and industrial applications. The former concerns aspects of advanced and dexterous robotics and mainly deals with the problem of planning grasps for multi"ngered robot hands, including very complicated procedures concerning the selection of grasp postures based on attempts to mimic human grasping behavior [1,2]. The latter includes the present work and considers the grasp planning problem restricted to industrial cases, i.e. use of industrial grippers, grasps on object of industrial interest, grasps to perform industrial tasks, etc. This `restrictiona leads to a deep simpli"cation of computation algorithms with respect to the non-industrial application "eld: the structures of grippers, the limited number of "ngers (usually two and sometimes three), the simple shape of "nger-tips (usually #at or V-shaped) and the simple motion needed to approach the grasping surfaces allow a strong reduction of the number of feasible grasps. The problem of selecting grasping con"gurations has been studied by many researchers under di!erent points of view. Some of them [3,4] concentrate their e!orts to the theoretical analysis of grasp stability by identifying the combination of forces and moments exerted on the object to keep it in a stable equilibrium ( force-closure condition); two problems are usually dealt with: a forward problem, consisting of analyzing whether a grasp, de"ned
G. Dini, F. Failli / Robotics and Computer Integrated Manufacturing 16 (2000) 451}463
by a set of contacts, is force closure or not; a reverse problem, regarding the selection of places to put the "nger tips so that the grasp is force closure. Another important stability criterion concerns the resistance of a grip to slippage and twisting due to the weight of the object, to the inertial forces or to the loads exerted during the contact of the part with the external environment (typically in peg-in-hole operations). This last aspect is discussed in [5], where the resistance to slippage is expressed in terms of friction between surfaces, shape of contact areas and distance of the grasp from the centroid of the object, and in [6] where the distance of the grasp from the resultant of the external forces is also taken into account. The problem of accessibility is discussed in many contributions which emphasize the high complexity of this aspect. In [7,8] this problem is approached in connection with the planning of collision-free trajectories and kinematically feasible motions of the robot arm. In particular, in [7] the approach is based on a kinematic analysis of the con"guration-space representation of the motion constraints; a simple and e$cient algorithm calculating the possible collision-free paths in pick-andplace operations on polyhedrical objects is proposed. Meaningful examples of grasp planning contributions speci"cally focused to industrial cases are reported in [9,10]. The former addresses the task of automatic grasp planning for assembly operations, presenting a system for selecting grasps and parallel two-jaw grippers according to the object geometry and assembly task speci"cations (e.g. placing an object on another, inserting a peg into a hole, grasping a screwdriver to turn a screw, turning a nut onto a bolt, "tting a part into a slot, etc.). The latter deals with the same speci"c problem (i.e. assembly operations by means of parallel two-jaw grippers), concentrating its e!orts in grasping rotational components and using three di!erent heuristics to determine feasible grasps, gripper functional attributes and areas available for grasping a component. Another e$cient way to deal with the problem of grasp planning is the application of Artixcial Intelligence software tools [6,11}13]. The use of these methodologies, in fact, allows an e!ective management of the knowledge needed to perform the optimal choice among the feasible solutions. Moreover, they allow a representation of a knowledge di$cult to be formally de"ned and resulting from experiences of many years of pick-and-place activities. This knowledge can be also modi"ed and adapted to the actual requirements of the user. Last but not least, these methods usually give the outputs in short computational times with respect to other more traditional techniques. From this point of view, Artixcial Neural Networks (ANNs) can be surely considered a very attractive programming tool. The particular procedure of knowledge acquisition and the capability to face unknown
453
situations, without having the explicit rule for solution, make ANNs an e!ective tool for some typical problems of computer-aided planning in manufacturing engineering [14]. Using this approach, in fact, the technological knowledge can be easily expressed through a set of representative examples. In this way, a knowledge based on theories partially resulted from empirical rules can be easily implemented in a computer; this is the case of problems such as the sequencing of machining operations, the selection of cutting parameters, the selection of cutting tools and, also, the selection of grasps. A further advantage related to the use of ANNs in grasp planning is the high computing speed allowed by this programming tool, especially if the network is implemented in hardware platforms based on parallel architectures. This important feature can be used in applications where a fast response on some aspects of the production process is requested, for example during a CAD session for a real implementation of the concepts of concurrent engineering. The innovative contribution of this work is therefore represented by testing the feasibility of using the neural network software tool in the research area of grasp planning. The networks will be properly tailored and trained in order to solve the problems related to the choice of grasping surfaces using di!erent types of industrial grippers.
3. Overall description of the proposed grasp planning system The previous considerations have led to the development of a computer-aided system for the detection of grasps of 3D objects, using ANNs, in order to evaluate and compare the di!erent alternatives. This evaluation is performed considering the grasps obtained by the most common classes of grippers used in industrial applications: E two-jaw grippers, with pivoting or parallel motion of "ngers; E three-jaw grippers, with self-centering motion of "ngers; E magnetic grippers, with a single circular magnetic plate; E vacuum grippers, with a single circular vacuum cup; E expandable grippers, such as collet or rubber gripper used for internal grasps of parts with holes. Fig. 2 gives a schematical overview of the system, emphasizing the following steps needed to generate the "nal response: (a) inputs: the inputs of the system are divided in two groups. The former includes morphological and technological aspects of the object, without any reference to the speci"c operation to be performed; these data can be introduced manually, through a proper
454
G. Dini, F. Failli / Robotics and Computer Integrated Manufacturing 16 (2000) 451}463
Fig. 2. Schematic overview of the proposed grasp planning system.
interactive session, otherwise they can be automatically extracted from the 3D CAD model of the object. The latter includes all the aspects characterizing the robotized operation such as object orientation, accessible surfaces, direction of insertion, etc.; (b) pre-processing of inputs: the inputs are pre-processed in order to select feasible and potential grasps for each class of grippers. This "rst selection is made by rules and conditions described in detail in Section 4. Successively, the geometrical and technological aspects of each potential grasp are extracted and translated in proper inputs for the ANNs; (c) evaluation of grasps: each grasps previously selected is evaluated by ANNs, one for each class of grippers.
This step generates an output which represents a `judgementa on the quality of the grasp given on the basis of the knowledge acquired in training; (d) post-processing of ANN outputs: this part of the system has the objective of re-evaluating the grasps according to additional criteria concerning, for instance, the preference of using a certain type of gripper to perform a speci"c operation. This task is obtained by using proper parameters (ranging between 0 and 1) which weigh the results generated by the system. For instance, in high-precision grasping, jaw or expandable grippers are preferred to magnetic or vacuum ones due to their intrinsic accuracy in grasping an object (the following weights have been adopted: 1.0
G. Dini, F. Failli / Robotics and Computer Integrated Manufacturing 16 (2000) 451}463
455
evaluating grasps by ANNs. In this way, each ANN does not process all the combinations of surfaces, but only those ones which guarantee a grasp from a purely geometrical point of view. This selection is performed through a two-step procedure. First of all, only certain types of surface combinations are taken into account for each class of gripper (#at, curved and cylindrical surfaces). Table 1 shows the combinations considered in this system (at the moment, other combinations have not been taken into account being quite unusual in real mechanical parts). Moreover, the following restrictions have been introduced:
for grasps with two- and three-jaw grippers, 0.8 for grasps with expandable grippers, 0.5 for grasps with magnetic and vacuum grippers); (e) xnal results: at the end of the process, the system produces a list of preferred grasps, each of them represented by the type of gripper to be used and the set of gripping surfaces of the object. The various grasps are sorted according to the judgements assigned by the ANNs: the grasp with the best judgement represents the best solution generated by the system; the others can be valid alternatives if other constraints prevent the adoption of the best solution (e.g. minimization of gripper changes in successive operations).
E the term curved surface identi"es single-curvature surfaces having constant radius, subdivided in concave and convex ones; E the term cylindrical surface identi"es complete rotational surfaces, subdivided in external (e.g. a peg) and internal ones (e.g. a hole).
The system does not perform any collision check among the gripper, the object and the external environment. This control, in fact, can be accomplished only if the geometrical aspects of the gripper, the "xtures and the devices used in the surroundings and the trajectory of the robot are completely known. For this reason, it has been decided to postpone this control to a successive simulation module (not described in this paper), where the alternatives proposed by the grasp planning system can be analyzed, under this point of view, and "nally selected.
The second step of this selection consists of applying and verifying geometrical conditions speci"c for the different types of grippers. In the next paragraphs these conditions are shown and discussed. 4.1. Two-jaw grippers Using two-jaw grippers, a correct and stable grasping cannot leave out of consideration two geometrical requirements of surfaces: mutual orientation and mutual visibility. A pair of #at surfaces S and S satis"es the require ment of mutual orientation when the angle 0 between the normal n and n agrees with the following condition (Fig. 3a): 2
(1) "1803!0 ") , k Q
4. Selection of potential grasps In this context, a potential grasp is identi"ed by one or more object surfaces which must satisfy, at least, the minimum geometrical requirements for a correct grasping by a given type of gripper. The selection of all the potential grasps is therefore an important step which has to be accomplished before Table 1 Surface combinations considered for each type of gripper Gripper
Surface combinations Single #at Single surface curved surface
Two-jaw Three-jaw Magnetic Vacuum Expandable
Single cylindrical surface (external)
Single cylindrical surface (internal)
Two #at surfaces
Two curved surfaces
One #at surface and one curved surface
One #at surface and one cylindrical surface (external or internal)
One curved surface and one cylindrical surface (external or internal)
Two or more co-axial surfaces.
456
G. Dini, F. Failli / Robotics and Computer Integrated Manufacturing 16 (2000) 451}463
Fig. 3. Concepts of mutual orientation (a) and mutual visibility (b) for a pair of #at surfaces.
being the friction angle between the surface of the object and the "nger-tip and k a safety factor. Obviously, Q the optimal condition is obtained when the surfaces are parallel and opposed (i.e. 0"1803), but small variations from this value are allowed if the "nger-tips are properly shaped or self-aligning jaws are used. It has to be noticed that the analysis of n and n directions, together with their application points, allows a distinction between external and internal grasps. A pair of #at surfaces S and S satis"es the requirement of mutual visibility where exists a surface S obtained as S "SH5SH, (2) where SH and SH are, respectively, the projections of the surfaces S and S on the bisector plane (Fig. 3b). As far as curved surfaces are concerned, the previous conditions (1) and (2) can be still veri"ed considering as normal of the surface that vector, whose direction passes through the center of the surface curvature, and by which the angle 0 assumes a value as near as possible to 1803. An example of this concept is given in Fig. 4a, where d represents the minimum distance from the edge of KGL the surface at which the grasp acts. The directions of n and n also lead to the identi"cation of the bisector plane and therefore to the veri"cation of mutual visibility condition. Cylindrical surfaces present a much higher #exibility in terms of grasping, since they can be approached by the "ngers from any direction (in the plane of grasping). For this reason, a grasp performed on one cylindrical surface (external or internal) intrinsically satis"es both mutual orientation and mutual visibility conditions. If two cylindrical surfaces are used to perform a grasp, or a cylindrical surface is used together a #at or a curved surface, the veri"cation of the orientation and visibility conditions presents similarities with respect to the previous cases and an example is depicted in Fig. 4b. 4.2. Three-jaw grippers If the grasp is performed on a single cylindrical surface (external or internal), no speci"c condition has to be
Fig. 4. Examples of di!erent situations in grasping curved and cylindrical surfaces.
veri"ed. In this situation, in fact, an in"nite number of grasps assuring three points of contact spaced at 1203 exists. If the grasp is performed on a single curved surface, in order to assure the presence of the aforesaid three points of contact, the following condition, referred to the arc obtained by sectioning the curved surface with a plane perpendicular to its axis, has to be satis"ed: l!2d * pR,
where l is the length of the arc and R is its radius (Fig. 4c). If the grasp is performed on two or more curved surfaces, the following conditions have to be satis"ed: E co-axiality: the surfaces must have the same axis; E homogeneity: the surfaces must have the same radius R and must belong to the same category (external or internal); E mutual visibility: this condition takes into account, for each surface, the segment s of the axis obtained G through the radial projection of the surface (Fig. 4d); the condition is satis"ed if exists a segment s given by L s "s 5s 525s , L L E mutual orientation: the set of surfaces must include three points disposed at 1203 from each other having a distance from the edges of surface at least of d .
G. Dini, F. Failli / Robotics and Computer Integrated Manufacturing 16 (2000) 451}463
457
4.3. Magnetic grippers The potential grasps are selected verifying the material and the orientation of the surface. First of all, the surface must be made of ferromagnetic material; secondly, being 0 the angle formed between its normal and the gravity vector, the following condition has to be satis"ed: d "1803!0 ") , k Q where k is a safety factor and d represents the maximum Q angle of surface inclination, with respect to a horizontal plane, which allows a correct grasp with a magnetic gripper. 4.4. Vacuum grippers The conditions are quite similar to those presented for magnetic grippers. A "rst control is made on the surface quality features: threaded or porous surfaces are discarded due to the impossibility of a correct gripping through a vacuum cup. A second control is made on the surface orientation: being 0 the angle formed between its normal and the gravity vector, the following condition has to be satis"ed: f "1803!0 ") , k Q where k is a safety factor and f represents the maximum Q angle of surface inclination, with respect to a horizontal plane, which allows a correct grasp with a vacuum gripper. 4.5. Expandable grippers No particular condition has to be satis"ed in this case, being internal cylindrical surfaces the unique possibility of grasping for this kind of gripper.
5. Neural network architectures The neural networks are used to evaluate the feasible grasps previously selected, implementing a concept schematically shown in Fig. 5. The inputs of the ANNs are represented by the di!erent grasp parameters p . These parameters are G summarized in Table 2 for each kind of gripper and, considering for brevity only the grasps performed by a two-jaw gripper, are evaluated as follows: E surface combinations (p ): it indicates the kind of surfa ces used in the grasp (#at, curved or cylindrical) and it is represented by a numerical value ranging between 0 and 1 in order to distinguish the di!erent combinations listed in Table 1;
Fig. 5. General structure of a neural network and its use in the grasp planning system.
E available area (p ): it is proportional to the area avail able for grasping and calculated through the expression p "A/A , being A the area of the surface S (Fig. 6)
and A the maximum area among all the surfaces
S obtained from each pair of object surfaces; E slenderness of available area (p ): it is calculated as p "4(A/P, where P is the perimeter of the surface S ; E distance between surfaces (p ): it is proportional to the distance u existing between the centroids G and G of the portions of the surfaces S and S obtained pro jecting to themselves the surface S (Fig. 6). It is calculated as p "u/u , being u the maximum
distance evaluated among all the possible pairs of surfaces; E mutual orientation (p ): it is calculated as p " "n ) n ", where n and n are the normal unit vectors of the surfaces S and S (Fig. 6); E external/internal grasp (p ): it distinguishes between external (p "1) and internal grasps ( p "0); E distance from gravity axis (p ): it is evaluated in func tion of the distance v between the center point of the segment G G (indicated in Fig. 6 as GCP: grasp center point) and the line of action of the gravity vector (gravity axis). The parameter is calculated as p "0.1(A/v if v*0.1(A, otherwise p "1; E orientation with respect to the gravity axis (p ): it is calculated as p ""q ) r", where q is the unit vector having the same direction of the segment G G and r is the unit vector perpendicular to the gravity axis and having the direction joining the GCP and the gravity axis; E distance from assembly axis (p ): it is calculated in the same way of p , but the distance v is between the GCP and the line, parallel to the assembly direction of the object, that coincides with the axis of the surfaces in contact during insertion with the object positioned on the workplace (assembly axis in Fig. 6);
Orientation Surface respect to roughness assembly (p ) axis (p ) Orientation Distance respect to from gravity (p ) assembly axis (p )
Two-jaw gripper Three-jaw gripper Magnetic gripper Vacuum gripper Expandable gripper
Distance from gravity axis (p ) External/ internal grasp (p ) Distance Mutual between orientation surfaces (p ) (p ) Available area (p )
Slenderness of available area (p )
Fig. 6. Nomenclature adopted for the de"nition of grasp parameters p . G
Surface combinations (p )
Inputs (grasp parameters p ) ANN
Table 2 Inputs of the neural networks for each type of gripper
G. Dini, F. Failli / Robotics and Computer Integrated Manufacturing 16 (2000) 451}463 Surface regularity (p )
458
E orientation with respect to the assembly axis (p ): it is calculated in the same way of p , but considering the assembly axis instead of the gravity axis; E surface roughness (p ): it indicates the status of surfa ces ("nished, roughed, etc.) and it is calculated as a numerical value ranging between 0 and 1 in order to distinguish the di!erent possibilities; E surface regularity (p ): it takes into account the pres ence and the distribution of holes and/or protruding parts on the surfaces. The parameter is calculated as p "p ) p . The single terms p and p are FMJC NPMR FMJC NPMR evaluated in function of four variables: the ratio r"A /A, being A the area of the available surface GPP GPP covered by the irregularities; the kind of gripper; the number of irregularities; the distribution of irregularities on the surface. As far as this last aspect is concerned, three types of distributions have been considered: centered distribution, when the irregularities are mainly concentrated in the center of the surface; peripheral distribution, when the irregularities are mainly concentrated near the border of the surface; intermediate distribution, when the distribution cannot be classi"ed in one of the previous types. Table 3 shows the di!erent expressions adopted to calculate the parameters p and p . For example, considerFMJC NPMR ing a grasp performed by a vacuum gripper on a surface having a centered/intermediate distribution of holes (very critical situation for this kind of gripper), the parameter p is calculated by the function f . The FMJC concave shape of this function, in fact, allows a strong distinction between surfaces having very low values of r (grasping is still possible) and surfaces having medium or high values of r (grasping is very di$cult).
G. Dini, F. Failli / Robotics and Computer Integrated Manufacturing 16 (2000) 451}463
459
Table 3 Determination of the parameters related to the irregularities of a grasped surface
The output of each ANN is a linguistic variable which can assume one of the following terms referred to the `qualitya of grasp: optimum, good, quite good, acceptable, bad. The ANN architecture adopted in this work has been the radial basis function (RBF) [15,16], formed by the two layers of neurons: a hidden layer, containing neurons with Gaussian transfer functions, and an output layer, containing neurons with linear transfer functions (Fig. 5). The output layer, in particular, includes "ve neurons, one for each linguistic variable previously mentioned: the activation of an output neuron therefore means the assignment of the corresponding linguistic variable to the examined grasp. The activation of two contiguous outputs can also be accepted (e.g. good}quite good). This means that the network has not given a result univocally set to one output, but it has generated an intermediate response between two outputs. Accepting this kind of results, the possible levels of evaluation increase from 5 to 9.
The adoption of RBF architecture is justi"ed by the following two reasons: E it presents a high training speed, especially if it is compared with ANNs based on back propagation training methods; E it allows an easier optimization of performances, since the only parameter which can be used to modify its structure is the number of neurons in the hidden layer. Furthermore, this number is automatically set in the training procedure. The key aspect in training and validation is represented by the creation of examples. The basic idea is to create examples from real cases adopting the following procedure: E one or more objects are proposed to the system which extracts from them the potential grasps using the same selection procedure described in Section 4; E for each potential grasp, the parameters p are then G evaluated, using the same criteria previously explained;
460
G. Dini, F. Failli / Robotics and Computer Integrated Manufacturing 16 (2000) 451}463
6. Example of application and results The behavior of the system has been tested in di!erent situations. In the next sections, an example of application is given both for the training and validation phase and for the detection of grasps in a case study. 6.1. Training and validation of ANNs
Fig. 7. Training and validation procedure.
E the system proposes the grasps to a human expert (the program user), together with the corresponding parameters and the localization of surfaces on the object. The user gives a judgement taken from the list of linguistic expressions previously mentioned; E the system stores the examples formed by the parameters p and the corresponding judgements, in order G to use them in training and validation. The training and validation are performed through an iterative procedure which intrinsically allows a satisfactory optimization of the ANN structure. This procedure can be described as follows (Fig. 7): (a) a "rst training set (TS ) formed by a limited number of examples is used to train the ANN; (b) a "rst validation set (VS ) formed by examples con cerning grasps, di!erent from TS , are used to vali date the ANN; (c) the examples of VS which give unexpected outputs are extracted and added to TS in order to create a new training set TS ; (d) the examples of VS which give correct outputs are extracted and added to other di!erent examples in order to create a new validation set VS ; (e) the training and validation sets TS and VS are used for a new procedure, repeating the steps from (a) until no error (or a limited number of errors) occurs in step (b).
Training and validation of ANNs have been performed by using a set of examples extracted from di!erent families of very simple objects, such as boxes, cylinders, etc. The di!erent objects belonging to each family are obtained starting from a representative object and varying di!erent geometrical and technological aspects within a predetermined range. It is obvious that the choice of these objects strongly a!ects the training and therefore the behavior of the networks. This critical operation has to be performed in function of the range of situations to be covered or, in other words, in function of the technological knowledge domain which has to be implemented in the system. Table 4 reports some examples used in training and validation for grasps performed by two-jaw grippers. The examples are created, by the procedure described in Section 5, from a family of hollow cylinders obtained starting from a representative object and varying the following aspects: E E E E E
y"e\ G N \UG
outer diameter D (from 10 to 40 mm); inner diameter d (from 8 to 30 mm); height h (from 10 to 80 mm); orientation of simmetry axis (horizontal or vertical); position of center of gravity (centered or shifted of 0.5D); E position of assembly axis (centered or shifted of 0.5D); E regularity of surfaces (without or with a presence of protruding parts so that A "0.1A). GPP On the contrary, the surface roughness has been considered unchanged. In this way, 128 examples of grasp evaluations have been obtained. Other examples of twojaw gripper grasps have been created considering families of boxes (228 examples) and prisms (48 examples), for a total of 404 examples. Considering, for brevity, the training and validation of three ANNs for two-jaw, magnetic and vacuum grippers, Table 5 shows the data concerning:
where p (with i"1, 2,2, n) are the ANN inputs and G w (with i"1, 2,2, n) are the parameters belonging to G the example which gives the maximum error. This procedure allows, for each iteration, the introduction of an entity which gives the best result in situations where the previous structure failed, increasing rapidly the performance of the network. Neurons are added until the sum-squared errors falls beneath an error goal or a maximum number of neurons has been reached.
E the number of examples used in training: it can be noticed that the ANN for two-jaw grippers has required the highest number of examples being this kind of grasp characterized by the highest number of di!erent alternatives; E the maximum sum-squared error: this parameter has been set, for each network, to a compromise value in order to obtain acceptable results and, at the same time, to keep low the total computational time;
In training, a further iterative procedure is used to optimize the design of the RBF network, creating successive structures by adding one neuron at a time in the hidden layer. This neuron has a transfer function given by L
G. Dini, F. Failli / Robotics and Computer Integrated Manufacturing 16 (2000) 451}463
461
Table 4 Set of 128 examples extracted from a family of hollow cylinders and used in training and validation phases of the ANN for two-jaw grippers
E the maximum number of neurons accepted in the hidden layer: this parameter has been set su$ciently high in order to obtain the termination of training procedure due to the reaching of the error goal; E the number of neurons in the hidden layer obtained at the end of the process: the obtained values demonstrate the `di$cultya encountered in training the ANN for twojaws grippers with respect to the others.
6.2. Case study The object illustrated in Fig. 8 is proposed to the system. It can be noticed that the gravity axis coincides with the assembly axis; furthermore, the surface 6 is not accessible, being the object placed on a workbench. No particular constraint concerning the accuracy of grasping has been given.
462
G. Dini, F. Failli / Robotics and Computer Integrated Manufacturing 16 (2000) 451}463
Table 5 Data concerning the training phase of ANNs used for the evaluation of grasps performed by two-jaw, magnetic and vacuum grippers ANN
Size of training set (no. of examples)
Maximum sum-squared error
Maximum no. of neurons in hidden layer
No. of neurons in hidden layer
Two-jaw gripper Magnetic gripper Vacuum gripper
326 80 100
4.00 1.00 1.00
250 150 150
191 64 85
The results, obtained considering ANNs trained by the procedure described in Section 6.1, are summarized in Table 6. In particular, the following considerations can be made: E the best grasp is represented by the cylindrical surface 1 performed by a two-jaw gripper. The reason of this choice can be explained considering the wide area of grasping, the low value of slenderness, the external approaching of gripper and the central position of grasp with respect to the gravity and assembly axis; E just below the best one, three grasps can be found: an internal grasp performed by a two-jaw gripper on cylindrical surface 5; an external grasp performed by a two-jaw gripper on curved surfaces 3 and 15; a grasp on the #at surface 2 performed by a magnetic gripper. The di!erent evaluation given to these grasps with respect to the "rst choice can be explained considering: the internal approach to surface 5, which can introduce accessibility problems in grasping the object; a lower grasping area and a higher slenderness of surfaces 3 and 15; the presence of the hole in the center of surface 2; E a lower value has been given to other grasps with two-jaw grippers characterized by a GCP not positioned on the axis of the object. As examples taken among them, the grasp performed on #at surface 8 and cylindrical surface 5 is considered better than the grasp performed on #at surfaces 7 and 9. This fact can be explained by the di!erent orientation of these grasps with respect to the object axis: the former has p "1 and p "1 and therefore presents a more stable gras ping, the latter has p "0 and p "0 which can increase the possibility of slippage during grasping due to the weight of the object; E grasps performed by magnetic devices, except that one acting on surface 2, have not received a good evaluation due to their small and slender area as well as their distance from the axis of the object; E the same reasoning can be done for grasps performed by vacuum grippers. All these grasps present acceptable or bad evaluation due to their small and slender area as well as their distance from the axis of the object. In addition, surface 2, although it is centered with respect to the axis, has a big hole which prevents a good evaluation by the system.
Fig. 8. Case study: a valve housing.
Table 6 Grasps detected by the system for the object illustrated in Fig. 7 and sorted in order of importance No. of grasp
Grasping surfaces
Type of gripper
Evaluation by ANNs
1 2 3 4 5 6 7 8 10 11 12 13 14 15 16 17 18 19 20 21 22
1}1 5}5 3}15 2 8}5 12}5 5}3 5}15 8}12 1}5 10 14 10 14 7}9 11}13 4 16 4 16 2
Two-jaw Two-jaw Two-jaw Magnetic Two-jaw Two-jaw Two-jaw Two-jaw Two-jaw Two-jaw Magnetic Magnetic Vacuum Vacuum Two-jaw Two-jaw Magnetic Magnetic Vacuum Vacuum Vacuum
Optimum Good-optimum Good Good Quite good-good Quite good-good Quite good Quite good Quite good Acceptable-quite good Bad-acceptable Bad-acceptable Bad-acceptable Bad-acceptable Bad Bad Bad Bad Bad Bad Bad
G. Dini, F. Failli / Robotics and Computer Integrated Manufacturing 16 (2000) 451}463
The results shown in Table 6 can be post-processed according to di!erent instances concerning task requirements. If, for example, a high-precision positioning of the object is requested, the system re-arranged the outputs penalizing those solutions which do not guarantee a high accuracy such as the grasps performed by magnetic or vacuum grippers.
463
Acknowledgements The authors would like to express sincere thanks to Prof. B. Lazzerini and Dr. F. Marcelloni of the Department of Information Engineering of the University of Pisa for their technical support, and to Dr. M. Montanari for the active contribution to this work. References
7. Concluding remarks The feasibility of implementing a neural algorithm in a computer-aided grasp planning system has been successfully demonstrated. This new approach, in particular, gives the chance to acquire the technological knowledge without using complicated heuristic rules, usually di$cult to be formulated and managed, but presenting to the system a set of grasping examples together with their evaluation given by a human expert. This aspect, although representing the most evident advantage of the proposed technique, at the same time is the most critical point: it is important to pay attention on the selection of proper examples which have to be focused to a given technological and geometrical domain and equally distributed within it. The procedure proposed to train and validate the ANNs works in this direction allowing, through its iterative process, a self-calibrating detection of the training set size. Future developments of this research project will follow two main directions: improvement of system performances and integration of the system in a more complex assembly planning package. The former could include improvements concerning some aspects such as capability to deal with more complex object surfaces than those ones considered in the system (at present, only #at, cylindrical and curved surfaces are taken into account); extension of surface combinations considered for each type of gripper; capability of grasping on edges or vertices of the object. The latter could involve aspects related to the selection of gripper features (dimensions, "nger strokes, grasping force, "nger-tip shape, etc.) and integration problems related to sequencing and path planning activities, such as selection of grasps according to assembly sequence of components in order to minimize gripper changes; evaluation of the exact "nger-tip positions on selected surfaces; veri"cation of surface accessibility by means of collision check and path planning procedures.
[1] Kawarazaki N, Hasegawa T, Nishihara K. Grasp planning algorithm for a multi-"ngered hand-arm robot. Proceedings of IEEE International Conference of Robotics and Automation, 1998. p. 933}9. [2] Bekey GA, Liu H, Tomovic R, Karplus WJ. Knowledge-based control of grasping in robot hands using heuristics from human motor skills. IEEE Trans Robotics Automat 1993;9(6):709}22. [3] Xiong YL, Sanger DJ, Kerr DR. Geometric modelling of bounded and frictional grasps. Robotics 1993;11:185}92. [4] Montana DJ. The conditions for contact grasp stability. Proceedings of IEEE International Conference on Robotics and Automation, 1991. p. 412}7. [5] Wolter JD, Volz RA, Woo AC. Automatic generation of gripping positions. IEEE Trans Systems Man Cybernet 1985;SMC15(2):204}13. [6] Dini G. A module for the automated planning of grasps in robotized assembly operations. Ann CIRP 1993;42(1):1}4. [7] Jones JL, Lozano-Perez T. Planning two-"ngered grasps for pickand-place operations on polyhedra. Proceeding of IEEE International Conference, 1990. p. 683}8. [8] Terasaki H, Hasegawa T, Takahashi H, Arakawa T. Automatic grasping and regrasping by space characterization for pick-andplace operations. Proceeding of IEEE/RSJ International Workshop on Intelligent Robots and Systems, 1991. p. 58}63. [9] Dubey A, Nnaji BO. Task-oriented and feature-based grasp planning. Robotics Comput Integrated Manuf 1992;9(6):471}84. [10] Boubekri N, Ramanujam G. Design of grasping methodologies for rotational assembly components. Int J Flexible Manuf Systems 1995;7:373}88. [11] Pham DT, Yeo SH. A knowledge-based system for robot gripper selection: criteria for choosing grippers and surfaces for gripping. Int J Mach Tools Manuf 1988;28(4):301}13. [12] Canzi U, Cugini U, Rigillo F. Kali: a knowledge-based system for grasp planning. Proceedings of 5th International Conference on Advanced Robotics, 1991. [13] Pham DT, Tacgin E. Grippex: a hybrid expert system for selecting robot gripper types. Int J Mach Tools Manuf 1992;32(3):349}60. [14] Dagli CH, editor. Arti"cial neural networks for intelligent manufacturing. London: Chapman & Hall, 1994. [15] Wasserman PD. Advanced methods in neural computing. New York: Van Nostrand Reinhold, 1993. [16] Chen S, Cowan CFN, Grant PM. Orthogonal least squares learning algorithm for radial basis function networks. IEEE Trans Neural Networks 1991;2(2):302}9.