Assembly simulations in virtual environments with optimized haptic path and sequence

Robotics and Computer-Integrated Manufacturing 27 (2011) 306–317

Contents lists available at ScienceDirect

Robotics and Computer-Integrated Manufacturing journal homepage: www.elsevier.com/locate/rcim

Full length Article

Assembly simulations in virtual environments with optimized haptic path and sequence Christiand, Jungwon Yoon n School of Mechanical and Aerospace Engineering and ReCAPT, Gyeongsang National University, Jinju, Republic of Korea

a r t i c l e in f o

a b s t r a c t

Article history: Received 12 September 2008 Received in revised form 2 July 2010 Accepted 27 July 2010

Virtual assembly and disassembly simulations can be accomplished in intuitive and effective ways using haptic information in virtual environments (VEs). Potential problems in a given assembly scheme can be predicted by a user who may be able to suggest an alternative scheme in the VE. This paper describes an intelligent virtual assembly system in which an optimal assembly algorithm is used to allow haptic interactions during virtual assembly operations. This algorithm provides optimal paths for haptic guidance as well as an assembly sequence of the parts to be assembled. The performance of the given assembly schemes was simulated using a virtual assembly system. Experimental results showed that the haptic-path sequence-guidance (HSG) mode gave the best performance improvement in terms of accumulated assembly time (28.33%) and travel distance (15.05%) compared to the unguided mode, while the sequence-guidance (SG) mode alone increased performance by 15.33% for assembly time and 11.36% for travel distance. The experimental results were analyzed by the sub-tasks of gripper selection, inter-part movement, and part assembly. For the HSG mode, the greatest contributor to the time and distance reductions was the optimized haptic path, while for the SG mode, the reduced numbers of gripper exchanges and orientation change made the greatest contributions to reducing the assembly time and the travel distance. As a result, the optimized haptic path, as well as sequence guidance, enhanced the working performance of virtual assembly tasks. & 2010 Elsevier Ltd. All rights reserved.

Keywords: Assembly/disassembly (A/D) Haptic guidance Path planning Virtual reality Sequence optimization

1. Introduction Assembly planning can be simulated by means of virtual reality (VR) technologies to increase the efficiency of a real assembly process. In this way, potential problems in a given assembly scheme can be predicted by a user who may be able to suggest alternative methods in virtual environments (VEs). Since physical objects are represented as virtual objects inside VEs, real objects are not necessary for the simulation of a process. From this point of view, VR technologies have definite benefits in terms of cost and time. Several VR-based applications for assembly planning purposes have been proposed. Jayaram et al. [1] developed virtual assembly design environment (VADE), a VR-system that is capable to evaluate the assembly planning for design and manufacturing. This system gives not only features for standard assembly planning simulation but also other features such as dynamic simulation of parts and dexterous manipulation with a haptic glove. As the result, the assembly planning simulation could be held in detail and intuitive ways. Yand et al. [2] and Yao et al. [3]

n

Corresponding author. Tel.: +82 55 751 6078; fax: + 82 55 762 0277. E-mail address: [email protected] (J. Yoon).

0736-5845/$ - see front matter & 2010 Elsevier Ltd. All rights reserved. doi:10.1016/j.rcim.2010.07.015

have developed successful VR-based simulation systems that feature interactive constraint recognition and parts paths to assist the virtual assembly work. Their works were emphasized on the detail of constraint recognition and management in virtual assembly (VA). Through these systems, assembly sequence planning can be established after the assembly planner conducts several assembly simulations to verify the overall process of the specific assembly work. Raghavan et al. [4] has also developed an assembly planning simulator using an augmented reality (AR)-based application. During the assembly simulation, the assembly planner interacts with real assembled parts while a user can watch a virtual guidance image from a head-mounted device (HMD). Since physical objects are also involved in the system, this has the benefit of combining physical and VEs in the assembly planning process. On the other hand, changing the assembly simulation is not easy since the physical objects comprising the prototype must always be provided. Ye et al. [5] and Chryssolouris et al. [6] discussed the performance of VR-based assembly planning systems based on assembly time evaluation. Ye et al. evaluated the quality data in assembly planning, including items such as the number of difficult assembly operations and dissimilar assembly operations. The performance of VR-based assembly planning systems was compared to traditional manual physical assembly task simulations.

Christiand, J. Yoon / Robotics and Computer-Integrated Manufacturing 27 (2011) 306–317

Two studies [5,6] concluded that a VR-based assembly planning system improves the performance efficiency. However, most of VR assembly planning systems neither provide sufficient object tangibility for active assembly help nor evaluate assembly performances with optimal assembly schemes for guiding the best assembly conditions for a user. The growing number of haptic technology advances in the field of VR is contributing to the development of new haptic applications in such diverse fields as medical diagnosis, surgical simulation, and games. Through haptic interfaces, users have a sense of tangible reality of the virtual objects they interact with in VEs. Borro et al. [7] developed a haptic system for an aeronautical maintenance simulation with a haptic device that ensured the users’ movements were the same as those that occur when testing a physical mock-up. A similar study was described by Savall et al. [8], while Kuang et al. [9] focused on building a virtual fixture assembly language. This library could be used to construct various virtual fixture series with graphic and force guidance rules, making the low-level haptic and graphic rendering details transparent to the developers. For more active haptic guidance as well as object tangibility in the field of medical haptics, Bettini et al. [10] successfully developed a haptic guidance application for microsurgical purposes. Their mechanism guides users along a path by making the movements parallel to the path easier than movements in a perpendicular direction. Teo et al. [11] have developed a 6-degree-of-freedom (DOF) haptic interface for teaching Chinese handwriting. Moreover, haptic guidance can be applied to each mechanical part during a maintenance simulation to use haptic information more fully for an assembly and disassembly (A/D) process [12]. This means that A/D simulation can be performed in ways that are more intuitive and effective. To improve the performance of a virtual assembly simulation, the system itself can have intelligent algorithms so that the users are able to optimize the assembly tasks. Several Refs. [13–15] describe various techniques to achieve the optimal A/D sequence. In those methods, the number of gripper exchanges and the number of orientation changes were typically the basis for determining an optimal solution through an evaluation process. Several researchers [16–18] have also shown many valuable results for optimal path planning of robotic manipulators. Bobrow [16] and Shiller and Dubowsky [17] has proposed a path planning optimization for a manipulator robot which is optimal in time domain. Garg and Kumar [18] have paid great attentions on genetic algorithm (GA) to develop an optimal trajectory for a multiple robotic configuration. However, it is difficult to find optimization schemes to interconnect sequence and path planning optimizations. The optimal assembly algorithms presented in the past did not consider the path for parts assembly and vice versa. This makes haptic guidance difficult to apply in assembly simulations because there is no way to combine the haptic information with the simulation. VR-based applications, such as a virtual assembly tasks, require more information about path planning to use haptic guidance in an intelligent manner. To cope with this problem, we suggested a new assembly algorithm [19], which can generate simultaneously optimal paths as well as an optimal assembly sequence for a haptic guidance application. Based upon the VE we have developed, an assembly operator can simulate the process of moving parts from their initial to their final positions following an assembly sequence with the haptic guidance. This enhances the efficiency of an assembly task in terms of time and energy. In this paper, we describe an enhanced assembly simulation system in intelligent VEs with optimized haptic paths and sequences. The proposed simulation system allows users to use a haptic interface in an A/D simulation, and allows a user to be effectively guided by an optimized haptic path and an optimal

307

sequence. In this research, our main objective is to enhance the assembly works by the optimal assembly algorithm while the haptic sensation is used as a means to increase the user’s reality of virtual object and to prove the effectiveness of the suggested system through extensive user studies. For these purposes, the paper content is divided into six sections. We introduce the virtual assembly simulation system in Section 2. Section 3 describes the proposed optimization scheme for intelligent haptic interaction. Section 4 describes user studies of a virtual assembly simulation. Section 5 discusses the suggested system, while Section 6 contains our conclusions and a discussion of future work.

2. Intelligent virtual assembly simulation system 2.1. System overview Since the ability of assembly operators to perform repetitive assembly tasks depends on their experience, they must be familiar with the basic skills of the tasks themselves. Due to the advances of VR technologies, VE simulations in a manner similar to the physical assembly work will eliminate the need for a physical model and make it easier to change the simulation scheme. Our intelligent virtual assembly simulation system was built to enhance the benefits of virtual assembly simulations. The proposed system consists of a VE, a haptic device, and a user, as shown in Fig. 1. The VE includes manipulated objects (i.e., virtual objects inside the VE), a graphics engine, and a haptic rendering engine. The files for manipulated objects are generated using a CAD software package and converted to object files, so that they can be read by the graphics engine. 2.2. Virtual environment for assembly simulation The VE for the assembly simulation consists of obstacles, parts in initial positions, final position marks, and suggested paths, as shown in Fig. 3. All the objects in the VE are virtual objects. The existence of virtual objects in the VE depends on several modes of assembly simulation that will be explained in Section 4. Even though the workspace is three-dimensional (3D), the assembly task is based on planar assembly with the final arrangement of parts, as shown in Fig. 2. To add a constraint for user movements

Fig. 1. Virtual assembly simulation system overview.

Fig. 2. Assembled part.

308

Christiand, J. Yoon / Robotics and Computer-Integrated Manufacturing 27 (2011) 306–317

Fig. 3. GUI components and VE simulation (unguided mode).

in the Z + direction (toward the user’s face), an invisible object is placed parallel to the floor at some offset distance (Fig. 3). Thus, the pointer can move only in the space between the floor and the invisible object. This scheme was chosen due to the special assembly case that is limited to planar assembly (2D). Adding constraint on Z + direction makes user motion nearly planar motion (X, Y), which will allow easier analysis with the absence of user motion on Z + direction. 2.3. Haptic interfaces The user communicates with the VE using Sensable’s Phantom Omni, a 3-DOF haptic device that provides six inputs (force and torque components) and three outputs (force components). In the VE, the haptic interaction point (HIP) is represented by the user-controlled pointer that is represented as a sphere. Since we use only point-to-mesh haptic rendering, users can feel a haptic sense through the pointer. When the user’s HIP reaches the region of a part’s initial point, a snapping mechanism is activated to attach the part to the HIP. At the same time, the center point of the manipulated object is automatically located in line with the HIP at some offset distance, as shown in Fig. 3. 2.4. Haptic rendering software We used the CHAI3D application programming interface (API) [20] for the graphics and haptic rendering engines. CHAI3D provides the most basic functions for developing haptic applications, including collision detection and haptic rendering. The graphical user interface (GUI) was designed to provide an interface between

the operator and the VE (Fig. 3). The operator can manipulate objects in a window that contains the virtual assembly scheme. The haptic rendering loop runs at 1 kHz. 2.5. System algorithm As shown in Fig. 4, there are two types of processes inside the system: a real-time process and an offline process. The offline process deals with assembly optimization. The data needed for an optimal assembly algorithm are the part geometries and the gripper associated with each part. Outputs of the assembly algorithm are used to construct the path for the haptic guidance and the assembly sequence that will be explained in Section 3. The real-time process includes a haptic rendering loop and an assembly procedure. The haptic rendering loop provides haptic feedback during the simulation process to make all virtual objects tangible. The assembly procedure ensures that the assembly task is performed with an optimal assembly path and sequence. 2.6. Virtual assembly simulation scheme During virtual assembly operations, the operator must move parts from their initial positions to their final positions according to the optimal assembly sequence. The operator is guided by a haptic guidance force along a prescribed path. The assembly sequence and its corresponding prescribed path are generated by an offline optimal assembly algorithm (see Section 3). The prescribed path is represented as a solid object in the shape of the path with boundaries on either sides of the path. The operator must ensure that the parts pointer (Fig. 3) always remains inside the path

Christiand, J. Yoon / Robotics and Computer-Integrated Manufacturing 27 (2011) 306–317

309

assembly sequence has a significant impact on the assembly time for both robot and human operator assembly processes. Moreover, path optimization can influence the efficiency of the overall assembly process. Obstacles and locations of the parts can be considered as path-planning factors. In an assembly process, the paths that are generated from a formal path-planning analysis contribute greatly to the efficiency of an assembly task since a long path consumes more travel time and energy. Therefore, the path-planning optimization formulization seeks an optimal setting for the path-planning variable. We selected the repulsive force value at the boundary (r0) as our path-planning variable and implemented the potential field method [21] described in Section 3.2. The correlation between the path planning and the assembly sequence must also be considered in the form of assembly rules implementation [19] (see Section 3.3). Finally, we combined path-planning optimization with assembly sequence optimization. The cost function components were the number of gripper exchanges (G), the number of assembly direction changes (O), and the actual distance of the path (Dact). The number of gripper exchanges and direction changes contributes to the assembly sequence optimization, while the distance of the path for each part contributes to the path-planning optimization. 3.2. Path planning of the parts for haptic guidance

Fig. 4. System algorithm flowchart.

boundary to avoid collisions with obstacles. When the operator begins to move away from the path boundary, a haptic guidance force is generated to ensure that the operator’s movement stays inside the path boundary. Haptic guidance exists in the form of a haptic reaction force when the collisions occur between the pointer and the path boundary. This scheme enhances the efficiency of the assembly process by guiding the assembly operator to follow the desired path with haptic feedback. We used the potential field method [21] for the path-planning algorithm. Repulsive and attractive forces were combined to create a path that avoids obstacles and guides the part to its final position. To generate both the optimal assembly sequence and the path simultaneously, an optimal assembly algorithm was developed using a simple genetic algorithm (GA) [22].

A part is represented as a polygonal object with its center located at the center of the object boundary box (2D object), as shown in Fig. 5. Even though 3D parts will more closely approximate a real assembly scenario, the framework for, and the efficiency of, the proposed assembly optimization can be verified through the movement of 2D parts. The axis-aligned boundary box method is used to create the boundary box. The positions of a part and its boundary box in mobile frame o1 with respect to the reference frame o0 can be represented as the transformed points of part vertexes using a 2D homogenous transformation matrix. The configuration of the parts is represented by q¼ [dxi dyi]. The potential field method uses simple design variables and is adequate for real-time applications. This method makes use of attractive and repulsive forces; the attractive force pulls the part toward its final position while the repulsive force repels the part from a collision with obstacles. The repulsive and attractive forces are represented as follows:if rðoi ðqÞÞ r r0 , then   1 1 1  rrðoi ðqÞÞ ð1Þ Frep,i ðqÞ ¼ Zi rðoi ðqÞÞ r0 r2 ðoi ðqÞÞ else if r(oi(q))4 r0, then Frep,i ðqÞ ¼ 0

ð2Þ

3. Optimal assembly algorithm for the haptic path 3.1. Effective factors for assembly optimization The optimization of the assembly sequence represents the problem of finding the part sequence that will bring about the most efficient assembly process. Previous studies considered the number of gripper exchanges and direction changes of mechanical parts during assembly or disassembly processes [13,14,23]. A higher number of gripper exchanges requires an increased time to change from one gripper to another. Similarly, many direction or orientation changes during assembly tasks require more effort to perform. One of the optimization objectives is to determine the optimal assembly sequence that has the least number of gripper exchanges and orientation changes. Pan and Smith [24] showed that the number of reorientations in an

Fig. 5. Parts and obstacles composed of polygonal objects in a 2D workspace.

310

Christiand, J. Yoon / Robotics and Computer-Integrated Manufacturing 27 (2011) 306–317

where q is the configuration of the parts, Frep the repulsive force, r0 the repulsive force radius, oi(q) the point on the workspace, Zi the scale factor, and i the index of the ith part. If Joi ðqÞoi ðqf ÞJ rd, then Fatt,i ðqÞ ¼ zi ðoi ðqÞoi ðqf ÞÞ

ðoi ðqÞoi ðqf ÞÞ :ðoi ðqÞoi ðqf ÞÞ:

Rule 1. ‘‘A part that has reached its final position will become an obstacle for the next part’’.

ð3Þ

else if :oi ðqÞoi ðqf Þ: 4 d, then Fatt,i ðqÞ ¼ dzi

directions when parts enter the final position area. The following are the assembly rules and their descriptions:

ð4Þ

where qf is the final configuration of a part at its final position, d the distance from the current position to the final position, z the scale factor, Fatt the attractive force. The summation of the attractive and repulsive forces gives the direction for movement as a normalized vector: ! Fatt þFrep US ð5Þ f ¼ :Fatt þFrep : ! where f is the force applied to move a part, S the step between path points The assembly operator will follow the path guidance from the ! part’s initial position to its final position (Fig. 6). The vector f can be directly used to produce path guidance. We use the resulting path to construct the guidance path in the form of solid objects. The repulsive force has a conditional value r(oi(q)), which is the distance from the center of a part to the center of an obstacle. The assignment of repulsive forces from obstacles must be handled carefully. During an assembly operation, user wants to have an optimal path for the best efficiency. Therefore, it is necessary to know how to plan optimal path for every single part in one assembly sequence, so that the total distance is optimal. To achieve an optimal path, the value of r0 must not be too large or too small. The value of r0 in Eqs. (1) and (2) will affect the overall results of the path-planning algorithm. The value of r0 must be predetermined to achieve an efficient assembly process by optimization. 3.3. Assembly rules and gripper selection We proposed new assembly rules [19] to establish a relationship between the assembly sequence and path planning. One example of an optimal assembly of parts is shown in Fig. 3 (cyan). The given workspace consists of parts in their initial positions (red), marks of final part position areas (cyan), obstacles (green), and floor (brown). The path is represented as a solid object (yellow cylinder) that is developed from the path points. In path planning, the assembly rules control the part movements, repulsive forces, and assembly

This rule ensures that an arriving part will not collide with the previous part when it enters the final position area. The path of each part will be directly affected by the accumulated repulsive forces that are generated based on the other parts that have already arrived. Since the path depends on the assembly sequence, the final path for the sequence will depend on which parts have already arrived. Rule 2. ‘‘After selecting a possible assembly direction, the part must move in that direction when assembled to avoid interference between parts’’. In an assembly process, each part must have a predefined assembly direction to prevent interference between parts. A connection matrix (Table 1) is used to select a feasible assembly direction by observing the relationship between successive parts. As shown in Fig. 3, the initial positions of all the parts were located in the Y-direction and the search order for the assembly direction was selected using a counterclockwise rule in the following order: Y  , X +, Y+ , and X  . In Table 1, 0 indicates that the part can be assembled in the given direction, while 1 indicates that the part cannot be assembled in the given direction because other parts will interfere with the movement. The proposed direction-finding algorithm only considers the directions of the principal axes of the part, which is sufficient for this case. In numerical calculations, the assembly directions were represented as integer values: X + ¼1, Y + ¼2, X  ¼3, and Y  ¼ 4. Rule 3. ‘‘If the path point (oi) is perpendicular to the face of the part in a given assembly direction during the path search, all repulsive forces that are generated from existing parts will disappear automatically’’. If the final position of a part is located inside the radius of the accumulated repulsive force, the next part cannot reach its final position. Hence, the existence of a repulsive force in the final area must be controlled by the third assembly rule. Parts can reach their final position if the third rule is followed. The gripper that is used to grasp and move a part from one position to another is also an important factor in a real assembly process. We assumed that four types of grippers are used. The Table 1 Connection matrix. X+

Y+

Parts

1

2

3

4

5

6

Parts

1

2

3

4

5

6

1 2 3 4 5 6

0 0 0 0 0 0

1 0 0 0 0 0

1 1 0 1 0 0

0 0 0 0 0 0

0 0 0 1 0 1

0 0 0 0 0 0

1 2 3 4 5 6

0 1 1 0 0 0

0 0 1 0 0 0

0 0 0 0 0 0

0 0 1 0 1 0

0 0 0 0 0 0

0 0 0 0 1 0

X

Fig. 6. Direction vector for the next path point (oi + 1) toward the final position.

Y

Parts

1

2

3

4

5

6

Parts

1

2

3

4

5

6

1 2 3 4 5 6

0 0 0 0 0 0

1 0 0 0 0 0

0 1 0 0 0 0

1 1 0 0 0 0

1 1 1 1 0 0

1 0 0 1 0 0

1 2 3 4 5 6

0 1 0 1 1 1

0 0 1 1 1 0

0 0 0 0 1 0

0 0 0 0 1 1

0 0 0 0 0 0

0 0 0 0 0 0

Christiand, J. Yoon / Robotics and Computer-Integrated Manufacturing 27 (2011) 306–317

gripper assigned to each part is indicated as G1, G2, G3, or G4. In some cases, it is possible to use the same gripper for several parts; this possibility must be considered in the optimization. The gripper used for each part is determined by checking a gripper table that corresponds to the assembly sequence. The grippers are listed in Table 2. Grippers in column P1 are preferable to those in column P2. A planner will use a gripper in column P1 if the previous gripper has no similarity or is unsuitable for the current part. For example, if the sequence of parts is [623 451], the G2 gripper from column P1 will be used for Part 6. Then we have two gripper options for Part 2: G1 and G2. Since G2 was used for the previous part, it will also be used for Part 2. If a single gripper cannot be used for both Parts 6 and 2, gripper G1 will be selected from column P1 for Part 2.

3.4. Optimization assembly algorithm incorporating the assembly sequence and path planning factors Our optimization problem consists of finding the optimal sequence and appropriate repulsive force value r0 to achieve an efficient assembly task under the condition that the optimized variables (optimal sequence and repulsive force radius) satisfy the assembly rules. Fig. 7 shows the relations between all components in optimization. The assembly sequence (the order when we assemble the parts) affects the following three variables: (1) number of gripper exchanges, (2) number of orientation, and (3) total path distance. In particular, the assembly sequence affects the repulsive force at final position that finally affects the total path distance as well. This is regulated by our established rules (assembly rule no. 1). In addition, the total path distance is also affected by r0 value. Thus, we can get optimal solution of combined optimization (assembly sequence optimization + r0 optimization) by defining a cost function that is based on number of gripper exchanges, number of orientation, and total path distance. The optimization method used the simple GA introduced by Holland [22] that considers optimized variables as a chromosome; Table 2 Gripper matrix. Part

P1

P2

1 2 3 4 5 6

G1 G1 G3 G3 G3 G2

G2 G2

Fig. 7. Relationships between all components in optimization.

G4

311

in our work, the population consisted of 50 chromosomes. A cost value was calculated for each chromosome in the population using   Pn   o  G i ¼ 1 ððDref Þi Þ=ððDact Þi Þ Cost ¼ w1 1 þw2 1 þ w3 n1 n1 n ð6Þ where w1, w2, and w3 are the weighting factors. n is the number of parts. The actual path distance (Dact) is the sum of the single distances from one path point oi to another to reach the final position. Dref is the shortest distance from the initial position to the final position. A comparison of Dact with Dref gives the efficiency of the path. The number of orientation changes was based on how many times the assembly directions changed in one sequence. The number of gripper exchanges was based on how many times a planner changed the gripper. For the path planning optimization, the size of the tools is not considered. Only the obstacles and part sizes (bounding box) are considered. We do not use any particular collision detection method to guarantee the path is collision free. Instead of using collision detector, our optimization scheme only finds the optimal r0 value in the bounded range that has minimum and maximum value. The minimum (r0 lower) and maximum (r0 upper) bounds are calculated by taking into account the parts and obstacles size (bounding box) as shown in Fig. 8. For every single part, there is a unique bounded range which is assigned to that part. This unique bounded range is the solution space for r0. When we do the path planning for one part, the followings procedure/calculation is repeated: (i) The minimum bound is calculated by finding the longest distance among all distances between part’s bounding boxes and obstacle’s bounding boxes. (ii) The maximum bound is calculated by finding the shortest distance among all the half distances between all obstacles. The minimum and the maximum bounds of the above optimization constraint guarantees collision-free path planning and can reduce local minima occurrence during the part movement from initial position to final position, respectively. In one assembly sequence, the sequence of part corresponds to the sequence of r0. Then, the optimal solution (best assembly sequence and its r0) is found through the GA process. For one part, there is one unique r0 which will be applied to all obstacles in the environment and to create the repulsive forces for that part. If there is more than one obstacle, automatically the part will get the accumulated repulsive force from the surrounding obstacles to which part’s position is inside their repulsive area. All the optimization was performed as an offline process following the sequence shown in Fig. 9.

Fig. 8. Solution space of r0 optimization.

312

Christiand, J. Yoon / Robotics and Computer-Integrated Manufacturing 27 (2011) 306–317

3.5. Optimal assembly results Two optimization processes were performed by varying the weighting values on the cost function to determine the contributions of the assembly sequence and path planning objectively. In Eq. (6), the weighting values on the first optimization process were set to w1 ¼ 0.5, w2 ¼0.5, and w3 ¼0 to implement pure sequence optimization. The weighting values on the second optimization process were set to w1 ¼0.25, w2 ¼0.25, and w3 ¼0.5 to consider both the sequence and path planning. The results of these optimization processes are shown in Table 3. The first optimization only used the number of gripper exchanges and assembly orientation changes to determine the best sequence for the assembly process. Therefore in the optimized results, the number of orientation changes, O, is 0 and the number of gripper exchanges, G, was 2, which resulted in the sequence [6, 5, 3, 4, 2, 1]. On the other hand, since the second optimization result used the path information (r0) as well as the number of gripper exchanges and orientation changes, O and G both increased to 3. The second optimization results, however, can be used to allow haptic guidance during the assembly of mechanical parts. The path planning optimization results for the second optimization were obtained for the 6 parts, as shown in Fig. 10. Using this information, the assembly operators can be guided through the optimal assembly task in the VE. The optimal r0 values are used to

develop the optimal path, while the optimal sequence contributes to optimizing the numbers of gripper and assembly orientation changes during the assembly task. The quantitative effects of two optimal solutions will be explained in the user-studies section (Section 4) with real-time virtual assembly simulations.

4. User studies for the virtual A/D system 4.1. Modes for user studies 4.1.1. Virtual assembly procedure An experiment was designed to evaluate the effectiveness of the intelligent virtual assembly simulation with haptic path and sequence guidance from the suggested optimal assembly results. The process of moving parts from initial to final positions was the basic A/D task to be performed by the assembly operators participating in the experiment. Simulation schemes were prepared in three modes to evaluate the A/D task with the assistance of sequence and path planning (haptic guidance). The modes were classified as unguided (U), sequence-guidance (SG), and haptic sequence-guidance (HSG). The operators followed four main assembly steps in each mode: (1) choosing a part, (2) choosing a gripper, (3) selecting the part, and (4) assembling the part. When performing the A/D task, the operator used certain assistance features, such as the optimized sequence of parts with guidance for gripper selection and assembly orientation, and the optimized assembly path. The availability of the help features depended on the selected modes.

Fig. 10. A path planning optimization result using the second optimization scheme.

Fig. 9. Optimization process flowchart.

Table 3 Optimization results. A. Result of the first optimization process (w1 ¼ 0.5, w2 ¼0.5, w3 ¼ 0) Sequence

Orientation

O

Grippers

G

6, 5, 3, 4, 2, 1

4, 4, 4, 4, 4, 4

0

G2, G3, G3, G3, G1, G1

2

B. Result of the second optimization process (w1 ¼0.25, w2 ¼0.25, w3 ¼0.5) Sequence

r0

Orientation

O

Grippers

G

1, 2, 3, 5, 6, 4

3.69, 3.41, 4.03, 4.41, 3.69, 3.64

4, 1, 1, 1, 2, 3

3

G1, G1, G3, G3, G2, G3

3

Christiand, J. Yoon / Robotics and Computer-Integrated Manufacturing 27 (2011) 306–317

4.1.2. Unguided (U) mode In this mode (Fig. 11), the operator selects and moves the parts one-by-one to their final positions without any guidance information (i.e., no help features). All parts appear in the workspace simultaneously. To select a part, the operator is required to choose an appropriate gripper from the available selection. Once the user selects a gripper, the constraints of the optimization process follow. Based on the selected sequence, the user does not need to make a gripper selection if the current gripper can handle the next part. The possible orientation directions from Table 1 will be appeared to a user by multiple arrow-shaped indicators for satisfying feasible assembly between parts. During assembly operations, the part should also avoid collisions with any objects in the VE. A haptic force is generated to the HIP point to prevent parts from passing through obstacles. In this mode, the assembly path and orientation are fully under the operator’s control without any assistance from the help features. The snapping-into-place mechanism when a part reaches its final position is the only extra feature available.

4.1.3. Sequence-guidance (SG) mode In this mode (Fig. 12), the optimized sequence and guidance for gripper selection and assembly orientation are provided to the operator as help features. The results of the first optimization process from Section 3.5 are used in this mode as help features. The operator simply selects the parts and performs the assembly tasks based on the results of the optimal assembly sequence in Table 3A. The suggested assembly sequence affects the order of each part’s appearance in the VE as well as the gripper suggestion. After a part appears in the VE, the operator should choose a gripper. In this mode, the operator does have to use the recommended gripper to increase the possibility of using the same gripper for several parts that we had identified in advance. In this mode, the optimal orientation is also suggested during the assembly operation by an arrow-shaped indicator as shown in Fig. 12. The operator can assemble the part by orienting it as indicated by the arrow-shaped indicator. The assembly sequence is implemented such that only one part appears to a user for each

313

assembly task. After the assembly of one part is complete, another part will appear in its initial position. This happens repeatedly until the assembly of all parts is complete. 4.1.4. Haptic sequence-guidance (HSG) mode In this mode (Fig. 13), the help features (optimized sequence of parts, gripper suggestion, and assembly orientation) are similar to those of the SG mode, with the addition of the optimal assembly path. The results of the second optimization process in Table 3B are used in this mode. Similar to the SG mode, parts appear one at a time in the VE based on the optimal sequence, and are accompanied by the suggested path. For each part, the operator chooses the suggested gripper by touching it. During part assembly operation, the operator follows the path to assemble a part, orienting it as shown by the arrow-shaped indicator. After finishing assembling one part, the operator uses the optimal path to return to an initial position for the next part. The path is constructed in the form of a solid-body object in the VE based on the result of the optimal assembly algorithm. Movement is limited to the path boundary. Whenever movement begins to cross the path boundary, the operator feels haptic reaction forces between the pointer and the path boundaries. This haptic reaction feedback is expected to direct the operator’s movements to the optimal assembly path. This scheme for passive haptic guidance with the optimized path planning results is considered as the optimized haptic path. 4.2. Assembly evaluation parameters Based on the assembly optimization schemes, we measured the assembly time and travel distance as final outputs of the virtual assembly simulation of the participants in our experiment using the method shown in Fig. 14. The participants included 8 men and 4 women, aged 25–29 years. The assigned task (assembly case) was a planar assembly of the objects shown in Fig. 2. Each participant performed the assembly task in the three modes. The total assembly time and total distance traveled were measured for each participant during the virtual assembly

Fig. 11. Unguided mode.

314

Christiand, J. Yoon / Robotics and Computer-Integrated Manufacturing 27 (2011) 306–317

Fig. 12. Sequence-guidance mode.

Fig. 13. Haptic sequence-guidance mode.

operations. The total assembly time was the time to finish the assembly tasks of all parts, including gripper selection and part transition. The total travel distance was the overall distance traveled by the user from the initial to the final positions for all the parts, including any additional movements for gripper selection and parts transition. A short total assembly time and small total travel distance are desirable for effective assembly. Each participant was allowed to try the system several times to become familiar with the VE and the assembly algorithm process.

4.3. Evaluation results 4.3.1. Evaluation of overall system performance Fig. 15 shows that the performance of each participant depends on the mode. The results in Table 4 are the average values for all participants. We used the U mode results as the base value against which to compare the others. Compared to the U mode, the time to finish the 6 assembly tasks was reduced by 15.53% for the SG mode and 28.33% for the HSG mode. The average distance traveled was reduced by 11.36% for the SG mode

Christiand, J. Yoon / Robotics and Computer-Integrated Manufacturing 27 (2011) 306–317

Fig. 14. Measuring the performance of the intelligent virtual assembly system.

Fig. 15. Evaluation results for each participant.

Table 4 Participant performance.

Items Total Std. Dev. Improvement (%) Gripper Orientation Average Velocity

Mode SG D

U T

D

T

32.21 6.23 -

566.01 39.72 -

27.21 4.71 15.53

501.73 35.73 11.36

HSG T

D

23.09 3.66 28.33

480.82 31.20 15.05

3

2

3

3 17.57

0 18.44

3 20.83

* Improvement is shown compared to nominal value (U); T = time (s), D = distance (units)

315

and 15.05% for the HSG mode. In addition, standard deviations were reduced as the mode progressed from U to SG to HSG, which means that the performance results of participants in the HSG and SG modes were more uniform than in the U mode. In the U mode, since the participants were likely to choose a non-optimal sequence and a non-optimal path, the numbers of gripper exchanges and orientation changes were larger than in the SG mode, and the assembly time and the distance traveled were longer than in the HSG mode. In the HSG mode, the participants tended to use a higher velocity for the assembly task due to the optimized haptic path and the predetermined sequence. This can be observed in the comparison of average velocities, as shown in the last column of Table 4.

4.3.2. Evaluation analysis To illustrate the differences of the contributing components in the three modes (U, SG, and HSG) clearly, we divided the assembly task into three sub-tasks: gripper selection, inter-part movement, and part assembly. Fig. 16 shows the sub-tasks in the virtual assembly simulation, and Table 5 summarizes the average operator performance for each. Table 5 shows that for gripper selection, the SG mode gave better results than the HSG and U modes in terms of time and distance since it had fewer gripper exchanges. Note that the operators using the HSG and U modes had to make one more gripper exchange, requiring more time and distance. Therefore in the gripper sub-task, the differences (18.6% for time and 18.9% for distance) between the SG and the HSG modes represented the effect of one more gripper selection. However, even though the number of gripper exchange was equal for the U and the HSG modes, the performances of the HSG mode were slightly better in the U mode since the gripper selection of the HSG mode was automatically supported by the helping feature. The effect of the gripper selection guidance on the overall assembly process is clear from the results of this sub-task. The inter-part movements section of Table 5 shows that the HSG mode was better than the other two modes in this sub-task. This was because in the HSG mode, the optimized path, which was shown for the next corresponding part, could be used as the haptic path guidance to reach the initial position for the next part. In the SG and U modes, the results for the inter-part path sub-task were not significantly different since the paths were fully under the control of the operator. The effect of haptic path guidance on the overall assembly process is clear from the results of this sub-task. In the assembly sub-task, the HSG mode gave the best results in terms of time and distance. This was due to the optimization process that included path planning. The optimal path planning with the desired assembly direction minimized the necessary distance and time during the assembly task. The SG mode was better than the U mode since it is used the optimized orientation information in the form of the arrow-shaped indicator. Therefore for the assembly sub-task, the differences between the HSG and the SG modes (33% for time and 14.6% for distance) represented the effects of the optimized haptic path, while the differences between the SG and the U modes (14.8% for time and 12.7% for distance) represented the effects orientation guidance help feature. In this case, the U mode had three more orientation changes than the SG mode, as shown in Table 4. Overall, the HSG mode had the best virtual assembly results in terms of time and distance. The detailed analyses of the assembly sub-tasks show that the performance improvements were mainly due to the reduced travel distance during parts assembly and inter-part movements with the haptic path guidance. Also, compared to the U mode, the SG mode had better results due to

316

Christiand, J. Yoon / Robotics and Computer-Integrated Manufacturing 27 (2011) 306–317

Fig. 16. Assembly sub-tasks.

Table 5 Performance analysis for each assembly sub-task. Contributor

Mode U

Gripper Inter-part Assembly Total

SG

HSG

T

D

T

D

T

D

6.70 10.04 15.47 32.21

127.67 193.81 244.53 566.01

4.66 9.38 13.17 27.21

97.08 191.27 213.39 501.73

5.53 8.74 8.82 23.09

115.43 183.30 182.09 480.82

T¼ time (s) and D ¼distance (units).

the optimal sequence with the gripper and orientation guidance. Thus, the optimized helping features enhanced the effectiveness of the virtual assembly simulation.

5. Discussions In all modes, the grippers are located at some fixed distance from each part’s initial position. This scheme was chosen since the optimization process does not consider the gripper’s position. In this way, the effects of the number of gripper changes can be clearly investigated. In addition, to avoid the tendency to choose only the part nearest the initial HIP location, the initial HIP position is selected randomly in each trial from among the 5 possible positions shown in Fig. 11. Several haptic features have been considered for effective use of the optimized haptic path. The influences of the operator’s movement in the Z + direction should be minimized in evaluating the assembly task for planar assembly. By adding an invisible object parallel to the floor, the pointer can only move in the space between the floor and that invisible object (Fig. 3). The crosssectional shape of the path boundary is also taken into consideration. In the suggested 2D assembly haptic interaction, the wall-type (rectangular) shape provides a better haptic sensation than the semi-circular shape. Finally, the width of the path boundary can affect the performance of the operator following that path. Wide path boundaries provide more freedom to the operator, but the operator movements are not uniform. On the other hand, when the path boundary is too narrow, the HIP is difficult to move. However, the width of the path boundary is not considered in the optimization process since it is difficult to model haptic effects in the optimization process. Thus, we

carefully selected the width of the path boundary to keep it sufficiently small while still allowing free operator movement. With careful haptic path designs, the operator can move freely inside the path without problems such as binding or sticking. The repetition of virtual assembly tasks by the haptic feedback is expected similar to the effect of real repetitive training. The initial motivation was to develop an intelligent assembly simulation system, which combines assembly sequence optimization and path planning optimization. This scheme can increase the efficiency of an assembly task through an assembly simulation system. For limitations, as already known, the path planning process might consume a lot of resources due to its complexity. As a rule of thumb, the complexity of path planning problem will arise with increase in the workspace dimension, number of parts, and other additional constraints. These facts make the optimization which involves path planning difficult to be performed. Also, an error computation algorithm may be required because the process of combining two optimizations has its inherent complexion. The detail validation for each step of procedure is also fairly difficult to be achieved. However, the results can be tested through feasible solutions to validate that whether it is an optimal solution or not.

6. Conclusions We have described an intelligent virtual assembly system with an optimized haptic path and sequence. The virtual assembly system included an intelligent algorithm to enhance the effectiveness of the assembly task. The proposed intelligent assembly scheme allowed an operator to achieve the optimal assembly process by following an optimal assembly sequence and optimal haptic paths. Moreover, the results of the proposed optimal assembly algorithm were used as help features to assist the assembly task process in a virtual assembly simulation, and were assessed through user experiments. The experimental results showed that compared to the unguided mode, the HSG mode was 28.33% better for assembly time and 15.05% for travel distance, while the SG mode was 15.53% better for assembly time and 11.36% for travel distance. The detailed analyses of each assembly mode showed that for the HSG mode, the performance improvements were mainly due to the reduced travel distance during parts assembly and inter-part movements, while for the SG mode, mainly due to the reduced numbers of gripper exchanges and orientation change. We can conclude that the optimized haptic path, as well as sequence guidance, enhances the working

Christiand, J. Yoon / Robotics and Computer-Integrated Manufacturing 27 (2011) 306–317

performance of virtual assembly tasks. In addition, the developed virtual assembly simulation system can be a framework to verify the effectiveness of optimal assembly algorithms. In future work, the virtual assembly system and the assembly algorithm will be extended to a more complex simulation with 3D mechanical parts to apply the proposed virtual assembly scheme to more practical applications, such as aircraft parts assembly.

Acknowledgements This work was supported by Priority Research Centers Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education, Science and Technology (2009-0094016) and supported by 2nd stage BK21 (Brain Korea 21).

Appendix A. Supporting information

[7]

[8]

[9]

[10]

[11]

[12]

[13] [14]

Supplementary data associated with this article can be found in the online version at doi:10.1016/j.rcim.2010.07.015.

[15] [16] [17]

References [18] [1] Jayaram S, Jayaram U, Wang Y, Tirumali H, Lyons K, Hart P, et al. Virtual assembly design environment. Computer Graphics and Applications 1999;19(6):44–50. [2] Yand RD, Fan XM, Wu DL, Yan JQ. Virtual assembly technologies based on constraints and DOF analysis. Robotics and Computer-integrated Manufacturing 2006;23(4):447–56. [3] Yao YX, Xia PJ, Liu JS, Li JG. A pragmatic system to support interactive assembly planning and training in an immersive virtual environment (I-VAPTS). International Journal of Advanced Manufacturing Technology 2006;30:959–67. [4] Raghavan V, Molineros J, Sharma R. Interactive evaluation of assembly sequences using augmented reality. IEEE Transactions on Robotics and Automation 1999;15(3):435–49. [5] Ye N, Banerjee P, Banerjee A, Dech FA. Comparative study of assembly planning in traditional and virtual environments. IEEE Transactions on Robotics and Automation 1999;29(4):546–55. [6] Chryssolouris G, Mavrikios D, Fragos D, Karabatsou V. A virtual reality-based experimentation environment for the verification of human-related factors in

[19]

[20]

[21] [22] [23]

[24]

317

assembly processes. Robotics and Computer Integrated Manufacturing 2000;16:267–76. Borro D, Savall J, Amundarain A, Gil JJ, Garcia-Alonso A, Matey L. A large haptic device for aircraft engine maintainability. IEEE Computer Graphics and Applications 2004;24(6):70–4. Savall J, Borro D, Gil JJ, Matey L. Description of a haptic system for virtual maintainability in aeronautics. Proceeding of IEEE international conference on intelligent robots and systems. 2002. 2887–92. Kuang AB, Payandeh S, Zheng B, Henigman F, MacKenzie CL. Assembling virtual fixtures for guidance in training environments. Proceeding of international symposium on haptic interfaces for virtual environment and teleoperator systems. 2004. 367–74. Bettini A, Lang S, Okamura A, Hager G. Vision assisted control for manipulation using virtual fixtures: experiments at macro and micro scales. Proceeding IEEE international conference on robotics and automation. 2002. 3354–61. Teo CL, Burdet E, Lim HP. A robotic teacher of Chinese handwriting. In: Proceeding of symposium of haptic interfaces for virtual environment and teleoperator systems, 24–25 March 2002, Orlando, USA. . p.335–41. Christiand Yoon J. Intelligent assembly/disassembly system with a haptic device for aircraft parts maintenance. Lecture Notes in Computer Science—ICCS 2007;4488:760–7. Lazzerini B, Marcelloni F. A genetic algorithm for generating optimal assembly plans. Artificial Intelligence in Engineering 2000;14:319–29. Li JR, Khoo LP, Tor SB. An object-oriented intelligent disassembly sequence planner for maintenance. Computers in Industry 2005;56:699–718. Tseng HE. Guided genetic algorithm for solving a larger constraint assembly problem. International Journal of Production Research 2006;44(3):601–25. Bobrow JE. Optimal robot path planning using the minimum-time criterion. IEEE Journal of Robotics and Automation 1988;4(4):443–50. Shiller Z, Dubowsky S. On computing the global time-optimal motions of robotic manipulators in the presence of obstacles. IEEE Transaction on Robotics and Automation 1991;7(6):785–97. Garg PD, Kumar M. Optimization techniques applied to multiple manipulators for path planning and torque minimization. Engineering Application of Artificial Intelligence 2002;15:241–52. Christiand Yoon J, Kumar PA. Novel optimal assembly algorithm for the haptic interface application of a virtual maintenance system. Journal of Mechanical Science and Technology. 2009;23(1):183–94. Conti F., Barbagli F., Morris D., Sewell C. CHAI: an open-source library for the rapid development of haptic scenes. Proceeding of IEEE World Haptics. Pisa, Italy. 2005 (in demo paper). Spong MW, Hutchinson S, Vidyasagar M. Robot modeling and control. USA: John Willey and Sons; 2006. Holland JH. Adaptation in natural and artificial systems. Cambridge, USA: MIT Press; 1992. Galantucci LM, Percoco G, Spina R. Assembly and disassembly planning by using fuzzy logic & genetic algorithms. International Journal of Advanced Robotic Systems 2004;1(2):66–74. Pan C, Smith S. Case study: the impact of assembly reorientations on assembly time. International Journal of Production Research 2006;44(21): 4569–4585.