- Email: [email protected]
Assembly process monitoring algorithm using force data and deformation data
Robotics and Computer Integrated Manufacturing 56 (2019) 149–156
Contents lists available at ScienceDirect
Robotics and Computer Integrated Manufacturing journal homepage: www.elsevier.com/locate/rcim
Assembly process monitoring algorithm using force data and deformation data
T
⁎
Dong-Hyeong Leea, Min-Woo Nab, Jae-Bok Songa, , Chan-Hun Parkc, Dong-Il Parkc a
Department of Mechanical Engineering, Korea University, Seoul 02841, Republic of Korea Department of Smart Convergence, Korea University, Seoul 02841, Republic of Korea c Department of Robotics and Mechatronics, Korea Institute of Machinery & Materials (KIMM), Daejeon 305-343, Republic of Korea b
A R T I C LE I N FO
A B S T R A C T
Keywords: Deformation sensor Force/torque sensor Robotic assembly Assembly state estimation
In robotic assembly with smaller repeatability than the assembly tolerance, failure should not occur. However, in the industrial field, assemblies may fail because of positional errors in the assembled parts and other factors. Owing to the characteristics of position control, the robot tries to move to the desired position irrespective of the failure of assembly. This situation causes excessive contact force, which can lead to the damage of parts and robots. To prevent this, an assembly process monitoring algorithm is proposed in this study. The role of this algorithm is to monitor, from the sensor information measured in the assembly process, whether the assembly state is formed; thus, the robot may recognize whether the assembly is normally performed or not. In this study, the monitoring performance was verified by applying the algorithm to the process of assembling the parts of a tablet PC.
1. Introduction Recently, robotic assembly has been expanded to the assembly of tablet PCs and smart phones with high difficulty because the assembly parts and tolerances are relatively small [1–3]. In the industrial field, robotic assembly is mostly conducted through position control of a robot manipulator. In theory, such assembly must always be successful because the robot's repeatability is smaller than the assembly tolerance of the components. However, assembly failure frequently occurs because of position errors in the part supply process, inflow of foreign matter, and other factors. Owing to the nature of position control, the robot tries to reach the commanded position irrespective of the success or failure of the assembly. In case of failure, excessive insertion force is often applied to the parts, which may damage the parts and/or the robot [4]. An algorithm for monitoring the assembly process is needed to prevent this. An interaction between workers and the environment of the assembly line is becoming increasingly important [5,6]. For collaborative human-robot manufacturing cell, a control framework for the assembly of automobile parts was proposed in [7]. In addition, a system for flexible assembly process with the worker using a dual arm robot is proposed, which allows fenceless human robot cooperation [8]. But the scope of the research in [7,8] is different from our scope. The goal of this study is to make the assembly process completely automated
⁎
without the aid of a worker. Numerous studies on robotic assembly have proposed various methods to search for the assembly position through force control schemes [9,10]. A force control-based hole detection algorithm that is easily applicable to the square peg-in-hole assembly was proposed in [9]. A force-controlled hybrid system that allows the controller to recognize changes in the state of the assembly step was proposed in [10]. Although it is possible to perform precision assembly using force control with a high success rate, this process takes too much time and thus cannot replace human workers. Hardware such as the remote center of compliance (RCC) is used to compensate passively for the position error generated during the assembly process [11]. However, there is a limit in the position error that can be compensated by the RCC and its performance deteriorates sharply as the position error increases. Many studies adopted vision sensors to improve the assembly performance. A micro vision system was used for precision assembly but its performance was adversely affected by illumination changes in [12]. A stereo camera was used for welding car doors in [13] and visual servoing was implemented to obtain the pose information of a product in the 3D space in [14], but both methods failed to obtain sufficient accuracy for assembly. The assembly of a back shell with a smartphone was conducted using a closed-loop approach that uses eye-in-hand visual servoing with two monocular cameras in [15], but this method has a disadvantage that the field of view (FOV) of the vision system must be
Corresponding author. E-mail address: [email protected] (J.-B. Song).
https://doi.org/10.1016/j.rcim.2018.09.008 Received 12 February 2018; Received in revised form 21 September 2018; Accepted 21 September 2018 0736-5845/ © 2018 Elsevier Ltd. All rights reserved.
Robotics and Computer Integrated Manufacturing 56 (2019) 149–156
D.-H. Lee et al.
changed according to the size of the part to be assembled. Many attempts have been made to develop the algorithms that detects failure in the assembly process [16–20]. A recovery method using a F/T sensor data that diagnosed the position errors occurring in the assembly process of electronic connectors was proposed in [16]. Also, the peg-in-hole assembly was conducted using a method to guide the searching of the hole by mathematically modeling each contact using the sensor data [17]. However, these methods in [16,17] have a disadvantage of requiring additional mathematical models which are difficult to be applied to various assembly parts. The support vector machine (SVM), which is a powerful binary classifier, was adopted, but did not show satisfactory results due to its fundamental limitation in [18]. The relative change-based hierarchical taxonomy (RCBHT) was used to abstract force/torque data, but this increased the computational load significantly, thus degrading real-time performance. In [19], a fault detection and isolation (FDI) strategy was used to detect the assembly failure and compensate for the errors by approximating the assembly process of an electric connector with a linear model. However, it was difficult to linearize most parts to be assembled because of the complexity of the contact force generated in the assembly process. A machine learning based system to detect a slip of grasped parts from a gripper using a F/T sensor was proposed in [20]. This machine learning based approach requires datasets for each state, which makes it difficult to cope with various parts in an assembly line. In this study, we propose an assembly process monitoring algorithm that is easy to apply in industrial sites. The main contribution of this study is that the proposed algorithm enables the robots to recognize the success or failure of the assembly in real time while performing a highspeed assembly process based on position control. In addition, the proposed simple models, based on force or deformation, make this algorithm convenient for field applications where various parts should be assembled. Another contribution is that this algorithm is versatile, and thus it can be used in combination with either 6-axis force/torque sensors or deformation sensors attached at the robot's wrist. This paper is organized as follows. Section 2 introduces the object to be assembled and the assembly process, and presents the concept of assembly state defined in this study. Section 3 discusses the modeling of the assembly process. Section 4 presents the procedure to monitor the assembly process based on the proposed models. In Section 5, the effectiveness of the proposed algorithm is verified through various experiments. Finally, in Section 6, conclusions are presented.
Fig. 1. Assembly parts and experimental setup: (a) base frame, (b) side frame, and (c) system configuration.
at the earphone terminal of the base frame so that the pose information of this frame can be known from the pose information of the tool coordinate system relative to the base coordinate system {W} of the robot. Note that pose means position and orientation. In addition, SCHMALZ's suction pads (FM-SW 76 × 22) and a vacuum pump (EVE-TR 10) were used to grasp the flat base frame. Two sensors were used to monitor the assembly process. One is ATI's Gamma, a commercial 6-axis force/torque sensor, which is mounted at the wrist of the robot to measure forces and torques in the x, y, and z directions. The other is a Magic Gripper, developed by the Korea Institute of Machinery and Materials [21], which has a passive stiffness function, like the RCC, to prevent the buildup of excessive contact force owing to the abnormal contact that may occur during the assembly process. It also has a function to quantitatively measure the degree of its deformation in response to external forces. As shown in Fig. 2, the Magic Gripper is in the form of a Stewart platform and can measure the displacement between the upper and lower plates. The displacement measuring part consists of three compliance bars and six linear variable differential transformer (LVDT) modules. The linear displacements measured by the LVDTs are converted into three linear deformations and three rotational deformations in the x, y, and z axes of the lower plate coordinate system with respect to the upper plate coordinate system. In this study, the force/torque sensor is attached to the upper plate of the Magic Gripper, and the suction gripper module is attached
2. Analysis of assembly process 2.1. Experiment environments The assembly of the base frame and side frame of a tablet PC shown in Fig. 1, was selected for the verification of the proposed assembly process monitoring algorithm in this study. From the perspective of assembly, the two parts are characterized as follows. First, they are too flat to be grasped with a finger-type gripper. Second, they are vulnerable to an external force because the base frame is mounted on the side frame although they are made of plastic, and thus have elasticity. Third, the assembly process is more complicated than a simple peg-in-hole process because an earphone terminal is protruded. Fourth, there is an assembly tolerance of 0.5 mm in the left and right directions, and 1.0 mm in the forward and backward directions. Fig. 1(c) shows the system configuration for assembly experiments. In this study, a DENSO's VM-6083G industrial robot, which was connected to its controller and the PC, was used to implement position control with a repeatability of 0.1 mm. An external timer (RTX) was used to ensure the real-time performance that was not guaranteed by the Windows operating system. The control period for position control and assembly state monitoring algorithms was set to 10 ms. The computation time for the proposed algorithm is 1.5 ms which is much less than the control period. The tool coordinate system of the robot was set
Fig. 2. Magic gripper. 150
Robotics and Computer Integrated Manufacturing 56 (2019) 149–156
D.-H. Lee et al.
Fig. 3. Three different robot motions and their corresponding contact states. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)
to the lower plate. As a result, the contact behavior during the assembly process can be monitored from either the force data or the deformation data in the x, y, and z directions. 2.2. Analysis of assembly process Generally, the robotic assembly process can be divided into grasping, transferring, assembling, and returning to the initial position. In addition, the assembly work can be subdivided according to the operation of the robot. In this study, the assembly of the base frame, which is divided into tilting, pushing, and inserting, is performed while the contact between the objects to be assembled is sequentially formed. Fig. 3 shows these operations, and the resulting point contact, line contact, and surface contact. In Fig. 4, the operation of the robot can be known through the changes in Ry, Px, and Pz. The assembly operation of the robot can be divided into three stages according to the contact occurrence and the change of contact pattern. The first is the stage where the robot operates but a new contact does not occur. The second is the stage where the contact has occurred but the contact state keeps changing because the robot is still operating. The third is the stage where the contact state is maintained. The second and third stages are shown in Fig. 4 as the shaded areas in yellow and red. It is seen from the measured force data (i.e., Fx, …, Tz) and the deformation data (i.e., Dx, …, Drz) that transitions occur in the second stage and the measured values remain constant in the third stage.
Fig. 4. Measured data during the assembly process.(For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)
Fig. 5. Contact between assembled parts.
point, line, and surface contacts in Fig. 3 and maintain the robot pose in the section indicated by the red shade in Fig. 4. 3. Modeling of assembly process
2.3. Definition of assembly state 3.1. Data collection and analysis In general, contact is defined as the encounter of point, line, and surface elements of two or more objects. However, the definition of mathematical contact states is somewhat ambiguous. For example, in Fig. 5, the fixed object F and the object M operated by the robot can take many poses (positions and orientations) while maintaining the contact state as object M is moving. In this study, we select one pose, Xi, which has the highest assembly success rate among many possible poses, and define it as the corresponding assembly state Ai, as follows:
Ai = {Xi }, i ∈ [1, ⋯, nA]
In this study, the force or deformation data were collected and analyzed to model the assembly process. Fig. 6 shows the force and deformation data collected from repeated assembly processes. In the figure, the data of 10 successful assemblies are in red, and the data of the two failed ones are in blue and black. It is seen that the data of the successful assemblies show a certain pattern, whereas the data of the failed ones are different from the successful data, and even different from each other. Therefore, in this study, the assembly model was established based on the data of the region where the assembly states are formed among the data collected by the repeated successful assembly processes. Furthermore, to select the data components that appropriately reflect the assembly states, we analyzed the discrimination between the data of the assembled state and the data of the non-assembled state, and that between the data collected in different assembled states. In this study, the region where each assembly state is formed is referred to as the Ai
(1)
where nA is the number of assembly states that occur during the assembly process, and Xi is a 6 × 1 vector, which includes either the force components (i.e., Fx, …, Tz) in the case of the force data, or the deformation components (i.e., Dx, …, Drz), in the case of the deformation data. For the base frame assembly process, there are three assembly states of tilting, pushing, and inserting, and thus, nA = 3. As a result, the three assembly states A1, A2, and A3 of the base frame form the 151
Robotics and Computer Integrated Manufacturing 56 (2019) 149–156
D.-H. Lee et al.
Fig. 8. Histograms of deformation data according to the assembly states and remaining regions: (a) A1 vs R1, (b) A2 vs R2, and (c) A3 vs R3.(For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)
Fig. 6. Measured data during successful assembly and failed assembly processes.(For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)
region, whereas all the remaining portions, except for the Ai region, are designated as the Ri region. Figs. 7 and 8 show the histograms of force and deformation data measured in the Ai region (red) and Ri region (green). For example, Ty is the most discriminating component in the histogram of Fig. 7(a), in which the force data measured in the A1 and R1 regions are compared. The distances between the distributions of the force data measured in the A2 and R2 regions of Fig. 7(b) are short, which means that it is difficult to distinguish A2 from R2. Because the force data distributions are well separated, except for Tz in Fig. 7(c), it is rather easy to distinguish A3 from R3. The distributions of the deformation data measured in the assembled and non-assembled regions in Fig. 8 are similar to those of the force data in Fig. 7. However, the data distribution of Ai is closer to that of Ri for the deformation data than for the force data,
Fig. 9. Histogram data according to three different assembly states: (a) force histograms, and (b) deformation histograms.(For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)
which means that it is more difficult to discriminate the assembly states based on the deformation data than based on the force data. Fig. 9 shows the histograms of force data and deformation data measured in different assembly states: A1 (red), A2 (green), and A3 (blue). The histogram of Fx, Fy, Fz, and Tx in Fig. 9(a) indicates that the data distribution for A3 is far from the distributions for A1 and A2, whereas the distributions for A1 and A2 are close to each other. The distributions for Dy and Dz in Fig. 9(b) overlap each other, and thus, it is difficult to distinguish one assembly state from the others. In this study, the Bhattacharyya distance, an index that quantitatively evaluates the statistical distance between two different distributions, was introduced to evaluate both the difference in the data
Fig. 7. Histograms of force data according to the assembly states and remaining regions: (a) A1 vs R1, (b) A2 vs R2, and (c) A3 vs R3.(For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.) 152
Robotics and Computer Integrated Manufacturing 56 (2019) 149–156
D.-H. Lee et al.
Table 1 Bhattacharyya distance of the force data distribution between Ai and Ri.
Fx Fy Fz Tx Ty Tz
dB(A1,R1)
dB(A2,R2)
dB(A3,R3)
1.01 1.72 1.09 1.43 2.06 0.78
1.56 1.69 1.08 1.03 1.11 0.64
3.68 6.34 3.54 4.65 3.87 0.65
Table 3 Bhattacharyya distance of the force data distribution between three different assembly states.
Fx Fy Fz Tx Ty Tz
distributions measured in the Ai and Ri regions and the difference in the data distribution measured in the different Ai states. This distance is used to find the components that have little impact on the discrimination of the assembly states and to exclude them in the evaluation process. The Bhattacharyya distance dB between the two different distributions p and q is given by
2 2 σq2 1 ⎛ 1 ⎛ σp 1 ⎛ (μp − μq ) ⎞ ⎞⎞ ln + 2 + 2⎟ ⎟ + ⎜ 2 2 ⎟ 4 ⎜ 4 ⎜⎝ σq2 4 σp ⎝ σp + σq ⎠ ⎠⎠ ⎝
Dx Dy Dz Drx Dry Drz
(2)
where μp, σp, μq, and σq are the means and standard deviations of the distributions p and q, respectively. The value of dB(p,q) becomes larger as the separation between the two distributions becomes larger. The Bhattacharyya distances for the force and deformation data measured in Ai and Ri are summarized in Tables 1 and 2, respectively. It is clear that dB(A3, R3) in Tables 1 and 2 is larger than dB(A1, R1) and dB(A2, R2), which means that it is easier to determine the assembly state with the data at the time of A3 formation than with those of A1 or A2. This is also shown in Figs. 7 and 8, in which the data distributions of A3 and R3 are farther apart than the other data distributions. Tables 3 and 4 summarize the Bhattacharyya distances based on the measured force and deformation data, respectively, when the three assembly states are formed. As a result of the evaluation, dB(A1, A2) is smaller than dB (A2, A3) and dB(A1, A3) in both force and deformation data. This means that the distribution of the measured data in A1 is close to that in A2, and the distributions in these two regions is far from that in A3, as already confirmed in the histogram of Fig. 9. In the case of dB(A1, A2), the force component Tx and deformation component Drx are 0.66 and 0.03, respectively, which are very small compared with other components; thus, the data distributions of the corresponding components in the histogram overlap each other in Fig. 9. If the distributions of the measured data are similar to each other when different assembly states are formed, as described above, the inclusion of these data has an adverse effect on the algorithm's ability to determine the assembly state.
Mf = [μij ], Md = [μij ],
Dx Dy Dz Drx Dry Drz
0.59 0.53 0.17 0.75 1.39 0.91
1.16 0.69 0.34 0.92 0.87 1.09
1.25 0.81 0.49 6.61 5.60 0.72
125.73 282.10 2726.64 1453.54 144.31 1.09
128.93 174.58 2237.29 1106.37 644.53 6.77
dB(A1,A2)
dB(A2,A3)
dB(A1,A3)
5.01 2.26 1.06 0.03 267.11 0.18
15.39 0.04 2.12 88.59 320.53 0.07
6.87 1.27 0.33 70.37 750.47 0.29
∑ = [σij](i = 1, ⋯, nA, j = 1, ⋯, 6) (3)
∑d
= [σij] (i = 1, ⋯, nA , j = 1, ⋯, 6)
(4)
4. Assembly process monitoring algorithm 4.1. Discrimination of assembly state The data constituting the model matrices reflect the contact pattern when the assembly states are successfully formed. Therefore, it is possible to estimate whether the assembly state is formed by comparing the model data with the data measured in real time in the assembly process. In this study, the Mahalanobis distance is used to evaluate the similarity of the data. The Mahalanobis distance dM(x) of the measured value x in the onedimensional space is obtained by using the mean μ and the standard deviation σ of the corresponding data as follows:
(x − μ)2 / σ 2
dM (x ) =
(5)
It is noted that the Mahalanobis distance represents the degree of separation of the data from the mean by a multiple of the standard deviation. Based on Eq. (5), the similarity for evaluating the formation of the three assembly states of the base frame assembly process is defined by
Table 2 Bhattacharyya distance of the deformation data distribution between Ai and Ri. dB(A3,R3)
1.67 2.80 16.39 0.66 141.34 2.99
where μij and σij are the mean and standard deviation of the data, respectively. In this study, Eqs. (3) and (4) are referred to as the forcebased model matrix and deformation-based model matrix, respectively. The size of each matrix varies depending on the number of assembly states defined in the assembly process and the number of collected data. In the case of the base frame assembly process, there are six force components and six deformation components for three assembly states (i.e., nA = 3), and thus, the model matrices of mean and standard deviation are all 3 × 6 matrices.
An assembly model is created based on the statistical parameters of the force and deformation data collected in the regions where the assembly state is formed as follows:
dB(A2,R2)
dB(A1,A3)
f
3.2. Modeling of assembly process
dB(A1,R1)
dB(A2,A3)
Table 4 Bhattacharyya distance of the deformation data distribution between three different assembly states.
dB (p , q) =
dB(A1,A2)
Si =
1 6
∑ j = 1 wij [(x j − μij )/ σij]2
(i = 1, 2, 3) (6)
where Si denotes the similarity between the data components of the measured value x and the data (μij, σij) reflecting Ai. Note that the similarity is defined as the inverse of the sum of the Mahalanobis distances for the data components of the measured value. Therefore, the 153
Robotics and Computer Integrated Manufacturing 56 (2019) 149–156
D.-H. Lee et al.
S1, S2, and S3 shown in red, green, and blue. In this case, because the similarities of (b), in which the weights are applied, are larger than those of (a), in which the weights are not applied, it is possible to evaluate with greater certitude whether the assembly is successful in the case of (b) than in (a). Fig. 12 shows the simulation results of the assembly process monitoring algorithm based on the deformation data. It is shown that when the weights are not applied, in Fig. 12(a), the discrimination of A1 and A2 of the successful assembly process is somewhat low. This is because both S1 and S2 are high in the formation of A1 and A2 owing to the similar data distribution. However, when the weight matrix Wd is applied, the discrimination of A1 and A2 increases owing to the improved similarity. In this study, we define two discrimination indexes, as shown below, to quantify the performance of the assembly process monitoring algorithm before and after weighting. First, the discrimination index Di of a successful assembly process is defined as
Mahalanobis distance becomes smaller, and thus the similarity becomes higher, as the measured value gets closer to the assembly model data. Furthermore, since the Mahalanobis distance has a standardization effect, the similarity is not affected by the difference in scale between the components of the measured value. Based on the results of the evaluation of the Bhattacharyya distance in Section 3, the following weighting matrices are introduced to exclude components that adversely affect the similarity evaluation.
⎡1 ⎢1 1 Wf = ⎢ ⎢0 ⎢1 ⎢0 ⎣
1 1 1 1 1 0
1⎤ ⎡1 1 1⎥ ⎢1 0 1⎥ 0 0 ,W =⎢ 1⎥ d ⎢0 1 ⎢1 1 1⎥ ⎢0 0 0⎥ ⎦ ⎣
1⎤ 1⎥ 0⎥ 0⎥ 1⎥ 0⎥ ⎦
(7)
where Wf and Wd are the weight matrices to exclude the force and deformation components that are near zero in the Bhattacharyya distance. For example, the Tx component in A1 is excluded from the similarity calculation because the 1st column and the 4th row in Wf is zero.
Di =
min (Si ) Ai max (Si ) Ri
( i = 1, 2, 3)
(8)
This index quantitatively indicates to what extent the similarity evaluated when forming the assembly state is larger than the similarity when it is not formed. In addition, the discrimination index D′i between success and failure of the assembly process is defined as
4.2. Assembly process monitoring algorithm The success or failure of the entire assembly process can be determined based on whether the individual assembly state was normally formed. Fig. 10 shows a flowchart of an algorithm for monitoring the assembling process of the base frame. When the assembly is started, the robot holding the base frame triggers the formation of A1, A2, and A3 through tilting, pushing, and inserting operations. The algorithm then calculates the similarities S1, S2, and S3 and compares them with the thresholds C1, C2, and C3 set by the user to determine whether the three assembly states are formed. Refer to Eq. (10) to set the size of the threshold. For example, if S1 > C1, S2 < C2, and S3 < C3 during the tilting operation, then the algorithm determines that A1 has been formed normally, otherwise it determines that A1 is abnormally formed. Moreover, if it is judged that all the three assembly states are normally formed, the algorithm regards the entire assembly process as a success.
D ′i =
min (Si ) Ai max (S ′i) Ai
( i = 1, 2, 3)
(9)
This index quantifies the difference in the similarity evaluated in Ai of the successful and failed assembly processes. Therefore, the discrimination index D′ quantitatively indicates how well the evaluated similarity reflects the success or failure of the assembly. Table 5 summarizes the force and deformation-based discrimination indices. As seen from the table, in both cases the discrimination indices are sufficiently large; thus, it is possible to judge whether the assembly state is formed or not and whether the assembly is successful or not using the concept of similarity. Furthermore, if the force data is used instead of the deformation data, then the discrimination index would be generally increased, and thus, better performance can be obtained.
4.3. Algorithm optimization through simulation 5. Experimental verification The performance of the proposed assembly process monitoring algorithm was evaluated through MATLAB simulations. Fig. 11 shows the similarity evaluated based on the force data measured on success and failure of the assembly process. It is possible to know whether the three assembled states A1, A2, and A3 are formed from the three similarities
Various experiments were conducted to investigate the monitoring performance of the proposed algorithm. As shown in Fig. 13, the first experiment was performed 100 times when the base frame was normally supplied. In the second experiment, the base frame was offset from the normal position by 3–5 mm in the front, rear, left, and right directions to induce failure and was performed 25 times for each case. In addition, the force-based and deformation-based assembly process monitoring algorithms were tested with identical assembly processes and their performance was evaluated independently. The size of the threshold value used to determine whether the assembly state is formed and the success or failure of the assembly process is calculated as follows:
Ci =
1 [min (Si ) Ai + max (Si ) Ri] ( i = 1, 2, 3) 2
(10)
Because Ci is calculated as the average of the minimum value of the similarity evaluated in Ai of the successful assembly process and the maximum value of the similarity evaluated in Ri, it should be between these two values. Thus, it is possible to distinguish whether the assembly state is formed or not. In addition, as the similarity evaluated in Ai of the failed assembly process is smaller than the similarity evaluated in Ri of the successful assembly process, it is also possible to estimate the success or failure of the assembly process using the above average value. Table 6 summarizes the results of the assembly process monitoring
Fig. 10. Flowchart of assembly process monitoring algorithm. 154
Robotics and Computer Integrated Manufacturing 56 (2019) 149–156
D.-H. Lee et al.
Fig. 11. Simulation results of the assembly process monitoring algorithm based on the force data: (a) without weighting, and (b) with weighting.(For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)
Fig. 12. Simulation results of the assembly process monitoring algorithm using the deformation data: (a) without weighting, and (b) with weighting. Table 5 Discrimination indexes based on force and deformation data. Force-based
A1
A2
A3
Di D′i
3.2 5.4
8.2 9.8
5.2 10.4
Deformation-based
A1
A2
A3
Di D′i
2.8 4.2
3.8 6.8
5.0 5.1
based on the force and deformation data, respectively. In both cases, the proposed algorithm succeeded in determining 100 successes of the actual assembly process as a success, and a false negative error (i.e., actually a success but judged as a failure) did not occur. In addition, the proposed algorithm estimated all the 100 processes that failed as a failure, and a false positive error, which is a failure mistakenly estimated as a success, did not occur. As a result, the algorithm predicted whether the assembly state was formed in the actual assembly process and the success or failure of the assembly.
Fig. 13. Intentional position error for experiments.
Table 6 Results of experiment using force and deformation data.
Success Failure
6. Conclusion In this study, we propose an assembly process monitoring algorithm 155
Estimated to success
Estimated to failure
100 0
0 100
Robotics and Computer Integrated Manufacturing 56 (2019) 149–156
D.-H. Lee et al.
and verify its performance through various experiments. The following conclusions were drawn.
[4] A. Stolt, On Robotic Assembly using Contact Force Control and Estimation, Ph.D Thesis Lund University, 2015. [5] J. Krüger, L. Wang, A. Verl, T. Bauernhansl, E. Carpanzano, S. Makris, J. Fleischer, G. Reinhart, J. Franke, S. Pellegrinelli, Innovative control of assembly systems and lines, CIRP Ann. Manuf. Tech. 66 (2017) 707–730. [6] R. Meziane, M.J.-D. Otis, H. Ezzaidi, Human-robot collaboration while sharing production activities in dynamic environment: SPADER system, Robot. Comput. Integr. Manuf. 48 (2017) 243–253. [7] A. Cherubini, R. Passama, A. Crosnier, A. Lasnier, P. Fraisse, Collaborative manufacturing with physical human-robot interaction, Robot. Comput. Integr. Manuf. 40 (2016) 1–13. [8] S. Makris, P. Tsarouchi, A.-S. Matthaiakis, A. Athanasatos, X. Chatzigeorgiou, M. Stefos, K. Giavridis, S. Aivaliotis, Dual arm robot in cooperation with humans for flexible assembly, CIRP Ann. Manuf. Tech. 66 (2017) 13–16. [9] Y.L. Kim, H.C. Song, J.-B. Song, Hole detection algorithm for chamferless square peg-in-hole based on shape recognition using F/T sensor, Int. J. Precis. Eng. Manuf. 15 (2014) 425–432. [10] S. Chhatpar, M. Branicky, A hybrid systems approach to force-guided robotic assemblies, Proc. Intl. Symp. Assembly Task Planning, 1999, pp. 301–306. [11] D.E. Whitney, Quasi-static assembly of compliantly supported rigid parts, J. Dyn. Syst. Meas. Control. 104 (1) (1982) 65–77. [12] C. Delprete, C. Rosso, G. Savino, C. Scarzella, Advanced vision approach applied to non-contact micro-measurements: a practical application, Int. J. Adv. Manuf. Technol. 88 (2017) 471–481. [13] G. Michalos, S. Makris, A. Eytan, S. Matthaiakis, G. Chryssolouris, Robot path correction using stereo vision system, Procedia CIRP 3 (2012) 352–357. [14] P. Tsarouchi, G. Michalos, S. Makris, G. Chryssolouris, Vision system for robotic handling of randomly placed objects, Procedia CIRP 9 (2013) 61–66. [15] W.C. Chang, Robotic assembly of smartphone back shells with eye-in-hand visual servoing, Robot. Comput. Integr. Manuf. 50 (2018) 102–113. [16] F. Chen, F. Cannella, J. Huang, H. Sasaki, T. Fukuda, A study on error recovery search strategies of electronic connector mating for robotic fault-tolerant assembly, J. Intell. Robot. Syst. Theory Appl. 81 (2016) 257–271. [17] W.S. Newman, Y. Zhao, Y.-H. Pao, Interpretation of force and moment signals for compliant peg-in-hole assembly, IEEE Intl. Conf. Robot. Autom. 2001, pp. 571–576. [18] W. Luo, J. Rojas, T. Guan, K. Harada, K. Nagata, Cantilever snap assemblies failure detection using SVMs and the RCBHT, Intl. Conf. Mechatron. Autom. 2014, pp. 384–389. [19] J. Huang, Y. Wang, T. Fukuda, Set-Membership-based fault detection and isolation for robotic assembly of electrical connectors, IEEE Trans. Autom. Sci. Eng. 15 (1) (2016) 160–171. [20] E. Moreira, L.F. Rocha, A.M. Pinto, A.P. Moreira, G. Veiga, Assessment of robotic picking operations using a 6 axis force/torque Sensor, IEEE Robot. Autom. Lett. 1 (2016) 768–775. [21] D.I. Park, H. Kim, C. Park, B. Kim, D. Kim, J.H. Kyung, Variable Passive Compliance Device for the Robotic Assembly, Intl. Conf. Ubiquitous Robot. Ambient Intell. 2017, pp. 753–754.
(1) The assembly process was modeled based on the data collected in the successful assembly process. (2) On performing the assembly, based on high-speed position control, and monitoring a success or failure of the assembly process by using the proposed assembly process monitoring algorithm, both successful and failed assembly processes were correctly identified. (3) The proposed assembly monitoring algorithm showed satisfactory performance not only for the force data obtained by a force/torque sensor but also for the deformation data obtained by the deformation sensor. It is confirmed that damage of the parts and robot can be prevented using the assembly process monitoring algorithm. The modeling of the assembly process proposed in this study is simple, and thus, it could be easily applied in industrial sites where various parts are assembled in large quantities and a short time. In addition, it is possible to monitor the assembling process using a low-cost deformation sensor, without requiring an expensive force/torque sensor. A re-assembly strategy after the detection of assembly failure will be developed in the future. Acknowledgment This study was supported by the MOTIE under the Industrial Foundation Technology Development Program, which is supervised by the KEIT (No. 10060110) References [1] K.S. Chin, M.M. Ratnam, R. Mandava, Force-guided robot in automated assembly of mobile phone, Assem. Autom. 23 (1) (2003) 75–86. [2] J.L. Navarro-Gonzalez, I. Lopez-Juarez, R. Rios-Cabrera, K. Ordaz-Hernández, Online knowledge acquisition and enhancement in robotic assembly tasks, Robot. Comput. Integr. Manuf 33 (2015) 78–89. [3] H.M. Do, C. Park, J.H. Kyung, Dual Arm Robot for Packaging and Assembling of IT Products, Int. Conf. Autom. Sci. Eng. (2012) 1067–1070.
156
Contents lists available at ScienceDirect
Robotics and Computer Integrated Manufacturing journal homepage: www.elsevier.com/locate/rcim
Assembly process monitoring algorithm using force data and deformation data
T
⁎
Dong-Hyeong Leea, Min-Woo Nab, Jae-Bok Songa, , Chan-Hun Parkc, Dong-Il Parkc a
Department of Mechanical Engineering, Korea University, Seoul 02841, Republic of Korea Department of Smart Convergence, Korea University, Seoul 02841, Republic of Korea c Department of Robotics and Mechatronics, Korea Institute of Machinery & Materials (KIMM), Daejeon 305-343, Republic of Korea b
A R T I C LE I N FO
A B S T R A C T
Keywords: Deformation sensor Force/torque sensor Robotic assembly Assembly state estimation
In robotic assembly with smaller repeatability than the assembly tolerance, failure should not occur. However, in the industrial field, assemblies may fail because of positional errors in the assembled parts and other factors. Owing to the characteristics of position control, the robot tries to move to the desired position irrespective of the failure of assembly. This situation causes excessive contact force, which can lead to the damage of parts and robots. To prevent this, an assembly process monitoring algorithm is proposed in this study. The role of this algorithm is to monitor, from the sensor information measured in the assembly process, whether the assembly state is formed; thus, the robot may recognize whether the assembly is normally performed or not. In this study, the monitoring performance was verified by applying the algorithm to the process of assembling the parts of a tablet PC.
1. Introduction Recently, robotic assembly has been expanded to the assembly of tablet PCs and smart phones with high difficulty because the assembly parts and tolerances are relatively small [1–3]. In the industrial field, robotic assembly is mostly conducted through position control of a robot manipulator. In theory, such assembly must always be successful because the robot's repeatability is smaller than the assembly tolerance of the components. However, assembly failure frequently occurs because of position errors in the part supply process, inflow of foreign matter, and other factors. Owing to the nature of position control, the robot tries to reach the commanded position irrespective of the success or failure of the assembly. In case of failure, excessive insertion force is often applied to the parts, which may damage the parts and/or the robot [4]. An algorithm for monitoring the assembly process is needed to prevent this. An interaction between workers and the environment of the assembly line is becoming increasingly important [5,6]. For collaborative human-robot manufacturing cell, a control framework for the assembly of automobile parts was proposed in [7]. In addition, a system for flexible assembly process with the worker using a dual arm robot is proposed, which allows fenceless human robot cooperation [8]. But the scope of the research in [7,8] is different from our scope. The goal of this study is to make the assembly process completely automated
⁎
without the aid of a worker. Numerous studies on robotic assembly have proposed various methods to search for the assembly position through force control schemes [9,10]. A force control-based hole detection algorithm that is easily applicable to the square peg-in-hole assembly was proposed in [9]. A force-controlled hybrid system that allows the controller to recognize changes in the state of the assembly step was proposed in [10]. Although it is possible to perform precision assembly using force control with a high success rate, this process takes too much time and thus cannot replace human workers. Hardware such as the remote center of compliance (RCC) is used to compensate passively for the position error generated during the assembly process [11]. However, there is a limit in the position error that can be compensated by the RCC and its performance deteriorates sharply as the position error increases. Many studies adopted vision sensors to improve the assembly performance. A micro vision system was used for precision assembly but its performance was adversely affected by illumination changes in [12]. A stereo camera was used for welding car doors in [13] and visual servoing was implemented to obtain the pose information of a product in the 3D space in [14], but both methods failed to obtain sufficient accuracy for assembly. The assembly of a back shell with a smartphone was conducted using a closed-loop approach that uses eye-in-hand visual servoing with two monocular cameras in [15], but this method has a disadvantage that the field of view (FOV) of the vision system must be
Corresponding author. E-mail address: [email protected] (J.-B. Song).
https://doi.org/10.1016/j.rcim.2018.09.008 Received 12 February 2018; Received in revised form 21 September 2018; Accepted 21 September 2018 0736-5845/ © 2018 Elsevier Ltd. All rights reserved.
Robotics and Computer Integrated Manufacturing 56 (2019) 149–156
D.-H. Lee et al.
changed according to the size of the part to be assembled. Many attempts have been made to develop the algorithms that detects failure in the assembly process [16–20]. A recovery method using a F/T sensor data that diagnosed the position errors occurring in the assembly process of electronic connectors was proposed in [16]. Also, the peg-in-hole assembly was conducted using a method to guide the searching of the hole by mathematically modeling each contact using the sensor data [17]. However, these methods in [16,17] have a disadvantage of requiring additional mathematical models which are difficult to be applied to various assembly parts. The support vector machine (SVM), which is a powerful binary classifier, was adopted, but did not show satisfactory results due to its fundamental limitation in [18]. The relative change-based hierarchical taxonomy (RCBHT) was used to abstract force/torque data, but this increased the computational load significantly, thus degrading real-time performance. In [19], a fault detection and isolation (FDI) strategy was used to detect the assembly failure and compensate for the errors by approximating the assembly process of an electric connector with a linear model. However, it was difficult to linearize most parts to be assembled because of the complexity of the contact force generated in the assembly process. A machine learning based system to detect a slip of grasped parts from a gripper using a F/T sensor was proposed in [20]. This machine learning based approach requires datasets for each state, which makes it difficult to cope with various parts in an assembly line. In this study, we propose an assembly process monitoring algorithm that is easy to apply in industrial sites. The main contribution of this study is that the proposed algorithm enables the robots to recognize the success or failure of the assembly in real time while performing a highspeed assembly process based on position control. In addition, the proposed simple models, based on force or deformation, make this algorithm convenient for field applications where various parts should be assembled. Another contribution is that this algorithm is versatile, and thus it can be used in combination with either 6-axis force/torque sensors or deformation sensors attached at the robot's wrist. This paper is organized as follows. Section 2 introduces the object to be assembled and the assembly process, and presents the concept of assembly state defined in this study. Section 3 discusses the modeling of the assembly process. Section 4 presents the procedure to monitor the assembly process based on the proposed models. In Section 5, the effectiveness of the proposed algorithm is verified through various experiments. Finally, in Section 6, conclusions are presented.
Fig. 1. Assembly parts and experimental setup: (a) base frame, (b) side frame, and (c) system configuration.
at the earphone terminal of the base frame so that the pose information of this frame can be known from the pose information of the tool coordinate system relative to the base coordinate system {W} of the robot. Note that pose means position and orientation. In addition, SCHMALZ's suction pads (FM-SW 76 × 22) and a vacuum pump (EVE-TR 10) were used to grasp the flat base frame. Two sensors were used to monitor the assembly process. One is ATI's Gamma, a commercial 6-axis force/torque sensor, which is mounted at the wrist of the robot to measure forces and torques in the x, y, and z directions. The other is a Magic Gripper, developed by the Korea Institute of Machinery and Materials [21], which has a passive stiffness function, like the RCC, to prevent the buildup of excessive contact force owing to the abnormal contact that may occur during the assembly process. It also has a function to quantitatively measure the degree of its deformation in response to external forces. As shown in Fig. 2, the Magic Gripper is in the form of a Stewart platform and can measure the displacement between the upper and lower plates. The displacement measuring part consists of three compliance bars and six linear variable differential transformer (LVDT) modules. The linear displacements measured by the LVDTs are converted into three linear deformations and three rotational deformations in the x, y, and z axes of the lower plate coordinate system with respect to the upper plate coordinate system. In this study, the force/torque sensor is attached to the upper plate of the Magic Gripper, and the suction gripper module is attached
2. Analysis of assembly process 2.1. Experiment environments The assembly of the base frame and side frame of a tablet PC shown in Fig. 1, was selected for the verification of the proposed assembly process monitoring algorithm in this study. From the perspective of assembly, the two parts are characterized as follows. First, they are too flat to be grasped with a finger-type gripper. Second, they are vulnerable to an external force because the base frame is mounted on the side frame although they are made of plastic, and thus have elasticity. Third, the assembly process is more complicated than a simple peg-in-hole process because an earphone terminal is protruded. Fourth, there is an assembly tolerance of 0.5 mm in the left and right directions, and 1.0 mm in the forward and backward directions. Fig. 1(c) shows the system configuration for assembly experiments. In this study, a DENSO's VM-6083G industrial robot, which was connected to its controller and the PC, was used to implement position control with a repeatability of 0.1 mm. An external timer (RTX) was used to ensure the real-time performance that was not guaranteed by the Windows operating system. The control period for position control and assembly state monitoring algorithms was set to 10 ms. The computation time for the proposed algorithm is 1.5 ms which is much less than the control period. The tool coordinate system of the robot was set
Fig. 2. Magic gripper. 150
Robotics and Computer Integrated Manufacturing 56 (2019) 149–156
D.-H. Lee et al.
Fig. 3. Three different robot motions and their corresponding contact states. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)
to the lower plate. As a result, the contact behavior during the assembly process can be monitored from either the force data or the deformation data in the x, y, and z directions. 2.2. Analysis of assembly process Generally, the robotic assembly process can be divided into grasping, transferring, assembling, and returning to the initial position. In addition, the assembly work can be subdivided according to the operation of the robot. In this study, the assembly of the base frame, which is divided into tilting, pushing, and inserting, is performed while the contact between the objects to be assembled is sequentially formed. Fig. 3 shows these operations, and the resulting point contact, line contact, and surface contact. In Fig. 4, the operation of the robot can be known through the changes in Ry, Px, and Pz. The assembly operation of the robot can be divided into three stages according to the contact occurrence and the change of contact pattern. The first is the stage where the robot operates but a new contact does not occur. The second is the stage where the contact has occurred but the contact state keeps changing because the robot is still operating. The third is the stage where the contact state is maintained. The second and third stages are shown in Fig. 4 as the shaded areas in yellow and red. It is seen from the measured force data (i.e., Fx, …, Tz) and the deformation data (i.e., Dx, …, Drz) that transitions occur in the second stage and the measured values remain constant in the third stage.
Fig. 4. Measured data during the assembly process.(For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)
Fig. 5. Contact between assembled parts.
point, line, and surface contacts in Fig. 3 and maintain the robot pose in the section indicated by the red shade in Fig. 4. 3. Modeling of assembly process
2.3. Definition of assembly state 3.1. Data collection and analysis In general, contact is defined as the encounter of point, line, and surface elements of two or more objects. However, the definition of mathematical contact states is somewhat ambiguous. For example, in Fig. 5, the fixed object F and the object M operated by the robot can take many poses (positions and orientations) while maintaining the contact state as object M is moving. In this study, we select one pose, Xi, which has the highest assembly success rate among many possible poses, and define it as the corresponding assembly state Ai, as follows:
Ai = {Xi }, i ∈ [1, ⋯, nA]
In this study, the force or deformation data were collected and analyzed to model the assembly process. Fig. 6 shows the force and deformation data collected from repeated assembly processes. In the figure, the data of 10 successful assemblies are in red, and the data of the two failed ones are in blue and black. It is seen that the data of the successful assemblies show a certain pattern, whereas the data of the failed ones are different from the successful data, and even different from each other. Therefore, in this study, the assembly model was established based on the data of the region where the assembly states are formed among the data collected by the repeated successful assembly processes. Furthermore, to select the data components that appropriately reflect the assembly states, we analyzed the discrimination between the data of the assembled state and the data of the non-assembled state, and that between the data collected in different assembled states. In this study, the region where each assembly state is formed is referred to as the Ai
(1)
where nA is the number of assembly states that occur during the assembly process, and Xi is a 6 × 1 vector, which includes either the force components (i.e., Fx, …, Tz) in the case of the force data, or the deformation components (i.e., Dx, …, Drz), in the case of the deformation data. For the base frame assembly process, there are three assembly states of tilting, pushing, and inserting, and thus, nA = 3. As a result, the three assembly states A1, A2, and A3 of the base frame form the 151
Robotics and Computer Integrated Manufacturing 56 (2019) 149–156
D.-H. Lee et al.
Fig. 8. Histograms of deformation data according to the assembly states and remaining regions: (a) A1 vs R1, (b) A2 vs R2, and (c) A3 vs R3.(For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)
Fig. 6. Measured data during successful assembly and failed assembly processes.(For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)
region, whereas all the remaining portions, except for the Ai region, are designated as the Ri region. Figs. 7 and 8 show the histograms of force and deformation data measured in the Ai region (red) and Ri region (green). For example, Ty is the most discriminating component in the histogram of Fig. 7(a), in which the force data measured in the A1 and R1 regions are compared. The distances between the distributions of the force data measured in the A2 and R2 regions of Fig. 7(b) are short, which means that it is difficult to distinguish A2 from R2. Because the force data distributions are well separated, except for Tz in Fig. 7(c), it is rather easy to distinguish A3 from R3. The distributions of the deformation data measured in the assembled and non-assembled regions in Fig. 8 are similar to those of the force data in Fig. 7. However, the data distribution of Ai is closer to that of Ri for the deformation data than for the force data,
Fig. 9. Histogram data according to three different assembly states: (a) force histograms, and (b) deformation histograms.(For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)
which means that it is more difficult to discriminate the assembly states based on the deformation data than based on the force data. Fig. 9 shows the histograms of force data and deformation data measured in different assembly states: A1 (red), A2 (green), and A3 (blue). The histogram of Fx, Fy, Fz, and Tx in Fig. 9(a) indicates that the data distribution for A3 is far from the distributions for A1 and A2, whereas the distributions for A1 and A2 are close to each other. The distributions for Dy and Dz in Fig. 9(b) overlap each other, and thus, it is difficult to distinguish one assembly state from the others. In this study, the Bhattacharyya distance, an index that quantitatively evaluates the statistical distance between two different distributions, was introduced to evaluate both the difference in the data
Fig. 7. Histograms of force data according to the assembly states and remaining regions: (a) A1 vs R1, (b) A2 vs R2, and (c) A3 vs R3.(For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.) 152
Robotics and Computer Integrated Manufacturing 56 (2019) 149–156
D.-H. Lee et al.
Table 1 Bhattacharyya distance of the force data distribution between Ai and Ri.
Fx Fy Fz Tx Ty Tz
dB(A1,R1)
dB(A2,R2)
dB(A3,R3)
1.01 1.72 1.09 1.43 2.06 0.78
1.56 1.69 1.08 1.03 1.11 0.64
3.68 6.34 3.54 4.65 3.87 0.65
Table 3 Bhattacharyya distance of the force data distribution between three different assembly states.
Fx Fy Fz Tx Ty Tz
distributions measured in the Ai and Ri regions and the difference in the data distribution measured in the different Ai states. This distance is used to find the components that have little impact on the discrimination of the assembly states and to exclude them in the evaluation process. The Bhattacharyya distance dB between the two different distributions p and q is given by
2 2 σq2 1 ⎛ 1 ⎛ σp 1 ⎛ (μp − μq ) ⎞ ⎞⎞ ln + 2 + 2⎟ ⎟ + ⎜ 2 2 ⎟ 4 ⎜ 4 ⎜⎝ σq2 4 σp ⎝ σp + σq ⎠ ⎠⎠ ⎝
Dx Dy Dz Drx Dry Drz
(2)
where μp, σp, μq, and σq are the means and standard deviations of the distributions p and q, respectively. The value of dB(p,q) becomes larger as the separation between the two distributions becomes larger. The Bhattacharyya distances for the force and deformation data measured in Ai and Ri are summarized in Tables 1 and 2, respectively. It is clear that dB(A3, R3) in Tables 1 and 2 is larger than dB(A1, R1) and dB(A2, R2), which means that it is easier to determine the assembly state with the data at the time of A3 formation than with those of A1 or A2. This is also shown in Figs. 7 and 8, in which the data distributions of A3 and R3 are farther apart than the other data distributions. Tables 3 and 4 summarize the Bhattacharyya distances based on the measured force and deformation data, respectively, when the three assembly states are formed. As a result of the evaluation, dB(A1, A2) is smaller than dB (A2, A3) and dB(A1, A3) in both force and deformation data. This means that the distribution of the measured data in A1 is close to that in A2, and the distributions in these two regions is far from that in A3, as already confirmed in the histogram of Fig. 9. In the case of dB(A1, A2), the force component Tx and deformation component Drx are 0.66 and 0.03, respectively, which are very small compared with other components; thus, the data distributions of the corresponding components in the histogram overlap each other in Fig. 9. If the distributions of the measured data are similar to each other when different assembly states are formed, as described above, the inclusion of these data has an adverse effect on the algorithm's ability to determine the assembly state.
Mf = [μij ], Md = [μij ],
Dx Dy Dz Drx Dry Drz
0.59 0.53 0.17 0.75 1.39 0.91
1.16 0.69 0.34 0.92 0.87 1.09
1.25 0.81 0.49 6.61 5.60 0.72
125.73 282.10 2726.64 1453.54 144.31 1.09
128.93 174.58 2237.29 1106.37 644.53 6.77
dB(A1,A2)
dB(A2,A3)
dB(A1,A3)
5.01 2.26 1.06 0.03 267.11 0.18
15.39 0.04 2.12 88.59 320.53 0.07
6.87 1.27 0.33 70.37 750.47 0.29
∑ = [σij](i = 1, ⋯, nA, j = 1, ⋯, 6) (3)
∑d
= [σij] (i = 1, ⋯, nA , j = 1, ⋯, 6)
(4)
4. Assembly process monitoring algorithm 4.1. Discrimination of assembly state The data constituting the model matrices reflect the contact pattern when the assembly states are successfully formed. Therefore, it is possible to estimate whether the assembly state is formed by comparing the model data with the data measured in real time in the assembly process. In this study, the Mahalanobis distance is used to evaluate the similarity of the data. The Mahalanobis distance dM(x) of the measured value x in the onedimensional space is obtained by using the mean μ and the standard deviation σ of the corresponding data as follows:
(x − μ)2 / σ 2
dM (x ) =
(5)
It is noted that the Mahalanobis distance represents the degree of separation of the data from the mean by a multiple of the standard deviation. Based on Eq. (5), the similarity for evaluating the formation of the three assembly states of the base frame assembly process is defined by
Table 2 Bhattacharyya distance of the deformation data distribution between Ai and Ri. dB(A3,R3)
1.67 2.80 16.39 0.66 141.34 2.99
where μij and σij are the mean and standard deviation of the data, respectively. In this study, Eqs. (3) and (4) are referred to as the forcebased model matrix and deformation-based model matrix, respectively. The size of each matrix varies depending on the number of assembly states defined in the assembly process and the number of collected data. In the case of the base frame assembly process, there are six force components and six deformation components for three assembly states (i.e., nA = 3), and thus, the model matrices of mean and standard deviation are all 3 × 6 matrices.
An assembly model is created based on the statistical parameters of the force and deformation data collected in the regions where the assembly state is formed as follows:
dB(A2,R2)
dB(A1,A3)
f
3.2. Modeling of assembly process
dB(A1,R1)
dB(A2,A3)
Table 4 Bhattacharyya distance of the deformation data distribution between three different assembly states.
dB (p , q) =
dB(A1,A2)
Si =
1 6
∑ j = 1 wij [(x j − μij )/ σij]2
(i = 1, 2, 3) (6)
where Si denotes the similarity between the data components of the measured value x and the data (μij, σij) reflecting Ai. Note that the similarity is defined as the inverse of the sum of the Mahalanobis distances for the data components of the measured value. Therefore, the 153
Robotics and Computer Integrated Manufacturing 56 (2019) 149–156
D.-H. Lee et al.
S1, S2, and S3 shown in red, green, and blue. In this case, because the similarities of (b), in which the weights are applied, are larger than those of (a), in which the weights are not applied, it is possible to evaluate with greater certitude whether the assembly is successful in the case of (b) than in (a). Fig. 12 shows the simulation results of the assembly process monitoring algorithm based on the deformation data. It is shown that when the weights are not applied, in Fig. 12(a), the discrimination of A1 and A2 of the successful assembly process is somewhat low. This is because both S1 and S2 are high in the formation of A1 and A2 owing to the similar data distribution. However, when the weight matrix Wd is applied, the discrimination of A1 and A2 increases owing to the improved similarity. In this study, we define two discrimination indexes, as shown below, to quantify the performance of the assembly process monitoring algorithm before and after weighting. First, the discrimination index Di of a successful assembly process is defined as
Mahalanobis distance becomes smaller, and thus the similarity becomes higher, as the measured value gets closer to the assembly model data. Furthermore, since the Mahalanobis distance has a standardization effect, the similarity is not affected by the difference in scale between the components of the measured value. Based on the results of the evaluation of the Bhattacharyya distance in Section 3, the following weighting matrices are introduced to exclude components that adversely affect the similarity evaluation.
⎡1 ⎢1 1 Wf = ⎢ ⎢0 ⎢1 ⎢0 ⎣
1 1 1 1 1 0
1⎤ ⎡1 1 1⎥ ⎢1 0 1⎥ 0 0 ,W =⎢ 1⎥ d ⎢0 1 ⎢1 1 1⎥ ⎢0 0 0⎥ ⎦ ⎣
1⎤ 1⎥ 0⎥ 0⎥ 1⎥ 0⎥ ⎦
(7)
where Wf and Wd are the weight matrices to exclude the force and deformation components that are near zero in the Bhattacharyya distance. For example, the Tx component in A1 is excluded from the similarity calculation because the 1st column and the 4th row in Wf is zero.
Di =
min (Si ) Ai max (Si ) Ri
( i = 1, 2, 3)
(8)
This index quantitatively indicates to what extent the similarity evaluated when forming the assembly state is larger than the similarity when it is not formed. In addition, the discrimination index D′i between success and failure of the assembly process is defined as
4.2. Assembly process monitoring algorithm The success or failure of the entire assembly process can be determined based on whether the individual assembly state was normally formed. Fig. 10 shows a flowchart of an algorithm for monitoring the assembling process of the base frame. When the assembly is started, the robot holding the base frame triggers the formation of A1, A2, and A3 through tilting, pushing, and inserting operations. The algorithm then calculates the similarities S1, S2, and S3 and compares them with the thresholds C1, C2, and C3 set by the user to determine whether the three assembly states are formed. Refer to Eq. (10) to set the size of the threshold. For example, if S1 > C1, S2 < C2, and S3 < C3 during the tilting operation, then the algorithm determines that A1 has been formed normally, otherwise it determines that A1 is abnormally formed. Moreover, if it is judged that all the three assembly states are normally formed, the algorithm regards the entire assembly process as a success.
D ′i =
min (Si ) Ai max (S ′i) Ai
( i = 1, 2, 3)
(9)
This index quantifies the difference in the similarity evaluated in Ai of the successful and failed assembly processes. Therefore, the discrimination index D′ quantitatively indicates how well the evaluated similarity reflects the success or failure of the assembly. Table 5 summarizes the force and deformation-based discrimination indices. As seen from the table, in both cases the discrimination indices are sufficiently large; thus, it is possible to judge whether the assembly state is formed or not and whether the assembly is successful or not using the concept of similarity. Furthermore, if the force data is used instead of the deformation data, then the discrimination index would be generally increased, and thus, better performance can be obtained.
4.3. Algorithm optimization through simulation 5. Experimental verification The performance of the proposed assembly process monitoring algorithm was evaluated through MATLAB simulations. Fig. 11 shows the similarity evaluated based on the force data measured on success and failure of the assembly process. It is possible to know whether the three assembled states A1, A2, and A3 are formed from the three similarities
Various experiments were conducted to investigate the monitoring performance of the proposed algorithm. As shown in Fig. 13, the first experiment was performed 100 times when the base frame was normally supplied. In the second experiment, the base frame was offset from the normal position by 3–5 mm in the front, rear, left, and right directions to induce failure and was performed 25 times for each case. In addition, the force-based and deformation-based assembly process monitoring algorithms were tested with identical assembly processes and their performance was evaluated independently. The size of the threshold value used to determine whether the assembly state is formed and the success or failure of the assembly process is calculated as follows:
Ci =
1 [min (Si ) Ai + max (Si ) Ri] ( i = 1, 2, 3) 2
(10)
Because Ci is calculated as the average of the minimum value of the similarity evaluated in Ai of the successful assembly process and the maximum value of the similarity evaluated in Ri, it should be between these two values. Thus, it is possible to distinguish whether the assembly state is formed or not. In addition, as the similarity evaluated in Ai of the failed assembly process is smaller than the similarity evaluated in Ri of the successful assembly process, it is also possible to estimate the success or failure of the assembly process using the above average value. Table 6 summarizes the results of the assembly process monitoring
Fig. 10. Flowchart of assembly process monitoring algorithm. 154
Robotics and Computer Integrated Manufacturing 56 (2019) 149–156
D.-H. Lee et al.
Fig. 11. Simulation results of the assembly process monitoring algorithm based on the force data: (a) without weighting, and (b) with weighting.(For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)
Fig. 12. Simulation results of the assembly process monitoring algorithm using the deformation data: (a) without weighting, and (b) with weighting. Table 5 Discrimination indexes based on force and deformation data. Force-based
A1
A2
A3
Di D′i
3.2 5.4
8.2 9.8
5.2 10.4
Deformation-based
A1
A2
A3
Di D′i
2.8 4.2
3.8 6.8
5.0 5.1
based on the force and deformation data, respectively. In both cases, the proposed algorithm succeeded in determining 100 successes of the actual assembly process as a success, and a false negative error (i.e., actually a success but judged as a failure) did not occur. In addition, the proposed algorithm estimated all the 100 processes that failed as a failure, and a false positive error, which is a failure mistakenly estimated as a success, did not occur. As a result, the algorithm predicted whether the assembly state was formed in the actual assembly process and the success or failure of the assembly.
Fig. 13. Intentional position error for experiments.
Table 6 Results of experiment using force and deformation data.
Success Failure
6. Conclusion In this study, we propose an assembly process monitoring algorithm 155
Estimated to success
Estimated to failure
100 0
0 100
Robotics and Computer Integrated Manufacturing 56 (2019) 149–156
D.-H. Lee et al.
and verify its performance through various experiments. The following conclusions were drawn.
[4] A. Stolt, On Robotic Assembly using Contact Force Control and Estimation, Ph.D Thesis Lund University, 2015. [5] J. Krüger, L. Wang, A. Verl, T. Bauernhansl, E. Carpanzano, S. Makris, J. Fleischer, G. Reinhart, J. Franke, S. Pellegrinelli, Innovative control of assembly systems and lines, CIRP Ann. Manuf. Tech. 66 (2017) 707–730. [6] R. Meziane, M.J.-D. Otis, H. Ezzaidi, Human-robot collaboration while sharing production activities in dynamic environment: SPADER system, Robot. Comput. Integr. Manuf. 48 (2017) 243–253. [7] A. Cherubini, R. Passama, A. Crosnier, A. Lasnier, P. Fraisse, Collaborative manufacturing with physical human-robot interaction, Robot. Comput. Integr. Manuf. 40 (2016) 1–13. [8] S. Makris, P. Tsarouchi, A.-S. Matthaiakis, A. Athanasatos, X. Chatzigeorgiou, M. Stefos, K. Giavridis, S. Aivaliotis, Dual arm robot in cooperation with humans for flexible assembly, CIRP Ann. Manuf. Tech. 66 (2017) 13–16. [9] Y.L. Kim, H.C. Song, J.-B. Song, Hole detection algorithm for chamferless square peg-in-hole based on shape recognition using F/T sensor, Int. J. Precis. Eng. Manuf. 15 (2014) 425–432. [10] S. Chhatpar, M. Branicky, A hybrid systems approach to force-guided robotic assemblies, Proc. Intl. Symp. Assembly Task Planning, 1999, pp. 301–306. [11] D.E. Whitney, Quasi-static assembly of compliantly supported rigid parts, J. Dyn. Syst. Meas. Control. 104 (1) (1982) 65–77. [12] C. Delprete, C. Rosso, G. Savino, C. Scarzella, Advanced vision approach applied to non-contact micro-measurements: a practical application, Int. J. Adv. Manuf. Technol. 88 (2017) 471–481. [13] G. Michalos, S. Makris, A. Eytan, S. Matthaiakis, G. Chryssolouris, Robot path correction using stereo vision system, Procedia CIRP 3 (2012) 352–357. [14] P. Tsarouchi, G. Michalos, S. Makris, G. Chryssolouris, Vision system for robotic handling of randomly placed objects, Procedia CIRP 9 (2013) 61–66. [15] W.C. Chang, Robotic assembly of smartphone back shells with eye-in-hand visual servoing, Robot. Comput. Integr. Manuf. 50 (2018) 102–113. [16] F. Chen, F. Cannella, J. Huang, H. Sasaki, T. Fukuda, A study on error recovery search strategies of electronic connector mating for robotic fault-tolerant assembly, J. Intell. Robot. Syst. Theory Appl. 81 (2016) 257–271. [17] W.S. Newman, Y. Zhao, Y.-H. Pao, Interpretation of force and moment signals for compliant peg-in-hole assembly, IEEE Intl. Conf. Robot. Autom. 2001, pp. 571–576. [18] W. Luo, J. Rojas, T. Guan, K. Harada, K. Nagata, Cantilever snap assemblies failure detection using SVMs and the RCBHT, Intl. Conf. Mechatron. Autom. 2014, pp. 384–389. [19] J. Huang, Y. Wang, T. Fukuda, Set-Membership-based fault detection and isolation for robotic assembly of electrical connectors, IEEE Trans. Autom. Sci. Eng. 15 (1) (2016) 160–171. [20] E. Moreira, L.F. Rocha, A.M. Pinto, A.P. Moreira, G. Veiga, Assessment of robotic picking operations using a 6 axis force/torque Sensor, IEEE Robot. Autom. Lett. 1 (2016) 768–775. [21] D.I. Park, H. Kim, C. Park, B. Kim, D. Kim, J.H. Kyung, Variable Passive Compliance Device for the Robotic Assembly, Intl. Conf. Ubiquitous Robot. Ambient Intell. 2017, pp. 753–754.
(1) The assembly process was modeled based on the data collected in the successful assembly process. (2) On performing the assembly, based on high-speed position control, and monitoring a success or failure of the assembly process by using the proposed assembly process monitoring algorithm, both successful and failed assembly processes were correctly identified. (3) The proposed assembly monitoring algorithm showed satisfactory performance not only for the force data obtained by a force/torque sensor but also for the deformation data obtained by the deformation sensor. It is confirmed that damage of the parts and robot can be prevented using the assembly process monitoring algorithm. The modeling of the assembly process proposed in this study is simple, and thus, it could be easily applied in industrial sites where various parts are assembled in large quantities and a short time. In addition, it is possible to monitor the assembling process using a low-cost deformation sensor, without requiring an expensive force/torque sensor. A re-assembly strategy after the detection of assembly failure will be developed in the future. Acknowledgment This study was supported by the MOTIE under the Industrial Foundation Technology Development Program, which is supervised by the KEIT (No. 10060110) References [1] K.S. Chin, M.M. Ratnam, R. Mandava, Force-guided robot in automated assembly of mobile phone, Assem. Autom. 23 (1) (2003) 75–86. [2] J.L. Navarro-Gonzalez, I. Lopez-Juarez, R. Rios-Cabrera, K. Ordaz-Hernández, Online knowledge acquisition and enhancement in robotic assembly tasks, Robot. Comput. Integr. Manuf 33 (2015) 78–89. [3] H.M. Do, C. Park, J.H. Kyung, Dual Arm Robot for Packaging and Assembling of IT Products, Int. Conf. Autom. Sci. Eng. (2012) 1067–1070.
156