- Email: [email protected]
Vision-based tile recognition algorithms for robot grasping task in EAST
Fusion Engineering and Design 152 (2020) 111422
Contents lists available at ScienceDirect
Fusion Engineering and Design journal homepage: www.elsevier.com/locate/fusengdes
Vision-based tile recognition algorithms for robot grasping task in EAST Jiasheng Liu a b c
a,b,c
a,c
, Kun Lu , Hongtao Pan
a,c,
a,c
*, Yong Cheng , Tao Zhang
a,b,c
T
a,b,c
, Zhixin Yao
Institute of Plasma Physics, Chinese Academy of Sciences, Hefei, Anhui, China University of Science and Technology of China, Hefei, Anhui, China Huainan New Energy Research Center, Anhui Provincial Key Laboratory of Special Welding, Hefei, Anhui, China
A R T I C LE I N FO
A B S T R A C T
Keywords: EAST Vision-based recognition Tiles Grasping task
During the operation of the experimental advanced superconducting tokamak (EAST), tiles installed on the vacuum vessel break and fall onto the divertor. To ensure high experimental performance, a robot removes the broken tiles and replaces them with fine tiles. However, one of the key issues for this task is to distinguish fallen tiles from other tiles for grasping. In addition, discrete spots caused by burn and corrosion on the tiles’ surface also bring challenges in vision-based recognition. Here, a novel breakpoint detection algorithm is proposed that recognizes fallen tiles. Experiments are performed to demonstrate the recognition and catch processes of fallen tiles and to test the robustness of the algorithm.
1. Introduction During the operation of experimental advanced superconducting tokamak (EAST), tiles installed on the first wall of the vacuum vessel break and fall onto the divertor. As seen from Fig. 1, a tile is broken and falls from its original position. To ensure high experimental performance, an EAMA [1] robot removes a fallen tile and replaces it with a fine tile during the time gap between plasma shot of each experiment; this guarantees the integrity of the first wall. However, this process is time-consuming and it is difficult to manually manipulate the EAMA with 11 degrees of freedom to grasp a fallen tile. A vision-based control strategy is useful for detecting the contour and position of a fallen tile, which can be used to guide the robotic arm for grasping. Generally, a vision-based recognition task involves two pre-steps: feature extraction and representation. The most common visual features include color, texture, and shape [2]. In color feature extraction, a color space or model [3], such as RGB and HSV, is specified to describe images. Meanwhile, texture features show repeated local patterns and their arrangement rules in images. Approaches to texture analysis are usually categorized into structural [4], statistical [5], model-based [6], and transform [7]. Finally, shape features can be divided into contour-based and region-based. The former extracts features mainly from the external boundaries of objects, while the latter extracts features from the entire region such as the Hough transformation [8] and shape-invariant moment methods [9]. Feature representation, whose performance heavily depends on feature extraction, is used to describe the object and dictate the
⁎
recognition strategy, robustness, and efficiency [2]. Typically, a model in which there is a big difference between objects and background is built to recognize the objects. Even with the abovementioned techniques, there are still some difficulties in detecting fallen tiles. Firstly, all the tiles in a vacuum vessel have the same texture and color; hence, shape is the only feature that can be used to recognize fallen tiles. Meanwhile, damage events resulting in fallen tiles are random, with differences in fallen locations leading to diverse surface imperfections. Therefore, it is difficult to find an effective representation to detect fallen tiles due to differences in their appearance and shape characteristics. Although the tile quadrilateral approximation proposed by Wang [10] can be used to detect tiles, it cannot recognize fallen tiles. Secondly, corrosion, as shown in Fig. 2, may cause changes in the appearance of tiles but does not break them, which inevitably disturb the detection of the tiles’ shape features. Hence, in this study, an effective shape representation called breakpoint and the corresponding breakpoint detection algorithm are proposed to recognize fallen tiles in EAST. The rest of this paper is organized as follows. Section 2 introduces the image representation and breakpoint theory, which is the fundamental technology of the proposed algorithm. Section 3 details the method to detect and grasp fallen tiles from image preprocessing to contour recognition. Section 4 describes the experiments performed to demonstrate the recognition and catch processes of a fallen tile and to test the feasibility and robustness of the algorithm. Finally, Section 5 summarizes the work.
Corresponding author at: Institute of Plasma Physics, Chinese Academy of Sciences, Hefei, Anhui, China. E-mail address: [email protected] (H. Pan).
https://doi.org/10.1016/j.fusengdes.2019.111422 Received 10 July 2019; Received in revised form 27 October 2019; Accepted 21 November 2019 0920-3796/ © 2019 Elsevier B.V. All rights reserved.
Fusion Engineering and Design 152 (2020) 111422
J. Liu, et al.
Fig. 1. Broken and fallen tile.
Fig. 2. Corrosion of tiles.
2. Image representation The proposed algorithm is based on the special background constructed by rows of tiles in a vacuum vessel. The edges of the tiles form a network on the first wall. For any tile that has fallen without breaking into very small pieces, it can cover at least one edge of other fine tiles. Thus, the lines of the network will be broken and breakpoints will appear at the broken lines. The breakpoint is a point where a continuous line is broken into two parts, which usually exists in pairs. Fig. 3 shows a pair of breakpoints marked by two red circles and created by occlusion of a rectangular fallen tile. The method to detect the breakpoint will be discussed in detail in Section 3.2.
3. Methods The process of detecting fallen tiles is shown in Fig. 4. Firstly, an EAMA robot moves along a designated path and checks the first wall surface to find all fallen tiles. Then, the camera on the end effector captures real-time RGB images and transfers them to the PC to implement image preprocessing and breakpoint detection. Once breakpoints
Fig. 4. Process of detecting fallen tiles.
are found, the PC will send a stop signal to the EAMA to stop tracking. Meanwhile, the depth of the image is captured by a depth camera and sent to the PC. By combining breakpoints with the depth images, the PC can calculate a fallen tile’s position and use it to calculate the direction for the control of the EAMA to grasp the fallen tile automatically. The performance of the proposed breakpoint detection algorithm relies on the edge features of other fine tiles on the background. Thus, it is important to perform high-quality image preprocessing to extract the edge features. 3.1. Image preprocessing The image preprocessing for extracting edges of tiles involves three steps, as shown in Fig. 5. Binarization and connected-component labeling [11–13] are performed to filter corrosion traces and extract the tiles’ edges. Then, the edges are transformed into lines via the Hough transformation.
Fig. 3. A pair of breakpoints marked by the red circles (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.). 2
Fusion Engineering and Design 152 (2020) 111422
J. Liu, et al.
Fig. 7. Mask for connected-component labeling.
3.1.1. Image binarization Image binarization is used to extract the edge features of tiles. As can be seen in Fig. 6, the original RGB image in Fig. 6a is converted into a binary image in Fig. 6b through binarization. In this process, the RGB image is converted into a grayscale image. Each pixel in the image is represented by a number between 0 and 255. Then, to binarize an object pixel, the mean of all the pixels in an n×n block around the object pixel is used as the threshold such that if the object pixel is larger than the threshold, it becomes 0, otherwise, it becomes 1. In this article, the object pixels are 1 and the background pixels are 0. The value of n for the block is chosen to be an odd number and greater than or equal to 3, which is obtained by considering many values and finding the optimal value that gives the best results. 3.1.2. Connected-component labeling Connected-component labeling is used to filter out white discrete points caused by corrosion traces and extract more pure edge features of tiles from the binary image; the result of this process is shown in Fig. 6c. It marks all object pixels in the binarized image with labels such that all connected object pixels have the same label. Labeling an object pixel involves choosing one of the labels (if there are any) in a mask,
Fig. 5. Flow chart of steps involved in image preprocessing.
Fig. 6. a. Original image with a fallen tile. b. Binarized image. c. Result after performing connected-component labeling filtering. d. Result after performing Hough transformation. 3
Fusion Engineering and Design 152 (2020) 111422
J. Liu, et al.
discuss it in detail.
such as in Fig. 7, as the label or creating a new label. We use a fast connected-component labeling method to choose the labels [12]. The algorithm is as follows:
3.2. Breakpoint detection Although short lines representing edges were obtained, they cannot be directly used for breakpoint detection without further processing. Considering that the breakpoint detection theory is based on a continuous line, it is necessary to obtain practical lines in images rather than short lines. Practical lines are obtained by checking the positional relationship between short lines, i.e., whether they are close and have similar slopes, and then combining the lines meeting the conditions into a set. A line set represents a practical line. The algorithm is as follows.
In the above, rows×cols is the size of the binarized image. pi and ci represent the pixel value and the label of the ith pixel in the mask, respectively, as shown in Fig. 7. bxy represents the label of the pixel at (x, y). According to the traversal scan direction of this algorithm, all the pixels in the mask have already been processed before processing the current pixel of (x, y). For example, the algorithm processes the image in the order of (1, 1), (2, 1), … (5, 5), (6, 5), (7, 5), …(5, 6), …, (cols, rows). Before processing the pixel at (5, 6), the mask pixels (5, 5), (6, 5), (7, 5), and (5, 6) have been processed; thus, the function of the algorithm is achieved so that all the connected pixels of the mask pixels are marked by the same labels. Therefore, any connected pixels in the mask have the same label. Because the 3rd pixel at (x, y−1) is connected to all pixels in the mask, when it becomes the object pixel, other object pixels in the mask will have the same label as the 3rd pixel. Hence, in case 1 of the algorithm, c3 is chosen to be bxy. Then, the next three cases of the algorithm consider the 4th pixel at (x+1, y−1). Since the 4th pixel isn’t connected with the 1st and 2nd pixels, they may have different labels. However, they are now connected through the current pixel at (x, y). Thus, they must be combined into the same labels. It is also necessary to process the connected components of the 4th pixel at (x+1, y−1) and the 1st pixel at (x−1, y) or the 2nd pixel at (x−1, y−1) like that. The functions NewSet(L), CombineSet(c4, c2), and ConsecutiveLabel(L−1) are used to implement these steps [13]. Their simpler pseudocodes were then proposed in the equivalent-label-set strategy of Section 3.3.2 in Ref. [11], with their source code and the code for breakpoint detection in C++ stored at https://data.mendeley.com/datasets/hnkn8kgvsh/2. The abovementioned three functions are also used in the next breakpoint detection. Finally, when there is no pixel in the mask that is an object pixel, a new pixel is set as bxy. The function NewSet(L) is used to construct some arrays that store the connection relationship. The function ConsecutiveLabel(L−1) is used to make the labels of connected components consecutive. As can be seen from the binary image in Fig. 6b, the white discrete points are much smaller in area than the network formed by the tiles’ edges. Hence, the next task is to filter out the white discrete points through connected-component labelling. This is done by keeping the pixels with the label of the maximum quantity and removing the rest in the image.
In the above, the variable L represents the set of short lines detected by Hough transformation. Each line is represented as [x1, y1, x2, y2], corresponding to the coordinates of the end points of the line. The variable N is the size of L. In step 3, all short lines are labeled in set B, with the maximal label value m−1 obtained according to the following function.
In the above, Ai represents the angle between the short line Li and the transverse axis. The distance between the short lines Li and Lj can be computed according to Eq. (1).
d i, j =
(x i − x j )2 + (yi − yj )2 sin ⎜⎛ ⎝
Ai + Aj 2
− arctan
yi − yj ⎞ ⎟, xi − xj ⎠
(1)
where (xi, yi) and (xj, yj) are the positions of the center points of Li and Lj. di, j is the projection length of the line of centers in the vertical direction of the short lines Li and Lj, as shown in Fig. 8. Once the distance d between L1 and L2 is short enough, the two lines are considered to be connected and marked with the same label. In general, a new iteration short line Lj is marked as the label of its connected short line Li (Bj = Bi), or as a new label when its connected short lines have no labels (Bj = m). However, once Lj already had a label Bj before this iteration and it’s detected to be connected with Li in this iteration, labels Bi and Bj should be combined(CombineSet(Bi, Bj)). i.e., any short line with labels as the same as Bi or Bj should be marked
3.1.3. Hough transformation Hough transformation [8] transforms edges into short lines for subsequent steps. It projects the position of pixels into parameter space. Note that this is a standard method in the literature; hence, we do not
Fig. 8. Distance between two short lines. Note that o1 and o2 are the center points. o1p is a line with an average slope of L1 and L2. 4
Fusion Engineering and Design 152 (2020) 111422
J. Liu, et al.
as the smaller of Bi and Bj.The equivalent-label-set strategy was proposed to solve this problem [13], where three sets that store the relationship between labels are constructed using the function NewSet as follows.
In the above, R corresponds to the representative label of a label. e.g., Ri = j means that all short lines with label i are connected with lines with label j, and j is always less than i. Using the function CombineSet defined below, all connected short lines will have the same representative label.
Fig. 10. Detected breakpoints.
In step 4 of the combination algorithm, the function ConsecutiveLabel is used to generate consecutive final labels, which is defined as follows.
3.3. Detection of fallen tiles 3.3.1. Foundation Once the breakpoints are detected, the depth camera starts and the captured depth image is processed using the watershed segmentation algorithm [14] to get the contour of the fallen tile. This algorithm can separate an image into different regions according to the initial points in these regions and if there are obvious boundaries between these regions. Two groups of points, i.e., one inside the contour of the fallen tile and the other outside, can be used to separate the depth image into the fallen tile part and the rest part because there is an obvious boundary between the fallen tile and the tiles around it in the depth image. The next step is to get these two groups of points. In many cases, the breakpoints are on the contour of the fallen tile. Thus, a point on a line segment that links a pair of breakpoints must be within the contour, while the other point on the extension line of the line segment must be outside the contour. In general, to avoid the two points to be at singular positions, e.g., a location where the depth camera cannot measure the depth successfully, a points within a circle around each of the two points are used as the initial points in the watershed segmentation algorithm. An example of a depth image for executing the watershed segmentation algorithm is shown in Fig. 11a. For display purposes, the depth image was normalized from 0 to 255. Note, however, that most of the depth values were greater than 255 mm; thus, we used the depth values directly instead of the depth image. As shown in Fig. 11b, the fallen tile is extracted by watershed segmentation and is framed by its minimum circumscribed rectangle. The two white circles show the initial two points that are within and outside the contour of the fallen tile. During grasping, two long edges of the circumscribed rectangle are exactly the contact sides of the grasp. In a depth image, the position of the circumscribed rectangle can be computed using Eq. (2).
In step 5 of the combination algorithm, only connected short lines can be assigned to the same subset Cm of C. The judgment criterion to combine short lines into Cm is shown in Fig. 9. Firstly, short lines have similar slopes. Then, there is at least one straight line with a similar slope that could be constructed to pass through the short lines within a certain range of errors. Therefore, Cm could represent a practical line. The breakpoint detection algorithm is defined as follows. Am, 1 represents the angle between the short line Cm, 1 and the transverse axis. xm, 1, 1 and xm, 1, 2 represent the x-coordinates of two end points of the short line Cm, 1. max(xm, 1, 1, xm, 1, 2) returns the larger one between xm, 1, 1 and xm, 1, 2. Firstly, the algorithm sorts short lines from one end of a practical line Cm to the other. Then, the continuity of Cm is detected. As shown in Fig. 9, since p2 is on the right of p3, Cm is judged to be continuous between p1 and p4. However, p4 is on the left of p5; hence, Cm is not continuous between p4 and p5. Therefore, the practical line represented by Cm has no continuity, while the disconnected positions p4 and p5 are the pair of breakpoints. As shown in Fig. 10, four breakpoints are circled out, and the red lines are the short lines detected by Hough transformation.
Fig. 9. Three short lines in Cm. Note that L1, L2, and L3 are three adjacent short lines, and the dotted line represents the constructed line. p1 and p2 are the end points of L1. p3 and p4 are the end points of L2. p5 and p6 are the end points of L3. 5
Fusion Engineering and Design 152 (2020) 111422
J. Liu, et al.
Fig. 11. a. Normalized depth image. b. Fallen tile separated by watershed segmentation.
u ⎡X⎤ ⎡ ⎤ −1 ⎢Y ⎥ = Z × K × ⎢v⎥ ⎣1⎦ ⎣Z⎦ , 0 323.774472 ⎤ ⎡576.706989 K=⎢ 0 576.786077 251.742973⎥ 0 0 1 ⎣ ⎦
4. Experiment 4.1. Experimental set up To test the feasibility and robustness of the breakpoint detection algorithm, a platform was built to simulate the environment inside a vacuum vessel, as shown in Fig. 13. Some tiles collected from the EAST vacuum vessel were closely placed on the platform to simulate the divertor’s first wall. A universal robot (UR) with six degrees of freedom and a three-finger manipulator was mounted on the left side of the platform. A calibrated RGB+depth(RGB-D) camera was mounted at the end of the robot to form an eye-in-hand system.
(2)
where u and v are the pixel positions of a point, such as the center point of the minimum circumscribed rectangle in the depth image, and X, Y, and Z are the point’s position in the depth camera such that Z is the point’s depth value. In addition, K is the camera matrix of the depth camera. 3.3.2. Improvement In many cases, the breakpoints are on the contour of the fallen tile. However, in some other cases, the breakpoints may be on the edges of other tiles such as the unexpected pair of breakpoints shown in Fig. 12a. The unexpected pair of breakpoints appeared because one of the sides of the fallen tile was in line but not connected with that of another tile. Once this unexpected pair of breakpoints are used in watershed segmentation, two wrong groups of points will be chosen as the initial points, as we can see from the two white circles in Fig. 12b. Then, there will be an unexpected segmentation in the depth image. Thus, the unexpected pair of breakpoints need to be excluded. Since the unexpected segmentation makes the minimum circumscribed rectangle bigger than the rectangle by the expected segmentation, the area size of the minimum circumscribed rectangle can be used to exclude the unexpected pair of breakpoints. We use each pair of breakpoints as the initial points in watershed segmentation, and then record the area size of the minimum circumscribed rectangle. Finally, the watershed segmentation corresponding to the least area size is chosen as the valid watershed segmentation to detect the fallen tile, as shown in Fig. 12c. Therefore, once any valid pair of breakpoints are recognized in breakpoint detection, any other unexpected breakpoints cannot disturb the detection of the fallen tile.
4.2. Feasibility test In the experiment, the UR robot tracked a designated path, as shown in Fig. 14a. During the path, real-time RGB images were obtained similarly to Fig. 6a and then processed and extracted into lines similarly to Fig. 6d, and finally processed using the breakpoint detection algorithm described in Section 3.2. If there was no fallen tile in the visual field of the camera, the UR robot would continuously do the tracking and image preprocessing and breakpoint detection would keep working. However, once a fallen tile exists, the breakpoints would be obtained similarly to Fig. 10, which triggers the UR robot, the RGB camera, and image processing to stop, as shown in Fig. 14b. This triggers the start of the depth camera and watershed segmentation of the depth image to obtain the position of the fallen tile similarly to Fig. 11b. Finally, the UR robot grasps and takes away the fallen tile according to the obtained position, as shown in Fig. 14c and d. 4.3. Robustness test 4.3.1. Robustness to fallen location The location of the fallen tile on the divertor of the EAST can be
Fig. 12. a. Unexpected breakpoints. b. Unexpected watershed segmentation. c Valid watershed segmentation. 6
Fusion Engineering and Design 152 (2020) 111422
J. Liu, et al.
Fig. 13. Experimental platform.
Fig. 15. Construction of the coordinate system in the depth image.
random because the fallen tile can come from anywhere from the first wall of the EAST. In Section 3, only one location was investigated, which is not sufficient to prove the robustness of the proposed algorithm as applied in the EAST. Therefore, we carried out a series of experiments where the fallen tile is thrown down from 50 cm above the platform to produce random fallen locations. In the experiments, the robot remained still, while RGB and depth images were captured after each throw of the fallen tile. After 100 throws, all the RGB and depth images were processed in bulk. The processed images, including binarized images, images after connected-component labeling filtering, images of breakpoint detection, images of watershed segmentation, and poses, can be obtained at https://doi.org/10.17632/jrpn2w2dwj.3. The poses (positions+orientations) are presented in the format “i, x, y, z, ε1, ε2, ε3, ε4,” where i is the experiment’s throw number, x, y, z (in mm) are the coordinates of the position of the fallen tile, and ε1, ε2, ε3, ε4 (Euler parameters) are the orientations relative to the depth camera’s coordinate system. An example of constructing the coordinate system of the depth camera and that of the fallen tile for the first experiment is shown in Fig. 15; the pose is "1, 106, −28, 350, 0, 0, −0.38, 0.92.″
Because the camera is parallel to the platform, ε1 and ε2 are 0. The correct poses of the fallen tiles were successfully achieved in 98 out of 100 experiments; there were no breakpoints detected for the two failures. The images after connected-component labeling filtering, as shown in Fig. 16a and b, demonstrate that the two failures occurred because the upper-left part of the network was filtered out due to three disconnections, as shown in the binarized images in Fig. 16c and d. The first two disconnections were due to the fallen tile, while the last one was due to the narrow black gap because of the narrow spacing between the two tiles, as shown by the red circle in Fig. 16e or f. In conclusion, we found that the proposed algorithm is robust to the fallen location, but failures may occur when the spacing between tiles is too narrow or when the fallen tile is in the corner of the visual field of the camera. In addition, a special fallen location where the fallen tile overlaps with another tile, as shown in Fig. 16g, can also invalidate the proposed algorithm, but the probability of it occurring is small; in fact, we did not observe this special location in the 100 experiments performed.
Fig. 14. a. Tracking, b. stopping, c. grasping, and d. taking away processes of detecting and grasping a fallen tile. 7
Fusion Engineering and Design 152 (2020) 111422
J. Liu, et al.
Fig. 16. a.–b. Connected-component labeling filtering images. c.–d. Binarized images. e.–f. Original RGB images of the two failures. g. Special fallen location.
a light source was firstly shone on the center of the platform and then removed gradually. During the movement, RGB and depth images were captured and the illuminance was measured six times. Fig. 17a–f show the original RGB images. With the movement of the
4.3.2. Robustness to illumination 4.3.2.1. Illuminance. An experiment with varying illumination conditions but with the same fallen location was carried out to test the proposed algorithm’s robustness to illumination. In the experiment, 8
Fusion Engineering and Design 152 (2020) 111422
J. Liu, et al.
Fig. 17. a.–f. Original RGB images. g.–l. Binarized images. m.–r. Detected breakpoints. s.–w. Detected fallen tiles in the depth images.
former five images can almost be detected, while those in the last one cannot. According to the images of breakpoint detection in Fig. 17 m–r, a similar conclusion can be drawn where the breakpoints can be
light source, the illuminance on the surface of the fallen tile becomes darker, i.e., 132, 67, 52, 24, 20, and 15 lx in the images. According to the binarized images in Fig. 17g–l, the edge features of the tiles in the 9
Fusion Engineering and Design 152 (2020) 111422
J. Liu, et al.
Fig. 18. a. Original RGB image. b. Binarized image. c. Detected breakpoints. d. Detected fallen tile.
detected in the former five images, while those in the last one cannot. Fig. 17s–w show the fallen tiles detected by watershed segmentation in the depth images.
was carried out in the depth image, which is not influenced by illumination.
5. Conclusion
4.3.2.2. Shadow. To test if shadow affects the performance of the proposed algorithm, we carried out experiments where the fallen tile is covered with a shadow, as shown in Fig. 18a. The illuminance on the surface of the fallen tile in Fig. 18a is 17 lx. As shown in Fig. 18b, the contour of the shadow was detected by the algorithm, resulting in inaccuracies in the detection of the edges of the tiles. Moreover, the effect of the shadow resulted in many extra useless breakpoints as shown in Fig. 18c. However, despite these extra useless breakpoints, we still detected a pair of valid breakpoints; hence, the fallen tile can be detected, as shown in Fig. 18d.
This paper proposed a method to support the removal of fallen tiles in the EAST vacuum vessel. The method included a novel breakpoint detection algorithm to detect the existence of fallen tiles and a watershed segmentation algorithm to recognize the contour of fallen tiles. Experiments using a UR robot were performed to demonstrate the recognition and grabbing processes of fallen tiles and to test the robustness of the algorithm by considering the fallen locations of fallen tiles and illumination.
4.3.2.3. Results. To see if the illumination condition can influence the accuracy of the detection, the poses of the fallen tile in the experiments are listed in Table 1. Numbers 1–6 correspond to the experiments of Section 4.3.2.1, while number 7 corresponds to the experiment of Section 4.3.2.2. As shown in Table 1, the displacement standard deviation is less than 0.4 mm and the orientation variation is very small. Therefore, illumination does not affect the accuracy, only the rate of success, of finding the fallen tiles; it only influences the RGB image processed by the breakpoint detection algorithm. Moreover, breakpoint detection was used to find an approximate position of the fallen tile and provide a marker within the fallen tile’s contour for watershed segmentation. On the other hand, watershed segmentation
Funding This work was supported by the National Key R&D Program of China (CFETR Engineering Design Project, No. 2017YFE0300503) and the China Postdoctoral Science Foundation [Grant No. 2017M620260].
Declaration of Competing Interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Table 1 Poses of the fallen tile in the depth camera in different experiments. Number Illuminance/lx Pose
x/mm y/mm z/mm ε1 ε2 ε3 ε4
1
2
3
4
5
6
7
Standard Deviation
132 35.53 −31.30 349 0 0 −0.235 0.971
67 35.53 −30.40 349 0 0 −0.226 0.973
52 35.53 −31.30 349 0 0 −0.228 0.973
24 35.53 −31.00 349 0 0 −0.229 0.973
20 35.23 −30.70 349 0 0 −0.229 0.973
15 – – – – – – –
17 35.84 −30.40 349 0 0 −0.224 0.975
0.176 0.377 0 0 0 0.0034 0.0012
10
Fusion Engineering and Design 152 (2020) 111422
J. Liu, et al.
Acknowledgment
ijcseit.2012.2213. [6] Z. Kato, T.C. Pong, A Markov random field image segmentation model for color textured images, Image Vis. Comput. 24 (2006) 1103–1114, https://doi.org/10. 1016/j.imavis.2006.03.005. [7] S. Arivazhagan, L. Ganesan, S.P. Priyal, Texture classification using Gabor wavelets based rotation invariant features, Pattern Recognit. Lett. 27 (2006) 1976–1982, https://doi.org/10.1016/j.patrec.2006.05.008. [8] P. Hart, R.O. Duda, Use of the Hough transformation to detect lines and curves in pictures, Commun. ACM 15 (1972) 11–15 http://trac.astrometry.net/export/ 19652/trunk/documents/papers/dstn-review/papers/duda1972.pdf. [9] S. Li, M.C. Lee, C.M. Pun, Complex Zernike moments features for shape-based image retrieval, IEEE Trans. Syst. Man, Cybern. Part A Syst. Hum. 39 (2009) 227–237, https://doi.org/10.1109/TSMCA.2008.2007988. [10] H. Wang, L. Xu, W. Chen, Design and implementation of visual inspection system handed in tokamak flexible in-vessel robot, Fusion Eng. Des. 106 (2016) 21–28, https://doi.org/10.1016/j.fusengdes.2016.03.005. [11] L. He, X. Ren, Q. Gao, X. Zhao, B. Yao, Y. Chao, The connected-component labeling problem: a review of state-of-the-art algorithms, Pattern Recognit. (2017), https:// doi.org/10.1016/j.patcog.2017.04.018. [12] L. He, Y. Chao, K. Suzuki, K. Wu, Fast connected-component labeling, Pattern Recognit. 42 (2009) 1977–1987, https://doi.org/10.1016/j.patcog.2008.10.013. [13] L. He, Y. Chao, K. Suzuki, A linear-time two-scan labeling algorithm, Proc. - Int. Conf. Image Process. ICIP (2007), https://doi.org/10.1109/ICIP.2007.4379810. [14] L. Vincent, L. Vincent, P. Soille, Watersheds in digital spaces: an efficient algorithm based on immersion simulations, IEEE Trans. Pattern Anal. Mach. Intell. (1991), https://doi.org/10.1109/34.87344.
The authors would also like to express their thanks toeditor Akio Sagara of the journal and the anonymous reviewers for their valuable comments about the content of the paper and editor Jorge of Enago for providing language help. References [1] S. Shi, Y. Song, Y. Cheng, E. Villedieu, V. Bruno, H. Feng, H. Wu, P. Wang, Z. Hao, Y. Li, K. Wang, H. Pan, Conceptual design main progress of EAST Articulated Maintenance Arm (EAMA) system, Fusion Eng. Des. 104 (2016) 40–45, https://doi. org/10.1016/j.fusengdes.2016.02.005. [2] D. ping, T. ping Tian, A review on image feature extraction and representation techniques, Int. J. Multimed. Ubiquitous Eng. 8 (2013) 385–396, https://doi.org/ 10.1093/ecco-jcc/jjw221. [3] W.T. Chen, W.C. Liu, M.S. Chen, Adaptive color feature extraction based on image color distributions, IEEE Trans. Image Process. 19 (2010) 2005–2016, https://doi. org/10.1109/TIP.2010.2051753. [4] P. Cheng, J. Wang, C. Miao, C. Tu, Constrained Texture Mapping Via Approximate Voronoi Base Domain, (2018), pp. 19–22, https://doi.org/10.1145/3208159. 3208188. [5] B.S. V, Grey level co-occurrence matrices: Generalisation and some new features, Int. J. Comput. Sci. Eng. Inf. Technol. 2 (2012) 151–157, https://doi.org/10.5121/
11
Contents lists available at ScienceDirect
Fusion Engineering and Design journal homepage: www.elsevier.com/locate/fusengdes
Vision-based tile recognition algorithms for robot grasping task in EAST Jiasheng Liu a b c
a,b,c
a,c
, Kun Lu , Hongtao Pan
a,c,
a,c
*, Yong Cheng , Tao Zhang
a,b,c
T
a,b,c
, Zhixin Yao
Institute of Plasma Physics, Chinese Academy of Sciences, Hefei, Anhui, China University of Science and Technology of China, Hefei, Anhui, China Huainan New Energy Research Center, Anhui Provincial Key Laboratory of Special Welding, Hefei, Anhui, China
A R T I C LE I N FO
A B S T R A C T
Keywords: EAST Vision-based recognition Tiles Grasping task
During the operation of the experimental advanced superconducting tokamak (EAST), tiles installed on the vacuum vessel break and fall onto the divertor. To ensure high experimental performance, a robot removes the broken tiles and replaces them with fine tiles. However, one of the key issues for this task is to distinguish fallen tiles from other tiles for grasping. In addition, discrete spots caused by burn and corrosion on the tiles’ surface also bring challenges in vision-based recognition. Here, a novel breakpoint detection algorithm is proposed that recognizes fallen tiles. Experiments are performed to demonstrate the recognition and catch processes of fallen tiles and to test the robustness of the algorithm.
1. Introduction During the operation of experimental advanced superconducting tokamak (EAST), tiles installed on the first wall of the vacuum vessel break and fall onto the divertor. As seen from Fig. 1, a tile is broken and falls from its original position. To ensure high experimental performance, an EAMA [1] robot removes a fallen tile and replaces it with a fine tile during the time gap between plasma shot of each experiment; this guarantees the integrity of the first wall. However, this process is time-consuming and it is difficult to manually manipulate the EAMA with 11 degrees of freedom to grasp a fallen tile. A vision-based control strategy is useful for detecting the contour and position of a fallen tile, which can be used to guide the robotic arm for grasping. Generally, a vision-based recognition task involves two pre-steps: feature extraction and representation. The most common visual features include color, texture, and shape [2]. In color feature extraction, a color space or model [3], such as RGB and HSV, is specified to describe images. Meanwhile, texture features show repeated local patterns and their arrangement rules in images. Approaches to texture analysis are usually categorized into structural [4], statistical [5], model-based [6], and transform [7]. Finally, shape features can be divided into contour-based and region-based. The former extracts features mainly from the external boundaries of objects, while the latter extracts features from the entire region such as the Hough transformation [8] and shape-invariant moment methods [9]. Feature representation, whose performance heavily depends on feature extraction, is used to describe the object and dictate the
⁎
recognition strategy, robustness, and efficiency [2]. Typically, a model in which there is a big difference between objects and background is built to recognize the objects. Even with the abovementioned techniques, there are still some difficulties in detecting fallen tiles. Firstly, all the tiles in a vacuum vessel have the same texture and color; hence, shape is the only feature that can be used to recognize fallen tiles. Meanwhile, damage events resulting in fallen tiles are random, with differences in fallen locations leading to diverse surface imperfections. Therefore, it is difficult to find an effective representation to detect fallen tiles due to differences in their appearance and shape characteristics. Although the tile quadrilateral approximation proposed by Wang [10] can be used to detect tiles, it cannot recognize fallen tiles. Secondly, corrosion, as shown in Fig. 2, may cause changes in the appearance of tiles but does not break them, which inevitably disturb the detection of the tiles’ shape features. Hence, in this study, an effective shape representation called breakpoint and the corresponding breakpoint detection algorithm are proposed to recognize fallen tiles in EAST. The rest of this paper is organized as follows. Section 2 introduces the image representation and breakpoint theory, which is the fundamental technology of the proposed algorithm. Section 3 details the method to detect and grasp fallen tiles from image preprocessing to contour recognition. Section 4 describes the experiments performed to demonstrate the recognition and catch processes of a fallen tile and to test the feasibility and robustness of the algorithm. Finally, Section 5 summarizes the work.
Corresponding author at: Institute of Plasma Physics, Chinese Academy of Sciences, Hefei, Anhui, China. E-mail address: [email protected] (H. Pan).
https://doi.org/10.1016/j.fusengdes.2019.111422 Received 10 July 2019; Received in revised form 27 October 2019; Accepted 21 November 2019 0920-3796/ © 2019 Elsevier B.V. All rights reserved.
Fusion Engineering and Design 152 (2020) 111422
J. Liu, et al.
Fig. 1. Broken and fallen tile.
Fig. 2. Corrosion of tiles.
2. Image representation The proposed algorithm is based on the special background constructed by rows of tiles in a vacuum vessel. The edges of the tiles form a network on the first wall. For any tile that has fallen without breaking into very small pieces, it can cover at least one edge of other fine tiles. Thus, the lines of the network will be broken and breakpoints will appear at the broken lines. The breakpoint is a point where a continuous line is broken into two parts, which usually exists in pairs. Fig. 3 shows a pair of breakpoints marked by two red circles and created by occlusion of a rectangular fallen tile. The method to detect the breakpoint will be discussed in detail in Section 3.2.
3. Methods The process of detecting fallen tiles is shown in Fig. 4. Firstly, an EAMA robot moves along a designated path and checks the first wall surface to find all fallen tiles. Then, the camera on the end effector captures real-time RGB images and transfers them to the PC to implement image preprocessing and breakpoint detection. Once breakpoints
Fig. 4. Process of detecting fallen tiles.
are found, the PC will send a stop signal to the EAMA to stop tracking. Meanwhile, the depth of the image is captured by a depth camera and sent to the PC. By combining breakpoints with the depth images, the PC can calculate a fallen tile’s position and use it to calculate the direction for the control of the EAMA to grasp the fallen tile automatically. The performance of the proposed breakpoint detection algorithm relies on the edge features of other fine tiles on the background. Thus, it is important to perform high-quality image preprocessing to extract the edge features. 3.1. Image preprocessing The image preprocessing for extracting edges of tiles involves three steps, as shown in Fig. 5. Binarization and connected-component labeling [11–13] are performed to filter corrosion traces and extract the tiles’ edges. Then, the edges are transformed into lines via the Hough transformation.
Fig. 3. A pair of breakpoints marked by the red circles (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.). 2
Fusion Engineering and Design 152 (2020) 111422
J. Liu, et al.
Fig. 7. Mask for connected-component labeling.
3.1.1. Image binarization Image binarization is used to extract the edge features of tiles. As can be seen in Fig. 6, the original RGB image in Fig. 6a is converted into a binary image in Fig. 6b through binarization. In this process, the RGB image is converted into a grayscale image. Each pixel in the image is represented by a number between 0 and 255. Then, to binarize an object pixel, the mean of all the pixels in an n×n block around the object pixel is used as the threshold such that if the object pixel is larger than the threshold, it becomes 0, otherwise, it becomes 1. In this article, the object pixels are 1 and the background pixels are 0. The value of n for the block is chosen to be an odd number and greater than or equal to 3, which is obtained by considering many values and finding the optimal value that gives the best results. 3.1.2. Connected-component labeling Connected-component labeling is used to filter out white discrete points caused by corrosion traces and extract more pure edge features of tiles from the binary image; the result of this process is shown in Fig. 6c. It marks all object pixels in the binarized image with labels such that all connected object pixels have the same label. Labeling an object pixel involves choosing one of the labels (if there are any) in a mask,
Fig. 5. Flow chart of steps involved in image preprocessing.
Fig. 6. a. Original image with a fallen tile. b. Binarized image. c. Result after performing connected-component labeling filtering. d. Result after performing Hough transformation. 3
Fusion Engineering and Design 152 (2020) 111422
J. Liu, et al.
discuss it in detail.
such as in Fig. 7, as the label or creating a new label. We use a fast connected-component labeling method to choose the labels [12]. The algorithm is as follows:
3.2. Breakpoint detection Although short lines representing edges were obtained, they cannot be directly used for breakpoint detection without further processing. Considering that the breakpoint detection theory is based on a continuous line, it is necessary to obtain practical lines in images rather than short lines. Practical lines are obtained by checking the positional relationship between short lines, i.e., whether they are close and have similar slopes, and then combining the lines meeting the conditions into a set. A line set represents a practical line. The algorithm is as follows.
In the above, rows×cols is the size of the binarized image. pi and ci represent the pixel value and the label of the ith pixel in the mask, respectively, as shown in Fig. 7. bxy represents the label of the pixel at (x, y). According to the traversal scan direction of this algorithm, all the pixels in the mask have already been processed before processing the current pixel of (x, y). For example, the algorithm processes the image in the order of (1, 1), (2, 1), … (5, 5), (6, 5), (7, 5), …(5, 6), …, (cols, rows). Before processing the pixel at (5, 6), the mask pixels (5, 5), (6, 5), (7, 5), and (5, 6) have been processed; thus, the function of the algorithm is achieved so that all the connected pixels of the mask pixels are marked by the same labels. Therefore, any connected pixels in the mask have the same label. Because the 3rd pixel at (x, y−1) is connected to all pixels in the mask, when it becomes the object pixel, other object pixels in the mask will have the same label as the 3rd pixel. Hence, in case 1 of the algorithm, c3 is chosen to be bxy. Then, the next three cases of the algorithm consider the 4th pixel at (x+1, y−1). Since the 4th pixel isn’t connected with the 1st and 2nd pixels, they may have different labels. However, they are now connected through the current pixel at (x, y). Thus, they must be combined into the same labels. It is also necessary to process the connected components of the 4th pixel at (x+1, y−1) and the 1st pixel at (x−1, y) or the 2nd pixel at (x−1, y−1) like that. The functions NewSet(L), CombineSet(c4, c2), and ConsecutiveLabel(L−1) are used to implement these steps [13]. Their simpler pseudocodes were then proposed in the equivalent-label-set strategy of Section 3.3.2 in Ref. [11], with their source code and the code for breakpoint detection in C++ stored at https://data.mendeley.com/datasets/hnkn8kgvsh/2. The abovementioned three functions are also used in the next breakpoint detection. Finally, when there is no pixel in the mask that is an object pixel, a new pixel is set as bxy. The function NewSet(L) is used to construct some arrays that store the connection relationship. The function ConsecutiveLabel(L−1) is used to make the labels of connected components consecutive. As can be seen from the binary image in Fig. 6b, the white discrete points are much smaller in area than the network formed by the tiles’ edges. Hence, the next task is to filter out the white discrete points through connected-component labelling. This is done by keeping the pixels with the label of the maximum quantity and removing the rest in the image.
In the above, the variable L represents the set of short lines detected by Hough transformation. Each line is represented as [x1, y1, x2, y2], corresponding to the coordinates of the end points of the line. The variable N is the size of L. In step 3, all short lines are labeled in set B, with the maximal label value m−1 obtained according to the following function.
In the above, Ai represents the angle between the short line Li and the transverse axis. The distance between the short lines Li and Lj can be computed according to Eq. (1).
d i, j =
(x i − x j )2 + (yi − yj )2 sin ⎜⎛ ⎝
Ai + Aj 2
− arctan
yi − yj ⎞ ⎟, xi − xj ⎠
(1)
where (xi, yi) and (xj, yj) are the positions of the center points of Li and Lj. di, j is the projection length of the line of centers in the vertical direction of the short lines Li and Lj, as shown in Fig. 8. Once the distance d between L1 and L2 is short enough, the two lines are considered to be connected and marked with the same label. In general, a new iteration short line Lj is marked as the label of its connected short line Li (Bj = Bi), or as a new label when its connected short lines have no labels (Bj = m). However, once Lj already had a label Bj before this iteration and it’s detected to be connected with Li in this iteration, labels Bi and Bj should be combined(CombineSet(Bi, Bj)). i.e., any short line with labels as the same as Bi or Bj should be marked
3.1.3. Hough transformation Hough transformation [8] transforms edges into short lines for subsequent steps. It projects the position of pixels into parameter space. Note that this is a standard method in the literature; hence, we do not
Fig. 8. Distance between two short lines. Note that o1 and o2 are the center points. o1p is a line with an average slope of L1 and L2. 4
Fusion Engineering and Design 152 (2020) 111422
J. Liu, et al.
as the smaller of Bi and Bj.The equivalent-label-set strategy was proposed to solve this problem [13], where three sets that store the relationship between labels are constructed using the function NewSet as follows.
In the above, R corresponds to the representative label of a label. e.g., Ri = j means that all short lines with label i are connected with lines with label j, and j is always less than i. Using the function CombineSet defined below, all connected short lines will have the same representative label.
Fig. 10. Detected breakpoints.
In step 4 of the combination algorithm, the function ConsecutiveLabel is used to generate consecutive final labels, which is defined as follows.
3.3. Detection of fallen tiles 3.3.1. Foundation Once the breakpoints are detected, the depth camera starts and the captured depth image is processed using the watershed segmentation algorithm [14] to get the contour of the fallen tile. This algorithm can separate an image into different regions according to the initial points in these regions and if there are obvious boundaries between these regions. Two groups of points, i.e., one inside the contour of the fallen tile and the other outside, can be used to separate the depth image into the fallen tile part and the rest part because there is an obvious boundary between the fallen tile and the tiles around it in the depth image. The next step is to get these two groups of points. In many cases, the breakpoints are on the contour of the fallen tile. Thus, a point on a line segment that links a pair of breakpoints must be within the contour, while the other point on the extension line of the line segment must be outside the contour. In general, to avoid the two points to be at singular positions, e.g., a location where the depth camera cannot measure the depth successfully, a points within a circle around each of the two points are used as the initial points in the watershed segmentation algorithm. An example of a depth image for executing the watershed segmentation algorithm is shown in Fig. 11a. For display purposes, the depth image was normalized from 0 to 255. Note, however, that most of the depth values were greater than 255 mm; thus, we used the depth values directly instead of the depth image. As shown in Fig. 11b, the fallen tile is extracted by watershed segmentation and is framed by its minimum circumscribed rectangle. The two white circles show the initial two points that are within and outside the contour of the fallen tile. During grasping, two long edges of the circumscribed rectangle are exactly the contact sides of the grasp. In a depth image, the position of the circumscribed rectangle can be computed using Eq. (2).
In step 5 of the combination algorithm, only connected short lines can be assigned to the same subset Cm of C. The judgment criterion to combine short lines into Cm is shown in Fig. 9. Firstly, short lines have similar slopes. Then, there is at least one straight line with a similar slope that could be constructed to pass through the short lines within a certain range of errors. Therefore, Cm could represent a practical line. The breakpoint detection algorithm is defined as follows. Am, 1 represents the angle between the short line Cm, 1 and the transverse axis. xm, 1, 1 and xm, 1, 2 represent the x-coordinates of two end points of the short line Cm, 1. max(xm, 1, 1, xm, 1, 2) returns the larger one between xm, 1, 1 and xm, 1, 2. Firstly, the algorithm sorts short lines from one end of a practical line Cm to the other. Then, the continuity of Cm is detected. As shown in Fig. 9, since p2 is on the right of p3, Cm is judged to be continuous between p1 and p4. However, p4 is on the left of p5; hence, Cm is not continuous between p4 and p5. Therefore, the practical line represented by Cm has no continuity, while the disconnected positions p4 and p5 are the pair of breakpoints. As shown in Fig. 10, four breakpoints are circled out, and the red lines are the short lines detected by Hough transformation.
Fig. 9. Three short lines in Cm. Note that L1, L2, and L3 are three adjacent short lines, and the dotted line represents the constructed line. p1 and p2 are the end points of L1. p3 and p4 are the end points of L2. p5 and p6 are the end points of L3. 5
Fusion Engineering and Design 152 (2020) 111422
J. Liu, et al.
Fig. 11. a. Normalized depth image. b. Fallen tile separated by watershed segmentation.
u ⎡X⎤ ⎡ ⎤ −1 ⎢Y ⎥ = Z × K × ⎢v⎥ ⎣1⎦ ⎣Z⎦ , 0 323.774472 ⎤ ⎡576.706989 K=⎢ 0 576.786077 251.742973⎥ 0 0 1 ⎣ ⎦
4. Experiment 4.1. Experimental set up To test the feasibility and robustness of the breakpoint detection algorithm, a platform was built to simulate the environment inside a vacuum vessel, as shown in Fig. 13. Some tiles collected from the EAST vacuum vessel were closely placed on the platform to simulate the divertor’s first wall. A universal robot (UR) with six degrees of freedom and a three-finger manipulator was mounted on the left side of the platform. A calibrated RGB+depth(RGB-D) camera was mounted at the end of the robot to form an eye-in-hand system.
(2)
where u and v are the pixel positions of a point, such as the center point of the minimum circumscribed rectangle in the depth image, and X, Y, and Z are the point’s position in the depth camera such that Z is the point’s depth value. In addition, K is the camera matrix of the depth camera. 3.3.2. Improvement In many cases, the breakpoints are on the contour of the fallen tile. However, in some other cases, the breakpoints may be on the edges of other tiles such as the unexpected pair of breakpoints shown in Fig. 12a. The unexpected pair of breakpoints appeared because one of the sides of the fallen tile was in line but not connected with that of another tile. Once this unexpected pair of breakpoints are used in watershed segmentation, two wrong groups of points will be chosen as the initial points, as we can see from the two white circles in Fig. 12b. Then, there will be an unexpected segmentation in the depth image. Thus, the unexpected pair of breakpoints need to be excluded. Since the unexpected segmentation makes the minimum circumscribed rectangle bigger than the rectangle by the expected segmentation, the area size of the minimum circumscribed rectangle can be used to exclude the unexpected pair of breakpoints. We use each pair of breakpoints as the initial points in watershed segmentation, and then record the area size of the minimum circumscribed rectangle. Finally, the watershed segmentation corresponding to the least area size is chosen as the valid watershed segmentation to detect the fallen tile, as shown in Fig. 12c. Therefore, once any valid pair of breakpoints are recognized in breakpoint detection, any other unexpected breakpoints cannot disturb the detection of the fallen tile.
4.2. Feasibility test In the experiment, the UR robot tracked a designated path, as shown in Fig. 14a. During the path, real-time RGB images were obtained similarly to Fig. 6a and then processed and extracted into lines similarly to Fig. 6d, and finally processed using the breakpoint detection algorithm described in Section 3.2. If there was no fallen tile in the visual field of the camera, the UR robot would continuously do the tracking and image preprocessing and breakpoint detection would keep working. However, once a fallen tile exists, the breakpoints would be obtained similarly to Fig. 10, which triggers the UR robot, the RGB camera, and image processing to stop, as shown in Fig. 14b. This triggers the start of the depth camera and watershed segmentation of the depth image to obtain the position of the fallen tile similarly to Fig. 11b. Finally, the UR robot grasps and takes away the fallen tile according to the obtained position, as shown in Fig. 14c and d. 4.3. Robustness test 4.3.1. Robustness to fallen location The location of the fallen tile on the divertor of the EAST can be
Fig. 12. a. Unexpected breakpoints. b. Unexpected watershed segmentation. c Valid watershed segmentation. 6
Fusion Engineering and Design 152 (2020) 111422
J. Liu, et al.
Fig. 13. Experimental platform.
Fig. 15. Construction of the coordinate system in the depth image.
random because the fallen tile can come from anywhere from the first wall of the EAST. In Section 3, only one location was investigated, which is not sufficient to prove the robustness of the proposed algorithm as applied in the EAST. Therefore, we carried out a series of experiments where the fallen tile is thrown down from 50 cm above the platform to produce random fallen locations. In the experiments, the robot remained still, while RGB and depth images were captured after each throw of the fallen tile. After 100 throws, all the RGB and depth images were processed in bulk. The processed images, including binarized images, images after connected-component labeling filtering, images of breakpoint detection, images of watershed segmentation, and poses, can be obtained at https://doi.org/10.17632/jrpn2w2dwj.3. The poses (positions+orientations) are presented in the format “i, x, y, z, ε1, ε2, ε3, ε4,” where i is the experiment’s throw number, x, y, z (in mm) are the coordinates of the position of the fallen tile, and ε1, ε2, ε3, ε4 (Euler parameters) are the orientations relative to the depth camera’s coordinate system. An example of constructing the coordinate system of the depth camera and that of the fallen tile for the first experiment is shown in Fig. 15; the pose is "1, 106, −28, 350, 0, 0, −0.38, 0.92.″
Because the camera is parallel to the platform, ε1 and ε2 are 0. The correct poses of the fallen tiles were successfully achieved in 98 out of 100 experiments; there were no breakpoints detected for the two failures. The images after connected-component labeling filtering, as shown in Fig. 16a and b, demonstrate that the two failures occurred because the upper-left part of the network was filtered out due to three disconnections, as shown in the binarized images in Fig. 16c and d. The first two disconnections were due to the fallen tile, while the last one was due to the narrow black gap because of the narrow spacing between the two tiles, as shown by the red circle in Fig. 16e or f. In conclusion, we found that the proposed algorithm is robust to the fallen location, but failures may occur when the spacing between tiles is too narrow or when the fallen tile is in the corner of the visual field of the camera. In addition, a special fallen location where the fallen tile overlaps with another tile, as shown in Fig. 16g, can also invalidate the proposed algorithm, but the probability of it occurring is small; in fact, we did not observe this special location in the 100 experiments performed.
Fig. 14. a. Tracking, b. stopping, c. grasping, and d. taking away processes of detecting and grasping a fallen tile. 7
Fusion Engineering and Design 152 (2020) 111422
J. Liu, et al.
Fig. 16. a.–b. Connected-component labeling filtering images. c.–d. Binarized images. e.–f. Original RGB images of the two failures. g. Special fallen location.
a light source was firstly shone on the center of the platform and then removed gradually. During the movement, RGB and depth images were captured and the illuminance was measured six times. Fig. 17a–f show the original RGB images. With the movement of the
4.3.2. Robustness to illumination 4.3.2.1. Illuminance. An experiment with varying illumination conditions but with the same fallen location was carried out to test the proposed algorithm’s robustness to illumination. In the experiment, 8
Fusion Engineering and Design 152 (2020) 111422
J. Liu, et al.
Fig. 17. a.–f. Original RGB images. g.–l. Binarized images. m.–r. Detected breakpoints. s.–w. Detected fallen tiles in the depth images.
former five images can almost be detected, while those in the last one cannot. According to the images of breakpoint detection in Fig. 17 m–r, a similar conclusion can be drawn where the breakpoints can be
light source, the illuminance on the surface of the fallen tile becomes darker, i.e., 132, 67, 52, 24, 20, and 15 lx in the images. According to the binarized images in Fig. 17g–l, the edge features of the tiles in the 9
Fusion Engineering and Design 152 (2020) 111422
J. Liu, et al.
Fig. 18. a. Original RGB image. b. Binarized image. c. Detected breakpoints. d. Detected fallen tile.
detected in the former five images, while those in the last one cannot. Fig. 17s–w show the fallen tiles detected by watershed segmentation in the depth images.
was carried out in the depth image, which is not influenced by illumination.
5. Conclusion
4.3.2.2. Shadow. To test if shadow affects the performance of the proposed algorithm, we carried out experiments where the fallen tile is covered with a shadow, as shown in Fig. 18a. The illuminance on the surface of the fallen tile in Fig. 18a is 17 lx. As shown in Fig. 18b, the contour of the shadow was detected by the algorithm, resulting in inaccuracies in the detection of the edges of the tiles. Moreover, the effect of the shadow resulted in many extra useless breakpoints as shown in Fig. 18c. However, despite these extra useless breakpoints, we still detected a pair of valid breakpoints; hence, the fallen tile can be detected, as shown in Fig. 18d.
This paper proposed a method to support the removal of fallen tiles in the EAST vacuum vessel. The method included a novel breakpoint detection algorithm to detect the existence of fallen tiles and a watershed segmentation algorithm to recognize the contour of fallen tiles. Experiments using a UR robot were performed to demonstrate the recognition and grabbing processes of fallen tiles and to test the robustness of the algorithm by considering the fallen locations of fallen tiles and illumination.
4.3.2.3. Results. To see if the illumination condition can influence the accuracy of the detection, the poses of the fallen tile in the experiments are listed in Table 1. Numbers 1–6 correspond to the experiments of Section 4.3.2.1, while number 7 corresponds to the experiment of Section 4.3.2.2. As shown in Table 1, the displacement standard deviation is less than 0.4 mm and the orientation variation is very small. Therefore, illumination does not affect the accuracy, only the rate of success, of finding the fallen tiles; it only influences the RGB image processed by the breakpoint detection algorithm. Moreover, breakpoint detection was used to find an approximate position of the fallen tile and provide a marker within the fallen tile’s contour for watershed segmentation. On the other hand, watershed segmentation
Funding This work was supported by the National Key R&D Program of China (CFETR Engineering Design Project, No. 2017YFE0300503) and the China Postdoctoral Science Foundation [Grant No. 2017M620260].
Declaration of Competing Interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Table 1 Poses of the fallen tile in the depth camera in different experiments. Number Illuminance/lx Pose
x/mm y/mm z/mm ε1 ε2 ε3 ε4
1
2
3
4
5
6
7
Standard Deviation
132 35.53 −31.30 349 0 0 −0.235 0.971
67 35.53 −30.40 349 0 0 −0.226 0.973
52 35.53 −31.30 349 0 0 −0.228 0.973
24 35.53 −31.00 349 0 0 −0.229 0.973
20 35.23 −30.70 349 0 0 −0.229 0.973
15 – – – – – – –
17 35.84 −30.40 349 0 0 −0.224 0.975
0.176 0.377 0 0 0 0.0034 0.0012
10
Fusion Engineering and Design 152 (2020) 111422
J. Liu, et al.
Acknowledgment
ijcseit.2012.2213. [6] Z. Kato, T.C. Pong, A Markov random field image segmentation model for color textured images, Image Vis. Comput. 24 (2006) 1103–1114, https://doi.org/10. 1016/j.imavis.2006.03.005. [7] S. Arivazhagan, L. Ganesan, S.P. Priyal, Texture classification using Gabor wavelets based rotation invariant features, Pattern Recognit. Lett. 27 (2006) 1976–1982, https://doi.org/10.1016/j.patrec.2006.05.008. [8] P. Hart, R.O. Duda, Use of the Hough transformation to detect lines and curves in pictures, Commun. ACM 15 (1972) 11–15 http://trac.astrometry.net/export/ 19652/trunk/documents/papers/dstn-review/papers/duda1972.pdf. [9] S. Li, M.C. Lee, C.M. Pun, Complex Zernike moments features for shape-based image retrieval, IEEE Trans. Syst. Man, Cybern. Part A Syst. Hum. 39 (2009) 227–237, https://doi.org/10.1109/TSMCA.2008.2007988. [10] H. Wang, L. Xu, W. Chen, Design and implementation of visual inspection system handed in tokamak flexible in-vessel robot, Fusion Eng. Des. 106 (2016) 21–28, https://doi.org/10.1016/j.fusengdes.2016.03.005. [11] L. He, X. Ren, Q. Gao, X. Zhao, B. Yao, Y. Chao, The connected-component labeling problem: a review of state-of-the-art algorithms, Pattern Recognit. (2017), https:// doi.org/10.1016/j.patcog.2017.04.018. [12] L. He, Y. Chao, K. Suzuki, K. Wu, Fast connected-component labeling, Pattern Recognit. 42 (2009) 1977–1987, https://doi.org/10.1016/j.patcog.2008.10.013. [13] L. He, Y. Chao, K. Suzuki, A linear-time two-scan labeling algorithm, Proc. - Int. Conf. Image Process. ICIP (2007), https://doi.org/10.1109/ICIP.2007.4379810. [14] L. Vincent, L. Vincent, P. Soille, Watersheds in digital spaces: an efficient algorithm based on immersion simulations, IEEE Trans. Pattern Anal. Mach. Intell. (1991), https://doi.org/10.1109/34.87344.
The authors would also like to express their thanks toeditor Akio Sagara of the journal and the anonymous reviewers for their valuable comments about the content of the paper and editor Jorge of Enago for providing language help. References [1] S. Shi, Y. Song, Y. Cheng, E. Villedieu, V. Bruno, H. Feng, H. Wu, P. Wang, Z. Hao, Y. Li, K. Wang, H. Pan, Conceptual design main progress of EAST Articulated Maintenance Arm (EAMA) system, Fusion Eng. Des. 104 (2016) 40–45, https://doi. org/10.1016/j.fusengdes.2016.02.005. [2] D. ping, T. ping Tian, A review on image feature extraction and representation techniques, Int. J. Multimed. Ubiquitous Eng. 8 (2013) 385–396, https://doi.org/ 10.1093/ecco-jcc/jjw221. [3] W.T. Chen, W.C. Liu, M.S. Chen, Adaptive color feature extraction based on image color distributions, IEEE Trans. Image Process. 19 (2010) 2005–2016, https://doi. org/10.1109/TIP.2010.2051753. [4] P. Cheng, J. Wang, C. Miao, C. Tu, Constrained Texture Mapping Via Approximate Voronoi Base Domain, (2018), pp. 19–22, https://doi.org/10.1145/3208159. 3208188. [5] B.S. V, Grey level co-occurrence matrices: Generalisation and some new features, Int. J. Comput. Sci. Eng. Inf. Technol. 2 (2012) 151–157, https://doi.org/10.5121/
11