- Email: [email protected]
A new pixel matching method using the modulation of shadow areas in online 3D measurement
Optics and Lasers in Engineering 51 (2013) 1078–1084
Contents lists available at SciVerse ScienceDirect
Optics and Lasers in Engineering journal homepage: www.elsevier.com/locate/optlaseng
A new pixel matching method using the modulation of shadow areas in online 3D measurement Kuang Peng, Yiping Cao n, Yingchun Wu, Yanshan Xiao Opto-Electronics Department, Sichuan University, Chengdu 610064, China
art ic l e i nf o
a b s t r a c t
Article history: Received 6 December 2012 Received in revised form 17 February 2013 Accepted 15 March 2013 Available online 23 April 2013
A new pixel matching method based on phase measuring profilometry (PMP) using the modulation of shadow areas in online 3D measurement is proposed and verified by experiments. The modulation of the shadow areas representing low reliability of the phase of the object which is usually avoided as much as possible is now used to match pixels of the measured object on pipe-line as the reason for its distinguished feature in this paper. In this method based on the modulation of shadow areas, we use the Otsu method to find out a threshold value and then use it to realize the binarization for the modulation image of the object extracted from the deformed fringes. As there are only 0 and 1 in the binarization image so that the calculation is fast as the product of 0 and anther nonzero number is 0, the speed of pixel matching can be promoted with the precision guaranteed. With more complex objects, the saved time will be more. The experiments prove it. & 2013 Elsevier Ltd. All rights reserved.
Keywords: Modulation Otsu method Binarization Shadow areas Online 3D measurement
1. Introduction With the fast development of the modern industry and a large number of applications of the pipe-lines, online 3D measurement technology [1,2] is used in many kind of fields such as robotics, component quality control, medicine and solid modeling applications [3], and in the meanwhile the demands for online 3D measurement become higher and higher. Traditional methods of online 3D measurement are mainly based on machine vision [4,5], and the precision is affected by the image processing algorithm. The precision is limited by different kinds of algorithms applied for the objects with the condition of different complexities and the algorithms usually need a large amount of calculations. Other three dimensional measurement methods, such as Fourier transform profilometry (FTP) [6–10], phase measurement profilometry (PMP) [11–16] and modulation measurement profilometry (MMP) [17–19] etc., obtain the three dimensional contour information of the object by analyzing the spatial structure light field modulated by the object's surface. FTP, proposed by Takeda and Mutoh [6], is suited for online 3D measurement as it requires only one deformed fringe recorded. Lurong Guo and Su [7] proposed an improved FTP method using a defocused optical field with a Ronchi grating and a grating π phase shifting technique, which can extend the measurable slope of height variation by nearly three times. Although FTP has several
n
Corresponding author. E-mail address: [email protected] (Y. Cao).
0143-8166/$ - see front matter & 2013 Elsevier Ltd. All rights reserved. http://dx.doi.org/10.1016/j.optlaseng.2013.03.008
advantages of easy collection, simple data processing and fast speed, high frequency information of the object is missing during the spatial filtering process and so the information of the object is not complete and the precision cannot be ensured. PMP proposed by V. Srinivasan can satisfy the demands of online measurement due to its characteristic high precision. In PMP, more than three deformed fringe patterns with phase shifting will be recorded so that the coordinates in each pattern do not correspond as the object is moving. So pixel matching is needed. In the conventional method in PMP, pixel points are matched with certain characteristics marked on the measured object. Modulation is important information of the deformed patterns, and it is the basis of the reliability of phase unwrapping in PMP. On the basis of it, Yingchun Wu et al. proposed a pixel matching method based on modulation [20] and a processed modulation using delamination for pixel matching [21]. The measurement profilometry based on modulation (MMP) proposed by Xianyu Su can realize the three dimensional surface shape measurement of the object. Modulation, the basis of weighted approach in the phase unwrapping, is usually applied in phase unwrapping of the equal step phase shifting algorithm. In this paper, a new pixel matching method based on the modulation of shadow areas in online 3D measurement is proposed. The modulation information of the object can be obtained by using the method of Fourier transform analysis easily, and the modulation of the shadow areas representing low reliability which should be avoided generally is used to do pixel matching as its distinguished feature. In the processing, for further improvement in the accuracy and the speed of pixel matching in 3D
K. Peng et al. / Optics and Lasers in Engineering 51 (2013) 1078–1084
DLP
Then through pixel matching and following the Stoilov algorithm, phase function can be calculated using Eq. (1): 2ðI′2 ðx,yÞ−I′4 ðx,yÞÞ ϕ ¼ arctan sinϕ0 , ð2Þ 2I′3 ðx,yÞ−I′1 ðx,yÞ−I′5 ðx,yÞ
D
CCD d
Q
P
1079
sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi I′1 ðx,yÞ−I′5 ðx,yÞ 2 sinϕ0 ¼ 1− , 2½I′2 ðx,yÞ−I′4 ðx,yÞ
z
L
D Object h C
B
A
O
Reference Plane
ð3Þ
In Eqs. (2) and (3), I′n ðx,yÞ, n ¼ 1,2,3,4,5, are the deformed patterns captured from I n ðx,yÞ, n ¼ 1,2,3,4,5, which will be described in Section 3. As Eq. (2) shows arctan calculation which makes the phase truncated in the scope from −π to π, the phase is discrete. But in fact the surface phase distribution is continuous, so the truncated phase should be unwrapped. In the paper we use the diamond phase unwrapping algorithm. And then by the mapping algorithm between phase and height, the height distribution information of the object hðx,yÞ can be reconstructed: 1 1 1 ¼ aðx,yÞ þ bðx,yÞ þ cðx,yÞ 2 , hðx,yÞ ϕðx,yÞ ϕ ðx,yÞ
Fig. 1. On-line PMP setup.
measurement, we use the Otsu method for finding out a threshold value and binarization processing for shorting the spending time in correlation operation. Then the modulation of the shadow areas can be extracted from deformed fringe patterns. By setting one pattern as a template, pixel matching which sets the pixel coordinates of each deformed fringe image with one-to-one correspondence can be done. Then we can obtain the phase information of the object by using the Stoilov algorithm [22,23] in PMP. And through the phase unwrapping and calibration processing [24–30], the object surface shape can be reconstructed.
2. Principle of PMP Phase measurement profilometry (PMP) is a 3D measurement method using the phase information to reconstruct the object surface shape and the Stoilov algorithm is a new phase shift algorithm with equal steps. A schematic of the basic measuring system in PMP based on the Stoilov algorithm is illustrated in Fig. 1. The point O is the intersection between the optic axis of digital light processing (DLP) and that of the imaging system (CCD). The constant and the variable representing the distance are signed in Fig. 1. A computer can compile a sinusoidal grating fringe suitably and then it is projected onto the tested object mounted on the pipe-line by DLP. The length of the sinusoidal grating fringe spreading from DLP to the surface of the object is changing caused by the movement of the measured object so that the original phase pattern without the object on the reference plane is modulated by the height of the measured object. Then the deformed patterns are recorded by CCD. By the Stoilov algorithm, on the condition of satisfying equal steps even without knowing its magnitude, it is possible to calculate the truncated phase of the object surface accurately. When the amount of phase shift ϕ0 is equal, five frame deformed patterns I n ðx,yÞ, n ¼ 1,2,3,4,5 acquired in the five steps with equal distance by CCD are shown below: I n ðx,yÞ ¼ Rðx,yÞ Aðx,yÞ þ Bðx,yÞcos½ϕðx,yÞ þ ðn−1Þϕ0 , n ¼ 1,2,3,4,5 ð1Þ where Rðx,yÞ is the surface reflectance of the object, Aðx,yÞ is background light intensity, Bðx,yÞ represents the fringe contrast, and ϕðx,yÞ indicates phase information modified by the height of the object.
ð4Þ
where aðx,yÞ, bðx,yÞ and cðx,yÞ in the Eq. (4) are acquired by plane calibration.
3. A pixel matching method using the modulation of shadow areas Pixel matching is a process which can make the points of each frame correspond one-to-one so that the phase shifting can be calculated. Without this processing, using the Stoilov algorithm would lead the error as the object is not static. Modulation information of the object can be used to realize the detection of the object displacement, and so the corresponding cut patterns from five deformed patterns can be obtained. After the Fourier transformation of both sides of Eq. (1) Gn ðf x ,f y Þ ¼ Gn0 ðf x ,f y Þ þ Gn1 ðf x ,f y Þ þG−n1 ðf x ,f y Þ,
n ¼ 1,2,3,4,5
ð5Þ
where Gn0 ðf x ,f y Þ, Gn1 ðf x ,f y Þ, and G−n1 ðf x ,f y Þ represent the deformed patterns of zero level and positive and negative first level spectra respectively. Using the spatial filtering method to filter out the first level spectrum Gn1 ðf x ,f y Þ and carrying out inverse Fourier transform: h i þ∞ P n ðx,yÞ ¼ ∬ Gn1 ðf x ,f y Þexp i2πðf x x þf y yÞ df x df y −∞
1 ¼ Rðx,yÞBðx,yÞexp i2πf x x þ ϕ þ ϕ0 2
ð6Þ
Modulation distributions are defined as the modulus of P n ðxn ,yn Þ, n ¼ 1,2,3,4,5: 1 ð7Þ M n ðx,yÞ ¼ ½P n ðx,yÞ ¼ Rðx,yÞBðx,yÞ, n ¼ 1,2,3,4,5 2 In the equation, the modulation information of the deformed patterns contains the surface reflectance of the object Rðx,yÞ and the fringe contrast of projected fringe Bðx,yÞ. If the uniformity of the projected fringe patterns can be guaranteed, Bðx,yÞ could be regarded as a constant, so the modulation can reflect the gray information of the measured object well. When the object is moving online on the pipe-line, the modulation of the object corresponding to each point would be moving accordingly, so the modulation can be regarded as the marks of the object movement. In the modulation patterns, the most distinct part is the shadow areas where the value of the modulation is low. As usual, the low modulation represents low reliability which will make phase unwrapping inaccurate and
1080
K. Peng et al. / Optics and Lasers in Engineering 51 (2013) 1078–1084
time-consuming and so we should avoid it as much as possible. Meanwhile, the most distinct characteristic parts which are moving with the object in five patterns are the shadow areas, and they can be used in pixel matching. When the object is moving, the angle from CCD to the measured object varies and the shadow areas will be changing. But at the same time, the speed for recording five deformed fringe patterns is so large that the angle will change slightly. These changes have few effects on the precision and efficiency of the method because we can get accurate one-to-one corresponding deformed patterns as shown later. To extract the shadow areas from five deformed fringe patterns, the Otsu method is used to find out a threshold value K
and then we use it to realize the binarization for the modulation pattern as shown follows. ( M n ðx,yÞ ¼
1,M n ðx,yÞ o K 0,others
,
n ¼ 1,2,3,4,5
ð8Þ
In the binarization, only values M n ðx,yÞ, n ¼ 1,2,3,4,5, smaller than K are set to be 1 and other values are set to be 0. As there are only 0 and 1 in the binarization patterns and the product of 0 and any other nonzero number is 0, the speed of correlation operation and pixel matching will be promoted with the precision guaranteed.
Fig. 2. Simulated construction: (a) simulation object, (b)the first frame deformed fringe pattern, and (c) the second frame deformed fringe pattern.
Fig. 3. Binarization processing: (a) the modulation of the first frame, (b) the modulation of the second frame, (c) the binarization pattern of the first frame, and (d) the binarization pattern of the second frame.
K. Peng et al. / Optics and Lasers in Engineering 51 (2013) 1078–1084
1081
Fig. 4. Experimental results: (a) template, (b) cut pattern in the first frame pattern after pixel matching, (c) cut pattern in the second frame pattern after pixel matching, (d) reconstructed object, and (e) error distribution.
4 frame deformed patterns by moving S pixels along the direction of the object movement on the pipe-line:
Table 1 The time spent with and without the binarization. Measuring sequence
1
2
3
Average
Time spent with the binarization (s) Time spent without the binarization (s)
1.2415 1.5994
1.2666 1.6073
1.2638 1.6082
1.2573 1.6050
Work plane
X
3D measuring device
I′i ðx,yÞ ¼ ∏I i ðx,yÞI′1 ðx,yÞ,
i ¼ 2,3,4,5
ð9Þ
In Eq. (9), ∏ is the pixel matching operation cutting the corresponding deformed patterns at the position shifting nS (n ¼1,2,3,4) in the moving direction by the template's size from I n ðx,yÞ, n ¼ 2,3,4,5, respectively. Pixel matching, by obtaining the object equal step distance by the modulation of the shadow areas and using the image clipping method, can make the pixel coordinates in five deformed patterns cut down I′n ðx,yÞ, n ¼ 1,2,3,4,5 one-to-one correspondence.
4. Numerical simulations and experiments
Z Measured Object Step-motor
Fig. 5. The setup of the experiment.
Then the entire modulation distribution of the shadow areas M 1 ðx,yÞ is cut from the first frame deformed fringe and taken to be a template. The modulations of the Nth (N ¼2,3,4,5) frame deformed fringe pattern can be matched by carrying out correlation operation so that the shift of the object S can be measured. The deformed pattern I′1 ðx,yÞ, including the areas where the object is located, is cut from the first frame deformed pattern I 1 ðx,yÞ, and then the shift S can be used to obtain the phase shift fringe patterns at the corresponding coordinates of the other
To verify the feasibility of proposed online 3D measurement and check its speed, a series of experiments have been done. Firstly an experiment is simulated where the sample of the experiment is half of an ellipsoid, the long and short axis of which are 230 mm and 130 mm respectively and the height of which is 30 mm as shown in Fig. 2(a). When the ellipsoid on the pipe-line is moving, the sinusoidal grating fringe is modified by the height of the object, and five frame deformed patterns are captured at five positions with equal distance and two of them are shown in Fig. 2(b) and (c). Fig. 3 (a) and (b) show the modulation of two frame deformed patterns, and the corresponding binarization patterns are shown in (c) and (d). The shadow areas in the binarization pattern of the first frame are cut down to be regarded as a template as shown in Fig. 4(a). Then on the basis of the template, pixel matching can be done after carrying out the correlation operation and the distance between two adjacent patterns S can be calculated. Then according
1082
K. Peng et al. / Optics and Lasers in Engineering 51 (2013) 1078–1084
to the distance S, the cut parts in two patterns after pixel matching are shown in Fig. 4(b) and (c). The simulated reconstruction results are shown in Fig. 4(d). Fig. 4(e) shows the absolute error distribution of 3D reconstruction, the biggest error of which is less than 0.07 mm and the RMS error of which is 0.0221 mm. The precision of this method is guaranteed. We have done large numbers of simulations both with binarization processing and without it as shown in Table 1; the average time spent in the pixel matching with the binarization processing
is about 28% faster than that without this processing. From the simulation results and the analysis, the high accuracy and efficiency of the proposed method are confirmed. Further verification of the precision and efficiency of the method proposed in this paper, a rabbit-like object which is 50.82 mm 60.01 mm 10.20 mm in volume had been measured. Fig. 5 shows the setup of the experiment. In Fig. 5, the white box is a 3D measuring device integrating the functions of both DLP and CCD for 3D online measurement. The
Fig. 6. Initial data: (a) rabbit-like object, (b) added noise, (c)the first frame deformed fringe pattern, and (d) the second frame deformed fringe pattern.
Fig. 7. Binarization processing: (a) the modulation of the first frame, (b) the modulation of the second frame, (c) the binarization pattern of the first frame, and (d) the binarization pattern of the second frame.
K. Peng et al. / Optics and Lasers in Engineering 51 (2013) 1078–1084
projection and the camera on it are on a straight line along the movement's direction X of the measured object. The Z axis is used to change the distance between the 3D measuring device and the work plane to control the stripe projected onto the work place. One sinusoidal grating image is vertically projected onto the measured object and then five deformed patterns modulated by the height of the rabbit-like object are captured by the 3D measuring device while the measured object moves along the X axis at equivalent steps controlled by a set of step-motors. The projected grating pitch is about 3.34 mm and the moving step is about 2.33 mm. And then the five deformed patterns are transmitted to a computer to be processed with the above proposed method.
1083
Fig. 6 shows the initial data in the experiment. In Fig. 6, (a) is the rabbit-like object, and (c) and (d) are the first and second frame deformed fringe patterns respectively after modification by the object. And as the captured deformed patterns are in good quality, noise is added into the deformed patterns and its RMS is about 0.5 mm as shown in Fig. 6(b). Following the processing introduced above, Fig. 7(c) and (d) show the binarization results of two frame deformed patterns corresponding to Fig. 7(a) and (b). We can see the shadow areas after the binarization processing. The template is shown in Fig. 8(a). Then through correlation operation and pixel matching, the one-to-one coordinate
Fig. 8. Experimental results: (a) template, (b) cut pattern in the first frame pattern after pixel matching (c) cut pattern in the second frame pattern after pixel matching, (d) reconstructed object, and (e) a cross section of the reconstructed object.
1084
K. Peng et al. / Optics and Lasers in Engineering 51 (2013) 1078–1084
Table 2 The time spent with and without the binarization. Measuring sequence
1
2
3
Average
Time spent with the binarization (s) Time spent without the binarization (s)
1.3281 1.8473
1.3278 1.7926
1.3637 1.8182
1.3399 1.8194
corresponding parts are cut from the two patterns are shown in Fig. 8(b) and (c). Fig. 8(d) shows the reconstructed rabbit-like object which is about 10.11 mm in height and the object's real height is 10.20 mm. Fig. 8(e) is a cross section of the constructed object and it is smooth. We have done lots of experiments without the binarization processing and there is analysis on the time spent in the pixel matching with and without the binarization processing. Rather than the average time without the processing as shown in Table 2, the average time spent in the pixel matching with the binarization processing is about 35% faster. From the results of the simulation and the experiment, we can confirm that the proposed method is accurate and efficient. And with a more complex measured object, the saved time can be more. 5. Conclusions In this paper, a new pixel matching method using the modulation of shadow areas is proposed. The entire modulation distribution of the object can be used in pixel matching instead of the traditional marking method and we finish the binarization for the modulation patterns with the Otsu method finding out the threshold value. By the correlation operation between the template cut from the first binarization pattern and the modulation of the other frame binarization patterns, the amount of the object shifting can be calculated. On this basis, pixel matching has been accomplished and the measured object can be reconstructed finally. As experimental results show, this way can satisfy the demand of high accuracy and fast measurement in online 3D measurement on the pipe-line in the industry. At the same time, it presents a new idea using the entire modulation distribution of the shadow areas to match pixels in online 3D measurement. Acknowledgment This work is supported by 863 National Plan Foundation of China under Grant no. 2007AA01Z333. References [1] Song Zhang. Recent progresses on real-time 3D shape measurement using digital fringe projection techniques. Opt Laser Eng 2010;48(2):58–149.
[2] Xianyu Su, Qican Zhang. Dynamic 3-D shape measurement method: a review. Opt Laser Eng 2010;48(2):191–204. [3] Gorthi SS, Rastogi P. Fringe projection technique: whither we are? Opt Laser Eng 2010;48(2):133–40. [4] Cubero Sergio, Aleixos Nuria. Advances in machine vision applications for automatic inspection and quality evaluation of fruits and vegetables. Food Bioprocess Technol 2011;4(4):487–504. [5] Ho Chao-Ching, He Jun-Jia, Liao Te-Ying. On-line estimation of laser-drilled hole depth using a machine vision method. Sensors 2012;12(8):10148–62. [6] Takeda M, Mutoh K. Fourier transform profilometry for the automatic measurement of 3-D object shape. Appl Opt 1983;22(24):3977–82. [7] Guo Lurong, Su Xianyu. Improved Fourier transform profilometry for the automatic measurement of 3D object shapes. Opt Eng 1990;29(12):1439–44. [8] Zappa Emanuele, Busca Giorgio. Static and dynamic features of Fourier transform profilometry: a review. Opt Laser Eng 2012;50(8):1140–51. [9] Cobelli Pablo Javior. Global measurement of water waves by Fourier transform profilometry. Exp Fluids 2009;46(6):1037–47. [10] Qian Zeng, Fourier transform profilometry using linear CCD array In: Proceedings of the symposium on photonics and optoelectronics (SOPO), 2012. [11] Halioua M, Liu H C. Optical three-dimensional sensing by phase measuring profilometry. Opt Lasers Eng 1989;11(3):185–215. [12] Yoneyama S, Morimoto Y. Phase-measuring profilometry of moving object without phase-shifting device. Opt Laser Eng 2003;40:153–61. [13] Hu Eryi, He Yuming. Profile measurement of a moving object using an improved projection grating phase-shifting profilometry. Opt Commun 2009;282(15):3047–51. [14] Chen Feng, Su Xianyu. Analysis and identification of phase error in phase measuring profilometry. Opt Express 2010;18(11):11300–7. [15] Tang Yan, Su Xianyu. 3D sharp measurement of the aspheric mirror by advanced phase measuring deflectometry. Opt Express 2008;16(19):15090–6. [16] Srinivasan V, Liu HC. Automated phase-measuring profilometry of 3-D diffuse objects. Appl Opt 1984;23(18):3105–8. [17] Su Likun, Su Xianyu. Application of modulation measurement profilometry to objects with surface holes. Appl Opt 1999;38(7):1153–8. [18] Dou Yunfu, Su Xianyu. A flexible fast 3D profilometry based on modulation measurement. Opt Lasers Eng 2011;49(3):376–83. [19] Xianyu Su, Likun Su New Fourier transform profilometry based on modulation measurement, Proc. SPIE 3749 Proceedings of the 18th congress of the international commission for optics, 1999. [20] Yingchun Wu, Yiping Cao, Mingteng Lu, Kun Li. An on-line phase measuring profilometry based on modulation. Opt Appl 2010;XLII(1):31–41. [21] Yingchun Wu, Yiping Cao, Yanshan Xiao, Xu Zheng. An on-line phase measuring profilometry: processed modulation using for pixel matching. Optik 2012;4 (Online, 4 June). [22] Stoilov G, Dragostinov T. Phase-stepping interferometry�five-frame algorithm with an arbitrary step. Opt Laser Eng 1997;28(1):61–9. [23] Xu Xinfen, Cao Yiping. An improved Stoilov algorithm based on statistical approach. Acta Opt Sin 2009;29(3):733–7. [24] Guo HW, Chen MY. Least-squares fitting of carrier phase distribution by using a rational function in profilometry fringe projection. Opt Lett 2006;31 (24):3588–90. [25] Tavares PJ, Vaz MA. Linear calibration procedure for the phase-to-height relationship in phase measurement profilometry. Opt Commun 2007;274 (2):307–14. [26] Xiao Yanshan, Cao Yiping. Improved algorithm for phase-to-height mapping in phase measuring profilometry. Appl Opt 2012;51(8):1149–55. [27] Chen Yaoyao, Zhou X, Xiao YL. An improved watermarking method based on double random phase encoding technique. Opt Laser Technol 2010;42 (4):617–23. [28] Zhong Lijun, Cao Yiping. Online phase measuring profilometry with phase shifting perpendicular to moving direction of measured object. Acta Opt Sin 2009;29(2):417–20. [29] Yu Chunsheng, Peng Qingjin. A unified-calibration method in FTP-based 3D data acquisition for reverse engineering. Opt Laser Eng 2007;45(3):396–404. [30] Zappa Emanuele, Busca Giorgio. Fourier-transform profilometry calibration based on an exhaustive geometric model of the system. Opt Laser Eng 2009;47 (7-8):754–67.
Contents lists available at SciVerse ScienceDirect
Optics and Lasers in Engineering journal homepage: www.elsevier.com/locate/optlaseng
A new pixel matching method using the modulation of shadow areas in online 3D measurement Kuang Peng, Yiping Cao n, Yingchun Wu, Yanshan Xiao Opto-Electronics Department, Sichuan University, Chengdu 610064, China
art ic l e i nf o
a b s t r a c t
Article history: Received 6 December 2012 Received in revised form 17 February 2013 Accepted 15 March 2013 Available online 23 April 2013
A new pixel matching method based on phase measuring profilometry (PMP) using the modulation of shadow areas in online 3D measurement is proposed and verified by experiments. The modulation of the shadow areas representing low reliability of the phase of the object which is usually avoided as much as possible is now used to match pixels of the measured object on pipe-line as the reason for its distinguished feature in this paper. In this method based on the modulation of shadow areas, we use the Otsu method to find out a threshold value and then use it to realize the binarization for the modulation image of the object extracted from the deformed fringes. As there are only 0 and 1 in the binarization image so that the calculation is fast as the product of 0 and anther nonzero number is 0, the speed of pixel matching can be promoted with the precision guaranteed. With more complex objects, the saved time will be more. The experiments prove it. & 2013 Elsevier Ltd. All rights reserved.
Keywords: Modulation Otsu method Binarization Shadow areas Online 3D measurement
1. Introduction With the fast development of the modern industry and a large number of applications of the pipe-lines, online 3D measurement technology [1,2] is used in many kind of fields such as robotics, component quality control, medicine and solid modeling applications [3], and in the meanwhile the demands for online 3D measurement become higher and higher. Traditional methods of online 3D measurement are mainly based on machine vision [4,5], and the precision is affected by the image processing algorithm. The precision is limited by different kinds of algorithms applied for the objects with the condition of different complexities and the algorithms usually need a large amount of calculations. Other three dimensional measurement methods, such as Fourier transform profilometry (FTP) [6–10], phase measurement profilometry (PMP) [11–16] and modulation measurement profilometry (MMP) [17–19] etc., obtain the three dimensional contour information of the object by analyzing the spatial structure light field modulated by the object's surface. FTP, proposed by Takeda and Mutoh [6], is suited for online 3D measurement as it requires only one deformed fringe recorded. Lurong Guo and Su [7] proposed an improved FTP method using a defocused optical field with a Ronchi grating and a grating π phase shifting technique, which can extend the measurable slope of height variation by nearly three times. Although FTP has several
n
Corresponding author. E-mail address: [email protected] (Y. Cao).
0143-8166/$ - see front matter & 2013 Elsevier Ltd. All rights reserved. http://dx.doi.org/10.1016/j.optlaseng.2013.03.008
advantages of easy collection, simple data processing and fast speed, high frequency information of the object is missing during the spatial filtering process and so the information of the object is not complete and the precision cannot be ensured. PMP proposed by V. Srinivasan can satisfy the demands of online measurement due to its characteristic high precision. In PMP, more than three deformed fringe patterns with phase shifting will be recorded so that the coordinates in each pattern do not correspond as the object is moving. So pixel matching is needed. In the conventional method in PMP, pixel points are matched with certain characteristics marked on the measured object. Modulation is important information of the deformed patterns, and it is the basis of the reliability of phase unwrapping in PMP. On the basis of it, Yingchun Wu et al. proposed a pixel matching method based on modulation [20] and a processed modulation using delamination for pixel matching [21]. The measurement profilometry based on modulation (MMP) proposed by Xianyu Su can realize the three dimensional surface shape measurement of the object. Modulation, the basis of weighted approach in the phase unwrapping, is usually applied in phase unwrapping of the equal step phase shifting algorithm. In this paper, a new pixel matching method based on the modulation of shadow areas in online 3D measurement is proposed. The modulation information of the object can be obtained by using the method of Fourier transform analysis easily, and the modulation of the shadow areas representing low reliability which should be avoided generally is used to do pixel matching as its distinguished feature. In the processing, for further improvement in the accuracy and the speed of pixel matching in 3D
K. Peng et al. / Optics and Lasers in Engineering 51 (2013) 1078–1084
DLP
Then through pixel matching and following the Stoilov algorithm, phase function can be calculated using Eq. (1): 2ðI′2 ðx,yÞ−I′4 ðx,yÞÞ ϕ ¼ arctan sinϕ0 , ð2Þ 2I′3 ðx,yÞ−I′1 ðx,yÞ−I′5 ðx,yÞ
D
CCD d
Q
P
1079
sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi I′1 ðx,yÞ−I′5 ðx,yÞ 2 sinϕ0 ¼ 1− , 2½I′2 ðx,yÞ−I′4 ðx,yÞ
z
L
D Object h C
B
A
O
Reference Plane
ð3Þ
In Eqs. (2) and (3), I′n ðx,yÞ, n ¼ 1,2,3,4,5, are the deformed patterns captured from I n ðx,yÞ, n ¼ 1,2,3,4,5, which will be described in Section 3. As Eq. (2) shows arctan calculation which makes the phase truncated in the scope from −π to π, the phase is discrete. But in fact the surface phase distribution is continuous, so the truncated phase should be unwrapped. In the paper we use the diamond phase unwrapping algorithm. And then by the mapping algorithm between phase and height, the height distribution information of the object hðx,yÞ can be reconstructed: 1 1 1 ¼ aðx,yÞ þ bðx,yÞ þ cðx,yÞ 2 , hðx,yÞ ϕðx,yÞ ϕ ðx,yÞ
Fig. 1. On-line PMP setup.
measurement, we use the Otsu method for finding out a threshold value and binarization processing for shorting the spending time in correlation operation. Then the modulation of the shadow areas can be extracted from deformed fringe patterns. By setting one pattern as a template, pixel matching which sets the pixel coordinates of each deformed fringe image with one-to-one correspondence can be done. Then we can obtain the phase information of the object by using the Stoilov algorithm [22,23] in PMP. And through the phase unwrapping and calibration processing [24–30], the object surface shape can be reconstructed.
2. Principle of PMP Phase measurement profilometry (PMP) is a 3D measurement method using the phase information to reconstruct the object surface shape and the Stoilov algorithm is a new phase shift algorithm with equal steps. A schematic of the basic measuring system in PMP based on the Stoilov algorithm is illustrated in Fig. 1. The point O is the intersection between the optic axis of digital light processing (DLP) and that of the imaging system (CCD). The constant and the variable representing the distance are signed in Fig. 1. A computer can compile a sinusoidal grating fringe suitably and then it is projected onto the tested object mounted on the pipe-line by DLP. The length of the sinusoidal grating fringe spreading from DLP to the surface of the object is changing caused by the movement of the measured object so that the original phase pattern without the object on the reference plane is modulated by the height of the measured object. Then the deformed patterns are recorded by CCD. By the Stoilov algorithm, on the condition of satisfying equal steps even without knowing its magnitude, it is possible to calculate the truncated phase of the object surface accurately. When the amount of phase shift ϕ0 is equal, five frame deformed patterns I n ðx,yÞ, n ¼ 1,2,3,4,5 acquired in the five steps with equal distance by CCD are shown below: I n ðx,yÞ ¼ Rðx,yÞ Aðx,yÞ þ Bðx,yÞcos½ϕðx,yÞ þ ðn−1Þϕ0 , n ¼ 1,2,3,4,5 ð1Þ where Rðx,yÞ is the surface reflectance of the object, Aðx,yÞ is background light intensity, Bðx,yÞ represents the fringe contrast, and ϕðx,yÞ indicates phase information modified by the height of the object.
ð4Þ
where aðx,yÞ, bðx,yÞ and cðx,yÞ in the Eq. (4) are acquired by plane calibration.
3. A pixel matching method using the modulation of shadow areas Pixel matching is a process which can make the points of each frame correspond one-to-one so that the phase shifting can be calculated. Without this processing, using the Stoilov algorithm would lead the error as the object is not static. Modulation information of the object can be used to realize the detection of the object displacement, and so the corresponding cut patterns from five deformed patterns can be obtained. After the Fourier transformation of both sides of Eq. (1) Gn ðf x ,f y Þ ¼ Gn0 ðf x ,f y Þ þ Gn1 ðf x ,f y Þ þG−n1 ðf x ,f y Þ,
n ¼ 1,2,3,4,5
ð5Þ
where Gn0 ðf x ,f y Þ, Gn1 ðf x ,f y Þ, and G−n1 ðf x ,f y Þ represent the deformed patterns of zero level and positive and negative first level spectra respectively. Using the spatial filtering method to filter out the first level spectrum Gn1 ðf x ,f y Þ and carrying out inverse Fourier transform: h i þ∞ P n ðx,yÞ ¼ ∬ Gn1 ðf x ,f y Þexp i2πðf x x þf y yÞ df x df y −∞
1 ¼ Rðx,yÞBðx,yÞexp i2πf x x þ ϕ þ ϕ0 2
ð6Þ
Modulation distributions are defined as the modulus of P n ðxn ,yn Þ, n ¼ 1,2,3,4,5: 1 ð7Þ M n ðx,yÞ ¼ ½P n ðx,yÞ ¼ Rðx,yÞBðx,yÞ, n ¼ 1,2,3,4,5 2 In the equation, the modulation information of the deformed patterns contains the surface reflectance of the object Rðx,yÞ and the fringe contrast of projected fringe Bðx,yÞ. If the uniformity of the projected fringe patterns can be guaranteed, Bðx,yÞ could be regarded as a constant, so the modulation can reflect the gray information of the measured object well. When the object is moving online on the pipe-line, the modulation of the object corresponding to each point would be moving accordingly, so the modulation can be regarded as the marks of the object movement. In the modulation patterns, the most distinct part is the shadow areas where the value of the modulation is low. As usual, the low modulation represents low reliability which will make phase unwrapping inaccurate and
1080
K. Peng et al. / Optics and Lasers in Engineering 51 (2013) 1078–1084
time-consuming and so we should avoid it as much as possible. Meanwhile, the most distinct characteristic parts which are moving with the object in five patterns are the shadow areas, and they can be used in pixel matching. When the object is moving, the angle from CCD to the measured object varies and the shadow areas will be changing. But at the same time, the speed for recording five deformed fringe patterns is so large that the angle will change slightly. These changes have few effects on the precision and efficiency of the method because we can get accurate one-to-one corresponding deformed patterns as shown later. To extract the shadow areas from five deformed fringe patterns, the Otsu method is used to find out a threshold value K
and then we use it to realize the binarization for the modulation pattern as shown follows. ( M n ðx,yÞ ¼
1,M n ðx,yÞ o K 0,others
,
n ¼ 1,2,3,4,5
ð8Þ
In the binarization, only values M n ðx,yÞ, n ¼ 1,2,3,4,5, smaller than K are set to be 1 and other values are set to be 0. As there are only 0 and 1 in the binarization patterns and the product of 0 and any other nonzero number is 0, the speed of correlation operation and pixel matching will be promoted with the precision guaranteed.
Fig. 2. Simulated construction: (a) simulation object, (b)the first frame deformed fringe pattern, and (c) the second frame deformed fringe pattern.
Fig. 3. Binarization processing: (a) the modulation of the first frame, (b) the modulation of the second frame, (c) the binarization pattern of the first frame, and (d) the binarization pattern of the second frame.
K. Peng et al. / Optics and Lasers in Engineering 51 (2013) 1078–1084
1081
Fig. 4. Experimental results: (a) template, (b) cut pattern in the first frame pattern after pixel matching, (c) cut pattern in the second frame pattern after pixel matching, (d) reconstructed object, and (e) error distribution.
4 frame deformed patterns by moving S pixels along the direction of the object movement on the pipe-line:
Table 1 The time spent with and without the binarization. Measuring sequence
1
2
3
Average
Time spent with the binarization (s) Time spent without the binarization (s)
1.2415 1.5994
1.2666 1.6073
1.2638 1.6082
1.2573 1.6050
Work plane
X
3D measuring device
I′i ðx,yÞ ¼ ∏I i ðx,yÞI′1 ðx,yÞ,
i ¼ 2,3,4,5
ð9Þ
In Eq. (9), ∏ is the pixel matching operation cutting the corresponding deformed patterns at the position shifting nS (n ¼1,2,3,4) in the moving direction by the template's size from I n ðx,yÞ, n ¼ 2,3,4,5, respectively. Pixel matching, by obtaining the object equal step distance by the modulation of the shadow areas and using the image clipping method, can make the pixel coordinates in five deformed patterns cut down I′n ðx,yÞ, n ¼ 1,2,3,4,5 one-to-one correspondence.
4. Numerical simulations and experiments
Z Measured Object Step-motor
Fig. 5. The setup of the experiment.
Then the entire modulation distribution of the shadow areas M 1 ðx,yÞ is cut from the first frame deformed fringe and taken to be a template. The modulations of the Nth (N ¼2,3,4,5) frame deformed fringe pattern can be matched by carrying out correlation operation so that the shift of the object S can be measured. The deformed pattern I′1 ðx,yÞ, including the areas where the object is located, is cut from the first frame deformed pattern I 1 ðx,yÞ, and then the shift S can be used to obtain the phase shift fringe patterns at the corresponding coordinates of the other
To verify the feasibility of proposed online 3D measurement and check its speed, a series of experiments have been done. Firstly an experiment is simulated where the sample of the experiment is half of an ellipsoid, the long and short axis of which are 230 mm and 130 mm respectively and the height of which is 30 mm as shown in Fig. 2(a). When the ellipsoid on the pipe-line is moving, the sinusoidal grating fringe is modified by the height of the object, and five frame deformed patterns are captured at five positions with equal distance and two of them are shown in Fig. 2(b) and (c). Fig. 3 (a) and (b) show the modulation of two frame deformed patterns, and the corresponding binarization patterns are shown in (c) and (d). The shadow areas in the binarization pattern of the first frame are cut down to be regarded as a template as shown in Fig. 4(a). Then on the basis of the template, pixel matching can be done after carrying out the correlation operation and the distance between two adjacent patterns S can be calculated. Then according
1082
K. Peng et al. / Optics and Lasers in Engineering 51 (2013) 1078–1084
to the distance S, the cut parts in two patterns after pixel matching are shown in Fig. 4(b) and (c). The simulated reconstruction results are shown in Fig. 4(d). Fig. 4(e) shows the absolute error distribution of 3D reconstruction, the biggest error of which is less than 0.07 mm and the RMS error of which is 0.0221 mm. The precision of this method is guaranteed. We have done large numbers of simulations both with binarization processing and without it as shown in Table 1; the average time spent in the pixel matching with the binarization processing
is about 28% faster than that without this processing. From the simulation results and the analysis, the high accuracy and efficiency of the proposed method are confirmed. Further verification of the precision and efficiency of the method proposed in this paper, a rabbit-like object which is 50.82 mm 60.01 mm 10.20 mm in volume had been measured. Fig. 5 shows the setup of the experiment. In Fig. 5, the white box is a 3D measuring device integrating the functions of both DLP and CCD for 3D online measurement. The
Fig. 6. Initial data: (a) rabbit-like object, (b) added noise, (c)the first frame deformed fringe pattern, and (d) the second frame deformed fringe pattern.
Fig. 7. Binarization processing: (a) the modulation of the first frame, (b) the modulation of the second frame, (c) the binarization pattern of the first frame, and (d) the binarization pattern of the second frame.
K. Peng et al. / Optics and Lasers in Engineering 51 (2013) 1078–1084
projection and the camera on it are on a straight line along the movement's direction X of the measured object. The Z axis is used to change the distance between the 3D measuring device and the work plane to control the stripe projected onto the work place. One sinusoidal grating image is vertically projected onto the measured object and then five deformed patterns modulated by the height of the rabbit-like object are captured by the 3D measuring device while the measured object moves along the X axis at equivalent steps controlled by a set of step-motors. The projected grating pitch is about 3.34 mm and the moving step is about 2.33 mm. And then the five deformed patterns are transmitted to a computer to be processed with the above proposed method.
1083
Fig. 6 shows the initial data in the experiment. In Fig. 6, (a) is the rabbit-like object, and (c) and (d) are the first and second frame deformed fringe patterns respectively after modification by the object. And as the captured deformed patterns are in good quality, noise is added into the deformed patterns and its RMS is about 0.5 mm as shown in Fig. 6(b). Following the processing introduced above, Fig. 7(c) and (d) show the binarization results of two frame deformed patterns corresponding to Fig. 7(a) and (b). We can see the shadow areas after the binarization processing. The template is shown in Fig. 8(a). Then through correlation operation and pixel matching, the one-to-one coordinate
Fig. 8. Experimental results: (a) template, (b) cut pattern in the first frame pattern after pixel matching (c) cut pattern in the second frame pattern after pixel matching, (d) reconstructed object, and (e) a cross section of the reconstructed object.
1084
K. Peng et al. / Optics and Lasers in Engineering 51 (2013) 1078–1084
Table 2 The time spent with and without the binarization. Measuring sequence
1
2
3
Average
Time spent with the binarization (s) Time spent without the binarization (s)
1.3281 1.8473
1.3278 1.7926
1.3637 1.8182
1.3399 1.8194
corresponding parts are cut from the two patterns are shown in Fig. 8(b) and (c). Fig. 8(d) shows the reconstructed rabbit-like object which is about 10.11 mm in height and the object's real height is 10.20 mm. Fig. 8(e) is a cross section of the constructed object and it is smooth. We have done lots of experiments without the binarization processing and there is analysis on the time spent in the pixel matching with and without the binarization processing. Rather than the average time without the processing as shown in Table 2, the average time spent in the pixel matching with the binarization processing is about 35% faster. From the results of the simulation and the experiment, we can confirm that the proposed method is accurate and efficient. And with a more complex measured object, the saved time can be more. 5. Conclusions In this paper, a new pixel matching method using the modulation of shadow areas is proposed. The entire modulation distribution of the object can be used in pixel matching instead of the traditional marking method and we finish the binarization for the modulation patterns with the Otsu method finding out the threshold value. By the correlation operation between the template cut from the first binarization pattern and the modulation of the other frame binarization patterns, the amount of the object shifting can be calculated. On this basis, pixel matching has been accomplished and the measured object can be reconstructed finally. As experimental results show, this way can satisfy the demand of high accuracy and fast measurement in online 3D measurement on the pipe-line in the industry. At the same time, it presents a new idea using the entire modulation distribution of the shadow areas to match pixels in online 3D measurement. Acknowledgment This work is supported by 863 National Plan Foundation of China under Grant no. 2007AA01Z333. References [1] Song Zhang. Recent progresses on real-time 3D shape measurement using digital fringe projection techniques. Opt Laser Eng 2010;48(2):58–149.
[2] Xianyu Su, Qican Zhang. Dynamic 3-D shape measurement method: a review. Opt Laser Eng 2010;48(2):191–204. [3] Gorthi SS, Rastogi P. Fringe projection technique: whither we are? Opt Laser Eng 2010;48(2):133–40. [4] Cubero Sergio, Aleixos Nuria. Advances in machine vision applications for automatic inspection and quality evaluation of fruits and vegetables. Food Bioprocess Technol 2011;4(4):487–504. [5] Ho Chao-Ching, He Jun-Jia, Liao Te-Ying. On-line estimation of laser-drilled hole depth using a machine vision method. Sensors 2012;12(8):10148–62. [6] Takeda M, Mutoh K. Fourier transform profilometry for the automatic measurement of 3-D object shape. Appl Opt 1983;22(24):3977–82. [7] Guo Lurong, Su Xianyu. Improved Fourier transform profilometry for the automatic measurement of 3D object shapes. Opt Eng 1990;29(12):1439–44. [8] Zappa Emanuele, Busca Giorgio. Static and dynamic features of Fourier transform profilometry: a review. Opt Laser Eng 2012;50(8):1140–51. [9] Cobelli Pablo Javior. Global measurement of water waves by Fourier transform profilometry. Exp Fluids 2009;46(6):1037–47. [10] Qian Zeng, Fourier transform profilometry using linear CCD array In: Proceedings of the symposium on photonics and optoelectronics (SOPO), 2012. [11] Halioua M, Liu H C. Optical three-dimensional sensing by phase measuring profilometry. Opt Lasers Eng 1989;11(3):185–215. [12] Yoneyama S, Morimoto Y. Phase-measuring profilometry of moving object without phase-shifting device. Opt Laser Eng 2003;40:153–61. [13] Hu Eryi, He Yuming. Profile measurement of a moving object using an improved projection grating phase-shifting profilometry. Opt Commun 2009;282(15):3047–51. [14] Chen Feng, Su Xianyu. Analysis and identification of phase error in phase measuring profilometry. Opt Express 2010;18(11):11300–7. [15] Tang Yan, Su Xianyu. 3D sharp measurement of the aspheric mirror by advanced phase measuring deflectometry. Opt Express 2008;16(19):15090–6. [16] Srinivasan V, Liu HC. Automated phase-measuring profilometry of 3-D diffuse objects. Appl Opt 1984;23(18):3105–8. [17] Su Likun, Su Xianyu. Application of modulation measurement profilometry to objects with surface holes. Appl Opt 1999;38(7):1153–8. [18] Dou Yunfu, Su Xianyu. A flexible fast 3D profilometry based on modulation measurement. Opt Lasers Eng 2011;49(3):376–83. [19] Xianyu Su, Likun Su New Fourier transform profilometry based on modulation measurement, Proc. SPIE 3749 Proceedings of the 18th congress of the international commission for optics, 1999. [20] Yingchun Wu, Yiping Cao, Mingteng Lu, Kun Li. An on-line phase measuring profilometry based on modulation. Opt Appl 2010;XLII(1):31–41. [21] Yingchun Wu, Yiping Cao, Yanshan Xiao, Xu Zheng. An on-line phase measuring profilometry: processed modulation using for pixel matching. Optik 2012;4 (Online, 4 June). [22] Stoilov G, Dragostinov T. Phase-stepping interferometry�five-frame algorithm with an arbitrary step. Opt Laser Eng 1997;28(1):61–9. [23] Xu Xinfen, Cao Yiping. An improved Stoilov algorithm based on statistical approach. Acta Opt Sin 2009;29(3):733–7. [24] Guo HW, Chen MY. Least-squares fitting of carrier phase distribution by using a rational function in profilometry fringe projection. Opt Lett 2006;31 (24):3588–90. [25] Tavares PJ, Vaz MA. Linear calibration procedure for the phase-to-height relationship in phase measurement profilometry. Opt Commun 2007;274 (2):307–14. [26] Xiao Yanshan, Cao Yiping. Improved algorithm for phase-to-height mapping in phase measuring profilometry. Appl Opt 2012;51(8):1149–55. [27] Chen Yaoyao, Zhou X, Xiao YL. An improved watermarking method based on double random phase encoding technique. Opt Laser Technol 2010;42 (4):617–23. [28] Zhong Lijun, Cao Yiping. Online phase measuring profilometry with phase shifting perpendicular to moving direction of measured object. Acta Opt Sin 2009;29(2):417–20. [29] Yu Chunsheng, Peng Qingjin. A unified-calibration method in FTP-based 3D data acquisition for reverse engineering. Opt Laser Eng 2007;45(3):396–404. [30] Zappa Emanuele, Busca Giorgio. Fourier-transform profilometry calibration based on an exhaustive geometric model of the system. Opt Laser Eng 2009;47 (7-8):754–67.