- Email: [email protected]
Line-based calibration of a micro-vision motion measurement system
Optics and Lasers in Engineering 93 (2017) 40–46
Contents lists available at ScienceDirect
Optics and Lasers in Engineering journal homepage: www.elsevier.com/locate/optlaseng
Line-based calibration of a micro-vision motion measurement system ⁎
Hai Li, Xianmin Zhang , Heng Wu, Jinqiang Gan
MARK
Guangdong Provincial Key Laboratory of Precision Equipment and Manufacturing Technology, School of Mechanical and Automotive Engineering, South China University of Technology, Guangzhou 510640, China
A R T I C L E I N F O
A B S T R A C T
Keywords: Micro-vision system Calibration Line-based method Resolution target Microscope Motion measurement
We propose a flexible and practical line-based method for calibrating a micro-vision motion measurement system by using a resolution target, considering the lens distortion. First, centerlines of the stripes in the image of resolution target are extracted with an improved center gravity approach. Then distortion coefficients are obtained through the nonlinear optimization, utilizing straightness of the lines. Finally, these lines are corrected and used for estimating parameters of the system. Two experiments are conducted to validate the effectiveness of the proposed method. The results demonstrate that our method is effective and practical for calibrating the micro-vision measurement system.
1. Introduction
MVS [13]. As can be seen, MVSs are widely used in the field of motion measurement with micro/nanoscale. However, an obvious drawback of MVS is its low absolute accuracy when used for high-precision measurement. This is due to the using of inaccurate geometric model of the system. In most cases, when a microscope is used for measurement, only the pixel representing distance (PRD) is roughly calibrated by using a stage micrometer or calculated from the theoretical model. In these situations, lens distortion is ignored, the intrinsic and extrinsic parameters of the system are either ignored or obtained according to the theoretical model. In order to improve the absolute measurement accuracy of the MVS and to obtain full intrinsic and extrinsic parameters, calibration procedures are indispensable. In this study, as the motions of the compliant mechanisms are planar movements, calibrating means finding the accurate mapping relationship between the motion plane and the image plane. In addition, both the PRD and magnification of the MVS need to be obtained. Currently, abundant algorithms for calibrating macro-vision measurement system have been proposed [14–16]. However, when it comes to microscale, two difficulties have made these commonly used methods out of action. The first is the lack of the calibration targets, which is caused by the small field of view (FOV). The other is that only a single image can be used during the calibration. The reasons are that the image plane and the object plane are almost parallel, and the depth of field of the microscope is narrow. In the domain of automatic micromanipulation, to realize the localization and the accurate measurement, several approaches have been proposed for the calibration of the microscopes. For instance, Zhou et al. proposed a special geometric
Compliant mechanisms are flexible mechanisms that transfer an input force or displacement to another point through elastic body deformation [1]. Compliance in design leads to joint-less, non-assembly, monolithic mechanical devices and thus is particularly suitable for applications with small range of motions [2], e.g., the bio-micromanipulations [3], the micro-electromechanical systems (MEMS) [4], and the precision positioning stages [5], etc. As more and more attentions have been paid to the design and apply of the compliant mechanisms [6–8], how to effectively measure motion of these mechanisms becomes more and more urgent. Due to the features of high resolution (from submicrometer to nanometer) and compact structure, detecting the motion of the compliant mechanisms is always a challenging work, especially when the mechanisms have multiple degrees of freedom (DOF). Owing to the advantages of non-contact, multi-DOF measurement ability, high resolution, etc., the microscopic vision system (MVS), which consists of optical microscope, camera and computer, provides an alternative for motion detection of compliant mechanisms. A number of micro-vision based approaches have been proposed for detecting micro/nanoscale motions in recent years [9–13]. Wu et al. realized the motion detection of an inverter and a three DOF positioning stage [9,10], utilizing the MVS. Assaf et al. presented an approach for measuring nanoscale displacements of the micro-devices by using an optical microscope [11]. Ouyang et al. proposed a visual-servo method that can improve the trajectory tracking precision of compliant micromanipulator [12]. Xia et al. achieved the three-dimensional surface displacement and shape measurements at nanoscale with the help of
⁎
Corresponding author. E-mail address: [email protected] (X. Zhang).
http://dx.doi.org/10.1016/j.optlaseng.2016.12.018 Received 27 August 2016; Received in revised form 25 December 2016; Accepted 26 December 2016 0143-8166/ © 2016 Elsevier Ltd. All rights reserved.
Optics and Lasers in Engineering 93 (2017) 40–46
H. Li et al.
model of the microscope and realized the calibration by using a 2D dot array-based calibration object [17]. Method proposed in this paper is commonly used for calibrating microscope. However, when the FOV of MVS keeps decreasing, the problem of short of calibration objects will become serious. In Ref. [18–20], researchers have presented several virtual-image-based methods, which mean that targets for calibration are obtained by virtue of precision mechanical setups. Such as the atomic force microscope (AFM) tips, the micromanipulators, etc. Nevertheless, these virtual-image-based approaches are much too complicated and lack of operational flexibility, especially when the magnification of the system needs to be changed frequently. To solve these problems, we propose a line-based calibration method for the MVS which uses the resolution target as the calibration object for the first time. As a generally used setup for the performance evaluation of microscope, resolution target normally has groups of linespace patterns with different spatial frequency, like the United States Air Force (USAF) 1951 (see in Section 2). Therefore, it is compatible for calibrating the MVSs with different magnifications and FOVs. There are four steps in this line-based method. Initially, centerlines of the stripes are extracted with the improved center of gravity method. Then distortion coefficients and the principle point are obtained through the nonlinear optimization of the projective invariant of the lines. After that, centerlines are corrected and fitted to estimate the homographic matrix. Finally, full intrinsic and extrinsic parameters are acquired from decomposing the homographic matrix. After implementing these steps, an accurate geometric model of the MVS can be obtained. The remainder of this paper is organized as follows. In Section 2, preliminary information about the MVS is given. Section 3 contains the detailed procedures of our line-based calibration method. In Section 4, a series of experiments are conducted to verify the effectiveness of the proposed method. Section 5 is the final conclusions.
Fig. 2. An example of the resolution target. (a) 1951 USAF resolution target, (b) an element of stripes in one group, (c) geometric characteristic of the stripes in one element.
In this study, a type of 1951 USAF resolution target is chosen as the calibration object, as shown in Fig. 2(a). This kind of target consists of groups of line-space patterns whose spatial frequency increases as the sixth root of two. A set of six elements (three horizontal lines and three vertical lines) is in one group (Fig. 2(b)), and which group will be used during the calibration is determined by the FOV of the MVS. The geometry characteristics of the stripes in one group is shown in Fig. 2(c). Since the largest spatial frequency of line-space patterns in this target can reach to 228 LP/mm (the corresponding value of X is 2.19 µm), this kind of resolution target can meet the needs for calibrating vision system with quite small FOV (high magnification). 2.2. Geometric model of the MVS The geometric model of the system is shown in Fig. 3. Suppose a 2D point on the image plane (IP) is denoted by p = (u , v )T , and the corresponding point on object plane (OP) is P = (x, y )T . Similarly, l and L represent the line pair on the IP and OP, respectively. We use x∼ to denote the normalized description of point vector or line vector. Since the IP and OP are parallel, the relationship between a 2D point M and m is expressed as
2. Preliminaries 2.1. System setup
∼ ∼ ∼ p = HP = AKP ,
The basic setup of the micro-vision measurement system is similar to that of our previous work [10], as shown in Fig. 1. The whole workstation includes an imaging system, a PC, and an adjustable platform (AP). The imaging system is consisted of a lighting source, an objective, a spectroscope, a tube lens, an adapter and a digital camera. The details of these setup are described in Section 4. The AP is used to ensure that the measured target is within the FOV of the camera. By modifying the tube-lens length or changing the objective lens, the FOV and magnification of the system can be easily adjusted according to the needs. Besides, by means of adopting proper algorithms, this system can realize online motion tracking of the compliant mechanisms.
Camera
⎡ f 0 u0 ⎤ ⎡ cos θ sin θ p ⎤ x ⎢ x ⎥ ⎢ ⎥ where A = ⎢ 0 fy v0 ⎥ , K = ⎢− sin θ cos θ py ⎥ . Here, A is the intrin⎢⎣ 0 0 1 ⎥⎦ ⎢⎣ 0 0 1 ⎥⎦ sic matrix, K is the extrinsic matrix which composes of a planar rotation angle θ with a translation vector (px , py ). (u 0 , v0 ) is the coordinate of principle point, fx and fy are the scale factors in image u-axis and v-axis, respectively. H is the homographic matrix between OP and IP. In this paper, since the shape of the pixel elements of the sensor is square and the physical dimensions of the pixels are equal, the skew factor is set as zero, and the scale factors fx and fy are set to be equal. According to the contravariant feature of point mapping [21], the mapping relationship of a line from OP to IP can be written as
Network cable
∼ ∼ l = H −T ⋅L ..
Adapter
(2)
Tube lens Spectroscope
Light source
(1)
PC
Objective
Motion control card Adjusting platform Fig. 1. The schematic diagram of the measuring system.
Fig. 3. The geometric model of the measuring system.
41
Optics and Lasers in Engineering 93 (2017) 40–46
H. Li et al.
2.3. Lens distortion model The model presented in Eq. (1) and Eq. (2) is an ideal pinhole model. However, there are always distortions that make the model nonlinear. For real lenses, the major type of distortion is usually radial distortion [14]. In order to make the distortion compensation more convenient, an inverse model is used to describe the radial distortion. Assuming resolution of the image is R × C . The distortion model used in this study is expressed as 2 4 ⎧ ⎪ u = u + (u − u )(k r d d 0 1 d + k 2rd ) ⎨ , 2 4 ⎪ ⎩ v = vd + (vd − v0 )(k1rd + k 2rd )
where rd2 =
2
(ud − u 0 )
(3)
2
(vd − v0 )
. Here, (u , v )T and (ud , vd )T represent the undistorted coordinate and the corresponding distorted coordinate, k1 and k 2 are the radial distortion coefficients. R2
+
C2
Fig. 4. Extraction of the centerlines. (a) Raw image; (b) Image after morphologic processes; (c) Extracted centerline; (d) Center of gravity method.
3. System calibration
the label value of t at row i can be expressed as
In this section, detailed descriptions about the line-based calibration of MVS are presented. The whole method consists of four procedures, including the detection of the centerlines, solving the lens distortion, estimation of the homographic matrix, and decomposition of the homographic matrix. The full intrinsic and extrinsic parameters can be obtained by performing these four procedures. According to Eq. (1) and Eq. (2), the parameters that need to be calibrated can be summarized as follows.
t
t
Yi =
x
y
y
0
0
1
t
M
∑ j =1 I (i , tT i(j ))
, (5)
where tM represents the number of column coordinates with the labeled value t . After performing this formula to each row of the image, the centerlines of the stripes are obtained as shown in Fig. 4(c).
• Extrinsic parameters: θ , p , p . • Intrinsic parameters: f , f , u , v , k , k , M . x
M
∑ j =1 I (i , tT i(j ))⋅ tT i(j )
3.2. Solving the radial distortion
2
According to the perspective of projective geometry, a line on the OP is still straight on the IP after the ideal pinhole projection. However, when lens distortion occurs, the projected line is not straight any more. Utilizing the property of straightness, as well as the explicit distortion model, the problem of solving lens distortion can be transformed to the question of nonlinear optimization [23–25]. After the camera's imaging, the line in space is discretized into a series of points. Suppose there are N stripes in an image, and the ideal equation (without considering the distortion) of the n−th centerline in the image frame is defined as
Here, M represents the magnification ratio of the MVS. Another parameter that is commonly used during the micro-vision measurement is PRD. The physical meaning of PRD is the space distance each pixel represents. It is the coefficient used to transform the image distance to the physical space distance during the measurement. The relationship between M, f and PRD is presented as follows:
PRD = 1/ f = PS / M ,
(4)
where PS is the physical size of a camera's pixel.
u cos α + v sin α + λ n = 0,
3.1. Extraction of the centerlines
(6)
where α and λ represent the tilt angle and intercept of the line, respectively. Through the introduction of the algorithm in Section 3.1, it can find that the maximum value of label and the number of centerlines are the same. Besides, the number of discrete points in each centerline is R , the distorted coordinate of points in the n−th centerline is (i , tY i ). For the convenience of expression, we assume the distorted coordinates and the corresponding undistorted coordinates of i−th point in the n−th centerline are (tuid , tvid )T and (tuiu , tviu )T , respectively. Then the nonlinear optimization question can be expressed as
It is well known that calibration benefits from sub-pixel accurate location of the features. In this work, we choose the stripe's centerline as the feature to perform the calibration. Utilizing an improved center of gravity method (ICGM), the locating accuracy of the discrete points on the centerline can reach to sub-pixels, as shown below. At the beginning of the calibration, an image of the resolution target is captured by the MVS. The basic requirement is that at least two parallel stripes are included in this image. Without loss of the generality, an image which includes three parallel stripes is used to illustrate the centerline extraction algorithm (see in Fig. 4(a)). After obtaining the image, the Ostu [22] optimal threshold segmentation algorithm is implemented, following with a batch of morphologic approaches to obtain the dilated regions of stripes (Fig. 4(b)). Finally, the ICGM, which is based on the one-dimensional run length coding, is used to obtain coordinates of the discrete points on each centerline. The details of the ICGM are given in the following. Suppose I is the source image, Id is the dilated image obtained after morphologic processing, and I (r , c ) is the gray value of a pixel having image coordinates (r , c ). First, the image Id is scanned row by row. The pixels that belong to the foreground (dilated stripes) is labeled, as shown in Fig. 4(d). Let tT i(j ) be the jth column coordinates with the labeled value t at row i . Then the coordinate of the stripe center with
Γ (k , u 0 , v0 , α , λ1, …, λ n ) N
R
=min ∑t =1 ∑i =1 (tuiu cos α + tviu sin α + λ n )2 ,
(7)
t u t d t d 2 4 ⎧ ⎪ u i = ui + ( ui − u 0 )(k1rd + k 2rd ) where ⎨ . Γ is the optimal objective t d t d 2 4 ⎪ t u ⎩ vi = vi + ( vi − v0 )(k1rd + k 2rd ) function. To solve this equation, the Levenberg-Marquardt nonlinear optimization method is adopted. Since the initial values usually have large effect on the results, the approximate initial values for solving Eq. (7) is given as following. The initial value of the distortion coefficients k1 and k2 are set as zero. The principle point is initialized to be the image center. The initial values of the centerlines’ parameters are obtained by fitting the obtained discrete points into lines.
42
Optics and Lasers in Engineering 93 (2017) 40–46
H. Li et al.
3.3. Estimation of the homographic matrix
3.5. Discussions of this calibration method
After obtaining the distortion coefficients and the principle point, the distorted centerlines can be immediately corrected. Then these corrected lines are used as the elements for estimating the homographic matrix [26]. Since the skew factor of the camera is zero and the planes of OP and IP are parallel, the projective model of the MVS can be simplified as the similarity transformation. According to the projective geometry, the similarity transformation is an isometry composed with an isotropic scaling. Therefore, the homographic matrix can be represented as
For the proposed calibration method, line features are used during the calibration process. Line features are more robust to the image noise than point-based methods, which is important since images from microscope with high magnification are always heavily polluted by various noises. In addition, utilizing the straightness of the line, the distortion coefficients and the principle point can be solved independently through the nonlinear optimization, which makes the solving of other parameters become possible. However, there are some limitations in this method that need to be improved in the future. The first is the centerline extraction algorithm. In the present method, the precision of the extracted center point is easily affected by the uneven illumination and the large inclination of the stripes. The second is the mapping model. In the current method, based on the facts that the depth of field of the objective is quite small and the installation accuracy is very high, we assume that the IP and OP are parallel. In order to further improve the measurement accuracy, nonparallel angular errors need to be considered. This can be achieved by using a projective transformation model replaces the currently used similar transformation model. To solve the projective transformation model, calibration image which includes more lines in the normal position will be needed.
⎡ sx sy tx ⎤ ⎢ ⎥ H = ⎢−sy sx ty ⎥ , ⎢⎣ 0 0 1 ⎥⎦
(8)
where sx , sy , tx , and ty are the parameters that need to be estimated. Let ∼ li = (lai, lbi, lci )T be the corrected centerline in image frame, and ∼ Li = (Lai, Lbi, Lci )T be the corresponding line in the objective frame. ∼ ∼ According to the Eq. (2) and Eq. (8), at least two pairs of l and L are needed to estimate the matrix H since there are four unknown parameters in H. Normally, more group of lines will make the solution more accurate and reliable. During the estimation of H, as the last element in H is set as one, the lines must be normalized to have the form of l = (la, lb, 1)T to hold Eq. (2). Nevertheless, when a line crosses over the origin of the frame, the last element will be zero. This makes it impossible to normalize the last ∼ ∼ elements in l and L . To figure out this problem, the normalization should be implemented on the non-zero elements of the lines. This results in an increase in the number of parameters in Eq. (2). In this study, the origin of the object frame is set to the first detected point on the first centerline, and the y-axis direction is set to coincide with the centerline. Therefore, these centerlines can be normalized as ∼ ∼ Li′ = (Lai′ , 1, Lci′ )T and li′ = (lai′ , 1, lci′ )T , with i = 1, 2, 3. The projective ∼ ∼‘ relationship between L and l ′ is then expressed as
⎡ sx′ −sy′ 0 ⎤ ⎢ ⎥∼ ∼′ T ∼′ Li = H l = ⎢ sy′ sx′ 0 ⎥⋅li′, ⎢ ⎥ ⎣ tx′ ty′ ρ ⎦
4. Experiments and results To demonstrate the practicability and reliability of the proposed method, two experiments are carried out. The first is to demonstrate the feasibility and effectiveness of the proposed method in calibrating the MVS. The other is to evaluate the performance improvement of the MVS in measuring motions of the compliant mechanisms. In the first experiment, the MVS is calibrated using the method proposed in this paper and the method proposed in Ref. [17], respectively. In the second experiment, the MVS and a LIM are used to synchronously measure motion of a single DOF compliant positioning stage, so that to evaluate performance of the MVS. 4.1. Calibration of the MVS
(9)
Experimental setup used in this experiment is shown in Fig. 5. The imaging system consists of an objective with a controllable tube (Navitar 12X UltraZoom-Motorized lens, magnification ranges from 0.714 to 3.330 when combines with micro objectives), a CCD camera (AVT Manta-G201, resolution is 1624×1234 pixels) with an adapter (Navitar 0.5× standard adapter). An USAF 1951 resolution target (Edmund, 3" x 3" Positive) is used, as shown in Fig. 5(c). A computer (Intel(R) Core(TM) i7-4570, CPU 2.40 GHz, RAM 4 GB) is dedicated to implement the proposed methods. The algorithms are developed by
where ρ is the additional parameter. Through the linear transformation of Eq. (9), a new equation can be obtained as
⎡ Lai′ ⎤ ⎡ lai′ −1 0 0 0 ⎤ ⎥ ⎢ ⎥ ⎢ 1 l′ 0 0 0 ⎥h = Bh, ai ⎢1⎥=⎢ ⎢⎣ Lci′ ⎥⎦ ⎢ 0 0 l ′ 1 l ′ ⎥ ci ⎦ ai ⎣
(10)
where h = (sx′, sy′, tx′, ty′, ρ) . Since there are five unknown parameters in ∼ ∼ H and each line pair (l ↔ L ) generates three constrains, at least two pair of lines are needed to solve out all the parameters. T
3.4. Decomposition of the homographic matrix According to Eq. (1) and Eq. (8), the relation between the homographic matrix H, intrinsic matrix A, and extrinsic matrix K is expressed as
⎡ f cos θ f sin θ f p + u 0 ⎤ x x x ⎢ x ⎥ H = AK = ⎢−fy sin θ fy cos θ fy py + v0 ⎥ . ⎢⎣ ⎥⎦ 0 0 1
(11)
Once the principle point (u 0 , v0 ) and the homographic matrix H are obtained, the remainder intrinsic and extrinsic parameters can be easily solved through decomposition of H.
Fig. 5. The experimental setup of the micro-vision system.
43
Optics and Lasers in Engineering 93 (2017) 40–46
H. Li et al.
Table 1 Calibration results using the proposed method. Calibration results
Unit
fx , fy
pixel /μm −2
k1
pixel
k2
pixel −4 1 pixel ° μm
M (u 0 , v0 ) θ
(px , py )
Table 3 Error from the lens distortion.
10X
50X
10X objective
3.574
17.653
L1
L2
L3
L1
L2
L3
0.0234
0.0230
1.35 0.09 2.67 0.73
0.23 0.17 1.53 0.28
2.16 0.11 4.06 0.88
1.54 0.11 3.74 0.93
0.52 0.20 1.64 0.38
2.33 0.24 3.95 0.96
−0.0193
0.0100
15.736 (614.5, 810.6) −0.169
77.673 (615.2, 812.8) 0.906
(38.56, −126.26)
(−44.14, −30.14)
Mean DD (MeDD) MeDD After calibration Max DD (MaDD) MaDD After calibration
50X objective
(Unit: pixel)
decreased after calibration. An un-normal result in this table is that MeDD of L2 is larger than L1 and L3. This is caused by the inaccurate extraction of L2. In the experiment, a LED-point lighting source is used, which has caused the non-uniform illumination in the FOV of the camera. As a result, center part of the image is much brighter than the peripheral part. Since we cannot make sure that influence from nonuniform illumination are symmetric about the second centerline, the extracted coordinates of centerline will be affected, especially the second centerline. This is a disadvantage that we need to improve in the future work.
Matlab. Tube-length is set to the maximum so that it has the theoretical magnification of 3.33. In this experiment, a 10× objective (Mitutoyo, 0.28 NA) and a 50× objective (Mitutoyo, 0.55 NA) are mounted and calibrated, respectively. By multiplying the magnifications of the objective, adapter and tube lens, we can obtain the theoretical magnification of the system. When the 10× and 50× objectives are mounted, the calculated magnifications are 16.65 and 83.25, respectively. The corresponding FOVs are 429 µm ×326 µm and 86 µm ×65 µm. In the experiment, three stripes are included in each calibration image. The distribution of these stripes in the image is similar to the one present in Section 3.1. Table 1 shows the calibration results. From this table, we can find that all the parameters in the projective model are obtained, including the magnification of the system. To show the effectiveness of the proposed method, the method proposed in Ref. [17] is implemented for comparison. Since a dot array is required in this method, a test target R1L1S1P from THORLABS that includes grids with different scales is selected and used in this test. To increase the comparability, we have normalized the coordinates of the distortion model used in Ref. [17], so that the distortion coefficients in these two methods are in unison. Experimental results about the related intrinsic parameters are shown in Table 2. The principle point is not included in this table, because it is not calibrated and is set as (617, 814) in Ref. [17]. Comparing the intrinsic parameters shown in table1 and Table 2, it can be seen that results of magnification M and scale factor f are quite close. In our method, two order distortion model is used so that there are two distortion coefficients. According to the results of k2 shown in Table 1 and the model presented in Eq. (3), it can be seen that distortion from the second order coefficient is quite small. Comparing the first order distortion coefficients in Table 1 and Table 2, we can see that distortion results from these two different methods are also coincident. Hence, it can be seen that the method proposed in this paper is quite effective. To show the influence from the lens distortion, centerlines are fitted by the corrected points and then used as the ideal equations of the lines; afterwards, distance from the corrected point to the ideal lines and distance from extracted points to the ideal lines are both calculated. The distance is named as the distortion deviation (DD). The statistical results are shown in Table 3. From the table, we can find that DD of the points from the first line and third line are larger, this is reasonable since these two lines are further from the principle point than the second line. According to the lens distortion model, the further from the principle point, the bigger the DD. From the third and fourth row of the table, it can be seen that the maximum distortion has apparently
4.2. Performance evaluation In this experiment, the professional measuring equipment of LIM is used. The whole experiment setup is shown in Fig. 6. The LIM is Renishaw XL-80 that has a linear measurement accuracy of ± 0.5 ppm. Because the LIM cannot measure the X and Y direction displacements at the same time, a 1-DOF compliant positioning stage is chosen as the measured object, as shown in Fig. 6. The designed ratio of input displacement and output displacement of this stage is one. The detail experimental scheme is given as below. First, the reflector of the LIM is rigidly connected to the output part of the compliant stage. Then a template that extracts from the surface of the reflector is tracked by the MVS with the method presented in Ref. [9]. Subsequent, a set of input is given in order, as shown in Table 4. After each output of the compliant stage is stable, its displacement is measured by the LIM and the MVS, separately. During each measurement of the compliant stage, 30 times of measurement from the MVS are recorded, the average value is used as the measuring value of the MVS. The measuring results by the LIM system and the MVS are presented in Fig. 7, the detailed data is shown Table 4. Three approximate straight lines are included in this figure. Except the data from LIM and MVS, SMVS data represent the source data from MVS that have not been calibrated and corrected. As can be seen, measurement results from LIM and MVS are almost coincide with each other. However, the SMVS data is obviously deviating from the other two line. Besides, the bigger the input value, the larger the deviation. The reason is that without calibration, the PRD is obtained by the theoretical calculation, as
Table 2 Calibration results using method proposed in [17]. Calibration results
Unit
fx , fy
pixel /μm
k1 M
pixel 1
−2
10X
50X
3.520
17.629
0.0213
0.0189
15.488
77.566 Fig. 6. The experimental setup of the LIM system.
44
Optics and Lasers in Engineering 93 (2017) 40–46
H. Li et al.
Table 4 Source data from the measurement.
Table 5 Source data of the AE and STD.
Number
Input
LIM
MVS
SMVS
1 2 3 4 5 6 7 8 9 10 11 12 13
0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 20.0 22.5 25.0 27.5 30.0
0 2.5562 5.0698 7.5731 10.0675 12.5776 15.0869 17.6219 20.1399 22.6933 25.2870 27.8411 30.4153
0 2.461 5.129 7.586 10.136 12.405 14.959 17.517 20.121 22.677 25.198 27.884 30.461
0 2.376 4.950 7.320 9.777 11.963 14.421 16.882 19.384 21.836 24.254 26.825 29.288
Number
1 2 3 4 5 6 7 8 9 10 11 12 13
(Unit: μm).
MVS
SMVS
AE
STD
AE
STD
0 0.095 0.059 0.013 0.068 0.172 0.124 0.105 0.019 0.018 0.089 0.043 0.045
0.020 0.037 0.018 0.019 0.021 0.022 0.024 0.021 0.027 0.031 0.017 0.021 0.039
0 0.181 0.120 0.253 0.291 0.614 0.666 0.740 0.756 0.857 1.033 1.019 1.128
0.019 0.035 0.017 0.018 0.020 0.020 0.023 0.020 0.026 0.029 0.016 0.020 0.037
(Unit: μm).
results of image processing. During the distortion compensation, the dispersion characteristics of the results in image frame will not change, so the STD results are not influenced in this step. However, during the processing of transforming result from image frame to space frame, the dispersion characteristics of the result is influenced by the PRD, and this kind of influence is linear. As the PRD used in MVS is bigger than that used in SMVS, so the trends of STD results are accordance, and the value from MVS is always larger than that of SMVS, as shown in Fig. 8(b). However, the STD deviations between MVS and SMVS are very small. Comparing Fig. 8(a) and Fig. 8(b), it can be seen that the MVS has high measurement repeatability (small STD) and high measurement accuracy after calibration. From the above two experiments, it can be found that the presented approach can successfully realize the calibration of the MVS with different magnifications and FOV. Compared with the method proposed in Ref. [17], the geometric model present in this paper is simpler, but the calibration results are coincident. It also shows that lens distortion will generate deviations to the measurement results, especially when the measured target is far away from the principle point. Besides, absolute accuracy of this measurement system has been improved from 1.2 to 0.18 µm after calibration. Moreover, accuracy of the MVS still has the potential to be further improved since the STD of the system is very small.
Fig. 7. Experimental result comparisons between the proposed method and the LIM.
shown in Eq. (4). Nevertheless, the magnification used in the calculation is always not accurate. Moreover, influence from lens distortion will generate deviations to the measurement results, and this kind of deviations is varying along with the movement of the target. To further show the improvement of the MVS after calibration, we use the data from LIM as the reference value to calculate the absolute error (AE), the results is shown in Fig. 8(a). The standard deviation (STD) of MVS and SMVS are also given in Fig. 8(b). Table 5 is the detailed data about AE and STD. From Fig. 8(a), it can be seen that after calibration, the AE has decreased from 1.2 to 0.18 µm. Hence, absolute accuracy has apparently improved after calibration. In Fig. 8(b), it shows that STD values from MVS are always bigger than values of SMVS and the ratio between these corresponding data is constant. This is reasonable because both the MVS data and SMVS data are coming from
5. Conclusions This paper presents a flexible and practical line-based method to calibrate a MVS. The resolution target USAF 1951 is firstly proposed
Fig. 8. AE and STD comparisons of MVS before and after calibration.
45
Optics and Lasers in Engineering 93 (2017) 40–46
H. Li et al.
and used as the calibration target in this method. Utilizing the property that there are groups of line-space patterns with different spatial frequency in the target, the proposed method can well meet the need for calibrating MVS with changeable tube length. Hence, the flexibility of the proposed method is high. In addition, the cost of the resolution target is low, so the practicality of this method is high, too. Experimental results demonstrate that the proposed method can successfully realize calibration of the MVS. Except for the magnification and PRD, full intrinsic and extrinsic parameters can be obtained, including the lens distortion. Compared with LIM, it shows that the absolute measuring accuracy of MVS has been apparently improved after calibration. Furthermore, the proposed method possesses the capability for calibrating multi-camera system when more lines with normal positions are included in the calibration image. Improvements that need to be done in the future studies are: 1) improve the robustness and accuracy of the centerline extraction algorithm; 2) more factors need to be considered to further improve the absolute measuring accuracy of the system.
[7] [8] [9] [10]
[11]
[12]
[13] [14] [15]
[16]
Acknowledgments
[17]
This work was supported by the National Natural Science Foundation of China (U1501247); the Scientific and Technological Research Project of Guangdong Province (201604010100, 2015B020239001).
[18]
[19] [20]
References
[21] [22]
[1] Howell LL. Compliant mechanisms. John Wiley & Sons; 2001. [2] Kota S. , et al. Design of compliant mechanisms: applications to MEMS. Analog Integr Circuits Signal Process 2001;29(1–2):7–15. [3] Fukuda T, Arai F, Nakajima M. Micro-nanorobotic manipulation systems and their applications. Springer Science & Business Media; 2013. [4] Kim YS, Yoo JM, Yang SH, Choi YM, Dagalakis NG, Gupta SK. Design, fabrication and testing of a serial kinematic MEMS XY stage for multifinger manipulation. J Micromech Microeng 2012;22(8):085029. [5] Wang H, Zhang X. Input coupling analysis and optimal design of a 3-DOF compliant micro-positioning stage. Mech Mach Theory 2008;43(4):400–10. [6] Zhu B, Zhang X, Fatikow S. Structural topology and shape optimization using a level
[23] [24] [25] [26]
46
set method with distance-suppression scheme. Comput Methods Appl Mech Eng 2015;283:1214–39. Liu P, Yan P. A new model analysis approach for bridge-type amplifiers supporting nano-stage design. Mech Mach Theory 2016;99:176–88. Jin M, Zhang X. A new topology optimization method for planar compliant parallel mechanisms. Mech Mach Theory 2016;95:42–58. Wu H, Zhang X, Gan J, Li H, Ge P. Displacement measurement system for inverters using computer micro-vision. Opt Lasers Eng 2016;81:113–8. Wu H, Zhang X, Gan J, Li H, He Z. High-precision displacement measurement method for three degrees of freedom-compliant mechanisms based on computer micro-vision. Appl Opt 2016;55(10):2594–600. Ya Akobovitz A, Krylov S, Hanein Y. Nanoscale displacement measurement of electrostatically actuated micro-devices using optical microscopy and digital image correlation. Sens Actuators A Phys 2010;162(1):1–7. Ouyang PR, Zhang WJ, Gupta MM, Zhao W. Overview of the development of a visual based automated bio-micromanipulation system. Mechatronics 2007;17(10):578–88. Xia S, Pan Z, Zhang J. Optical microscope for three-dimensional surface displacement and shape measurements at the microscale. Opt Lett 2014;39(14):4267–70. Zhang Z. A flexible new technique for camera calibration. Pattern Anal Mach Intell IEEE Trans on 2000;22(11):1330–4. Heikkila J, Silvén O. A four-step camera calibration procedure with implicit image correction. in Computer Vision and Pattern Recognition, 1997. Proceedings., 1997 IEEE Computer Society Conference on: IEEE; 1997. Hong Y, Ren G, Liu E. Non-iterative method for camera calibration. Opt Express 2015;23(18):23992–4003. Zhou Y, Nelson BJ. Calibration of a parametric model of an optical microscope. Opt Eng 1999;38(12):1989–95. Ammi M, Frémont V, Ferreira A. Automatic camera-based microscope calibration for a telemicromanipulation system using a virtual pattern. Robot IEEE Trans on 2009;25(1):184–91. Kawaji A, Arai F, Fukuda T. 3D calibration for micro-manipulation with precise position measurement. J Micromech 2001;1(2):117–30. Ren L, Wang L, Mills JK, Sun D. Vision-based 2-D automatic micrograsping using coarse-to-fine grasping strategy. IEEE Trans Ind Electron 2008;55(9):3324–31. Hartley R, Zisserman A. Multiple view geometry in computer vision. Cambridge university press; 2003. Abutaleb AS. Automatic thresholding of gray-level pictures using two-dimensional entropy. Comput Vis Graph Image Process 1989;47(1):22–32. Prescott B, McLean GF. Line-based correction of radial lens distortion. Graph Models Image Process 1997;59(1):39–47. Zhou F, Cui Y, Gao H, Wang Y. Line-based camera calibration with lens distortion correction from a single image. Opt Lasers Eng 2013;51(12):1332–43. Jiang S, Cao D, Wu Y, Zhu S, Hu P. Efficient line-based lens distortion correction for complete distortion with vanishing point constraint. Appl Opt 2015;54(14):4432–8. Zeng H, Deng X, Hu Z. A new normalized method on line-based homography estimation. Pattern Recognit Lett 2008;29(9):1236–44.
Contents lists available at ScienceDirect
Optics and Lasers in Engineering journal homepage: www.elsevier.com/locate/optlaseng
Line-based calibration of a micro-vision motion measurement system ⁎
Hai Li, Xianmin Zhang , Heng Wu, Jinqiang Gan
MARK
Guangdong Provincial Key Laboratory of Precision Equipment and Manufacturing Technology, School of Mechanical and Automotive Engineering, South China University of Technology, Guangzhou 510640, China
A R T I C L E I N F O
A B S T R A C T
Keywords: Micro-vision system Calibration Line-based method Resolution target Microscope Motion measurement
We propose a flexible and practical line-based method for calibrating a micro-vision motion measurement system by using a resolution target, considering the lens distortion. First, centerlines of the stripes in the image of resolution target are extracted with an improved center gravity approach. Then distortion coefficients are obtained through the nonlinear optimization, utilizing straightness of the lines. Finally, these lines are corrected and used for estimating parameters of the system. Two experiments are conducted to validate the effectiveness of the proposed method. The results demonstrate that our method is effective and practical for calibrating the micro-vision measurement system.
1. Introduction
MVS [13]. As can be seen, MVSs are widely used in the field of motion measurement with micro/nanoscale. However, an obvious drawback of MVS is its low absolute accuracy when used for high-precision measurement. This is due to the using of inaccurate geometric model of the system. In most cases, when a microscope is used for measurement, only the pixel representing distance (PRD) is roughly calibrated by using a stage micrometer or calculated from the theoretical model. In these situations, lens distortion is ignored, the intrinsic and extrinsic parameters of the system are either ignored or obtained according to the theoretical model. In order to improve the absolute measurement accuracy of the MVS and to obtain full intrinsic and extrinsic parameters, calibration procedures are indispensable. In this study, as the motions of the compliant mechanisms are planar movements, calibrating means finding the accurate mapping relationship between the motion plane and the image plane. In addition, both the PRD and magnification of the MVS need to be obtained. Currently, abundant algorithms for calibrating macro-vision measurement system have been proposed [14–16]. However, when it comes to microscale, two difficulties have made these commonly used methods out of action. The first is the lack of the calibration targets, which is caused by the small field of view (FOV). The other is that only a single image can be used during the calibration. The reasons are that the image plane and the object plane are almost parallel, and the depth of field of the microscope is narrow. In the domain of automatic micromanipulation, to realize the localization and the accurate measurement, several approaches have been proposed for the calibration of the microscopes. For instance, Zhou et al. proposed a special geometric
Compliant mechanisms are flexible mechanisms that transfer an input force or displacement to another point through elastic body deformation [1]. Compliance in design leads to joint-less, non-assembly, monolithic mechanical devices and thus is particularly suitable for applications with small range of motions [2], e.g., the bio-micromanipulations [3], the micro-electromechanical systems (MEMS) [4], and the precision positioning stages [5], etc. As more and more attentions have been paid to the design and apply of the compliant mechanisms [6–8], how to effectively measure motion of these mechanisms becomes more and more urgent. Due to the features of high resolution (from submicrometer to nanometer) and compact structure, detecting the motion of the compliant mechanisms is always a challenging work, especially when the mechanisms have multiple degrees of freedom (DOF). Owing to the advantages of non-contact, multi-DOF measurement ability, high resolution, etc., the microscopic vision system (MVS), which consists of optical microscope, camera and computer, provides an alternative for motion detection of compliant mechanisms. A number of micro-vision based approaches have been proposed for detecting micro/nanoscale motions in recent years [9–13]. Wu et al. realized the motion detection of an inverter and a three DOF positioning stage [9,10], utilizing the MVS. Assaf et al. presented an approach for measuring nanoscale displacements of the micro-devices by using an optical microscope [11]. Ouyang et al. proposed a visual-servo method that can improve the trajectory tracking precision of compliant micromanipulator [12]. Xia et al. achieved the three-dimensional surface displacement and shape measurements at nanoscale with the help of
⁎
Corresponding author. E-mail address: [email protected] (X. Zhang).
http://dx.doi.org/10.1016/j.optlaseng.2016.12.018 Received 27 August 2016; Received in revised form 25 December 2016; Accepted 26 December 2016 0143-8166/ © 2016 Elsevier Ltd. All rights reserved.
Optics and Lasers in Engineering 93 (2017) 40–46
H. Li et al.
model of the microscope and realized the calibration by using a 2D dot array-based calibration object [17]. Method proposed in this paper is commonly used for calibrating microscope. However, when the FOV of MVS keeps decreasing, the problem of short of calibration objects will become serious. In Ref. [18–20], researchers have presented several virtual-image-based methods, which mean that targets for calibration are obtained by virtue of precision mechanical setups. Such as the atomic force microscope (AFM) tips, the micromanipulators, etc. Nevertheless, these virtual-image-based approaches are much too complicated and lack of operational flexibility, especially when the magnification of the system needs to be changed frequently. To solve these problems, we propose a line-based calibration method for the MVS which uses the resolution target as the calibration object for the first time. As a generally used setup for the performance evaluation of microscope, resolution target normally has groups of linespace patterns with different spatial frequency, like the United States Air Force (USAF) 1951 (see in Section 2). Therefore, it is compatible for calibrating the MVSs with different magnifications and FOVs. There are four steps in this line-based method. Initially, centerlines of the stripes are extracted with the improved center of gravity method. Then distortion coefficients and the principle point are obtained through the nonlinear optimization of the projective invariant of the lines. After that, centerlines are corrected and fitted to estimate the homographic matrix. Finally, full intrinsic and extrinsic parameters are acquired from decomposing the homographic matrix. After implementing these steps, an accurate geometric model of the MVS can be obtained. The remainder of this paper is organized as follows. In Section 2, preliminary information about the MVS is given. Section 3 contains the detailed procedures of our line-based calibration method. In Section 4, a series of experiments are conducted to verify the effectiveness of the proposed method. Section 5 is the final conclusions.
Fig. 2. An example of the resolution target. (a) 1951 USAF resolution target, (b) an element of stripes in one group, (c) geometric characteristic of the stripes in one element.
In this study, a type of 1951 USAF resolution target is chosen as the calibration object, as shown in Fig. 2(a). This kind of target consists of groups of line-space patterns whose spatial frequency increases as the sixth root of two. A set of six elements (three horizontal lines and three vertical lines) is in one group (Fig. 2(b)), and which group will be used during the calibration is determined by the FOV of the MVS. The geometry characteristics of the stripes in one group is shown in Fig. 2(c). Since the largest spatial frequency of line-space patterns in this target can reach to 228 LP/mm (the corresponding value of X is 2.19 µm), this kind of resolution target can meet the needs for calibrating vision system with quite small FOV (high magnification). 2.2. Geometric model of the MVS The geometric model of the system is shown in Fig. 3. Suppose a 2D point on the image plane (IP) is denoted by p = (u , v )T , and the corresponding point on object plane (OP) is P = (x, y )T . Similarly, l and L represent the line pair on the IP and OP, respectively. We use x∼ to denote the normalized description of point vector or line vector. Since the IP and OP are parallel, the relationship between a 2D point M and m is expressed as
2. Preliminaries 2.1. System setup
∼ ∼ ∼ p = HP = AKP ,
The basic setup of the micro-vision measurement system is similar to that of our previous work [10], as shown in Fig. 1. The whole workstation includes an imaging system, a PC, and an adjustable platform (AP). The imaging system is consisted of a lighting source, an objective, a spectroscope, a tube lens, an adapter and a digital camera. The details of these setup are described in Section 4. The AP is used to ensure that the measured target is within the FOV of the camera. By modifying the tube-lens length or changing the objective lens, the FOV and magnification of the system can be easily adjusted according to the needs. Besides, by means of adopting proper algorithms, this system can realize online motion tracking of the compliant mechanisms.
Camera
⎡ f 0 u0 ⎤ ⎡ cos θ sin θ p ⎤ x ⎢ x ⎥ ⎢ ⎥ where A = ⎢ 0 fy v0 ⎥ , K = ⎢− sin θ cos θ py ⎥ . Here, A is the intrin⎢⎣ 0 0 1 ⎥⎦ ⎢⎣ 0 0 1 ⎥⎦ sic matrix, K is the extrinsic matrix which composes of a planar rotation angle θ with a translation vector (px , py ). (u 0 , v0 ) is the coordinate of principle point, fx and fy are the scale factors in image u-axis and v-axis, respectively. H is the homographic matrix between OP and IP. In this paper, since the shape of the pixel elements of the sensor is square and the physical dimensions of the pixels are equal, the skew factor is set as zero, and the scale factors fx and fy are set to be equal. According to the contravariant feature of point mapping [21], the mapping relationship of a line from OP to IP can be written as
Network cable
∼ ∼ l = H −T ⋅L ..
Adapter
(2)
Tube lens Spectroscope
Light source
(1)
PC
Objective
Motion control card Adjusting platform Fig. 1. The schematic diagram of the measuring system.
Fig. 3. The geometric model of the measuring system.
41
Optics and Lasers in Engineering 93 (2017) 40–46
H. Li et al.
2.3. Lens distortion model The model presented in Eq. (1) and Eq. (2) is an ideal pinhole model. However, there are always distortions that make the model nonlinear. For real lenses, the major type of distortion is usually radial distortion [14]. In order to make the distortion compensation more convenient, an inverse model is used to describe the radial distortion. Assuming resolution of the image is R × C . The distortion model used in this study is expressed as 2 4 ⎧ ⎪ u = u + (u − u )(k r d d 0 1 d + k 2rd ) ⎨ , 2 4 ⎪ ⎩ v = vd + (vd − v0 )(k1rd + k 2rd )
where rd2 =
2
(ud − u 0 )
(3)
2
(vd − v0 )
. Here, (u , v )T and (ud , vd )T represent the undistorted coordinate and the corresponding distorted coordinate, k1 and k 2 are the radial distortion coefficients. R2
+
C2
Fig. 4. Extraction of the centerlines. (a) Raw image; (b) Image after morphologic processes; (c) Extracted centerline; (d) Center of gravity method.
3. System calibration
the label value of t at row i can be expressed as
In this section, detailed descriptions about the line-based calibration of MVS are presented. The whole method consists of four procedures, including the detection of the centerlines, solving the lens distortion, estimation of the homographic matrix, and decomposition of the homographic matrix. The full intrinsic and extrinsic parameters can be obtained by performing these four procedures. According to Eq. (1) and Eq. (2), the parameters that need to be calibrated can be summarized as follows.
t
t
Yi =
x
y
y
0
0
1
t
M
∑ j =1 I (i , tT i(j ))
, (5)
where tM represents the number of column coordinates with the labeled value t . After performing this formula to each row of the image, the centerlines of the stripes are obtained as shown in Fig. 4(c).
• Extrinsic parameters: θ , p , p . • Intrinsic parameters: f , f , u , v , k , k , M . x
M
∑ j =1 I (i , tT i(j ))⋅ tT i(j )
3.2. Solving the radial distortion
2
According to the perspective of projective geometry, a line on the OP is still straight on the IP after the ideal pinhole projection. However, when lens distortion occurs, the projected line is not straight any more. Utilizing the property of straightness, as well as the explicit distortion model, the problem of solving lens distortion can be transformed to the question of nonlinear optimization [23–25]. After the camera's imaging, the line in space is discretized into a series of points. Suppose there are N stripes in an image, and the ideal equation (without considering the distortion) of the n−th centerline in the image frame is defined as
Here, M represents the magnification ratio of the MVS. Another parameter that is commonly used during the micro-vision measurement is PRD. The physical meaning of PRD is the space distance each pixel represents. It is the coefficient used to transform the image distance to the physical space distance during the measurement. The relationship between M, f and PRD is presented as follows:
PRD = 1/ f = PS / M ,
(4)
where PS is the physical size of a camera's pixel.
u cos α + v sin α + λ n = 0,
3.1. Extraction of the centerlines
(6)
where α and λ represent the tilt angle and intercept of the line, respectively. Through the introduction of the algorithm in Section 3.1, it can find that the maximum value of label and the number of centerlines are the same. Besides, the number of discrete points in each centerline is R , the distorted coordinate of points in the n−th centerline is (i , tY i ). For the convenience of expression, we assume the distorted coordinates and the corresponding undistorted coordinates of i−th point in the n−th centerline are (tuid , tvid )T and (tuiu , tviu )T , respectively. Then the nonlinear optimization question can be expressed as
It is well known that calibration benefits from sub-pixel accurate location of the features. In this work, we choose the stripe's centerline as the feature to perform the calibration. Utilizing an improved center of gravity method (ICGM), the locating accuracy of the discrete points on the centerline can reach to sub-pixels, as shown below. At the beginning of the calibration, an image of the resolution target is captured by the MVS. The basic requirement is that at least two parallel stripes are included in this image. Without loss of the generality, an image which includes three parallel stripes is used to illustrate the centerline extraction algorithm (see in Fig. 4(a)). After obtaining the image, the Ostu [22] optimal threshold segmentation algorithm is implemented, following with a batch of morphologic approaches to obtain the dilated regions of stripes (Fig. 4(b)). Finally, the ICGM, which is based on the one-dimensional run length coding, is used to obtain coordinates of the discrete points on each centerline. The details of the ICGM are given in the following. Suppose I is the source image, Id is the dilated image obtained after morphologic processing, and I (r , c ) is the gray value of a pixel having image coordinates (r , c ). First, the image Id is scanned row by row. The pixels that belong to the foreground (dilated stripes) is labeled, as shown in Fig. 4(d). Let tT i(j ) be the jth column coordinates with the labeled value t at row i . Then the coordinate of the stripe center with
Γ (k , u 0 , v0 , α , λ1, …, λ n ) N
R
=min ∑t =1 ∑i =1 (tuiu cos α + tviu sin α + λ n )2 ,
(7)
t u t d t d 2 4 ⎧ ⎪ u i = ui + ( ui − u 0 )(k1rd + k 2rd ) where ⎨ . Γ is the optimal objective t d t d 2 4 ⎪ t u ⎩ vi = vi + ( vi − v0 )(k1rd + k 2rd ) function. To solve this equation, the Levenberg-Marquardt nonlinear optimization method is adopted. Since the initial values usually have large effect on the results, the approximate initial values for solving Eq. (7) is given as following. The initial value of the distortion coefficients k1 and k2 are set as zero. The principle point is initialized to be the image center. The initial values of the centerlines’ parameters are obtained by fitting the obtained discrete points into lines.
42
Optics and Lasers in Engineering 93 (2017) 40–46
H. Li et al.
3.3. Estimation of the homographic matrix
3.5. Discussions of this calibration method
After obtaining the distortion coefficients and the principle point, the distorted centerlines can be immediately corrected. Then these corrected lines are used as the elements for estimating the homographic matrix [26]. Since the skew factor of the camera is zero and the planes of OP and IP are parallel, the projective model of the MVS can be simplified as the similarity transformation. According to the projective geometry, the similarity transformation is an isometry composed with an isotropic scaling. Therefore, the homographic matrix can be represented as
For the proposed calibration method, line features are used during the calibration process. Line features are more robust to the image noise than point-based methods, which is important since images from microscope with high magnification are always heavily polluted by various noises. In addition, utilizing the straightness of the line, the distortion coefficients and the principle point can be solved independently through the nonlinear optimization, which makes the solving of other parameters become possible. However, there are some limitations in this method that need to be improved in the future. The first is the centerline extraction algorithm. In the present method, the precision of the extracted center point is easily affected by the uneven illumination and the large inclination of the stripes. The second is the mapping model. In the current method, based on the facts that the depth of field of the objective is quite small and the installation accuracy is very high, we assume that the IP and OP are parallel. In order to further improve the measurement accuracy, nonparallel angular errors need to be considered. This can be achieved by using a projective transformation model replaces the currently used similar transformation model. To solve the projective transformation model, calibration image which includes more lines in the normal position will be needed.
⎡ sx sy tx ⎤ ⎢ ⎥ H = ⎢−sy sx ty ⎥ , ⎢⎣ 0 0 1 ⎥⎦
(8)
where sx , sy , tx , and ty are the parameters that need to be estimated. Let ∼ li = (lai, lbi, lci )T be the corrected centerline in image frame, and ∼ Li = (Lai, Lbi, Lci )T be the corresponding line in the objective frame. ∼ ∼ According to the Eq. (2) and Eq. (8), at least two pairs of l and L are needed to estimate the matrix H since there are four unknown parameters in H. Normally, more group of lines will make the solution more accurate and reliable. During the estimation of H, as the last element in H is set as one, the lines must be normalized to have the form of l = (la, lb, 1)T to hold Eq. (2). Nevertheless, when a line crosses over the origin of the frame, the last element will be zero. This makes it impossible to normalize the last ∼ ∼ elements in l and L . To figure out this problem, the normalization should be implemented on the non-zero elements of the lines. This results in an increase in the number of parameters in Eq. (2). In this study, the origin of the object frame is set to the first detected point on the first centerline, and the y-axis direction is set to coincide with the centerline. Therefore, these centerlines can be normalized as ∼ ∼ Li′ = (Lai′ , 1, Lci′ )T and li′ = (lai′ , 1, lci′ )T , with i = 1, 2, 3. The projective ∼ ∼‘ relationship between L and l ′ is then expressed as
⎡ sx′ −sy′ 0 ⎤ ⎢ ⎥∼ ∼′ T ∼′ Li = H l = ⎢ sy′ sx′ 0 ⎥⋅li′, ⎢ ⎥ ⎣ tx′ ty′ ρ ⎦
4. Experiments and results To demonstrate the practicability and reliability of the proposed method, two experiments are carried out. The first is to demonstrate the feasibility and effectiveness of the proposed method in calibrating the MVS. The other is to evaluate the performance improvement of the MVS in measuring motions of the compliant mechanisms. In the first experiment, the MVS is calibrated using the method proposed in this paper and the method proposed in Ref. [17], respectively. In the second experiment, the MVS and a LIM are used to synchronously measure motion of a single DOF compliant positioning stage, so that to evaluate performance of the MVS. 4.1. Calibration of the MVS
(9)
Experimental setup used in this experiment is shown in Fig. 5. The imaging system consists of an objective with a controllable tube (Navitar 12X UltraZoom-Motorized lens, magnification ranges from 0.714 to 3.330 when combines with micro objectives), a CCD camera (AVT Manta-G201, resolution is 1624×1234 pixels) with an adapter (Navitar 0.5× standard adapter). An USAF 1951 resolution target (Edmund, 3" x 3" Positive) is used, as shown in Fig. 5(c). A computer (Intel(R) Core(TM) i7-4570, CPU 2.40 GHz, RAM 4 GB) is dedicated to implement the proposed methods. The algorithms are developed by
where ρ is the additional parameter. Through the linear transformation of Eq. (9), a new equation can be obtained as
⎡ Lai′ ⎤ ⎡ lai′ −1 0 0 0 ⎤ ⎥ ⎢ ⎥ ⎢ 1 l′ 0 0 0 ⎥h = Bh, ai ⎢1⎥=⎢ ⎢⎣ Lci′ ⎥⎦ ⎢ 0 0 l ′ 1 l ′ ⎥ ci ⎦ ai ⎣
(10)
where h = (sx′, sy′, tx′, ty′, ρ) . Since there are five unknown parameters in ∼ ∼ H and each line pair (l ↔ L ) generates three constrains, at least two pair of lines are needed to solve out all the parameters. T
3.4. Decomposition of the homographic matrix According to Eq. (1) and Eq. (8), the relation between the homographic matrix H, intrinsic matrix A, and extrinsic matrix K is expressed as
⎡ f cos θ f sin θ f p + u 0 ⎤ x x x ⎢ x ⎥ H = AK = ⎢−fy sin θ fy cos θ fy py + v0 ⎥ . ⎢⎣ ⎥⎦ 0 0 1
(11)
Once the principle point (u 0 , v0 ) and the homographic matrix H are obtained, the remainder intrinsic and extrinsic parameters can be easily solved through decomposition of H.
Fig. 5. The experimental setup of the micro-vision system.
43
Optics and Lasers in Engineering 93 (2017) 40–46
H. Li et al.
Table 1 Calibration results using the proposed method. Calibration results
Unit
fx , fy
pixel /μm −2
k1
pixel
k2
pixel −4 1 pixel ° μm
M (u 0 , v0 ) θ
(px , py )
Table 3 Error from the lens distortion.
10X
50X
10X objective
3.574
17.653
L1
L2
L3
L1
L2
L3
0.0234
0.0230
1.35 0.09 2.67 0.73
0.23 0.17 1.53 0.28
2.16 0.11 4.06 0.88
1.54 0.11 3.74 0.93
0.52 0.20 1.64 0.38
2.33 0.24 3.95 0.96
−0.0193
0.0100
15.736 (614.5, 810.6) −0.169
77.673 (615.2, 812.8) 0.906
(38.56, −126.26)
(−44.14, −30.14)
Mean DD (MeDD) MeDD After calibration Max DD (MaDD) MaDD After calibration
50X objective
(Unit: pixel)
decreased after calibration. An un-normal result in this table is that MeDD of L2 is larger than L1 and L3. This is caused by the inaccurate extraction of L2. In the experiment, a LED-point lighting source is used, which has caused the non-uniform illumination in the FOV of the camera. As a result, center part of the image is much brighter than the peripheral part. Since we cannot make sure that influence from nonuniform illumination are symmetric about the second centerline, the extracted coordinates of centerline will be affected, especially the second centerline. This is a disadvantage that we need to improve in the future work.
Matlab. Tube-length is set to the maximum so that it has the theoretical magnification of 3.33. In this experiment, a 10× objective (Mitutoyo, 0.28 NA) and a 50× objective (Mitutoyo, 0.55 NA) are mounted and calibrated, respectively. By multiplying the magnifications of the objective, adapter and tube lens, we can obtain the theoretical magnification of the system. When the 10× and 50× objectives are mounted, the calculated magnifications are 16.65 and 83.25, respectively. The corresponding FOVs are 429 µm ×326 µm and 86 µm ×65 µm. In the experiment, three stripes are included in each calibration image. The distribution of these stripes in the image is similar to the one present in Section 3.1. Table 1 shows the calibration results. From this table, we can find that all the parameters in the projective model are obtained, including the magnification of the system. To show the effectiveness of the proposed method, the method proposed in Ref. [17] is implemented for comparison. Since a dot array is required in this method, a test target R1L1S1P from THORLABS that includes grids with different scales is selected and used in this test. To increase the comparability, we have normalized the coordinates of the distortion model used in Ref. [17], so that the distortion coefficients in these two methods are in unison. Experimental results about the related intrinsic parameters are shown in Table 2. The principle point is not included in this table, because it is not calibrated and is set as (617, 814) in Ref. [17]. Comparing the intrinsic parameters shown in table1 and Table 2, it can be seen that results of magnification M and scale factor f are quite close. In our method, two order distortion model is used so that there are two distortion coefficients. According to the results of k2 shown in Table 1 and the model presented in Eq. (3), it can be seen that distortion from the second order coefficient is quite small. Comparing the first order distortion coefficients in Table 1 and Table 2, we can see that distortion results from these two different methods are also coincident. Hence, it can be seen that the method proposed in this paper is quite effective. To show the influence from the lens distortion, centerlines are fitted by the corrected points and then used as the ideal equations of the lines; afterwards, distance from the corrected point to the ideal lines and distance from extracted points to the ideal lines are both calculated. The distance is named as the distortion deviation (DD). The statistical results are shown in Table 3. From the table, we can find that DD of the points from the first line and third line are larger, this is reasonable since these two lines are further from the principle point than the second line. According to the lens distortion model, the further from the principle point, the bigger the DD. From the third and fourth row of the table, it can be seen that the maximum distortion has apparently
4.2. Performance evaluation In this experiment, the professional measuring equipment of LIM is used. The whole experiment setup is shown in Fig. 6. The LIM is Renishaw XL-80 that has a linear measurement accuracy of ± 0.5 ppm. Because the LIM cannot measure the X and Y direction displacements at the same time, a 1-DOF compliant positioning stage is chosen as the measured object, as shown in Fig. 6. The designed ratio of input displacement and output displacement of this stage is one. The detail experimental scheme is given as below. First, the reflector of the LIM is rigidly connected to the output part of the compliant stage. Then a template that extracts from the surface of the reflector is tracked by the MVS with the method presented in Ref. [9]. Subsequent, a set of input is given in order, as shown in Table 4. After each output of the compliant stage is stable, its displacement is measured by the LIM and the MVS, separately. During each measurement of the compliant stage, 30 times of measurement from the MVS are recorded, the average value is used as the measuring value of the MVS. The measuring results by the LIM system and the MVS are presented in Fig. 7, the detailed data is shown Table 4. Three approximate straight lines are included in this figure. Except the data from LIM and MVS, SMVS data represent the source data from MVS that have not been calibrated and corrected. As can be seen, measurement results from LIM and MVS are almost coincide with each other. However, the SMVS data is obviously deviating from the other two line. Besides, the bigger the input value, the larger the deviation. The reason is that without calibration, the PRD is obtained by the theoretical calculation, as
Table 2 Calibration results using method proposed in [17]. Calibration results
Unit
fx , fy
pixel /μm
k1 M
pixel 1
−2
10X
50X
3.520
17.629
0.0213
0.0189
15.488
77.566 Fig. 6. The experimental setup of the LIM system.
44
Optics and Lasers in Engineering 93 (2017) 40–46
H. Li et al.
Table 4 Source data from the measurement.
Table 5 Source data of the AE and STD.
Number
Input
LIM
MVS
SMVS
1 2 3 4 5 6 7 8 9 10 11 12 13
0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 20.0 22.5 25.0 27.5 30.0
0 2.5562 5.0698 7.5731 10.0675 12.5776 15.0869 17.6219 20.1399 22.6933 25.2870 27.8411 30.4153
0 2.461 5.129 7.586 10.136 12.405 14.959 17.517 20.121 22.677 25.198 27.884 30.461
0 2.376 4.950 7.320 9.777 11.963 14.421 16.882 19.384 21.836 24.254 26.825 29.288
Number
1 2 3 4 5 6 7 8 9 10 11 12 13
(Unit: μm).
MVS
SMVS
AE
STD
AE
STD
0 0.095 0.059 0.013 0.068 0.172 0.124 0.105 0.019 0.018 0.089 0.043 0.045
0.020 0.037 0.018 0.019 0.021 0.022 0.024 0.021 0.027 0.031 0.017 0.021 0.039
0 0.181 0.120 0.253 0.291 0.614 0.666 0.740 0.756 0.857 1.033 1.019 1.128
0.019 0.035 0.017 0.018 0.020 0.020 0.023 0.020 0.026 0.029 0.016 0.020 0.037
(Unit: μm).
results of image processing. During the distortion compensation, the dispersion characteristics of the results in image frame will not change, so the STD results are not influenced in this step. However, during the processing of transforming result from image frame to space frame, the dispersion characteristics of the result is influenced by the PRD, and this kind of influence is linear. As the PRD used in MVS is bigger than that used in SMVS, so the trends of STD results are accordance, and the value from MVS is always larger than that of SMVS, as shown in Fig. 8(b). However, the STD deviations between MVS and SMVS are very small. Comparing Fig. 8(a) and Fig. 8(b), it can be seen that the MVS has high measurement repeatability (small STD) and high measurement accuracy after calibration. From the above two experiments, it can be found that the presented approach can successfully realize the calibration of the MVS with different magnifications and FOV. Compared with the method proposed in Ref. [17], the geometric model present in this paper is simpler, but the calibration results are coincident. It also shows that lens distortion will generate deviations to the measurement results, especially when the measured target is far away from the principle point. Besides, absolute accuracy of this measurement system has been improved from 1.2 to 0.18 µm after calibration. Moreover, accuracy of the MVS still has the potential to be further improved since the STD of the system is very small.
Fig. 7. Experimental result comparisons between the proposed method and the LIM.
shown in Eq. (4). Nevertheless, the magnification used in the calculation is always not accurate. Moreover, influence from lens distortion will generate deviations to the measurement results, and this kind of deviations is varying along with the movement of the target. To further show the improvement of the MVS after calibration, we use the data from LIM as the reference value to calculate the absolute error (AE), the results is shown in Fig. 8(a). The standard deviation (STD) of MVS and SMVS are also given in Fig. 8(b). Table 5 is the detailed data about AE and STD. From Fig. 8(a), it can be seen that after calibration, the AE has decreased from 1.2 to 0.18 µm. Hence, absolute accuracy has apparently improved after calibration. In Fig. 8(b), it shows that STD values from MVS are always bigger than values of SMVS and the ratio between these corresponding data is constant. This is reasonable because both the MVS data and SMVS data are coming from
5. Conclusions This paper presents a flexible and practical line-based method to calibrate a MVS. The resolution target USAF 1951 is firstly proposed
Fig. 8. AE and STD comparisons of MVS before and after calibration.
45
Optics and Lasers in Engineering 93 (2017) 40–46
H. Li et al.
and used as the calibration target in this method. Utilizing the property that there are groups of line-space patterns with different spatial frequency in the target, the proposed method can well meet the need for calibrating MVS with changeable tube length. Hence, the flexibility of the proposed method is high. In addition, the cost of the resolution target is low, so the practicality of this method is high, too. Experimental results demonstrate that the proposed method can successfully realize calibration of the MVS. Except for the magnification and PRD, full intrinsic and extrinsic parameters can be obtained, including the lens distortion. Compared with LIM, it shows that the absolute measuring accuracy of MVS has been apparently improved after calibration. Furthermore, the proposed method possesses the capability for calibrating multi-camera system when more lines with normal positions are included in the calibration image. Improvements that need to be done in the future studies are: 1) improve the robustness and accuracy of the centerline extraction algorithm; 2) more factors need to be considered to further improve the absolute measuring accuracy of the system.
[7] [8] [9] [10]
[11]
[12]
[13] [14] [15]
[16]
Acknowledgments
[17]
This work was supported by the National Natural Science Foundation of China (U1501247); the Scientific and Technological Research Project of Guangdong Province (201604010100, 2015B020239001).
[18]
[19] [20]
References
[21] [22]
[1] Howell LL. Compliant mechanisms. John Wiley & Sons; 2001. [2] Kota S. , et al. Design of compliant mechanisms: applications to MEMS. Analog Integr Circuits Signal Process 2001;29(1–2):7–15. [3] Fukuda T, Arai F, Nakajima M. Micro-nanorobotic manipulation systems and their applications. Springer Science & Business Media; 2013. [4] Kim YS, Yoo JM, Yang SH, Choi YM, Dagalakis NG, Gupta SK. Design, fabrication and testing of a serial kinematic MEMS XY stage for multifinger manipulation. J Micromech Microeng 2012;22(8):085029. [5] Wang H, Zhang X. Input coupling analysis and optimal design of a 3-DOF compliant micro-positioning stage. Mech Mach Theory 2008;43(4):400–10. [6] Zhu B, Zhang X, Fatikow S. Structural topology and shape optimization using a level
[23] [24] [25] [26]
46
set method with distance-suppression scheme. Comput Methods Appl Mech Eng 2015;283:1214–39. Liu P, Yan P. A new model analysis approach for bridge-type amplifiers supporting nano-stage design. Mech Mach Theory 2016;99:176–88. Jin M, Zhang X. A new topology optimization method for planar compliant parallel mechanisms. Mech Mach Theory 2016;95:42–58. Wu H, Zhang X, Gan J, Li H, Ge P. Displacement measurement system for inverters using computer micro-vision. Opt Lasers Eng 2016;81:113–8. Wu H, Zhang X, Gan J, Li H, He Z. High-precision displacement measurement method for three degrees of freedom-compliant mechanisms based on computer micro-vision. Appl Opt 2016;55(10):2594–600. Ya Akobovitz A, Krylov S, Hanein Y. Nanoscale displacement measurement of electrostatically actuated micro-devices using optical microscopy and digital image correlation. Sens Actuators A Phys 2010;162(1):1–7. Ouyang PR, Zhang WJ, Gupta MM, Zhao W. Overview of the development of a visual based automated bio-micromanipulation system. Mechatronics 2007;17(10):578–88. Xia S, Pan Z, Zhang J. Optical microscope for three-dimensional surface displacement and shape measurements at the microscale. Opt Lett 2014;39(14):4267–70. Zhang Z. A flexible new technique for camera calibration. Pattern Anal Mach Intell IEEE Trans on 2000;22(11):1330–4. Heikkila J, Silvén O. A four-step camera calibration procedure with implicit image correction. in Computer Vision and Pattern Recognition, 1997. Proceedings., 1997 IEEE Computer Society Conference on: IEEE; 1997. Hong Y, Ren G, Liu E. Non-iterative method for camera calibration. Opt Express 2015;23(18):23992–4003. Zhou Y, Nelson BJ. Calibration of a parametric model of an optical microscope. Opt Eng 1999;38(12):1989–95. Ammi M, Frémont V, Ferreira A. Automatic camera-based microscope calibration for a telemicromanipulation system using a virtual pattern. Robot IEEE Trans on 2009;25(1):184–91. Kawaji A, Arai F, Fukuda T. 3D calibration for micro-manipulation with precise position measurement. J Micromech 2001;1(2):117–30. Ren L, Wang L, Mills JK, Sun D. Vision-based 2-D automatic micrograsping using coarse-to-fine grasping strategy. IEEE Trans Ind Electron 2008;55(9):3324–31. Hartley R, Zisserman A. Multiple view geometry in computer vision. Cambridge university press; 2003. Abutaleb AS. Automatic thresholding of gray-level pictures using two-dimensional entropy. Comput Vis Graph Image Process 1989;47(1):22–32. Prescott B, McLean GF. Line-based correction of radial lens distortion. Graph Models Image Process 1997;59(1):39–47. Zhou F, Cui Y, Gao H, Wang Y. Line-based camera calibration with lens distortion correction from a single image. Opt Lasers Eng 2013;51(12):1332–43. Jiang S, Cao D, Wu Y, Zhu S, Hu P. Efficient line-based lens distortion correction for complete distortion with vanishing point constraint. Appl Opt 2015;54(14):4432–8. Zeng H, Deng X, Hu Z. A new normalized method on line-based homography estimation. Pattern Recognit Lett 2008;29(9):1236–44.