- Email: [email protected]
Calibration method for binocular vision with large FOV based on normalized 1D homography
Optik - International Journal for Light and Electron Optics 202 (2020) 163556
Contents lists available at ScienceDirect
Optik journal homepage: www.elsevier.com/locate/ijleo
Original research article
Calibration method for binocular vision with large FOV based on normalized 1D homography ⁎
Tao Jiang , Xiaosheng Cheng, Haihua Cui
T
⁎⁎
College of Mechanical and Electrical Engineering, Nanjing University of Aeronautics and Astronautics, Nanjing 210016, China
A R T IC LE I N F O
ABS TRA CT
Keywords: Camera calibration 1D target Normalized 1D target Large FOV Measurement
Calibration method for binocular vision with large field of view (FOV) is the key technology in visual measurement system. The method based on 1D targets has been widely studied due to its high flexibility and convenience. In this paper, a calibration method based on normalized 1D homography matrix is proposed for robust camera internal parameters calculation with 1D target. Furthermore, based on combined one-dimensional targets, the calibration method for external parameter of binocular vision system is proposed. The numerical simulation is carried out to verify our method and to analyze the noise sensitivity, and the real data experiments are also developed. The results show that the internal parameter based on the normalized homography matrix is better than 0.1mm when the noise level is 2, and the precision of the measurement system is about 0.1mm in the depth direction of 3m. The method based on 1D target in this paper is effective and practical in the calibration of large FOV measurement system.
1. Introduction Calibration of the vision measurement system is a key technology to ensure measurement accuracy, affecting the accuracy of 3D reconstruction. The vision system must be accurately calibrated for high-precision measurements, visual inspection, three-dimensional positioning, etc. The research of vision system calibration technology develops continuously, and a series of calibration methods are derived for different measurement fields and different calibration targets. Among calibration method based on precision targets, the 2D planar target calibration method has the advantages of high precision and high flexibility, and is widely used in research and application [1]. Zhang [2] also proposed a camera calibration method based on 1D target. The collinear feature points of at least three known positions are moved around a fixed point to establish an equation about the internal and external parameters of the camera. The advantages of 1D target calibration are convenience, flexibility, and ease of implementation, and are suitable for visual system composed of multiple cameras [3]. Followed by Zhang’s method, camera calibration with 1D precision object is further researched. Wu improves the case that the collinear rotation constraint in Zhang's method may cause solution failure. Wu et al. [4] then proposed a method for constructing an equivalent two-dimensional target by using a 1D target in plane motion to achieve camera calibration. In addition to the validity of calibration by moving the 1D target along the plane, Qi et al. [5] verified that the camera calibration can also be achieved by moving the 1D target along the parabola. All the above methods are based on the constraints of one-dimensional target motion for camera calibration. According to Zhang's method, the calibration of single camera cannot be completed without restricting the movement of
⁎
Corresponding author. Co-corresponding author. E-mail addresses: [email protected] (T. Jiang), [email protected] (X. Cheng), [email protected] (H. Cui).
⁎⁎
https://doi.org/10.1016/j.ijleo.2019.163556 Received 10 July 2019; Accepted 7 October 2019 0030-4026/ © 2019 Elsevier GmbH. All rights reserved.
Optik - International Journal for Light and Electron Optics 202 (2020) 163556
T. Jiang, et al.
1D target. Frana et al. [6,7]proposed a multi-camera external parameter calibration method based on 1D target. At the same time, the linear algorithm based on normalized image points significantly improved the calibration accuracy. The normalized image points can be robust in the solution process and are widely used in the method of solving image homogeneous coordinates, such as the normalized 8-point algorithm. Compared to the normalized 1D target calibration method, Shi et al. [8] proposed a weighted similarityinvariant linear algorithm with 1D targets, which achieves a more accurate calibration result. In addition to calibrating the internal parameters of the camera, for large FOV binocular vision system, one-dimensional targets can also be used to calibrate binocular structure parameters [9–12]. The structural parameter refers to external parameters of binocular vision. In the stereo vision theory, the fundamental matrix contains all the parameters of the binocular, from which the internal and external parameters can be decomposed. The 1D target can be used to solve the binocular fundamental matrix, obtain the normalized binocular external parameters, and then use the scale length of the 1D target to solve the accurate binocular external parameters. This method is especially suitable for large FOV vision system calibration. This paper mainly studies the calibration method of large FOV binocular vision system based on 1D target, and proposes a calibration method with normalized 1D homography matrix. The internal parameters of the camera are solved by the homography transformation between the one-dimensional target's own coordinate system and the image coordinate system, and then the external parameters of the dual purpose are calibrated and optimized with photogrammetry method. The article elaborates on the calibration principle, which is an improvement of Refs. [2,3], and has been experimentally verified. The Section 2 introduces the calibration principle, including the normalized homography matrix and camera parameter calculation method; the Section 3 is the simulation and experimental verification. The conclusion follows in Section 4. 2. Calibration principle 2.1. The normalized 1D homography One-dimensional target rotates around a fixed point at one end, and the coordinates of the fixed point remain unchanged relative to the camera. Taking the fixed point as the original, the direction of the remaining points of the one-dimensional target is the x direction, and a one-dimensional coordinate system can be established. As shown in Fig. 1, the one-dimensional coordinate system Ow-Xw is established for the one-dimensional target itself. In Fig. 1, m (u, v) and M(x, y, z) denote the points of the image coordinate system and the world coordinate system, respectively. It can be seen that y = z = 0 in the world coordinate system Ow-XwYwZw. Therefore, according to the principle of camera imaging,
x ⎡ ⎤ u 0 x ⎡ ⎤ s ⎢ v ⎥ = K [ r1 r2 r3 t ] ⎢ ⎥ = K [ r1 t ] ⎡ ⎤ ⎣1⎦ ⎢0⎥ ⎣1⎦ ⎣1⎦ T,
˜ = [u v 1] Let m
˜ = [ x 1] M
T,
(1)
we can get
˜ ˜ = HM sm
(2)
Where s1 H = K [ r1 t ]. This can be seen as a one-dimensional homography between the one-dimensional coordinate system and the u0 ⎤ x r ⎡ h1 h4 ⎤ ⎡ fu ⎡ c⎤ ⎡ 1⎤ image plane coordinate system. H is a 3 × 2 matrix. Let H = ⎢ h2 h5 ⎥, r1 = ⎢ r2 ⎥, t = ⎢ yc ⎥ and K = ⎢ fv v0 ⎥ 。The homography ⎢h 1 ⎥ ⎥ ⎢ ⎣ r3 ⎦ ⎣ zc⎦ 1⎦ ⎣ 3 ⎦ ⎣ matrix has 5 unknowns, and a pair of image points and spaces can provide 2 equations, so at least one dimension target requires at least 3 points. The homography matrix can be solved by the idea of DLT method. In order to improve the robustness of the algorithm, the image points and world points involved in the calculation are normalized. Tm and TM denote the normalized matrix of the image points and the word points. Then Eq. (2) can be expressed as
Fig. 1. Schematic diagram of one-dimensional target calibration. 2
Optik - International Journal for Light and Electron Optics 202 (2020) 163556
T. Jiang, et al.
ˆ TMM ˜ˆ ˆ mm ˜ˆ = H sT
(3)
ˆ is the normalized 1D homography matrix. Hence, Where H ˆ H=T−1 m HTM
(4)
The following is the solution process of the homography matrix. A homography matrix is solved for each data. Using the idea of DLT, the following equation can be obtained.
u ⎡ h1 h4 ⎤ v ⎤ = ⎢ h2 h5 ⎥ = ⎡ x ⎤ s1 ⎡ ⎢ ⎥ ⎢ ⎥ ⎣1⎦ ⎣1⎦ ⎣ h3 1 ⎦
(5)
That is,
⎧ s1 u = h1 x + h4 s1 v = h2 x + h5 ⎨ ⎩ s1 = h3 x + 1
(6)
Simplified
⎧u = ⎪ ⎨ ⎪v = ⎩
h1 x + h4 h3 x + 1 h2 x + h5 h3 x + 1
(7)
Thus the equation can be obtained.
⎡ h1 ⎤ ⎢ h2 ⎥ ⎢h ⎥ − − x 0 ux 1 0 u ⎡ ⎤ ⎢ 3 ⎥ = ⎡0⎤ − − 0 x vx 0 1 v ⎣ ⎦ ⎢ h4 ⎥ ⎣ 0 ⎦ ⎢ h5 ⎥ ⎢ ⎥ ⎣1⎦
(8)
At least 3 points make the above equations solutionable. In the calculation, firstly all the image points are normalized, the homography matrix is calculated by the above formula, and then the inverse homogenization is performed by Eq. (4) to obtain the true homography matrix. The optimization process is added during the homography matrix solution process. The objective function is as following,
˜ ||2 ˜ − HM H*=arg min || m
(9)
2.2. Parameter calculation based on normalized homography matrix According to the camera imaging model, s1 H = K [ r1 t ]. So,
u 0 ⎤ r1 x c ⎡ h1 h4 ⎤ ⎡ fu ⎡ ⎤ ⎡× × ⎤ s1 ⎢ h2 h5 ⎥ = ⎢ fv v0 ⎥ ⎢ r2 yc ⎥ = ⎢ × × ⎥ ⎥ ⎢h 1 ⎥ ⎢ × zc⎦ 1 ⎦ ⎣ r3 z c ⎦ ⎣ ⎦ ⎣ ⎣ 3
(10)
T
T,
From Eq. (10), s1 = z c . If h1 = [ h1 h2 h3 ] and h2 = [ h4 h5 1]
z c [ h1 h2 ] = K [ r1 t ]
(11)
According to the properties of rotation matrix, we can obtaine.
z c2 hT1 K−T K−1h1 = 1
(12)
Where z c is the fixed point in the 1D target. According to the above formula, a homography matrix can be obtained from an image, thus an equation about z c and K can be obtained. Since there are five unknowns in the Eq. (12), it needs at least five positions. According to Eq. (12) ⎡ B11 B12 B13 ⎤ Noting B = z c2 K−T K−1 = ⎢ B12 B22 B23 ⎥ and b = [ B11 B12 B22 B13 B23 B33 ]T , The above equation can be transformed into ⎢ B13 B23 B33 ⎥ hT1 B h1 = [ h12 2h1 h2 h⎣22 2h1 h3 2h2 h⎦3 h32 ] b = vT b (13) If I = [1 ... 1]T , we can obtain linear equations (14)
Vb = I 3
Optik - International Journal for Light and Electron Optics 202 (2020) 163556
T. Jiang, et al.
Hence, b = (V T V)−1V T I. K and z c can be decomposed from B. 2.3. Binocular external parameter calculation based on small 1D target There is an essential matrix E between the camera coordinate systems of the binocular camera, which describes the transformation relationship of the camera coordinate system, including R and t, and the essential matrix can be expressed as E = Rt× . The image points of the left and right cameras are q˜li and q˜ri , according to the imaging model:
⎧ λ1 q˜1i ′ = [I|0] q˜i = P1 q˜i ˜ ˜ ˜ ⎨ ⎩ λ2 q2i ′ = [R|t ] qi = P2 qi
(15)
q˜2i ′T Eq˜1i ′T
According to the propriety of the fundamental matrix, = 0 . R and t can be initially solved by matching points without scale. The ratio of the scale length to the actual length is k, and the actual t matrix can be obtained. Then we need to optimize the length to get the internal and external parameters. Minimizeing the following: n
min f (R, t) =
∑ ||d − di ||
(16)
i=1
The optimized internal and external parameters are the final parameters for the dual target. In this method, the length of the scale affects the calibration result, the longer scale requires higher machining accuracy, and the long scale calibration workload is large. Compared with the long ruler, this paper adopts a short ruler. The short ruler has high processing precision and can place multiple one-dimensional short rulers in the same field of view. The calibration work is flexible and can reduce the workload. At the same time, a large 2D calibration plate is used for accuracy verification. 3. Experiment and result analysis 3.1. Numerical simulation This is a simulation verification of the camera calibration method based on normalized one-dimensional homography. According to the model in Section 2, the homography matrix of the one-dimensional target and the image point should be calculated first, and then the internal reference K is calculated. The solution process can be represented as in Fig. 2. To verify the accuracy of the algorithm, simulation verification is performed. ⎡1500 0 1000 ⎤ Supposing the camera's internal reference matrix is K = ⎢ 0 1500 1000 ⎥. One-dimensional target has a total of 10 points, and 0 1 ⎦ ⎣ 0 the coordinates are, 100, 200, 300, 400, 500, 600, 700, 800, 900 mm, respectively. The one-dimensional target is rotated 20 times around a fixed point P0 = (0, 0, 2) m. The coordinates of the space points obtained according to the spherical coordinate system. For example, the point on the one-dimensional target is M1= (0.1,1), the coordinates in the camera system ⎧ X1 = 0.1 sin θ cos φ + 0 is Y1 = 0.1 sin θ sin φ + 0 .Where the θ and φ are randomly generated. Let r = [sin θ cos φ sin θ sin φ cos φ ]. The image points can ⎨ ⎩ Z1 = 0.1 cos φ + 2 then be obtained by re-projection. That is m=K(r*M+P0). The three-dimensional point map of the one-dimensional target and the coordinates of the re-projection to the image plane are shown in Fig. 3. ⎡1500 0 1000 ⎤ According to the proposed method, the internal parameter matrix of the camera is solved as K′ = ⎢ 0 1500 1000 ⎥. It can be 0 1 ⎦ ⎣ 0 seen that the result of the solution is completely consistent with the theoretical value. Gaussian noise is added to the image, and the result is optimized. The results of the experimental comparison are shown in Fig. 4. ⎡1494.1 − 0.0529 903.5⎤ The calibration result with the optimization process is K′1 = ⎢ 0 1397.1 940.5⎥, and the result of the solution without the 0 1 ⎦ ⎣
Fig. 2. Solution process. (a) 3D points of world coordinates (b) Image reprojection points. 4
Optik - International Journal for Light and Electron Optics 202 (2020) 163556
T. Jiang, et al.
Fig. 3. Simulation data generation. (a) Noise level and optimized results; (b) Noise levels and unoptimized results.
Fig. 4. Effect of noise on calibration results. (a) Source image (b) center detection. (c) Hough line transformation (d) Collinear feature extraction.
⎡1364.9 − 0.0761 952.3 ⎤ optimized internal parameter matrix is K′2 = ⎢ 0 1404.5 927.2 ⎥. It can be seen that the optimized internal reference matrix is 0 1 ⎦ ⎣ closer to the true value. It can be seen from Fig. 4 that the method proposed in this paper has a relative error of less than 0.1 mm in the case of a noise level of 2, which fully satisfies the visual measurement requirements. 3.2. Real data experiment Experiments were carried out on the calibration of the camera using the normalized homography matrix method. In order to ensure the constraint of the common point rotation in practice, multiple one-dimensional targets are drawn in a single image. In the experiment, the feature extraction and collinear point extraction process is shown in the Fig. 5 below. The one-dimensional target homography matrix of each collinear point is calculated separately, and then the above method is used for camera calibration. In addition, the binocular camera is calibrated using a small combined one-dimensional target. As shown in Fig. 3(a), the largefield binocular vision system built for this project. The system hardware includes: 2 IMAGEING cameras DMK 33GP031, 2 lenses with focal length of 8 mm, 2 sets of data control lines and power lines. The working distance of the vision system is about 3 m, the field of view is about 2 m × 1.8 m, and the depth of field is about 1 m. Fig. 6(b) is a schematic diagram of a small one-dimensional target with a center-to-center design dimension of d = 90 mm. A total 5
Optik - International Journal for Light and Electron Optics 202 (2020) 163556
T. Jiang, et al.
Fig. 5. Collinear feature extraction process.
Fig. 6. Structure parameters calibration for binocular vision. (a) Binocular vision setup; (b) Small 1D target.
of 80 coding circles make up 40 small one-dimensional targets with unique identifiability. Fig. 7 is a graph showing the results of corresponding point matching based on coding using 40 sets of small one-dimensional targets. Fig. 7 shows the results of binocular detection and matching. When the working distance is long, the binocular imaging is relatively fuzzy, and the coding points cannot be completely recognized, which affects the subsequent calibration procedure. When the working distance is 2 m, the binocular imaging is clearer, all the coding points are detected, and the double target is higher. The goal of optimization is the distance of the code point, and the distance of the code point is also used as the verification index of the double target accuracy. For the above two calibration cases, two groups were taken for calibration, and the third group was used for accuracy verification. The accuracy verification results obtained are shown in Fig. 8 below. 6
Optik - International Journal for Light and Electron Optics 202 (2020) 163556
T. Jiang, et al.
Fig. 7. Binocular code point matching with working distance of (a) 3 m and (b) 2 m.
Fig. 8. Calibration accuracy verification in different depth directions.
The distance in Fig. 8 is calculated from the two coding centers on the one-dimensional target. For different image quality at different depths, the number of targets that can be successfully identified is different, resulting in different target numbers. As can be seen in Fig. 8, the root mean square error of the evaluation distance error of the 1D target at a distance of 2 m in the depth direction is 0.0625 mm, and the root mean square error at 3 m is 0.1127 mm. It can be seen that the overall deviation is small and the accuracy is slightly reduced in the depth direction. These experiments verify the effectiveness of the one-dimensional target calibration method in this paper. 4. Conclusion The camera calibration technology based on one-dimensional target has a high flexibility and convenience, and is suitable for calibration of large-field binocular vision system. In this paper, a large field of view visual calibration method based on 1D target is proposed. A normalized one-dimensional homography matrix method and a binocular visual calibration method based on small 1D target are proposed. It verifies the calibration accuracy and noise impact through simulation and experiment, and analyzes it through 7
Optik - International Journal for Light and Electron Optics 202 (2020) 163556
T. Jiang, et al.
experiments. The proposed method is an improvement of the traditional one-dimensional target calibration, which can be applied to the binocular vision system calibration of a larger field of view. The calibration accuracy is about 0.1 mm in the depth direction of 2 m and 3 m, and the calibration accuracy meets the measurement requirements. Acknowledgement Our work is supported by Jiangsu Provincial Department of Education. References [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12]
Z. Zhang, A flexible new technique for camera calibration, IEEE Trans. Pattern Anal. Mach. Intell. 22 (11) (2000) 1330–1334. Z. Zhang, Camera calibration with one-dimensional objects, IEEE Trans. Pattern Anal. Mach. Intell. 26 (7) (2004) 0–899. L. Yaowen, L. Wei, X. Xiping, Methods based on 1D homography for camera calibration with 1D objects, Appl. Opt. 57 (9) (2018) 2155–2163. F. Wu, Z. Hu, H. Zhu, Camera calibration with moving one-dimensional objects, Pattern Recognit. 38 (5) (2005) 755–765. F. Qi, Q. Li, Y. Luo, et al., Camera calibration with one-dimensional objects moving under gravity, Pattern Recognit. 40 (1) (2007) 343–345. José Alexandre de Franca, M.R. Stemmer, A.M.B.D.M. Franc, et al., A new robust algorithmic for multi-camera calibration with a 1D object under general motions without prior knowledge of any camera intrinsic parameter, Pattern Recognit. 45 (10) (2012) 3636–3647. JoséA. de Franca, M.R. Stemmer, A.M.B.D.M. Franc, et al., Revisiting Zhang’s 1D calibration algorithm, Pattern Recognit. 43 (3) (2010) 1180–1187. K. Shi, Q. Dong, F. Wu, Weighted similarity-invariant linear algorithm for camera calibration with rotating 1-D objects, IEEE Trans. Image Process. 21 (8) (2012) 3806–3812. Z. Liu, X. Wei, G. Zhang, External parameter calibration of widely distributed vision sensors with non-overlapping fields of view, Opt. Lasers Eng. 51 (6) (2013) 643–650. Z. Xiao, L. Jin, D. Yu, et al., A cross-target-based accurate calibration method of binocular stereo systems with large-scale field-of-view, Measurement 43 (6) (2010) 747–754. J. Zou, J. Zhu, W. Wang, et al., Study on Camera Calibration Approach Using 1D Target in Wide Field of View// International Conference on Electric Information & Control Engineering, (2012). Y. Zhao, W. Li, Self-calibration of a binocular vision system based on a one-dimensional target, J. Mod. Opt. 61 (18) (2014) 1529–1537.
8
Contents lists available at ScienceDirect
Optik journal homepage: www.elsevier.com/locate/ijleo
Original research article
Calibration method for binocular vision with large FOV based on normalized 1D homography ⁎
Tao Jiang , Xiaosheng Cheng, Haihua Cui
T
⁎⁎
College of Mechanical and Electrical Engineering, Nanjing University of Aeronautics and Astronautics, Nanjing 210016, China
A R T IC LE I N F O
ABS TRA CT
Keywords: Camera calibration 1D target Normalized 1D target Large FOV Measurement
Calibration method for binocular vision with large field of view (FOV) is the key technology in visual measurement system. The method based on 1D targets has been widely studied due to its high flexibility and convenience. In this paper, a calibration method based on normalized 1D homography matrix is proposed for robust camera internal parameters calculation with 1D target. Furthermore, based on combined one-dimensional targets, the calibration method for external parameter of binocular vision system is proposed. The numerical simulation is carried out to verify our method and to analyze the noise sensitivity, and the real data experiments are also developed. The results show that the internal parameter based on the normalized homography matrix is better than 0.1mm when the noise level is 2, and the precision of the measurement system is about 0.1mm in the depth direction of 3m. The method based on 1D target in this paper is effective and practical in the calibration of large FOV measurement system.
1. Introduction Calibration of the vision measurement system is a key technology to ensure measurement accuracy, affecting the accuracy of 3D reconstruction. The vision system must be accurately calibrated for high-precision measurements, visual inspection, three-dimensional positioning, etc. The research of vision system calibration technology develops continuously, and a series of calibration methods are derived for different measurement fields and different calibration targets. Among calibration method based on precision targets, the 2D planar target calibration method has the advantages of high precision and high flexibility, and is widely used in research and application [1]. Zhang [2] also proposed a camera calibration method based on 1D target. The collinear feature points of at least three known positions are moved around a fixed point to establish an equation about the internal and external parameters of the camera. The advantages of 1D target calibration are convenience, flexibility, and ease of implementation, and are suitable for visual system composed of multiple cameras [3]. Followed by Zhang’s method, camera calibration with 1D precision object is further researched. Wu improves the case that the collinear rotation constraint in Zhang's method may cause solution failure. Wu et al. [4] then proposed a method for constructing an equivalent two-dimensional target by using a 1D target in plane motion to achieve camera calibration. In addition to the validity of calibration by moving the 1D target along the plane, Qi et al. [5] verified that the camera calibration can also be achieved by moving the 1D target along the parabola. All the above methods are based on the constraints of one-dimensional target motion for camera calibration. According to Zhang's method, the calibration of single camera cannot be completed without restricting the movement of
⁎
Corresponding author. Co-corresponding author. E-mail addresses: [email protected] (T. Jiang), [email protected] (X. Cheng), [email protected] (H. Cui).
⁎⁎
https://doi.org/10.1016/j.ijleo.2019.163556 Received 10 July 2019; Accepted 7 October 2019 0030-4026/ © 2019 Elsevier GmbH. All rights reserved.
Optik - International Journal for Light and Electron Optics 202 (2020) 163556
T. Jiang, et al.
1D target. Frana et al. [6,7]proposed a multi-camera external parameter calibration method based on 1D target. At the same time, the linear algorithm based on normalized image points significantly improved the calibration accuracy. The normalized image points can be robust in the solution process and are widely used in the method of solving image homogeneous coordinates, such as the normalized 8-point algorithm. Compared to the normalized 1D target calibration method, Shi et al. [8] proposed a weighted similarityinvariant linear algorithm with 1D targets, which achieves a more accurate calibration result. In addition to calibrating the internal parameters of the camera, for large FOV binocular vision system, one-dimensional targets can also be used to calibrate binocular structure parameters [9–12]. The structural parameter refers to external parameters of binocular vision. In the stereo vision theory, the fundamental matrix contains all the parameters of the binocular, from which the internal and external parameters can be decomposed. The 1D target can be used to solve the binocular fundamental matrix, obtain the normalized binocular external parameters, and then use the scale length of the 1D target to solve the accurate binocular external parameters. This method is especially suitable for large FOV vision system calibration. This paper mainly studies the calibration method of large FOV binocular vision system based on 1D target, and proposes a calibration method with normalized 1D homography matrix. The internal parameters of the camera are solved by the homography transformation between the one-dimensional target's own coordinate system and the image coordinate system, and then the external parameters of the dual purpose are calibrated and optimized with photogrammetry method. The article elaborates on the calibration principle, which is an improvement of Refs. [2,3], and has been experimentally verified. The Section 2 introduces the calibration principle, including the normalized homography matrix and camera parameter calculation method; the Section 3 is the simulation and experimental verification. The conclusion follows in Section 4. 2. Calibration principle 2.1. The normalized 1D homography One-dimensional target rotates around a fixed point at one end, and the coordinates of the fixed point remain unchanged relative to the camera. Taking the fixed point as the original, the direction of the remaining points of the one-dimensional target is the x direction, and a one-dimensional coordinate system can be established. As shown in Fig. 1, the one-dimensional coordinate system Ow-Xw is established for the one-dimensional target itself. In Fig. 1, m (u, v) and M(x, y, z) denote the points of the image coordinate system and the world coordinate system, respectively. It can be seen that y = z = 0 in the world coordinate system Ow-XwYwZw. Therefore, according to the principle of camera imaging,
x ⎡ ⎤ u 0 x ⎡ ⎤ s ⎢ v ⎥ = K [ r1 r2 r3 t ] ⎢ ⎥ = K [ r1 t ] ⎡ ⎤ ⎣1⎦ ⎢0⎥ ⎣1⎦ ⎣1⎦ T,
˜ = [u v 1] Let m
˜ = [ x 1] M
T,
(1)
we can get
˜ ˜ = HM sm
(2)
Where s1 H = K [ r1 t ]. This can be seen as a one-dimensional homography between the one-dimensional coordinate system and the u0 ⎤ x r ⎡ h1 h4 ⎤ ⎡ fu ⎡ c⎤ ⎡ 1⎤ image plane coordinate system. H is a 3 × 2 matrix. Let H = ⎢ h2 h5 ⎥, r1 = ⎢ r2 ⎥, t = ⎢ yc ⎥ and K = ⎢ fv v0 ⎥ 。The homography ⎢h 1 ⎥ ⎥ ⎢ ⎣ r3 ⎦ ⎣ zc⎦ 1⎦ ⎣ 3 ⎦ ⎣ matrix has 5 unknowns, and a pair of image points and spaces can provide 2 equations, so at least one dimension target requires at least 3 points. The homography matrix can be solved by the idea of DLT method. In order to improve the robustness of the algorithm, the image points and world points involved in the calculation are normalized. Tm and TM denote the normalized matrix of the image points and the word points. Then Eq. (2) can be expressed as
Fig. 1. Schematic diagram of one-dimensional target calibration. 2
Optik - International Journal for Light and Electron Optics 202 (2020) 163556
T. Jiang, et al.
ˆ TMM ˜ˆ ˆ mm ˜ˆ = H sT
(3)
ˆ is the normalized 1D homography matrix. Hence, Where H ˆ H=T−1 m HTM
(4)
The following is the solution process of the homography matrix. A homography matrix is solved for each data. Using the idea of DLT, the following equation can be obtained.
u ⎡ h1 h4 ⎤ v ⎤ = ⎢ h2 h5 ⎥ = ⎡ x ⎤ s1 ⎡ ⎢ ⎥ ⎢ ⎥ ⎣1⎦ ⎣1⎦ ⎣ h3 1 ⎦
(5)
That is,
⎧ s1 u = h1 x + h4 s1 v = h2 x + h5 ⎨ ⎩ s1 = h3 x + 1
(6)
Simplified
⎧u = ⎪ ⎨ ⎪v = ⎩
h1 x + h4 h3 x + 1 h2 x + h5 h3 x + 1
(7)
Thus the equation can be obtained.
⎡ h1 ⎤ ⎢ h2 ⎥ ⎢h ⎥ − − x 0 ux 1 0 u ⎡ ⎤ ⎢ 3 ⎥ = ⎡0⎤ − − 0 x vx 0 1 v ⎣ ⎦ ⎢ h4 ⎥ ⎣ 0 ⎦ ⎢ h5 ⎥ ⎢ ⎥ ⎣1⎦
(8)
At least 3 points make the above equations solutionable. In the calculation, firstly all the image points are normalized, the homography matrix is calculated by the above formula, and then the inverse homogenization is performed by Eq. (4) to obtain the true homography matrix. The optimization process is added during the homography matrix solution process. The objective function is as following,
˜ ||2 ˜ − HM H*=arg min || m
(9)
2.2. Parameter calculation based on normalized homography matrix According to the camera imaging model, s1 H = K [ r1 t ]. So,
u 0 ⎤ r1 x c ⎡ h1 h4 ⎤ ⎡ fu ⎡ ⎤ ⎡× × ⎤ s1 ⎢ h2 h5 ⎥ = ⎢ fv v0 ⎥ ⎢ r2 yc ⎥ = ⎢ × × ⎥ ⎥ ⎢h 1 ⎥ ⎢ × zc⎦ 1 ⎦ ⎣ r3 z c ⎦ ⎣ ⎦ ⎣ ⎣ 3
(10)
T
T,
From Eq. (10), s1 = z c . If h1 = [ h1 h2 h3 ] and h2 = [ h4 h5 1]
z c [ h1 h2 ] = K [ r1 t ]
(11)
According to the properties of rotation matrix, we can obtaine.
z c2 hT1 K−T K−1h1 = 1
(12)
Where z c is the fixed point in the 1D target. According to the above formula, a homography matrix can be obtained from an image, thus an equation about z c and K can be obtained. Since there are five unknowns in the Eq. (12), it needs at least five positions. According to Eq. (12) ⎡ B11 B12 B13 ⎤ Noting B = z c2 K−T K−1 = ⎢ B12 B22 B23 ⎥ and b = [ B11 B12 B22 B13 B23 B33 ]T , The above equation can be transformed into ⎢ B13 B23 B33 ⎥ hT1 B h1 = [ h12 2h1 h2 h⎣22 2h1 h3 2h2 h⎦3 h32 ] b = vT b (13) If I = [1 ... 1]T , we can obtain linear equations (14)
Vb = I 3
Optik - International Journal for Light and Electron Optics 202 (2020) 163556
T. Jiang, et al.
Hence, b = (V T V)−1V T I. K and z c can be decomposed from B. 2.3. Binocular external parameter calculation based on small 1D target There is an essential matrix E between the camera coordinate systems of the binocular camera, which describes the transformation relationship of the camera coordinate system, including R and t, and the essential matrix can be expressed as E = Rt× . The image points of the left and right cameras are q˜li and q˜ri , according to the imaging model:
⎧ λ1 q˜1i ′ = [I|0] q˜i = P1 q˜i ˜ ˜ ˜ ⎨ ⎩ λ2 q2i ′ = [R|t ] qi = P2 qi
(15)
q˜2i ′T Eq˜1i ′T
According to the propriety of the fundamental matrix, = 0 . R and t can be initially solved by matching points without scale. The ratio of the scale length to the actual length is k, and the actual t matrix can be obtained. Then we need to optimize the length to get the internal and external parameters. Minimizeing the following: n
min f (R, t) =
∑ ||d − di ||
(16)
i=1
The optimized internal and external parameters are the final parameters for the dual target. In this method, the length of the scale affects the calibration result, the longer scale requires higher machining accuracy, and the long scale calibration workload is large. Compared with the long ruler, this paper adopts a short ruler. The short ruler has high processing precision and can place multiple one-dimensional short rulers in the same field of view. The calibration work is flexible and can reduce the workload. At the same time, a large 2D calibration plate is used for accuracy verification. 3. Experiment and result analysis 3.1. Numerical simulation This is a simulation verification of the camera calibration method based on normalized one-dimensional homography. According to the model in Section 2, the homography matrix of the one-dimensional target and the image point should be calculated first, and then the internal reference K is calculated. The solution process can be represented as in Fig. 2. To verify the accuracy of the algorithm, simulation verification is performed. ⎡1500 0 1000 ⎤ Supposing the camera's internal reference matrix is K = ⎢ 0 1500 1000 ⎥. One-dimensional target has a total of 10 points, and 0 1 ⎦ ⎣ 0 the coordinates are, 100, 200, 300, 400, 500, 600, 700, 800, 900 mm, respectively. The one-dimensional target is rotated 20 times around a fixed point P0 = (0, 0, 2) m. The coordinates of the space points obtained according to the spherical coordinate system. For example, the point on the one-dimensional target is M1= (0.1,1), the coordinates in the camera system ⎧ X1 = 0.1 sin θ cos φ + 0 is Y1 = 0.1 sin θ sin φ + 0 .Where the θ and φ are randomly generated. Let r = [sin θ cos φ sin θ sin φ cos φ ]. The image points can ⎨ ⎩ Z1 = 0.1 cos φ + 2 then be obtained by re-projection. That is m=K(r*M+P0). The three-dimensional point map of the one-dimensional target and the coordinates of the re-projection to the image plane are shown in Fig. 3. ⎡1500 0 1000 ⎤ According to the proposed method, the internal parameter matrix of the camera is solved as K′ = ⎢ 0 1500 1000 ⎥. It can be 0 1 ⎦ ⎣ 0 seen that the result of the solution is completely consistent with the theoretical value. Gaussian noise is added to the image, and the result is optimized. The results of the experimental comparison are shown in Fig. 4. ⎡1494.1 − 0.0529 903.5⎤ The calibration result with the optimization process is K′1 = ⎢ 0 1397.1 940.5⎥, and the result of the solution without the 0 1 ⎦ ⎣
Fig. 2. Solution process. (a) 3D points of world coordinates (b) Image reprojection points. 4
Optik - International Journal for Light and Electron Optics 202 (2020) 163556
T. Jiang, et al.
Fig. 3. Simulation data generation. (a) Noise level and optimized results; (b) Noise levels and unoptimized results.
Fig. 4. Effect of noise on calibration results. (a) Source image (b) center detection. (c) Hough line transformation (d) Collinear feature extraction.
⎡1364.9 − 0.0761 952.3 ⎤ optimized internal parameter matrix is K′2 = ⎢ 0 1404.5 927.2 ⎥. It can be seen that the optimized internal reference matrix is 0 1 ⎦ ⎣ closer to the true value. It can be seen from Fig. 4 that the method proposed in this paper has a relative error of less than 0.1 mm in the case of a noise level of 2, which fully satisfies the visual measurement requirements. 3.2. Real data experiment Experiments were carried out on the calibration of the camera using the normalized homography matrix method. In order to ensure the constraint of the common point rotation in practice, multiple one-dimensional targets are drawn in a single image. In the experiment, the feature extraction and collinear point extraction process is shown in the Fig. 5 below. The one-dimensional target homography matrix of each collinear point is calculated separately, and then the above method is used for camera calibration. In addition, the binocular camera is calibrated using a small combined one-dimensional target. As shown in Fig. 3(a), the largefield binocular vision system built for this project. The system hardware includes: 2 IMAGEING cameras DMK 33GP031, 2 lenses with focal length of 8 mm, 2 sets of data control lines and power lines. The working distance of the vision system is about 3 m, the field of view is about 2 m × 1.8 m, and the depth of field is about 1 m. Fig. 6(b) is a schematic diagram of a small one-dimensional target with a center-to-center design dimension of d = 90 mm. A total 5
Optik - International Journal for Light and Electron Optics 202 (2020) 163556
T. Jiang, et al.
Fig. 5. Collinear feature extraction process.
Fig. 6. Structure parameters calibration for binocular vision. (a) Binocular vision setup; (b) Small 1D target.
of 80 coding circles make up 40 small one-dimensional targets with unique identifiability. Fig. 7 is a graph showing the results of corresponding point matching based on coding using 40 sets of small one-dimensional targets. Fig. 7 shows the results of binocular detection and matching. When the working distance is long, the binocular imaging is relatively fuzzy, and the coding points cannot be completely recognized, which affects the subsequent calibration procedure. When the working distance is 2 m, the binocular imaging is clearer, all the coding points are detected, and the double target is higher. The goal of optimization is the distance of the code point, and the distance of the code point is also used as the verification index of the double target accuracy. For the above two calibration cases, two groups were taken for calibration, and the third group was used for accuracy verification. The accuracy verification results obtained are shown in Fig. 8 below. 6
Optik - International Journal for Light and Electron Optics 202 (2020) 163556
T. Jiang, et al.
Fig. 7. Binocular code point matching with working distance of (a) 3 m and (b) 2 m.
Fig. 8. Calibration accuracy verification in different depth directions.
The distance in Fig. 8 is calculated from the two coding centers on the one-dimensional target. For different image quality at different depths, the number of targets that can be successfully identified is different, resulting in different target numbers. As can be seen in Fig. 8, the root mean square error of the evaluation distance error of the 1D target at a distance of 2 m in the depth direction is 0.0625 mm, and the root mean square error at 3 m is 0.1127 mm. It can be seen that the overall deviation is small and the accuracy is slightly reduced in the depth direction. These experiments verify the effectiveness of the one-dimensional target calibration method in this paper. 4. Conclusion The camera calibration technology based on one-dimensional target has a high flexibility and convenience, and is suitable for calibration of large-field binocular vision system. In this paper, a large field of view visual calibration method based on 1D target is proposed. A normalized one-dimensional homography matrix method and a binocular visual calibration method based on small 1D target are proposed. It verifies the calibration accuracy and noise impact through simulation and experiment, and analyzes it through 7
Optik - International Journal for Light and Electron Optics 202 (2020) 163556
T. Jiang, et al.
experiments. The proposed method is an improvement of the traditional one-dimensional target calibration, which can be applied to the binocular vision system calibration of a larger field of view. The calibration accuracy is about 0.1 mm in the depth direction of 2 m and 3 m, and the calibration accuracy meets the measurement requirements. Acknowledgement Our work is supported by Jiangsu Provincial Department of Education. References [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12]
Z. Zhang, A flexible new technique for camera calibration, IEEE Trans. Pattern Anal. Mach. Intell. 22 (11) (2000) 1330–1334. Z. Zhang, Camera calibration with one-dimensional objects, IEEE Trans. Pattern Anal. Mach. Intell. 26 (7) (2004) 0–899. L. Yaowen, L. Wei, X. Xiping, Methods based on 1D homography for camera calibration with 1D objects, Appl. Opt. 57 (9) (2018) 2155–2163. F. Wu, Z. Hu, H. Zhu, Camera calibration with moving one-dimensional objects, Pattern Recognit. 38 (5) (2005) 755–765. F. Qi, Q. Li, Y. Luo, et al., Camera calibration with one-dimensional objects moving under gravity, Pattern Recognit. 40 (1) (2007) 343–345. José Alexandre de Franca, M.R. Stemmer, A.M.B.D.M. Franc, et al., A new robust algorithmic for multi-camera calibration with a 1D object under general motions without prior knowledge of any camera intrinsic parameter, Pattern Recognit. 45 (10) (2012) 3636–3647. JoséA. de Franca, M.R. Stemmer, A.M.B.D.M. Franc, et al., Revisiting Zhang’s 1D calibration algorithm, Pattern Recognit. 43 (3) (2010) 1180–1187. K. Shi, Q. Dong, F. Wu, Weighted similarity-invariant linear algorithm for camera calibration with rotating 1-D objects, IEEE Trans. Image Process. 21 (8) (2012) 3806–3812. Z. Liu, X. Wei, G. Zhang, External parameter calibration of widely distributed vision sensors with non-overlapping fields of view, Opt. Lasers Eng. 51 (6) (2013) 643–650. Z. Xiao, L. Jin, D. Yu, et al., A cross-target-based accurate calibration method of binocular stereo systems with large-scale field-of-view, Measurement 43 (6) (2010) 747–754. J. Zou, J. Zhu, W. Wang, et al., Study on Camera Calibration Approach Using 1D Target in Wide Field of View// International Conference on Electric Information & Control Engineering, (2012). Y. Zhao, W. Li, Self-calibration of a binocular vision system based on a one-dimensional target, J. Mod. Opt. 61 (18) (2014) 1529–1537.
8