- Email: [email protected]
A camera calibration method based on plane mirror and vanishing point constraint
Optik 154 (2018) 558–565
Contents lists available at ScienceDirect
Optik journal homepage: www.elsevier.de/ijleo
Full length article
A camera calibration method based on plane mirror and vanishing point constraint Xiao-feng Feng a , Di-fu Pan b,∗ a b
Department of Traffic Management, Hunan Police Academy, Changsha, China School of Traffic & Transportation Engineering, Central South University, Changsha, China
a r t i c l e
i n f o
Article history: Received 26 July 2017 Accepted 18 October 2017 Keywords: Machine vision Parameter calibration Vanishing point pair Plane mirror
a b s t r a c t In order to avoid a cumbersome calibration of camera parameters, a method of selfcalibration using the imaging features of a plane mirror and the vanishing point pair constraint is proposed. A plane mirror and calibration template were placed at a certain angle and non-vertical geometric constraints were obtained from the calibration template, as well as from the virtual image of the calibration template. So all the internal parameters were calibrated at once. In order to reduce the influence on the calibration parameters of the vanishing point position, the least squares algorithm and the LM algorithm were used to optimize the parameters. Simulations and real experimental results show that the method has a high accuracy and robustness. © 2017 Elsevier GmbH. All rights reserved.
1. Introduction Camera calibration is the process of solving for the model parameters of the camera according to a camera model, which is based on the image information of the feature points and the 3D spatial information. At present, there are many camera calibration methods [1], which can be divided into two types: traditional calibration methods and camera self-calibration methods. A traditional calibration method [2] gets the internal and external parameters under a certain camera model, based on specific experimental conditions, such as the shape and size of the calibration reference, which should be known. It has stricter requirements for its calibration reference and image coordinates. In 1992, Hartley [3] and Faugeras [4] first proposed the idea of camera self-calibration. This method only depends on the corresponding relationship between image and image of the surrounding environment during the movement process of the camera. Many scholars have researched this aspect, among which the vanishing point of orthogonal characteristic calibration method has become a research hotspot. Caprile [5], as well as Beardsley [6] first proposed a method to calibrate the camera by using the geometric characteristics of the vanishing points. Wang [7] uses three sets of lines and vanishing points to realize the camera calibration, but the method requires that these three sets of line segments be the same length as well as being mutually perpendicular in the calibration space; these calibration conditions are quite harsh. Chen et al. [8] calibrate the parameters using 4 vanishing points produced by a square to establish the vanishing line constraint equation, which is more sensitive to noise due to the need to solve the Kruppa equation. Huo [9] generates a pair of orthogonal vanishing points using two pairs of orthogonal parallel lines, and establishes the constraint equations with a relationship to the origin of the camera. There are other calibration methods using the orthogonal vanishing point [10–12] generated by rectangular, cylindrical, or a cube (long).
∗ Corresponding author at: School of Traffic & Transportation Engineering, Central South University, Changsha, China. E-mail address: [email protected] (D.-f. Pan). https://doi.org/10.1016/j.ijleo.2017.10.086 0030-4026/© 2017 Elsevier GmbH. All rights reserved.
X.-f. Feng, D.-f. Pan / Optik 154 (2018) 558–565
559
Fig. 1. Relationship among camera coordinate systems.
As at least three images need to be taken in different positions to calibrate the camera parameters, it is quite cumbersome, whether moving the camera or moving the calibration template. In order to avoid the disadvantages of the traditional calibration method, a new calibration method is proposed based on the idea that the constraint equation can be used to obtain the intrinsic parameters of the intrinsic camera [13–16]. This method uses the principle of mirror reflection [17,18], puts the calibration template and plane mirror non-vertically, they only need to be located once, and there is just one shot: all parameters can be calibrated without moving the camera or the mirror. 2. Camera calibration model According to the pinhole model [19], as shown in Fig. 1, the mapping relationship of spatial coordinates P(XW , YW , ZW ) and pixel coordinates p(u, v)in the image coordinate system is
⎡ ⎤ u
⎡ ⎢
1/dx
⎣v ⎦ = ⎣0 1
0
⎤⎡
0
u0
1/dy
v0 ⎦ ⎣ 0 f
0
1
⎥
⎞ ⎤ ⎛⎡ r11 r12 r13 ⎤ XW t1 ⎥ ⎟ ⎜⎢ 0 ⎦ ⎝⎣ r21 r22 r23 ⎦ [ YW ] + [ t2 ]⎠ =A[
f
0
0
0
0
1
r31
r32
r33
ZW
R
|
T
]PW
(1)
t3
where (u0 , v0 ) are the coordinates of the image centre, f/dx , f/dy are separately normalized focal lengths; A = [f/dx 0 u0 ; 0 f/dy v0 ; 0 0 1] are the intrinsic parameters of the camera; R is a rotation matrix, and T is a translation vector, which forms the external parameters of the camera. This formula describes the transformation of a point in space from the world coordinate system to the camera coordinate system, then to the image plane coordinate system, and then to the pixel coordinate system. 3. Calibration principle based on plane mirror and orthogonal vanishing point 3.1. Properties of orthogonal vanishing point In Euclidean space two parallel lines will intersect at infinity, without considering the distortion of the pinhole camera model: according to the theory of projective transformations, the two parallel lines projected to the imaging plane will intersect at one point, which is called the vanishing point – it is the projection of the point at infinity to the imaging plane. According to [20], the line connecting the camera’s optical centre and the vanishing point formed in the projection surface by the spatial parallel lines will be parallel to the spatial parallel lines (Fig. 2). For a square ABCD in space, the image in the imaging plane is a quadrilateral abcd. According to the principle of the vanishing point, the image of the parallel sides ab and cd intersect at the vanishing pointv1 , while bc and ad intersect at the vanishing point v2 . Then ov1 AB CD and ov2 AD BC, as AB ⊥ AD, and so ov1 ⊥ ov2 . Here, v1 and v2 are a pair of orthogonal
560
X.-f. Feng, D.-f. Pan / Optik 154 (2018) 558–565
Fig. 2. Properties of vanishing point.
Fig. 3. Square and its projection image.
vanishing points: the line connecting these two points forms the vanishing lineL. Similarly, in the quadrilateralabcd, the extended line of diagonal ac and bd intersect with the vanishing line at op1 and op2 , respectively, and the two points also form a pair of orthogonal vanishing points; we also have op1 ⊥ op2 (Fig. 3). From [21], if x1 and x2 are two vanishing points of any two points on two orthogonal lines L1 and L2 separately, then x1 A−T A−1 x2 = 0, (A−1 x1 ) T (A−1 x2 ) = 0. Now the two constraint equations for the camera’s internal parameters are obtained from the two vanishing points:
⎧ ⎨ v1 A−T A−1 v2 = A−1 v1 T A−1 v2 = 0 −1 T −1 ⎩ −T −1 p1 A
A
p2 = A
p1
A
p2 = 0
(2)
X.-f. Feng, D.-f. Pan / Optik 154 (2018) 558–565
561
Fig. 4. Solution of Camera intrinsic parameters.
3.2. Solution for the camera’s intrinsic parameters From above equation, for the two orthogonal vanishing point pairs v1 (uv1 , vv1 ), v2 (uv2 , vv2 ), the coordinates of the two vanishing points in the camera coordinate system are
v1 [(uv1 − u0 )dx, (vv1 − v0 )dy, f ], v2 [(uv2 − u0 )dx, (vv2 − v0 )dy, f ]. From Eq. (2), {
[(uv1 − u0 )dx, (vv1 − v0 )dy, f]T [(uv2 − u0 ) dx, (vv2 − v0 )dy, f] = 0
(3)
[(up1 − u0 )dx, (vp1 − v0 )dy, f]T [ up2 − u0 dx, (vp2 − v0 )dy, f] = 0
As there are 4 unknown parameters in these equations, it is necessary to have at least another two equations to solve. This means that another square is needed. Using the imaging characteristics of a plane mirror, we can get two images of the square (Fig. 4). In order to avoid the degradation of the vanishing point caused by the corresponding parallel edges of the object and the image, the plane of the square should not be perpendicular to the plane of the plane mirror. E.g., ABCD, A B C D corresponding to a vanishing point is v2 , v1 , p2 , p1 . Another set of constraint equations can be obtained from the orthogonal relation of the vanishing point pair above. The nonlinear equations are as follows:
⎡
⎤
⎡ 2 ⎤ ⎡ 0 ⎤ ⎢ ⎥ dx/f ⎢ (up1 − u0 )(up2 − u0 ) (vp1 − v0 )(vp2 − v0 ) 1 ⎥ ⎢ ⎢ ⎥ ⎥ ⎢0⎥ ⎢ ⎥ ⎢ (u − u )(u − u ) (v − v )(v − v ) 1 ⎥ ⎣ dy/f 2 ⎦ = ⎣ 0 ⎦ 0 v2 0 v1 0 v2 0 ⎣ v1 ⎦ (uv1 − u0 )(uv2 − u0 )
(up 1 − u0 )(up 2 − u0 )
(vv1 − v0 )(vv2 − v0 )
1
1
(vp 1 − v0 )(vp 2 − v0 )
1
(4)
0
The above equations are 4 equations for the parameters u0 , v0 , (dx/f)2 , (dy/f)2 . In order to solve this system of linear equations, set x1 = u0 x3 ,
x2 = v0 ,
x3 = (dx/dy)
2
(5)
Subtracting the two corresponding equations yields
⎡
uv1 + uv2 −
up1 − up2
⎢ ⎣ uv1 + uv2 − uv 1 − uv 2 uv1 + uv2 −
up 1 − up 2
vv1 + vv2 − vp1 − vp2
vp1 vp2 − vv1 vv2
vv1 + vv2 − vv 1 − vv 2
vv 1 vv 2 − vv1 vv2
vv1 + vv2 − vp 1 − vp 2
vp 1 vp 2 − vv1 vv2
⎤
x1
⎥ ⎦ [ x2
x3
]=[
uv1 uv2 −
up1 up2
uv1 uv2 −
uv 1 uv 2 ]
uv1 uv2 −
up 1 up 2
Solving these equations forX = [x1 x2 x3 ], then putting Xinto Eq. (4), f/dx and f/dy can be obtained.
(6)
562
X.-f. Feng, D.-f. Pan / Optik 154 (2018) 558–565
Fig. 5. Three dimensional location of simulation square.
Table 1 The influence of different noise levels on the various parameters. noise levels
u0
v0
f/dx
f/dy
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 2.2 2.4 2.6
340 341.1849 336.4181 333.3004 341.9585 347.7529 331.5386 324.2629 347.1029 363.7561 309.8859 354.3151 329.9855 376.9032
270 269.9456 268.7604 270.3315 268.9674 267.9117 268.6557 268.4527 269.0365 271.7807 263.3261 274.7785 270.9449 275.4224
3000 3000.2 3003.7 3000.9 3002.1 2999.9 3005.0 3007.5 3004.3 2990.6 3017.8 2982.5 2994.9 2977.9
3000 2999.5 3004.1 3004.4 2999.8 2994.9 3008.7 3012.1 2999.2 2982.2 3030.2 2981.2 3005.0 2962.2
4. Simulation experiment In order to verify the effectiveness of the method, simulation experiments were carried out on the MATLAB platform. Firstly, there was established the simulation camera model. Assume that the camera’s internal parameters are u0 = 340, v0 = 270, f/dx = 3000, f/dy = 3000 , and the image size of the calibration template is 1600*1200 pixels. The plane x = 0 in threedimensional space is defined as the plane of the plane mirror. The angle between the imaging plane and the plane mirror is 70◦ while the virtual image plane and the imaging plane are symmetrical about the plane mirror. R and T denote the rotation matrix and translation vector of the camera in the world coordinate system. R is represented by its rotation angle relative to the three coordinate axes: R = [0◦ ,35◦ , 20◦ ]. Along the direction of the axis, a counterclockwise rotation angle is positive. The translation vector T = [2500, 2000, 2500]. For a square in three-dimensional space, the left side of the coordinate system is virtual square, its spatial position and shape are shown as follows (Fig. 5). At each image point of the image plane, add a random white noise with an amplitude of 0 ∼ 2pixel, Then carry out 50 independent experiments at each noise level. The variation of the internal parameters at different noise levels are shown in Table 1. The variation of the mean absolute error of the camera parameters with the noise is shown in Fig. 6. The simulation results show that the trend of calibration error is increasing with an increase of noise level, which shows the correctness of the method.
X.-f. Feng, D.-f. Pan / Optik 154 (2018) 558–565
563
Fig. 6. The relationship between the absolute error of internal parameters and the level of noise.
Fig. 7. The 4 corners used for intrinsic parameters Calibration.
5. Real experiment The calibration template uses a 6 × 6 chessboard, a grid size of 30 mm × 30 mm, and a camera with a resolution of 1024 pixel × 768 pixel to obtain the calibration template image. Then we selected 4 corners for the calibration target (A, B, C, D and their virtual images A’, B’, C’, D’ in Fig. 7) to calibrate (as shown in Fig. 7). The comparison of the calibration results obtained by the Matlab calibration toolbox and the method in this paper is shown in Table 2. Because of its mature and stable calibration results, the calibration results from the Matlab Calibration Toolbox were taken as standard. In this paper, the error of the 4 point calibration method is larger: this is mainly due to the distortion of the imaging and the error of the corner detection [22]: the two parallel lines in the same direction are not parallel, and the position of the corresponding vanishing points is not accurate. So, we extracted multiple corner points from two pairs
564
X.-f. Feng, D.-f. Pan / Optik 154 (2018) 558–565
Table 2 Camera intrinsic parameters calibrated by two methods. The units are pixels. Calibration method
u0
v0
f/dx
f/dy
Matlab Calibration tool box Proposed Calibration error
518.2573 400.3854 117.8719
392.1157 298.7370 93.3787
1084.7820 926.0545 158.7275
1083.9087 949.8721 134.0366
Fig. 8. Image used for Calibration. Table 3 Camera intrinsic parameters calibrated by different methods. The units are pixels. Calibration method
u0
v0
f/dx
f/dy
Matlab Calibration tool box Zhang’s method Multi-image calibration method Proposal after optimization Error of Multi-image method Error of Proposal
518.2573 519.1028 531.9585 532.1274 13.7012 13.8701
392.1157 394.6592 378.9449 378.2243 13.1708 13.8914
1084.7820 1085.5371 1072.2148 1074.0260 12.5672 10.7560
1083.9087 1086.1037 1077.6273 1076.8374 6.2814 7.0713
of edges and two diagonal directions, and then the least squares principle and Levenberg-Marquart algorithm were used to find the 4 optimal vanishing points in 4 directions (Fig. 8). For a number of corner points in a certain direction, linear equation fit through the least squares algorithm in this direction is kij u − v + bij = 0,i = 1, 2...,n;j = 1, 2, 3, 4;
(7)
Where i denotes the number of corner points in the direction of a straight line, j denotes the direction of two pairs of parallel lines and the direction of the two vertical diagonals. Corresponding to group j of parallel lines, the vanishing point p’s coordinates in the imaging plane isp(u, v), and the optimal location of the vanishing point should minimize d, the sum of the distances between the vanishing point and the projection lines, so the optimization function is defined as follows: n
Fj = arg min(
i=1
n |kij u − v + bij | )
d(p(u, v), lij (kij , −1, bij )) = arg min(
i=1
(8)
2
kij + 1
where d(p(u, v), lij (kij , −1, bij )) represents the distance between the vanishing point and the linelij . Taking the average of two projected points as the initial value for the iteration, we use the Levenberg–Marquart algorithm to carry out the optimization. In order to compare the results, this paper adopts the Matlab calibration toolbox, the Zhang-Zheng-You calibration method, the multiple image calibration method based on orthogonal vanishing points, and the method proposed in this paper, to calibrate parameters separately. The calibration results are shown in Table 3. It can be seen that the calibration results from the method proposed in this paper has a certain gap with the results by the Matlab Calibration Toolbox, but is similar to the results of the multiple image calibration method. As this method does not need any precise calibration object coordinates, and can calibrate all internal parameters linearly by just one imaging, it is easy to operate and realize, and has a certain application value in situations of low precision and fast calibration. 6. Conclusions (1) Making full use of the orthogonality properties of lines connected by orthogonal vanishing points and the optical center, a square generates two constraint equations, which enables the avoidance of tediously and repeatedly shooting and
X.-f. Feng, D.-f. Pan / Optik 154 (2018) 558–565
565
operating: employing the characteristics of the plane mirror, the intrinsic parameters of the camera can be calculated with just one imaging, with the help of the flat mirror. (2) The proposed method based on a planar mirror and vanishing point constraints, is simple, convenient, and has lenient accuracy requirements for the calibration target. Simulations and real experiments show that the method has good accuracy and robustness. Disclosure statement No potential conflict of interest was reported by the authors. Funding The work was supported by a program of Hunan Department of Education outstanding youth project (17B086), and a program of Hunan Police Academy Doctoral project (2016ZX02). Acknowledgement All members of the subject actively participated in the research, and we hereby thank them all. References [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20] [21] [22]
G.J. Zhang, Vision Measurement, Science Publisher, Beijing, 2008. J. Xue, X. Su, Camera calibration with single image based on two orthogonal one-dimensional objects, Acta Optica Sinica 32 (1) (2012) 0115001. R. Hartley, Self-calibration of stationary cameras, Int. J. Comput. Vision 22 (1) (1997) 5–23. S.J. Manbank, O.D. Faugeras, A theory of self-calibration of a moving camera, Int. J. Comput. Vision 8 (2) (1992) 123–151. B. Caprile, V. Torre, Using vanishing points for camera calibration, Int. J. Comput. Vision 4 (2) (1990) 127–140. P. Beardsley, D. Murray, Camera calibration using vanishing points, Proc. Br. Mach. Vision Conf. (1992) 416–425. G.H. Wang, H.T. Tsui, Z.Y. Hu, et al., Camera calibration and 3D reconstruction from a single view based on scene constraints, Image Vision Comput. 23 (2005) 311–323. A. Chen, C. Gao, B. He, Camera calibration method based on orthogonal vanishing point pair, Chin. J. Sci. Instrum. 33 (1) (2012) 161–166. J. Huo, W. Yang, M. Yang, A self-calibration technique based on the geometry property of the vanish point, Acta Optica Sinica 30 (2) (2010) 465–472. C. Ricolf-Viala, A.J. Sanchez-Salmeron, E. Martinez-Berti, Accurate calibration with highly distorted images, Appl. Opt. 51 (1) (2012) 89–101. F. Wu, Mathematical Methods in Computer Vision, Science Press, Beijing, 2008, pp. 86–111. B.W. He, Y.F. Li, A novel method for camera calibration using vanishing points, 14th International Conference on M2VIP (2007) 44–47. H. Yang, S. Zhang, C. Liu, Research into camera calibration methods based on the vanishing points theory and planar control fields, J. Image Graphics 15 (8) (2010) 1168–1174. Y. Hong, X. Sun, M. Cai, et al., An intrinsic parameters self-calibration technique based on infinite homography between orthogonal vanishing points, Acta Optica Sinica 42 (12) (2015) 1208001. M. Cai, X. Sun, S. Liu, et al., An accurate & real-time focal-length self-calibration method based on infinite homography between vanish points, Acta Optica Sinica 34 (5) (2014) 0515003. S. Xu, X. Sun, X. Liu, et al., Geometry method of camera self-calibration based on a rectangle, Acta Optica Sinica 34 (11) (2014) 1115002. F. Xiaofeng, P. Difu, Study of single camera stereo vision sensor based on plane mirror imaging, Acta Optica Sinica 34 (9) (2014) 0915002. F. Xiaofeng, P. Difu, Research on the application of single camera stereo vision sensor in three-dimensional point measurement, J. Modern Optics 62 (15) (2015) 1204–1210. S. Xu, X. Sun, S. Liu, et al., Model reference approaching method of camera distortion calibration, Acta Optica Sinica 33 (7) (2013) 0715001. H. Opower, Multiple view geometry in computer vision, Optics Lasers Eng. 37 (1) (2002) 85–86. R. Hartley, Multiple View Geometry in Computer Vision, 2nd ed., Cambridge University Press, 2003. J. Lu, H. Sun, Z. Chang, A novel method for camera calibration with orthogonal vanishing points, Chin. J. Lasers 41 (2) (2014) 0208001.
Contents lists available at ScienceDirect
Optik journal homepage: www.elsevier.de/ijleo
Full length article
A camera calibration method based on plane mirror and vanishing point constraint Xiao-feng Feng a , Di-fu Pan b,∗ a b
Department of Traffic Management, Hunan Police Academy, Changsha, China School of Traffic & Transportation Engineering, Central South University, Changsha, China
a r t i c l e
i n f o
Article history: Received 26 July 2017 Accepted 18 October 2017 Keywords: Machine vision Parameter calibration Vanishing point pair Plane mirror
a b s t r a c t In order to avoid a cumbersome calibration of camera parameters, a method of selfcalibration using the imaging features of a plane mirror and the vanishing point pair constraint is proposed. A plane mirror and calibration template were placed at a certain angle and non-vertical geometric constraints were obtained from the calibration template, as well as from the virtual image of the calibration template. So all the internal parameters were calibrated at once. In order to reduce the influence on the calibration parameters of the vanishing point position, the least squares algorithm and the LM algorithm were used to optimize the parameters. Simulations and real experimental results show that the method has a high accuracy and robustness. © 2017 Elsevier GmbH. All rights reserved.
1. Introduction Camera calibration is the process of solving for the model parameters of the camera according to a camera model, which is based on the image information of the feature points and the 3D spatial information. At present, there are many camera calibration methods [1], which can be divided into two types: traditional calibration methods and camera self-calibration methods. A traditional calibration method [2] gets the internal and external parameters under a certain camera model, based on specific experimental conditions, such as the shape and size of the calibration reference, which should be known. It has stricter requirements for its calibration reference and image coordinates. In 1992, Hartley [3] and Faugeras [4] first proposed the idea of camera self-calibration. This method only depends on the corresponding relationship between image and image of the surrounding environment during the movement process of the camera. Many scholars have researched this aspect, among which the vanishing point of orthogonal characteristic calibration method has become a research hotspot. Caprile [5], as well as Beardsley [6] first proposed a method to calibrate the camera by using the geometric characteristics of the vanishing points. Wang [7] uses three sets of lines and vanishing points to realize the camera calibration, but the method requires that these three sets of line segments be the same length as well as being mutually perpendicular in the calibration space; these calibration conditions are quite harsh. Chen et al. [8] calibrate the parameters using 4 vanishing points produced by a square to establish the vanishing line constraint equation, which is more sensitive to noise due to the need to solve the Kruppa equation. Huo [9] generates a pair of orthogonal vanishing points using two pairs of orthogonal parallel lines, and establishes the constraint equations with a relationship to the origin of the camera. There are other calibration methods using the orthogonal vanishing point [10–12] generated by rectangular, cylindrical, or a cube (long).
∗ Corresponding author at: School of Traffic & Transportation Engineering, Central South University, Changsha, China. E-mail address: [email protected] (D.-f. Pan). https://doi.org/10.1016/j.ijleo.2017.10.086 0030-4026/© 2017 Elsevier GmbH. All rights reserved.
X.-f. Feng, D.-f. Pan / Optik 154 (2018) 558–565
559
Fig. 1. Relationship among camera coordinate systems.
As at least three images need to be taken in different positions to calibrate the camera parameters, it is quite cumbersome, whether moving the camera or moving the calibration template. In order to avoid the disadvantages of the traditional calibration method, a new calibration method is proposed based on the idea that the constraint equation can be used to obtain the intrinsic parameters of the intrinsic camera [13–16]. This method uses the principle of mirror reflection [17,18], puts the calibration template and plane mirror non-vertically, they only need to be located once, and there is just one shot: all parameters can be calibrated without moving the camera or the mirror. 2. Camera calibration model According to the pinhole model [19], as shown in Fig. 1, the mapping relationship of spatial coordinates P(XW , YW , ZW ) and pixel coordinates p(u, v)in the image coordinate system is
⎡ ⎤ u
⎡ ⎢
1/dx
⎣v ⎦ = ⎣0 1
0
⎤⎡
0
u0
1/dy
v0 ⎦ ⎣ 0 f
0
1
⎥
⎞ ⎤ ⎛⎡ r11 r12 r13 ⎤ XW t1 ⎥ ⎟ ⎜⎢ 0 ⎦ ⎝⎣ r21 r22 r23 ⎦ [ YW ] + [ t2 ]⎠ =A[
f
0
0
0
0
1
r31
r32
r33
ZW
R
|
T
]PW
(1)
t3
where (u0 , v0 ) are the coordinates of the image centre, f/dx , f/dy are separately normalized focal lengths; A = [f/dx 0 u0 ; 0 f/dy v0 ; 0 0 1] are the intrinsic parameters of the camera; R is a rotation matrix, and T is a translation vector, which forms the external parameters of the camera. This formula describes the transformation of a point in space from the world coordinate system to the camera coordinate system, then to the image plane coordinate system, and then to the pixel coordinate system. 3. Calibration principle based on plane mirror and orthogonal vanishing point 3.1. Properties of orthogonal vanishing point In Euclidean space two parallel lines will intersect at infinity, without considering the distortion of the pinhole camera model: according to the theory of projective transformations, the two parallel lines projected to the imaging plane will intersect at one point, which is called the vanishing point – it is the projection of the point at infinity to the imaging plane. According to [20], the line connecting the camera’s optical centre and the vanishing point formed in the projection surface by the spatial parallel lines will be parallel to the spatial parallel lines (Fig. 2). For a square ABCD in space, the image in the imaging plane is a quadrilateral abcd. According to the principle of the vanishing point, the image of the parallel sides ab and cd intersect at the vanishing pointv1 , while bc and ad intersect at the vanishing point v2 . Then ov1 AB CD and ov2 AD BC, as AB ⊥ AD, and so ov1 ⊥ ov2 . Here, v1 and v2 are a pair of orthogonal
560
X.-f. Feng, D.-f. Pan / Optik 154 (2018) 558–565
Fig. 2. Properties of vanishing point.
Fig. 3. Square and its projection image.
vanishing points: the line connecting these two points forms the vanishing lineL. Similarly, in the quadrilateralabcd, the extended line of diagonal ac and bd intersect with the vanishing line at op1 and op2 , respectively, and the two points also form a pair of orthogonal vanishing points; we also have op1 ⊥ op2 (Fig. 3). From [21], if x1 and x2 are two vanishing points of any two points on two orthogonal lines L1 and L2 separately, then x1 A−T A−1 x2 = 0, (A−1 x1 ) T (A−1 x2 ) = 0. Now the two constraint equations for the camera’s internal parameters are obtained from the two vanishing points:
⎧ ⎨ v1 A−T A−1 v2 = A−1 v1 T A−1 v2 = 0 −1 T −1 ⎩ −T −1 p1 A
A
p2 = A
p1
A
p2 = 0
(2)
X.-f. Feng, D.-f. Pan / Optik 154 (2018) 558–565
561
Fig. 4. Solution of Camera intrinsic parameters.
3.2. Solution for the camera’s intrinsic parameters From above equation, for the two orthogonal vanishing point pairs v1 (uv1 , vv1 ), v2 (uv2 , vv2 ), the coordinates of the two vanishing points in the camera coordinate system are
v1 [(uv1 − u0 )dx, (vv1 − v0 )dy, f ], v2 [(uv2 − u0 )dx, (vv2 − v0 )dy, f ]. From Eq. (2), {
[(uv1 − u0 )dx, (vv1 − v0 )dy, f]T [(uv2 − u0 ) dx, (vv2 − v0 )dy, f] = 0
(3)
[(up1 − u0 )dx, (vp1 − v0 )dy, f]T [ up2 − u0 dx, (vp2 − v0 )dy, f] = 0
As there are 4 unknown parameters in these equations, it is necessary to have at least another two equations to solve. This means that another square is needed. Using the imaging characteristics of a plane mirror, we can get two images of the square (Fig. 4). In order to avoid the degradation of the vanishing point caused by the corresponding parallel edges of the object and the image, the plane of the square should not be perpendicular to the plane of the plane mirror. E.g., ABCD, A B C D corresponding to a vanishing point is v2 , v1 , p2 , p1 . Another set of constraint equations can be obtained from the orthogonal relation of the vanishing point pair above. The nonlinear equations are as follows:
⎡
⎤
⎡ 2 ⎤ ⎡ 0 ⎤ ⎢ ⎥ dx/f ⎢ (up1 − u0 )(up2 − u0 ) (vp1 − v0 )(vp2 − v0 ) 1 ⎥ ⎢ ⎢ ⎥ ⎥ ⎢0⎥ ⎢ ⎥ ⎢ (u − u )(u − u ) (v − v )(v − v ) 1 ⎥ ⎣ dy/f 2 ⎦ = ⎣ 0 ⎦ 0 v2 0 v1 0 v2 0 ⎣ v1 ⎦ (uv1 − u0 )(uv2 − u0 )
(up 1 − u0 )(up 2 − u0 )
(vv1 − v0 )(vv2 − v0 )
1
1
(vp 1 − v0 )(vp 2 − v0 )
1
(4)
0
The above equations are 4 equations for the parameters u0 , v0 , (dx/f)2 , (dy/f)2 . In order to solve this system of linear equations, set x1 = u0 x3 ,
x2 = v0 ,
x3 = (dx/dy)
2
(5)
Subtracting the two corresponding equations yields
⎡
uv1 + uv2 −
up1 − up2
⎢ ⎣ uv1 + uv2 − uv 1 − uv 2 uv1 + uv2 −
up 1 − up 2
vv1 + vv2 − vp1 − vp2
vp1 vp2 − vv1 vv2
vv1 + vv2 − vv 1 − vv 2
vv 1 vv 2 − vv1 vv2
vv1 + vv2 − vp 1 − vp 2
vp 1 vp 2 − vv1 vv2
⎤
x1
⎥ ⎦ [ x2
x3
]=[
uv1 uv2 −
up1 up2
uv1 uv2 −
uv 1 uv 2 ]
uv1 uv2 −
up 1 up 2
Solving these equations forX = [x1 x2 x3 ], then putting Xinto Eq. (4), f/dx and f/dy can be obtained.
(6)
562
X.-f. Feng, D.-f. Pan / Optik 154 (2018) 558–565
Fig. 5. Three dimensional location of simulation square.
Table 1 The influence of different noise levels on the various parameters. noise levels
u0
v0
f/dx
f/dy
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 2.2 2.4 2.6
340 341.1849 336.4181 333.3004 341.9585 347.7529 331.5386 324.2629 347.1029 363.7561 309.8859 354.3151 329.9855 376.9032
270 269.9456 268.7604 270.3315 268.9674 267.9117 268.6557 268.4527 269.0365 271.7807 263.3261 274.7785 270.9449 275.4224
3000 3000.2 3003.7 3000.9 3002.1 2999.9 3005.0 3007.5 3004.3 2990.6 3017.8 2982.5 2994.9 2977.9
3000 2999.5 3004.1 3004.4 2999.8 2994.9 3008.7 3012.1 2999.2 2982.2 3030.2 2981.2 3005.0 2962.2
4. Simulation experiment In order to verify the effectiveness of the method, simulation experiments were carried out on the MATLAB platform. Firstly, there was established the simulation camera model. Assume that the camera’s internal parameters are u0 = 340, v0 = 270, f/dx = 3000, f/dy = 3000 , and the image size of the calibration template is 1600*1200 pixels. The plane x = 0 in threedimensional space is defined as the plane of the plane mirror. The angle between the imaging plane and the plane mirror is 70◦ while the virtual image plane and the imaging plane are symmetrical about the plane mirror. R and T denote the rotation matrix and translation vector of the camera in the world coordinate system. R is represented by its rotation angle relative to the three coordinate axes: R = [0◦ ,35◦ , 20◦ ]. Along the direction of the axis, a counterclockwise rotation angle is positive. The translation vector T = [2500, 2000, 2500]. For a square in three-dimensional space, the left side of the coordinate system is virtual square, its spatial position and shape are shown as follows (Fig. 5). At each image point of the image plane, add a random white noise with an amplitude of 0 ∼ 2pixel, Then carry out 50 independent experiments at each noise level. The variation of the internal parameters at different noise levels are shown in Table 1. The variation of the mean absolute error of the camera parameters with the noise is shown in Fig. 6. The simulation results show that the trend of calibration error is increasing with an increase of noise level, which shows the correctness of the method.
X.-f. Feng, D.-f. Pan / Optik 154 (2018) 558–565
563
Fig. 6. The relationship between the absolute error of internal parameters and the level of noise.
Fig. 7. The 4 corners used for intrinsic parameters Calibration.
5. Real experiment The calibration template uses a 6 × 6 chessboard, a grid size of 30 mm × 30 mm, and a camera with a resolution of 1024 pixel × 768 pixel to obtain the calibration template image. Then we selected 4 corners for the calibration target (A, B, C, D and their virtual images A’, B’, C’, D’ in Fig. 7) to calibrate (as shown in Fig. 7). The comparison of the calibration results obtained by the Matlab calibration toolbox and the method in this paper is shown in Table 2. Because of its mature and stable calibration results, the calibration results from the Matlab Calibration Toolbox were taken as standard. In this paper, the error of the 4 point calibration method is larger: this is mainly due to the distortion of the imaging and the error of the corner detection [22]: the two parallel lines in the same direction are not parallel, and the position of the corresponding vanishing points is not accurate. So, we extracted multiple corner points from two pairs
564
X.-f. Feng, D.-f. Pan / Optik 154 (2018) 558–565
Table 2 Camera intrinsic parameters calibrated by two methods. The units are pixels. Calibration method
u0
v0
f/dx
f/dy
Matlab Calibration tool box Proposed Calibration error
518.2573 400.3854 117.8719
392.1157 298.7370 93.3787
1084.7820 926.0545 158.7275
1083.9087 949.8721 134.0366
Fig. 8. Image used for Calibration. Table 3 Camera intrinsic parameters calibrated by different methods. The units are pixels. Calibration method
u0
v0
f/dx
f/dy
Matlab Calibration tool box Zhang’s method Multi-image calibration method Proposal after optimization Error of Multi-image method Error of Proposal
518.2573 519.1028 531.9585 532.1274 13.7012 13.8701
392.1157 394.6592 378.9449 378.2243 13.1708 13.8914
1084.7820 1085.5371 1072.2148 1074.0260 12.5672 10.7560
1083.9087 1086.1037 1077.6273 1076.8374 6.2814 7.0713
of edges and two diagonal directions, and then the least squares principle and Levenberg-Marquart algorithm were used to find the 4 optimal vanishing points in 4 directions (Fig. 8). For a number of corner points in a certain direction, linear equation fit through the least squares algorithm in this direction is kij u − v + bij = 0,i = 1, 2...,n;j = 1, 2, 3, 4;
(7)
Where i denotes the number of corner points in the direction of a straight line, j denotes the direction of two pairs of parallel lines and the direction of the two vertical diagonals. Corresponding to group j of parallel lines, the vanishing point p’s coordinates in the imaging plane isp(u, v), and the optimal location of the vanishing point should minimize d, the sum of the distances between the vanishing point and the projection lines, so the optimization function is defined as follows: n
Fj = arg min(
i=1
n |kij u − v + bij | )
d(p(u, v), lij (kij , −1, bij )) = arg min(
i=1
(8)
2
kij + 1
where d(p(u, v), lij (kij , −1, bij )) represents the distance between the vanishing point and the linelij . Taking the average of two projected points as the initial value for the iteration, we use the Levenberg–Marquart algorithm to carry out the optimization. In order to compare the results, this paper adopts the Matlab calibration toolbox, the Zhang-Zheng-You calibration method, the multiple image calibration method based on orthogonal vanishing points, and the method proposed in this paper, to calibrate parameters separately. The calibration results are shown in Table 3. It can be seen that the calibration results from the method proposed in this paper has a certain gap with the results by the Matlab Calibration Toolbox, but is similar to the results of the multiple image calibration method. As this method does not need any precise calibration object coordinates, and can calibrate all internal parameters linearly by just one imaging, it is easy to operate and realize, and has a certain application value in situations of low precision and fast calibration. 6. Conclusions (1) Making full use of the orthogonality properties of lines connected by orthogonal vanishing points and the optical center, a square generates two constraint equations, which enables the avoidance of tediously and repeatedly shooting and
X.-f. Feng, D.-f. Pan / Optik 154 (2018) 558–565
565
operating: employing the characteristics of the plane mirror, the intrinsic parameters of the camera can be calculated with just one imaging, with the help of the flat mirror. (2) The proposed method based on a planar mirror and vanishing point constraints, is simple, convenient, and has lenient accuracy requirements for the calibration target. Simulations and real experiments show that the method has good accuracy and robustness. Disclosure statement No potential conflict of interest was reported by the authors. Funding The work was supported by a program of Hunan Department of Education outstanding youth project (17B086), and a program of Hunan Police Academy Doctoral project (2016ZX02). Acknowledgement All members of the subject actively participated in the research, and we hereby thank them all. References [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20] [21] [22]
G.J. Zhang, Vision Measurement, Science Publisher, Beijing, 2008. J. Xue, X. Su, Camera calibration with single image based on two orthogonal one-dimensional objects, Acta Optica Sinica 32 (1) (2012) 0115001. R. Hartley, Self-calibration of stationary cameras, Int. J. Comput. Vision 22 (1) (1997) 5–23. S.J. Manbank, O.D. Faugeras, A theory of self-calibration of a moving camera, Int. J. Comput. Vision 8 (2) (1992) 123–151. B. Caprile, V. Torre, Using vanishing points for camera calibration, Int. J. Comput. Vision 4 (2) (1990) 127–140. P. Beardsley, D. Murray, Camera calibration using vanishing points, Proc. Br. Mach. Vision Conf. (1992) 416–425. G.H. Wang, H.T. Tsui, Z.Y. Hu, et al., Camera calibration and 3D reconstruction from a single view based on scene constraints, Image Vision Comput. 23 (2005) 311–323. A. Chen, C. Gao, B. He, Camera calibration method based on orthogonal vanishing point pair, Chin. J. Sci. Instrum. 33 (1) (2012) 161–166. J. Huo, W. Yang, M. Yang, A self-calibration technique based on the geometry property of the vanish point, Acta Optica Sinica 30 (2) (2010) 465–472. C. Ricolf-Viala, A.J. Sanchez-Salmeron, E. Martinez-Berti, Accurate calibration with highly distorted images, Appl. Opt. 51 (1) (2012) 89–101. F. Wu, Mathematical Methods in Computer Vision, Science Press, Beijing, 2008, pp. 86–111. B.W. He, Y.F. Li, A novel method for camera calibration using vanishing points, 14th International Conference on M2VIP (2007) 44–47. H. Yang, S. Zhang, C. Liu, Research into camera calibration methods based on the vanishing points theory and planar control fields, J. Image Graphics 15 (8) (2010) 1168–1174. Y. Hong, X. Sun, M. Cai, et al., An intrinsic parameters self-calibration technique based on infinite homography between orthogonal vanishing points, Acta Optica Sinica 42 (12) (2015) 1208001. M. Cai, X. Sun, S. Liu, et al., An accurate & real-time focal-length self-calibration method based on infinite homography between vanish points, Acta Optica Sinica 34 (5) (2014) 0515003. S. Xu, X. Sun, X. Liu, et al., Geometry method of camera self-calibration based on a rectangle, Acta Optica Sinica 34 (11) (2014) 1115002. F. Xiaofeng, P. Difu, Study of single camera stereo vision sensor based on plane mirror imaging, Acta Optica Sinica 34 (9) (2014) 0915002. F. Xiaofeng, P. Difu, Research on the application of single camera stereo vision sensor in three-dimensional point measurement, J. Modern Optics 62 (15) (2015) 1204–1210. S. Xu, X. Sun, S. Liu, et al., Model reference approaching method of camera distortion calibration, Acta Optica Sinica 33 (7) (2013) 0715001. H. Opower, Multiple view geometry in computer vision, Optics Lasers Eng. 37 (1) (2002) 85–86. R. Hartley, Multiple View Geometry in Computer Vision, 2nd ed., Cambridge University Press, 2003. J. Lu, H. Sun, Z. Chang, A novel method for camera calibration with orthogonal vanishing points, Chin. J. Lasers 41 (2) (2014) 0208001.