Camera calibration from very few images based on soft constraint optimization

ARTICLE IN PRESS

JID: FI

[m1+;February 25, 2020;13:48]

Available online at www.sciencedirect.com

Journal of the Franklin Institute xxx (xxxx) xxx www.elsevier.com/locate/jfranklin

Camera calibration from very few images based on soft constraint optimization Hongjun Zhu a,b,d,1,∗, Yan Li b,1, Xin Liu a,d, Xuehui Yin a,d, Yanhua Shao c, Ying Qian a,d,∗, Jindong Tan b a School

of Software Engineering, Chongqing University of Posts and Telecommunications, Chongqing City, 400065, China b Department of Mechanical, Aerospace, and Biomedical Engineering, University of Tennessee, Knoxville, 37919, USA c School of Information Engineering, Southwest University of Science and Technology, Mianyang, 621010, China d Chongqing Engineering Research Center of Software Quality Assurance, Testing and Assessment, Chongqing 400065, China Received 3 December 2018; received in revised form 16 December 2019; accepted 5 February 2020 Available online xxx

Abstract Camera calibration is a basic and crucial problem in photogrammetry and computer vision. Although existing calibration techniques exhibit excellent precision and flexibility in classical cases, most of them need from 5 to 10 calibration images. Unfortunately, only a limited number of calibration images and control points can be available in many application fields such as criminal investigation, industrial robot and augmented reality. For these cases, this paper presented a two-step calibration based on soft constraint optimization, which is motivated by "no free lunch" theorem and error analysis. The key steps include (1) homography estimation with weighting function, (2) Initialization based on a simplified model, and (3) soft constraint optimization in terms of reprojection error. The proposed method provides direct access to geometric information of the object from very few images. After extensive experiments, the results demonstrate that the proposed algorithm outperforms Zhang’s algorithms from the point of view of the success ratio, accuracy and precision. © 2020 The Franklin Institute. Published by Elsevier Ltd. All rights reserved.

∗ Corresponding authors at: School of Software Engineering, Chongqing University of Posts and Telecommunications, Chongqing City, 400065, China. E-mail addresses: [email protected] (H. Zhu), [email protected] (Y. Qian). 1 Both authors contributed equally to this work.

https://doi.org/10.1016/j.jfranklin.2020.02.006 0016-0032/© 2020 The Franklin Institute. Published by Elsevier Ltd. All rights reserved.

Please cite this article as: H. Zhu, Y. Li and X. Liu et al., Camera calibration from very few images based on soft constraint optimization, Journal of the Franklin Institute, https:// doi.org/ 10.1016/ j.jfranklin.2020.02.006

ARTICLE IN PRESS

JID: FI

2

[m1+;February 25, 2020;13:48]

H. Zhu, Y. Li and X. Liu et al. / Journal of the Franklin Institute xxx (xxxx) xxx

1. Introduction Camera calibration is the estimation process of the intrinsic and extrinsic camera parameters [1,2]. The extrinsic parameters represent the 3D position and orientation of the camera frame with respect to the world reference frame [3]. While the intrinsic parameters reflect the internal camera geometric and optical characteristics, which are invariant if the focal length is fixed [4]. With the development of theory and technique, camera calibration has been being widely applied in various fields such as biology, materials, photon imaging, physical measurement and object detection. At the same time, many problems emerged in actual conditions. One of them is calibration from a few images, which has received significant attention due to the fact that very small observed data can be available for many cases such as criminal investigation, industrial robot [5] and augmented reality [6], and that once the parameters are known, it is easy to obtain the position and geometry of the object from its image, or the relative position of the camera. Unfortunately, for the case that no enough data are available, most of conventional calibrations, which tend to focus on a general calibration purpose, do not perform well. According to "no free lunch" theorem [7], matching algorithms to problems gives higher average performance than applying a fixed algorithm to all. Following this idea, one of the solutions to calibration from very few images is to introduce constraints into the calibration process. To this end, some researchers brought geometric constraints into camera calibration through the scene that satisfies a Manhattan world assumption, that is, the scene contains three orthogonal, dominant directions [8]. For example, by using cuboids, de la Fraga et al. [9] proposed direct calibration method based on differential evolution. Similarly, Wilczkowiak et al. [10] took a parallelepiped as a calibration object. The results demonstrated that the relative errors of intrinsic parameters are no bigger than 7%. Using cylindrical object, Winkler and Zagar [11] fitted extrinsic parameters. The testing results showed that this method is valid. Chen [12] realized camera calibration from four coplanar corresponding points and a noncoplanar one, of which the performance is better than that of Miyagawa’s approach [13]. On the other hand, many people exploited coplanar calibration object to impose geometric constraints. Avinash and Murali [14] employed a rectangular prism to generate two vanishing points and then to determine focal length. The performance of this method compares favorably to the Zhang’s algorithm [15]. Similarly, Dan Liu et al. [16] presented a calibration method for a camera with lens distortion from a single image including vanishing points and ellipses. However, for such methods, the accuracy is dependent upon the vanishing points [16]. To resolve this dilemma, without vanishing points, Zhou et al. [17] achieved comparable measurement accuracy relative to the traditional method by adopting a single image including at least three squares with unknown length. Using two arbitrary coplanar circles, Zheng and Liu [18] proposed a closed-form solution of the focal length and the extrinsic parameters. The result can serve as the initial value of camera parameters for iterative optimization. Furthermore, Miyagawa et al. [13] proposed a technique to estimate parameters from five points on two orthogonal 1-D objects. The calibration results are close to those yielded by existing methods. In addition, many other constraints have been suggested, which can bring considerable benefits for camera calibration. For instance, a radial alignment constraint was enforced on camera calibration by Tsai [4]. A planarity constraint was employed by Zhang [15]. Miraldo and Araujo [19] introduced the smoothness constraint into camera calibration. Based on the Please cite this article as: H. Zhu, Y. Li and X. Liu et al., Camera calibration from very few images based on soft constraint optimization, Journal of the Franklin Institute, https:// doi.org/ 10.1016/ j.jfranklin.2020.02.006

JID: FI

ARTICLE IN PRESS

[m1+;February 25, 2020;13:48]

H. Zhu, Y. Li and X. Liu et al. / Journal of the Franklin Institute xxx (xxxx) xxx

3

epipolar constraint, Heller et al. [20] proposed a novel solution to the hand-eye calibration problem. To address the same issue, Liu et al. [21] presented a method based on rigidity constraints. In a word, despite a large amount of research appearing in the literature, accurate and flexible calibration from very few images has remained a difficult problem. This is essentially due to non-convex optimization arising from perspective distortions and insufficient data available to eliminate noise [22]. To solve this problem, constraint optimization is a promising way to achieve the global optimal solution. Based on the above idea, a novel calibration method is designed here especially for very few calibration images. This method involves three major steps: (1) homography matrix is estimated by the non-linear optimization with weighting function; (2) intrinsic and extrinsic camera parameters are initially computed based on the simplified model by decomposing homography matrix; (3) they are refined via a soft constraint optimization in terms of reprojection error. Comparative experiments demonstrate that the effectiveness of the proposed algorithm from the point of view of success ratio, precision and accuracy. In addition, the impact of shooting angle and control point number on parameter estimation was explored in this work. A pictorial display of error changes is presented with respect to the above factors. That can be employed as an action guide to tradeoff accuracy and cost for researchers and practitioners involved. The remainder of the paper is structured as follows: Section 2 introduces preliminary of camera calibration, Section 3 analyzes the main resource of calibration errors, Section 4 describes the camera calibration from very few images, Section 5 shows the experimental results, Section 6 discusses how the shooting angle of calibration objects and the number of control points impact the parameter estimation, and finally, Section 7 states the conclusions. 2. Preliminary of camera calibration In this section, we describe the perspective projection by using a pinhole camera model with geometrical distortion. 2.1. Pinhole camera model A camera is a mapping between a 3D object and its 2D image [23]. To introduce the fundamental concept on camera calibration, we consider a camera coordinate system (Xc ,Yc ,Zc ) shown in Fig. 1, in which the coordinate origin Oc is located at camera center, also known as the optical center, and Zc axis is perpendicular to the image plane. An arbitrary point in object space with coordinates Wc = (xc , yc , zc )T is mapped to the point w = (xic , yic )T on the image plane. The perspective projection based on the pinhole camera model without distortion can be formulated as follows: xc yc zc = = (1) xic yic f where f indicates the camera’s focal length, that is, a distance from the image plane to the optical center, and zfc is a scale factor that relates the object to its image. In practice, the coordinate origin in the image plane is not necessarily the principal point (the intersection of the image plane and the principal axis, i.e., the Zc axis). For example, image coordinates are frequently given with the origin at the top-left of the image. In addition, Please cite this article as: H. Zhu, Y. Li and X. Liu et al., Camera calibration from very few images based on soft constraint optimization, Journal of the Franklin Institute, https:// doi.org/ 10.1016/ j.jfranklin.2020.02.006

ARTICLE IN PRESS

JID: FI

4

[m1+;February 25, 2020;13:48]

H. Zhu, Y. Li and X. Liu et al. / Journal of the Franklin Institute xxx (xxxx) xxx

Fig. 1. Pinhole camera geometry. Oc is the camera center and p the principal point. The camera center is here placed at the origin of camera coordinate system. For clarity, the image plane is virtually positioned in front of the pinhole plane.

a pixel rather than a unit of length is conventionally adopted in the image coordinate system. For this reason, a more complex relationship needs to be expressed. Let m = (u, v )T and m0 = (u0 , v0 )T be the image of the object point Wc and the principal point in pixel unit, and the number of pixels per unit distance in image coordinates are mx and my in the x and y directions. If the x- and y-axes of the 2D image coordinate system (in Fig. 1) are parallel to the Xc - and Yc -axes, respectively, the central projection mapping from world coordinates to image coordinates can be expressed as follows: ⎛ ⎞ ⎡ ⎤⎛ x c ⎞ u f mx s uo 0 ⎜ ⎟ ⎢ ⎥⎜ y c ⎟ ⎟ f m y v o 0 ⎦⎜ zc ⎝ v ⎠ = ⎣ 0 (2) ⎝ z ⎠, c 1 0 0 1 0 1 Notice that zc in Eq. (2) cannot vanish anywhere. For added generality, we consider a skewing s of the pixel elements in the CCD array for the case that the x- and y-axes are not strictly perpendicular. By writing ⎡ ⎤ f mx s uo ⎢ ⎥ f m y v o ⎦, A=⎣ 0 (3) 0 0 1 expression (2) has the concise form ˜ c, zc m ˜ = A[I |0 ]W

(4)

˜ c = (xc , yc , zc , 1 )T are the homogeneous coordinates of m and where m ˜ = (u, v, 1 )T and W Wc . The matrix A is referred to as the camera intrinsic matrix, which encodes the intrinsic parameters [24–26]. As a matter of fact, the physical position of a 3D scene point is known not in the camera coordinate system but in the world coordinate system. In other words, in general, the origin of the camera coordinate system does not coincide with the one of the world coordinate system (X,Y,Z). At the same time, the optical axis is typically not exactly aligned with the Z-axis. For this reason, a transformation matrix is required to transform the points W = (x, y, z )T Please cite this article as: H. Zhu, Y. Li and X. Liu et al., Camera calibration from very few images based on soft constraint optimization, Journal of the Franklin Institute, https:// doi.org/ 10.1016/ j.jfranklin.2020.02.006

ARTICLE IN PRESS

JID: FI

[m1+;February 25, 2020;13:48]

H. Zhu, Y. Li and X. Liu et al. / Journal of the Franklin Institute xxx (xxxx) xxx

from the world ⎛ ⎞ ⎡ xc r11 ⎜yc ⎟ ⎢r21 ⎜ ⎟=⎢ ⎝z ⎠ ⎣r31 c 1 0 If we set ⎡ r11 r12 R = ⎣r21 r22 r31 r32

coordinates into the camera coordinates as follows ⎤⎛ ⎞ r12 r13 t1 x ⎥ ⎜ r22 r23 t2 ⎥⎜y ⎟ ⎟. r32 r33 t3 ⎦⎝ z ⎠ 0 0 1 1 r13 r23 r33

⎤ t1 t 2 ⎦, t3

5

(5)

(6)

and plug (5), (6) into Eq. (4), the perspective projection with an ideal camera can be rewritten as: ˜, zc m ˜ = ARW

(7)

where R is known as camera extrinsic matrix, which is composed by a 3 × 3 rotation matrix and a translation vector [27]. Here, H = AR is called perspective projection matrix [28]. If points W are located on the same plane, the perspective projection matrix can reduce to a 3 × 3 homography matrix [15]. 2.2. Geometrical distortion model Strictly speaking, cameras are not perfect and hence sustain a variety of aberrations. The expression (7), therefore, does not hold true for an actual camera. Unfortunately, it is difficult to design such a distortion model that is very well suited even for a given camera. One reason is that all kinds of distortion are not independent from one another. Another reason comes from the dilemma of optimal balance between performance and complexity of different models [29]. Specifically, a plain distortion model may fail to truly express the complex relationship between an object and its image, which lead to a high risk of suffering a big deviation of parameter estimation. On the other hand, a complicated expression usually involves high order equations as well as parameter redundancy, which can make the camera calibration lose uniqueness especially for small data sets. It is now widely accepted that lens distortion has to be considered to achieve an accurate calibration. The major distortion can be divided into three types: radial distortion, decentering distortion and thin prism distortion [2]. Among them, the last two components can be integrated into one expression [30]. For this reason, only two kinds of the distortions are considered here and then real (distorted) pixel image coordinates can be described analytically by 

u = u + ur + ud , 

v = v + vr + vd .

(8) (9)

Since the higher order coefficients of radial distortions are very small, which can be neglected without any significant accuracy loss, the amount of radial distortion can be defined by Liu et al. [31]     2  ur = (u − u0 ) k1 (u − u0 )2 + (v − v0 )2 + k2 (u − u0 )2 + (v − v0 )2 , (10) Please cite this article as: H. Zhu, Y. Li and X. Liu et al., Camera calibration from very few images based on soft constraint optimization, Journal of the Franklin Institute, https:// doi.org/ 10.1016/ j.jfranklin.2020.02.006

ARTICLE IN PRESS

JID: FI

6

[m1+;February 25, 2020;13:48]

H. Zhu, Y. Li and X. Liu et al. / Journal of the Franklin Institute xxx (xxxx) xxx

    2  vr = (v − v0 ) k1 (u − u0 )2 + (v − v0 )2 + k2 (u − u0 )2 + (v − v0 )2 ,

(11)

where k1 , k2 are radial distortion coefficients. The decentering distortion is considered here and defined by Wang et al. [30]:   ud = k3 3(u − u0 )2 + (v − v0 )2 + 2k4 (u − u0 )(v − v0 ),

(12)

  vd = k4 3(v − v0 )2 + (u − u0 )2 + 2k3 (u − u0 )(v − v0 ),

(13)

where k3 , k4 are decentering distortion coefficients. 3. Error analysis The calibration process is fraught with five potential sources of error: (1) suitability of the model for the given camera, (2) working accuracy of calibration objects, (3) positional accuracy of feature detection, (4) pose of calibration objects with respect to the camera, and (5) performance of calibration algorithm [32]. Camera models and distortion models have been discussed by Weng [2], Wang [30], and so on. The working accuracy depends on the manufacturing process and hence it is not energetically explored here. The measurement accuracy of calibration markers in the image has come up to 0.024 and 0.043 pixels for high and low contrast pictures, respectively [32]. As far as the pose is concerned, it is proved that the more parallel to the image plane the calibration plane is, the higher the calibration error is [33]. In this work, our discussion will be confined only to the calibration algorithm. Depending on the calculation strategy, the existing calibration algorithms can be classified roughly into three categories: (1) direct linear transformation, (2) direct nonlinear optimization, (3) two-step methods [2]. Direct linear transformation is of low computational complexity as well as has unique solution. However, camera calibration is an ill-posed problem, of which the solution is highly sensitive to the changes of observed data. And, the presence of noise is virtually inevitable. Therefore, the accuracy of the final solution may be relatively poor with the straightforward linear transformation. On the other side, direct nonlinear optimization is likely to achieve a high accuracy. Unfortunately, camera calibration is essentially a non-convex optimization problem. Consequently, the result tends to plunge into improper local optima and therefore diverges from the ground truth if the initial guess is unsuitable [34]. In reality, there is no effective method for solving the general nonlinear optimization problems, and even very simple problems can be extremely challenging [35]. Unlike previous strategy, two-step methods, see [2,4,15] for example, generally calculate the initial values of camera parameters by the direct linear transformation, and then they are refined by nonlinear optimization. The reasonableness of such methods can be demonstrated as follows. If we set fx = f mx and fy = f my , expression (3) is then rewritten as ⎡ ⎤ f x s uo f y vo ⎦ A = ⎣0 (14) 0 0 1 By abuse of notation, we still use fx and fy to denote the focal length in the x and y directions. Although their values are different from the focal length of the pin-hole camera model, they are usually in the same order. Please cite this article as: H. Zhu, Y. Li and X. Liu et al., Camera calibration from very few images based on soft constraint optimization, Journal of the Franklin Institute, https:// doi.org/ 10.1016/ j.jfranklin.2020.02.006

JID: FI

ARTICLE IN PRESS

[m1+;February 25, 2020;13:48]

H. Zhu, Y. Li and X. Liu et al. / Journal of the Franklin Institute xxx (xxxx) xxx

7

Fig. 2. The difference between the perspective projections of the object points and the corresponding image points: (a) the differences vary with fx and fy ; (b) local minima vary with fx .

If the distortion cannot be ignored, the camera calibration becomes a far more intricate problem. For simplicity, we assume that object points, corresponding image points and all the parameters except focal length are known and the calibration object remains stationary. When we changed fx and fy around the ground truth, the difference between the perspective projections of the object points and the corresponding image points was observed in the distortion condition. We found that the differences varying with focal length display a valley form as shown in Fig. 2(a). It is apparent that there exists a local minimum when fx varies and fy is fixed, or fy varies and fx is fixed. Moreover, when fx is fixed and fy changes, the local minima for various fx are shown in Fig. 2(b), and a similar profile for fy . Obviously, in the parameter space, there are several local minima. A local minimum does not necessarily correspond to the right solution. That is why the calibration method based on nonlinear optimization can easily get stuck into improper local minima. Unless a proper initial guess is provided, correct convergence is hard to reach. It is worth noting that a closer initial value does not mean a better result. Unlike direct nonlinear optimization, the two-step method initially estimates camera parameters in the first step, which determines a reasonable starting point of iterative optimization. Therefore the result is more likely to be globally optimal. By comparative experiment, Salvi et al. [36] demonstrated that the two-step calibration outperforms the direct linear transformation method and the direct nonlinear optimization method. That is why this work focuses only on the two- step calibration here. 4. Calibration from very few images In this section, we proposed a two-step method based on soft constraint optimization to achieve better performance of camera calibration from very few data. 4.1. Parameter initialization For nonlinear optimization, a robust initial estimation is desired. And, all the method based on Zhang’s algorithm need calculate the homography between the model plane and its image Please cite this article as: H. Zhu, Y. Li and X. Liu et al., Camera calibration from very few images based on soft constraint optimization, Journal of the Franklin Institute, https:// doi.org/ 10.1016/ j.jfranklin.2020.02.006

ARTICLE IN PRESS

JID: FI

8

[m1+;February 25, 2020;13:48]

H. Zhu, Y. Li and X. Liu et al. / Journal of the Franklin Institute xxx (xxxx) xxx

Fig. 3. Image points and the corresponding projections computed with homography matrix. Blue spots are image points with distortion and red circles indicate distortion-free projections of the control points. Note that the normalization procedure is applied to the coordinates of image points. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

at first. Then, the homography matrix can be decomposed into the intrinsic and extrinsic parameters. 4.1.1. Homography estimation There are many ways to estimate the homography matrix [15]. However, the resulting accuracy is heavily impacted by the lens distortion especially for the calibration from very few images. To alleviate this problem, a weighted optimization approach is presented here. Suppose that decentering distortion and thin prism distortion can be neglected, which is normally true for an ordinary camera [4,37,38]. Only radial distortion is considered here. To get an optimal estimation of homography H , it seems obvious that the objective function should be defined as the total distance between the image points and their corresponding projections of the control points. That causes every point to obtain a given deviation with an equal possibility. However, the radial distortion is positively related with the radial distance from the principle point [2], that is, the farther the distance from the principle point is, the more the deviation is. In this sense, objective function should be formalized as    2 n    mi − m i  (15)    c + ri  i=1 

where mi and mi are the image point and the distortion-free projection of the ith control point, ri is the distance from mi to (uo, vo), and c is a positive constant, which is small enough, to make sure the value of the denominator is not zero. If it is far larger than the mean of the  2  distance ri , the objective function degrades into ni=1 mi − mi  and then the information of the distance can’t be introduced into the algorithm. Using Levenberg–Marquardt method [39], the homography matrix can be estimated from object points and the corresponding image points. Fig. 3 shows image points and the corresponding projections computed with homography matrix. Here, a normalization procedure is Please cite this article as: H. Zhu, Y. Li and X. Liu et al., Camera calibration from very few images based on soft constraint optimization, Journal of the Franklin Institute, https:// doi.org/ 10.1016/ j.jfranklin.2020.02.006

ARTICLE IN PRESS

JID: FI

[m1+;February 25, 2020;13:48]

H. Zhu, Y. Li and X. Liu et al. / Journal of the Franklin Institute xxx (xxxx) xxx

9

applied to the image coordinates, that is, the abscissa and ordinate of the image points are divided by the width and height of the images, respectively. It is readily seen that the closer to center, the smaller the residual is. 4.1.2. Parameter estimation To obtain a perfect image quality, mx in expression (2) is typically as close to my as possible [3]. Moreover, the skew parameter s is very unlikely to be far away from 0 [33]. So, in the initial parameter estimation, one can set fx = fy = f and s = 0 in Eq. (10). Then fewer parameters need to be determined and hence less image data are required for the same accuracy level. Without loss of generality, we assume that the model plane is on z = 0 of the world coordinate system. Note that the Z- coordinate of an object in the world coordinate system differs from that in the camera coordinate system. From Eq. (7), we have ⎡ ⎤⎛ ⎞ ⎛ ⎞ ⎡ ⎤ r11 r12 r13 t1 x u f 0 uo 0 ⎢ ⎥ ⎜ ⎟ r r r t 21 22 23 2 ⎥⎜ y ⎟ z c ⎝ v ⎠ = ⎣ 0 f v o 0 ⎦⎢ (16) ⎣r31 r32 r33 t3 ⎦⎝0⎠. 1 0 0 1 0 0 0 0 1 1 That is equivalent to ⎛ ⎞ ⎡ ⎤⎡ u f 0 uo r11 zc ⎝v ⎠ = ⎣ 0 f vo ⎦⎣r21 1 0 0 1 r31

r12 r22 r32

⎤⎛ ⎞ t1 x t 2 ⎦⎝ y ⎠ . t3 1

˜ with z = 0 and Therefore, a control point W matrix H: ⎛ ⎞ ⎛ ⎞ ⎡ ⎤⎡ u x f 0 uo r11 1 ⎝v ⎠ = H ⎝y ⎠ with H = ⎣ 0 f vo ⎦⎣r21 zc 0 0 1 1 1 r31

(17) its image m ˜ are related by a homography r12 r22 r32

⎤ t1 t 2 ⎦. t3

(18)

The matrix H can be estimated with nonlinear optimization as described above. We assume the matrix H has previously been computed here. Let’s denote the ith column of rotation matrix and homography matrix by ri and hi , respectively. And, we set ⎡ ⎤ f 0 uo A = ⎣ 0 f v o ⎦. (19) 0 0 1 Then, Eq. (18) can be rewritten as  h1

h2

 1  h3 = A r 1 zc

r2

 t .

(20)

Since the column vectors of rotation matrix are orthonormal [15], we can derive hT1 A−T A−1 h2 = 0,

(21)

hT1 A−T A−1 h1 = hT2 A−T A−1 h2 .

(22)

Please cite this article as: H. Zhu, Y. Li and X. Liu et al., Camera calibration from very few images based on soft constraint optimization, Journal of the Franklin Institute, https:// doi.org/ 10.1016/ j.jfranklin.2020.02.006

ARTICLE IN PRESS

JID: FI

10

[m1+;February 25, 2020;13:48]

H. Zhu, Y. Li and X. Liu et al. / Journal of the Franklin Institute xxx (xxxx) xxx

Let the ith column vector of H be hi ⎡ 1 −u0 0 f2 f2 −v0 1 ⎢ 2 B = A−T A−1 = ⎣ 0 f f2 −u0 f2

⎡

1 1 = 2⎣ 0 f −u

u02 f2

−v0 f2

0

0 1 −v0

+

v02 f2

= [ hi1 ⎤

hi2

hi3 ]T . If we set

⎥ ⎦ +1

⎤ −u0 ⎦, −v0 2 2 2 u0 + v0 + f

(23)

⎛

⎞T hi3 h j3   ⎜− h h + h h ⎟ ⎜ i3 j1 i1 j3 ⎟ vi j = ⎜  ⎟ , ⎝− hi2 h j3 + hi3 h j2 ⎠ hi1 h j1 + hi2 h j2  2 b = u0 + v02 + f 2 u0

v0

(24)  1,

(25)

then Eqs. (21) and (22) can be rewritten as   v12 b = 0. v11 − v22

(26)

When only one image can be available, we set u0 = Sx /2, v0 = Sy /2, where [Sx , Sy ] is the image size [40]. Then, focal length can be determined by v12 b = 0. If n(n ≥ 2 ) images are used, by stacking n such equations as Eq. (26), we have an overdetermined system of equations V b = 0.

(27) T

It is well known that the nontrivial solution to Eq. (27) is the eigenvector of V V associated with the smallest eigenvalue. Once b is determined, all camera intrinsic parameters can be computed according to Eq. (25). And, one can calculate the extrinsic parameters for each image by Zhang [15] ⎧ r1 = τ A−1 h1 ⎪ ⎪ ⎨ r2 = τ A−1 h2 (28) r3 = r1 × r2 ⎪ ⎪ ⎩ −1 t = τ A h3 with τ = 1/A−1 h1 . 4.2. Parameter refinement To refine parameter estimation, a non-linear method is introduced by minimizing the geometric distance, which is typically defined as the sum of the squared residuals between image points and projection points of 3D control points. To enhance robustness, an additional constraint is enforced here. Given n (equal to 1 or 2) images of the chessboard with m points, the maximum-likelihood estimate can be obtained by minimizing the following function  2 n  m    2 fx −  fy mi j − m ˆ A, Ri , ti , k, M j + n · exp (29) 2a2 i=1 j=1 Please cite this article as: H. Zhu, Y. Li and X. Liu et al., Camera calibration from very few images based on soft constraint optimization, Journal of the Franklin Institute, https:// doi.org/ 10.1016/ j.jfranklin.2020.02.006

JID: FI

ARTICLE IN PRESS

[m1+;February 25, 2020;13:48]

H. Zhu, Y. Li and X. Liu et al. / Journal of the Franklin Institute xxx (xxxx) xxx

11

Fig. 4. Flowchart of the complete two-step calibration.

where m ˆ (A, Ri , ti , k, M j ) is the projection in image i of the control point M j ,  fx and  fy are the estimated values of fx and fy , respectively. Here, a is a constant, which allows a slight rather than significant difference between  fx and  fy . The value of a indicates the probable maximum of the difference. The non-linear optimization can be done by Levenberg–Marquardt method [39]. 4.3. Complete two-step calibration In this section, the complete process of the two-step calibration is presented here. The flowchart, as shown in Fig. 4, can be divided into four steps: (1) Detect the feature points in the images and measure the 3D coordinate of the points in the chess pattern. (2) Compute the homography H by the weighted optimization as described in Section 4.1. (3) Estimate the initial parameters with the simplified model presented in Section 4.1. Please cite this article as: H. Zhu, Y. Li and X. Liu et al., Camera calibration from very few images based on soft constraint optimization, Journal of the Franklin Institute, https:// doi.org/ 10.1016/ j.jfranklin.2020.02.006

ARTICLE IN PRESS

JID: FI

12

[m1+;February 25, 2020;13:48]

H. Zhu, Y. Li and X. Liu et al. / Journal of the Franklin Institute xxx (xxxx) xxx

(4) Refine the parameters by the nonlinear optimization with soft constraint described in Section 4.2. 5. Experimental results In order to assess the robustness of the calibration strategy described in the previous sections and establish performance limitations of calibration algorithm under the various conditions, the experiments were performed on both synthetic data and real data. The evaluation procedure is implemented in MATLAB. All the experiments were run on a PC with Intel Core i5 6200U PC, 4GB RAM, and 64-bit Windows 10 operating system. The flowchart of the calibration algorithms is the same as Fig. 4. For the purpose of comparison, Zhang’s algorithm is implemented according to the original paper [15]. In the programming, we referred to the code at the website https:// github.com/ simonwan1980/ A- flexible- new- technique- for- camera- calibration. An optimal calibration method should obtain unbiased and minimum variance estimates of the camera parameters [41]. In fact, calibration error, that is, the difference between the estimated parameters and the true parameters at a suitable scale, is inevitable [42]. Therefore, it may be used to assess the performance of a calibration method for the synthetic data. However, it is invalid for the real data because ground truth is unavailable. Thus, the assessment criteria will have to vary with experimental conditions. The control points are the known points that are provided by the calibration object, which play an important role in the camera calibration. In the test on the real data, corner points are used as control points. Their coordinates need to be determined by corner detector. However, the coordinates are directly given in the test on the synthetic data. In the experiment, constant c in formula (15) was set to 1 and constant a in formula (29) was 50. 5.1. Test on synthetic data The experiment on the synthetic data makes it possible for a thorough investigation of the calibration performance and makes it easy to assess the performance without introducing some additional error sources such as feature detection. In this test, we employed three performance parameters: accuracy, precision and success ratio. The accuracy and precision here are measured by the mean and standard deviation of the difference between estimated parameters and ground truth, respectively. The success ratio refers to the ratio between the number of successful trials and that of total trials. Here, a trial refers to an experimental run, in which purposeful changes are made to the input variables [43]. And, a successful trial is defined as the case where more than 70% of the parameters have a relative error less than 30% and the result has interpretable physical meaning, that is, the case that the values are without an imaginary part, which is very important for virtual reality. Note that the error of a parameter is allowed to be up to 0.1 if the ground truth of the parameter is zero. 5.1.1. Synthetic data For convenience, a virtual camera was developed here, which allow us to study various aspects of camera calibration without having to laboriously capture numerous images. In addition, it is easy for the virtual camera to change camera parameters, and to control pose to make sure the image of a calibration object is complete. Please cite this article as: H. Zhu, Y. Li and X. Liu et al., Camera calibration from very few images based on soft constraint optimization, Journal of the Franklin Institute, https:// doi.org/ 10.1016/ j.jfranklin.2020.02.006

JID: FI

ARTICLE IN PRESS

[m1+;February 25, 2020;13:48]

H. Zhu, Y. Li and X. Liu et al. / Journal of the Franklin Institute xxx (xxxx) xxx

13

To recover stably the internal and external camera parameters, it is required that the total number of control points is at least 60 and more than two images are nonparallel [4,33]. However, the test here is designed to explore the performance of the presented method on the bad conditions. For this reason, the trials are deliberately run under the worst conditions. Specifically, the number of control points started at 6 × 4 rather than 60 in the test. Moreover, the number of calibration images is no more than 2 for each trial. In this test, images were ‘taken’ from various angles ranged from 1.2 to 60° by the virtual camera. For a given shooting angle, that is, the angle between the principal axis and the normal of the calibration board, there are a series of images yielded, in which control points arranged into 6 rows while the number of points in every row varied from image to image. Notice that the shooting angle is not allowed to be zero, or else the performance would be seriously deteriorated [33]. In the test, the given parameters are set as close to real data as it can. The image resolution is 1200 × 1200. The simulated camera has the following property: fx = 4000, fy = 4010, s = 0, u0 = 600, v0 = 600, k1 = 2.3 × 10−7 , k2 = 2.8 × 10−15 , k3 = 0, k4 = 0. The intrinsic camera parameters and distortion parameters are fixed. The displacement which arises from the radial distortion is up to 35.67 pixels. 5.1.2. Test on a single view In this test, up to 1350 calibration trials ran on the test images without noise. The numbers of successful trials are 126 for Zhang’s algorithm and 608 for our algorithm. The success

Fig. 5. Success frequencies of camera calibration from a single image without noise: (a) distribution pattern of successful example of Zhang’s algorithm (bottom) and our algorithm (top); (b) frequency of the successful trials with regard to the number of points; (c) frequency of the successful trials with respect to various shooting angles. Note that orange points indicate success while blue ones represent failure in (a), and that, in (b) and (c), the black curve indicates Zhang’s algorithm while the blue one represents the algorithm proposed here. Please cite this article as: H. Zhu, Y. Li and X. Liu et al., Camera calibration from very few images based on soft constraint optimization, Journal of the Franklin Institute, https:// doi.org/ 10.1016/ j.jfranklin.2020.02.006

JID: FI

14

ARTICLE IN PRESS

[m1+;February 25, 2020;13:48]

H. Zhu, Y. Li and X. Liu et al. / Journal of the Franklin Institute xxx (xxxx) xxx

Fig. 6. Accuracy and precision of Zhang’s algorithm and our algorithm tested on a single image: (a) mean of calibration bias; (b) standard deviation of calibration bias. Note that yaw α, pitch β and tilt γ are Euler angles. OURS and ZHAL refer to our algorithm and Zhang’s algorithm, respectively.

ratios are 9.3% and 45%, respectively. Distribution patterns of successful example are depicted at Fig. 5(a), in which orange points indicate success while blue ones represent failure. It is readily seen that the success ratio of the proposed algorithm is much higher than that of Zhang’s algorithm. In order to investigate the dependency of success ratio, we carried out a set of single-factor experiments. The frequencies of successful trials vary with the number of control points as shown in Fig. 5(b). And, Fig. 5(c) presents the frequencies for a given shooting angle while the number of control points in each row varies from 4 to 30. From Fig. 5(a)–(c), it can be found that success ratio does not vary obviously with the shooting angle and the number of control points. To evaluate the accuracy and precision of the calibration algorithm, we calculated the mean and standard deviation of the difference between estimated parameters and ground truth, as shown in Fig. 6, from which it is readily seen that the proposed algorithm for a single view is more accurate and precise than Zhang’s algorithm. Note that yaw α, pitch β and tilt γ are Euler angles, which yield rotation matrix by the way described in detail by Tsai [4]. The mean and standard deviation of execution time of Zhang’s algorithm are 0.0286 s and 0.0105 s. Those of the proposed algorithm are 0.0804 s and 0.0222 s, respectively. Please cite this article as: H. Zhu, Y. Li and X. Liu et al., Camera calibration from very few images based on soft constraint optimization, Journal of the Franklin Institute, https:// doi.org/ 10.1016/ j.jfranklin.2020.02.006

JID: FI

ARTICLE IN PRESS

[m1+;February 25, 2020;13:48]

H. Zhu, Y. Li and X. Liu et al. / Journal of the Franklin Institute xxx (xxxx) xxx

15

Fig. 7. Success frequencies of camera calibration from two images without noise: (a) distribution pattern of successful example of Zhang’s algorithm (bottom) and our algorithm (top); (b) frequency of the successful trials with regard to number of points; (c) frequency of the successful trials with respect to various shooting angles. Note that orange points indicate success while blue points represent failure in (a), and that, in (b) and (c), black curve indicates Zhang’s algorithm while blue curve represents the algorithm proposed here. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

5.1.3. Test on two images In this test, two images are employed for every trial. And, one of them is taken from the shooting angle of 1.2°, but the other experimental parameters are the same as those in the preceding test. The testing results, as depicted in Fig. 7, show that the success rate of our algorithm is far better than that of Zhang’s algorithm. More specifically, our algorithm reaches 100% while Zhang’s is 42%. By comparing with the results tested on a single image, it is apparent that the number of images has large impact on the success ratio. However, shooting angle and number of control points are still not the influence factors of success ratio for camera calibration from two images. The mean and standard deviation of calibration bias of 19 parameters are shown in Fig. 8. The results demonstrate that the accuracy and precision of our algorithm are a lot higher than that of Zhang’s algorithm. It is worth noting that there are two sets of extrinsic parameters corresponding to two calibration images. The mean and standard deviation of run time of Zhang’s algorithm are 0.0220 s and 0.0215 s. Those of the proposed algorithm are 0.1817s and 0.0730 s, respectively. 5.1.4. Tests on quantization noise In a digital image, noise is inevitable. So, it is necessary to assess the availability of the algorithm proposed here in the noisy conditions. Although a digital image tends to be Please cite this article as: H. Zhu, Y. Li and X. Liu et al., Camera calibration from very few images based on soft constraint optimization, Journal of the Franklin Institute, https:// doi.org/ 10.1016/ j.jfranklin.2020.02.006

ARTICLE IN PRESS

JID: FI

16

[m1+;February 25, 2020;13:48]

H. Zhu, Y. Li and X. Liu et al. / Journal of the Franklin Institute xxx (xxxx) xxx

Fig. 8. Accuracy and precision of Zhang’s algorithm and our algorithm tested on two images without noise: (a) mean of calibration bias; (b) standard deviation of calibration bias. Note that yaw α, pitch β and tilt γ are Euler angles. OURS and ZHAL refer to our algorithm and Zhang’s algorithm, respectively.

corrupted by various types of noise, quantization noise is the major influence factor on the imaging location. In spatial quantization, the maximum error is half a pixel [44]. For this reason, we introduced quantization noise into the data by rounding to the nearest integer. In this test, two images are used every time. And, we also assessed the success ratio, precision and accuracy of camera calibration, as shown in Figs. 9 and 10. By comparison, the results on the noisy data are relatively similar to those on the noise-free data. However, the presence of noise significantly degrades the performance of calibration algorithms. The success frequencies of Zhang’s algorithm decreases from 567 to 540, but our algorithm stays the same. The mean and standard deviation of runtime of Zhang’s algorithm are 0.0287 s and 0.0213 s. Those of the proposed algorithm are 0.1925 s and 0.0784 s, respectively. 5.2. Test on real data Real data tend to differ from synthetic data. The experiments on real data can test the validity of the overall calibration method in the application field, where the data are captured Please cite this article as: H. Zhu, Y. Li and X. Liu et al., Camera calibration from very few images based on soft constraint optimization, Journal of the Franklin Institute, https:// doi.org/ 10.1016/ j.jfranklin.2020.02.006

JID: FI

ARTICLE IN PRESS

[m1+;February 25, 2020;13:48]

H. Zhu, Y. Li and X. Liu et al. / Journal of the Franklin Institute xxx (xxxx) xxx

17

Fig. 9. Success frequencies of camera calibration from two images with quantization noise: (a) distribution pattern of successful example of Zhang’s algorithm (bottom) and our algorithm (top); (b) frequency of the successful trials with regard to number of points; (c) frequency of the successful trials with respect to various shooting angles. Note that orange points indicate success while blue points represent failure in (a), and that, in (b) and (c), black curve indicates Zhang’s algorithm while blue curve represents the algorithm proposed here. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

with real setups [2].In this sense, the algorithm proposed here was tested under real conditions. The testing results are reported in this section. Without ground truth, it is difficult to measure directly the estimated errors. So, the accuracy of calibration algorithms are usually evaluated by means of the reprojection error, that is, the difference between the projection of each control point and its corresponding image [40]. And, the standard deviations of reprojection errors are measured to assess the precision of camera calibration. However, the statistical quantity of extrinsic parameters cannot serve as evaluation criteria for each image with separate extrinsic parameters. In this test, a digital camera Canon EOS 80D with the resolution 2976 × 1984 is utilized to take pictures. A chessboard with square size 12.7 mm, which has 15 × 15 corner patterns, was photographed from 20 different shooting angles. In this process, the focal length was fixed for all images. Fig. 11 shows one sample of the calibration pictures and the corners detected by the sub-pixel corner algorithm presented by Geiger et al. [45]. 5.2.1. Test on a single view In this test, we tried to calibrate a camera from a single image. Total 20 trials were conducted for each algorithm. The success ratio of our algorithm is 45%, which agrees with the result on the synthetic data. Unfortunately, all the trials on Zhang’s algorithm failed. This is essentially due to the fact that the solution to the system of Eq. (9) in Zhang’s paper [15] is Please cite this article as: H. Zhu, Y. Li and X. Liu et al., Camera calibration from very few images based on soft constraint optimization, Journal of the Franklin Institute, https:// doi.org/ 10.1016/ j.jfranklin.2020.02.006

JID: FI

18

ARTICLE IN PRESS

[m1+;February 25, 2020;13:48]

H. Zhu, Y. Li and X. Liu et al. / Journal of the Franklin Institute xxx (xxxx) xxx

Fig. 10. Accuracy and precision of Zhang’s algorithm and our algorithm tested on two images with quantization noise: (a) mean of calibration bias; (b) standard deviation of calibration bias. Note that yaw α, pitch β and tilt γ are Euler angles. OURS and ZHAL refer to our algorithm and Zhang’s algorithm, respectively.

Fig. 11. The chessboard with 15 × 15 corner patterns: (a) calibration pictures; (b) the detected sub-pixel corners.

unstably obtained through singular value decomposition. Because the total of unknowns is 5 in the equations, the solution of Eq. (9) becomes an underdetermined problem when the number of images is less than 3. Theoretically, the smallest eigenvalue of V T V should be very close to zero. In fact, it is frequently far away from zero when the number of images Please cite this article as: H. Zhu, Y. Li and X. Liu et al., Camera calibration from very few images based on soft constraint optimization, Journal of the Franklin Institute, https:// doi.org/ 10.1016/ j.jfranklin.2020.02.006

ARTICLE IN PRESS

JID: FI

[m1+;February 25, 2020;13:48]

H. Zhu, Y. Li and X. Liu et al. / Journal of the Franklin Institute xxx (xxxx) xxx

19

Table 1 Success frequency of camera calibration and statistical properties of reprojection errors. Parameters

Success frequency

Mean of errors

Standard deviation of errors

Zhang’s algorithm Our algorithm

0 9

6.67 4.47

1.245 0.661

Table 2 The mean and standard deviation of intrinsic parameters and distortion coefficients. Symbol

VMZ

VMO

VSZ

VSO

k1

2.41E − 05 −7.51E − 06i −5.60E − 11 +1.18E − 10i – – 1.87E + 04 +4.51E + 03i −2.46E + 03 +2.54E + 03i 2.88E + 04 +3.40E + 03i 1.39E + 03 +5.47E + 03i −3.92E + 03 +1.80E − 04i

6.18E − 07

1.13E − 04

5.30E − 06

−3.52E − 14

6.40E − 10

4.98E − 13

8.23E − 04 7.08E − 05 5.55E + 03

– – 3.15E + 04

1.89E − 03 1.01E − 03 8.00E + 02

−1.53E − 03

4.19E + 04

5.42E − 03

1.05E + 03

1.19E + 05

1.62E + 02

5.55E + 03

9.16E + 03

8.00E + 02

6.93E + 02

9.16E + 03

1.33E + 02

k2 k3 k4 fx s u0 fy v0

Note that subscript M, S, Z and O refer to mean, standard deviation, Zhang’s algorithm and our algorithm, respectively. Table 3 Success frequency of camera calibration and statistical properties of reprojection errors. Parameters

Success frequency

Mean of errors

Standard deviation of errors

Zhang’s algorithm Our algorithm

73 186

3.003 0.382

1.303 0.126

is very small. In that case, the corresponding eigenvector, typically with an imaginary part, is not the solution to the system of Eq. (9). For the purpose of comparison, the success frequency, mean and standard deviation of reprojection errors are given in Table 1. Table 2 tabulates the statistical properties of intrinsic parameters and distortion coefficients estimated by the proposed algorithm and Zhang’s algorithm, even Zhang’s results are out of physical meaning. The mean and standard deviation of execution time of Zhang’s algorithm are 0.0737 s and 0.0103 s. Those of the proposed algorithm are 0.1171s and 0.0251 s, respectively. 5.2.2. Test on two images In this part, we tested camera calibration from two images. Total 190 calibration trials were run. Among them, 186 trials are successful for our algorithm and 73 for Zhang’s algorithm. The success rate here is still very close to that on synthetic data. For clarity, the success frequency, mean and standard deviation of reprojection errors are tabulated in Table 3. And, Table 4 reports the mean and standard deviations of intrinsic parameters and distortion Please cite this article as: H. Zhu, Y. Li and X. Liu et al., Camera calibration from very few images based on soft constraint optimization, Journal of the Franklin Institute, https:// doi.org/ 10.1016/ j.jfranklin.2020.02.006

JID: FI

20

ARTICLE IN PRESS

[m1+;February 25, 2020;13:48]

H. Zhu, Y. Li and X. Liu et al. / Journal of the Franklin Institute xxx (xxxx) xxx

Fig. 12. A pictorial display of error changes with respect to shooting angle and control point number.

coefficients estimated by Zhang’s algorithm and our algorithm. From Tables 3 and 4, it is manifestly clear that our algorithm works better than Zhang’s algorithm in view of the success ratio, precision and accuracy. The mean and standard deviation of run time of Zhang’s algorithm are 0.5237 s and 0.9012 s. Those of the proposed algorithm are 0.4719 s and 0.0877 s, respectively. Please cite this article as: H. Zhu, Y. Li and X. Liu et al., Camera calibration from very few images based on soft constraint optimization, Journal of the Franklin Institute, https:// doi.org/ 10.1016/ j.jfranklin.2020.02.006

ARTICLE IN PRESS

JID: FI

[m1+;February 25, 2020;13:48]

H. Zhu, Y. Li and X. Liu et al. / Journal of the Franklin Institute xxx (xxxx) xxx

21

Table 4 The mean and standard deviation of intrinsic parameters and distortion coefficients. Symbol

VMZ

VMO

VSZ

VSO

k1 k2 k3 k4 fx s u0 fy v0

4.02E + 01 −2.32E + 02 – – 4.13E + 03 −8.34E + 02 1.35E + 03 3.11E + 03 −7.20E + 02

−3.56E − 07 1.54E − 14 8.95E − 05 1.90E − 04 5.41E + 03 2.37E − 06 1.50E + 03 5.41E + 03 1.00E + 03

3.12E + 03 2.18E + 03 – – 2.11E + 03 3.02E + 03 2.37E + 03 2.04E + 03 2.52E + 03

3.16E − 07 4.31E − 14 9.59E − 05 9.96E − 05 6.55E + 02 2.36E − 05 1.15E + 02 6.55E + 02 7.53E + 01

Note that subscript M, S, Z and O refer to mean, standard deviation, Zhang’s algorithm and our algorithm, respectively.

6. Discussion In this work, we also explored the impact of shooting angle and the number of control points on the accuracy of camera calibration by using synthetic data. To avoid additional calculation error, calibration parameters here are set to be as ‘simple’ as possible. The property of the simulated camera is specified as follows: fx = 1000, fy = 1050, s = 0, u0 = 1000, v0 = 1000, k1 = −2 × 10−7 , k2 = 2.5 × 10−15 , k3 = k4 = 0. The image resolution is 2000 × 2000. Calibration from two images carried out following the steps described in Section V-A. A pictorial display of the parameter errors is presented in Fig. 12. It can be observed that some of parameter errors decrease with the increase of the number of control points, as reported in [4]. However, the other parameters (for example, fx and fy ) are not affected. That is the reason why Zhang’s algorithm will not work if the other model planes are produced by a pure translation. In this case, the increase of calibration images is equivalent to increasing control points but in only one image. Note that, to make result clear, the testing images here are not corrupted by any kind of noise. For the noisy image, the results are similar but more fluctuant. Due to space limitations, they are not shown here.

7. Conclusion The flexible calibration from very few images has remained a difficult problem. To issue this problem, we presented a novel two-step calibration based on non-linear optimization. The algorithm exploits the weighted non-linear optimization to compute the homography and then the simplified camera model to get a reasonable initial guess. After that, the estimated parameters are refined by soft constraint optimization. To assess the performance, we carried out extensive experiments on both synthetic data and real data. The run time of the algorithm proposed here is statistically longer than that of Zhang’s algorithm. However, the results on a single image and two images demonstrate that the proposed algorithm performs much better than Zhang’s algorithm from the point of view of success ratio, accuracy and precision. In this sense, the increased reliability of camera calibration outweighs the added computational cost, especially for such application field as criminal investigation and industrial robot. Please cite this article as: H. Zhu, Y. Li and X. Liu et al., Camera calibration from very few images based on soft constraint optimization, Journal of the Franklin Institute, https:// doi.org/ 10.1016/ j.jfranklin.2020.02.006

ARTICLE IN PRESS

JID: FI

22

[m1+;February 25, 2020;13:48]

H. Zhu, Y. Li and X. Liu et al. / Journal of the Franklin Institute xxx (xxxx) xxx

An additional contribution of this paper is the finding that the influence factors of success ratio are not shooting angle and the number of control points, but the number of calibration images. That is why camera calibration from very few images is a challengeable problem. Moreover, this work presents a pictorial display of the impacts on the accuracy of camera calibration, which can provide reference for the researcher or practitioner applying camera calibration. Note that the method proposed here is limited to a common camera which satisfies fx ≈ fy and s ≈ 0. Although it is impossible to get an image with a pattern like chessboard in such applications as criminal investigation, a known object, like a coin, usually can be found in the image. It is easy to obtain the 2D relative coordinates of the features on the plane surface of the known object. In this sense, if a known object in an image has a plane surface, it is possible to obtain successful calibration based on only one image by using the proposed algorithm. Acknowledgments This work was sponsored by National Natural Science Foundation of China (No. 61701060, 61171060, 61801068); Natural Science Foundation of Chongqing, China (No.cstc2017jcyjAX0007, cstc2017jcyjAX0386, cstc2015jcyjA30001); Foundation of Chongqing Municipal Education Committee, China (No. 17SKG050, KJ1600410); and Sichuan Provincial Key Laboratory of Robot Technology Applied in Special Environment (Southwest University of Science and Technology), China (No. 17kftk02); Fundamental Research Funds for the Central Universities (No. 2019CDYGZD006). References [1] M.Z. Brown, D. Burschka, G.D. Hager, Advances in computational stereo, IEEE Trans. Pattern. Anal. Mach. Intell. 25 (8) (2003) 993–1008. [2] J. Weng, P. Cohen, M. Herniou, Camera calibration with distortion models and accuracy evaluation, IEEE Trans. Pattern. Anal. Mach. Intell. 14 (10) (1992) 965–980. [3] J.-.Y. Guillemaut, A.S. Aguado, J. Illingworth, Using points at infinity for parameter decoupling in camera calibration, IEEE Trans. Pattern. Anal. Mach. Intell. 27 (2) (2005) 265–270. [4] R. Tsai, A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf tv cameras and lenses, IEEE J. Robot. Autom. 3 (4) (1987) 323–344. [5] R.A. Boby, S.K. Saha, Single image based camera calibration and pose estimation of the end-effector of a robot, in: Proceedings of the IEEE International Conference on Robotics and Automation, Stockholm, SE, 2016. [6] I. Schillebeeckx, R. Pless, Single image camera calibration with lenticular arrays for augmented reality, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, 2016. [7] D.H. Wolpert, W.G. Macready, No free lunch theorems for optimization, IEEE Trans. Evol. Comput. 1 (1) (1997) 67–82. [8] J. Deutscher, M. Isard, J. Maccormick, Automatic camera calibration from a single manhattan image, in: Proceedings of the European Conference on Computer Vision, Copenhagen, DK, 2002. [9] L.G. de la Fraga, O. Schuetze, Direct calibration by fitting of cuboids to a single image using differential evolution, Int. J. Comput. Vis. 81 (2) (2009) 119–127. [10] M. Wilczkowiak, P. Sturm, E. Boyer, Using geometric constraints through parallelepipeds for calibration and 3D modeling, IEEE Trans. Pattern Anal. Mach. Intell. 27 (2) (2005) 194–207. [11] A.W. Winkler, B.G. Zagar, A curve fitting method for extrinsic camera calibration from a single image of a cylindrical object, Measur. Sci. Technol. 24 (8) (2013) 1–13. [12] H.T. Chen, Geometry-based camera calibration using five-point correspondences from a single image, IEEE Trans. Circ. Syst. Video Technol. 27 (12) (2017) 2555–2566. [13] I. Miyagawa, H. Arai, H. Koike, Simple camera calibration from a single image using five points on two orthogonal 1-D objects, IEEE Trans. Image Process. 19 (6) (2010) 1528–1538. Please cite this article as: H. Zhu, Y. Li and X. Liu et al., Camera calibration from very few images based on soft constraint optimization, Journal of the Franklin Institute, https:// doi.org/ 10.1016/ j.jfranklin.2020.02.006

JID: FI

ARTICLE IN PRESS

[m1+;February 25, 2020;13:48]

H. Zhu, Y. Li and X. Liu et al. / Journal of the Franklin Institute xxx (xxxx) xxx

23

[14] N. Avinash, S. Murali, Perspective geometry based single image camera calibration, J. Math. Imaging Vis. 30 (3) (2008) 221–230. [15] Z. Zhang, A flexible new technique for camera calibration, IEEE Trans. Pattern Anal. Mach. Intell. 22 (11) (2000) 1330–1334. [16] D. Liu, X. Liu, M. Wang, Camera self-calibration with lens distortion from a single image, Photogramm. Eng. Remote Sensing 82 (5) (2016) 325–334. [17] F. Zhou, et al., Line-based camera calibration with lens distortion correction from a single image, Opt. Lasers Eng. 51 (12) (2013) 1332–1343. [18] Y. Zheng, Y. Liu, Camera calibration using one perspective view of two arbitrary coplanar circles, Opt. Eng. 47 (6) (2008) 1–9. [19] P. Miraldo, H. Araujo, Calibration of smooth camera models, IEEE Trans. Pattern Anal. Mach. Intell. 35 (9) (2013) 2091–2103. [20] J. Heller, M. Havlena, T. Pajdla, Globally optimal hand-eye calibration using branch-and-bound, IEEE Trans. Pattern Anal. Mach. Intell. 38 (5) (2016) 1027–1033. [21] A. Liu, S. Marschner, N. Snavely, Caliber: camera localization and calibration using rigidity constraints, Int. J. Comput .Vis. 118 (1) (2016) 1–21. [22] C. Häne, et al., 3D visual perception for self-driving cars using a multi-camera system: calibration, mapping, localization, and obstacle detection, Image Vis. Comput. 68 (2017) 14–27. [23] R. Hartley, A. Zisserman, Multiple View Geometry in Computer Vision, Cambridge University Press, New York, 2004. [24] R. Szeliski, Computer vision: Algorithms and Applications, Springer, New York, 2010. [25] M. Pollefeys, R. Koch, L. Van Gool, Self-calibration and metric reconstruction inspite of varying and unknown intrinsic camera parameters, Int. J. Comput. Vis. 32 (1) (1999) 7–25. [26] B. Boudine, et al., A flexible technique based on fundamental matrix for camera self-calibration with variable intrinsic parameters from two views, J. Vis. Commun. Image Represent 39 (2016) 40–50. [27] L. Quan, Self-calibration of an affine camera from multiple views, Int. J. Comput. Vis. 19 (1) (1996) 93– 105. [28] Z. Zhang, Estimating motion and structure from correspondences of line segments between two perspective images, IEEE Trans. Pattern Anal. Mach. Intell. 17 (12) (1995) 1129–1139. [29] S. Tamaskar, K. Neema, D. Delaurentis, Framework for measuring complexity of aerospace systems, Res. Eng. Des. 25 (2) (2014) 125–137. [30] J. Wang, et al., A new calibration model of camera lens distortion, Pattern Recognit. 41 (2) (2008) 607–615. [31] Z. Liu, et al., High-accuracy calibration of low-cost camera using image disturbance factor, Opt. Express 24 (21) (2016) 24321–24336. [32] L. Krueger, C. Woehler, Accurate chequerboard corner localisation for camera calibration, Pattern Recognit. Lett. 32 (10) (2011) 1428–1435. [33] Z. Zhang, Flexible camera calibration by viewing a plane from unknown orientations, in: Proceedings of the IEEE International Conference on Computer Vision, Kerkyra , GR, IEEE, 1999. [34] J. Weng, T.S. Huang, N. Ahuja, Motion and structure from two perspective views: algorithms, error analysis, and error estimation, IEEE Trans. Pattern Anal. Mach. Intell. 11 (5) (1989) 451–476. [35] S. Boyd, L. Vandenberghe, Convex Optimization, Cambridge University Press, New York, 2004. [36] J. Salvi, X. Armangué, J. Batlle, A comparative review of camera calibrating methods with accuracy evaluation, Pattern Recognit. 35 (7) (2002) 1617–1635. [37] R. Hartley, S.B. Kang, Parameter-free radial distortion correction with center of distortion estimation, IEEE Trans. Pattern Anal. Mach. Intell. 29 (8) (2007) 1309–1321. [38] G.Q. Wei, S.D. Ma, Implicit and explicit camera calibration: theory and experiments, IEEE Trans. Pattern Anal. Mach. Intell. 16 (5) (1994) 469–480. [39] D.W. Marquardt, An algorithm for least-squares estimation of nonlinear parameters, J. Soc. Indust. Appl. Math. 11 (2) (1963) 431–441. [40] D. Douxchamps, K. Chihara, High-accuracy and robust localization of large control markers for geometric camera calibration, IEEE Trans. Pattern Anal. Mach. Intell. 31 (2) (2009) 376–383. [41] J. Heikkilä, Geometric camera calibration using circular control points, IEEE Trans. Pattern. Anal. Mach. Intell. 22 (10) (2000) 1066–1077. [42] P.R. Bevington, D.K. Robinson, Data Reduction and Error Analysis For the Physical Sciences, McGraw-Hill, New York, 2003.

Please cite this article as: H. Zhu, Y. Li and X. Liu et al., Camera calibration from very few images based on soft constraint optimization, Journal of the Franklin Institute, https:// doi.org/ 10.1016/ j.jfranklin.2020.02.006

JID: FI

24

ARTICLE IN PRESS

[m1+;February 25, 2020;13:48]

H. Zhu, Y. Li and X. Liu et al. / Journal of the Franklin Institute xxx (xxxx) xxx

[43] D.C. Montgomery, Design and Analysis of Experiments, John Wiley & Sons, Weinheim, 2008. [44] B. Kamgar-Parsi, Evaluation of quantization error in computer vision, IEEE Trans. Pattern Anal. Mach. Intell. 11 (9) (1989) 929–940. [45] A. Geiger, et al., Automatic camera and range sensor calibration using a single shot, in: Proceedings of the IEEE International Conference on Robotics and Automation, Paul, USA, 2012.

Please cite this article as: H. Zhu, Y. Li and X. Liu et al., Camera calibration from very few images based on soft constraint optimization, Journal of the Franklin Institute, https:// doi.org/ 10.1016/ j.jfranklin.2020.02.006