A spatial calibration method for master-slave surveillance system

A spatial calibration method for master-slave surveillance system

Optik 125 (2014) 2479–2483 Contents lists available at ScienceDirect Optik journal homepage: www.elsevier.de/ijleo A spatial calibration method for...

1MB Sizes 0 Downloads 80 Views

Optik 125 (2014) 2479–2483

Contents lists available at ScienceDirect

Optik journal homepage: www.elsevier.de/ijleo

A spatial calibration method for master-slave surveillance system Yu Liu ∗ , Hao Shi, Shiming Lai, Chenglin Zuo, Maojun Zhang College of Information Systems and Management, National University of Defense Technology, Deya Road, Changsha 410000, China

a r t i c l e

i n f o

Article history: Received 26 May 2013 Accepted 24 October 2013 Keywords: Master-slave camera Fish-eye panoramic camera PTZ dome camera Spatial calibration

a b s t r a c t The master-slave surveillance system uses stationary-dynamic camera assemblies to achieve wide-area surveillance and selective focus-of-interest. In such a system, a stationary fish-eye camera monitors a panoramic field of environment from a distance, while a dynamic PTZ (Pan–Tilt–Zoom) dome camera is commanded to acquire high-resolution images from a focused direction and provide multi-scale zoomedin information. In order to achieve the precise interaction, pre-processing spatial calibration between these two cameras is required. This paper introduces a novel vision based calibration approach to automatically calculate a transformation matrix model between the fish-eye coordinate system and the PTZ dome coordinate system by matching feature points in the scene. The experiment has demonstrated the effectiveness of using this method with high accuracy in short calibration time. Crown Copyright © 2013 Published by Elsevier GmbH. All rights reserved.

1. Introduction Recently, digital video surveillance has become prevalent in our daily life. Large numbers of monitoring cameras are used in public and private places such as government buildings, military bases, car parks, and jails. Traditional monitoring cameras can only cover a limited area, leading to “blind spots”. Developments in panoramic imaging technology offer significant advantages over traditional surveillance systems. For example, they can monitor an area that covers 180◦ or 360◦ and so replace several traditional monitoring cameras. However, this increase in area is at a cost of level of detail – images acquired with a panoramic camera have limited range of scale due to the wide view and the reduced resolution. Comparatively PTZ dome cameras can focus on object rapidly and scaling can be implemented by decreasing or increasing focal length. The master-slave camera composed of a fish-eye panoramic camera and PTZ dome camera combines the advantages of both approaches, and they have been applied in surveillance systems to achieve automatic multi-resolution data (Fig. 1). The fish-eye panoramic camera acts as the master, which is responsible for acquiring global, wide field-of-view information in the large surveillance area, as shown in Fig. 2(a). It is noted that upgrading from existing surveillance systems to a master-slave system is simple (shown in Fig. 2(b)). This system can automatically direct a slave PTZ dome camera(s) to zoom into target areas of

∗ Corresponding author. Tel.: +86 13975877517; fax: +86 731 82094657. E-mail addresses: [email protected] (Y. Liu), [email protected] (H. Shi), [email protected] (S. Lai), [email protected] (C. Zuo), [email protected] (M. Zhang).

interest, in which details of object appearance are available at a higher resolution. The main challenge in the application of the proposed system is to actively control a PTZ dome camera to correctly focus on the same target in the panoramic scene. The precision of this interaction largely depends of the accuracy of the spatial calibration, which can be considered as the mapping between each of pixels in fish-eye panoramic image and the Pan–Tilt angles of PTZ dome camera. A practical calibration method should not only be efficient and effective but also need no particular system setup or human intervention. Through calibration, we hope the target appointed by master camera can steer the slave camera to focus on the same position at the pixel level.

2. Related work A variety of master-slave camera systems have been proposed to facilitate video surveillance. These systems use various compositions of PTZ dome cameras and mixtures of other types of camera. Regardless of how the system is composed of camera types, the core issue is to develop a suitable calibration algorithm for accurate interaction between the cameras. The simplest and most direct way to precisely calibrate two cameras is to manually find every pixel in an image captured by one camera and correspond these with pixels in an image captured using another camera. A dense mapping such as this is impractical and it seriously limits the applicability. In practice, fewer points are required, which can be interpolated within some degree of accuracy. However, often the level of accuracy is not unacceptable. It has been shown experimentally that calibration based on 200 sample points may take around 2 h. Hence,

0030-4026/$ – see front matter. Crown Copyright © 2013 Published by Elsevier GmbH. All rights reserved. http://dx.doi.org/10.1016/j.ijleo.2013.10.100

2480

Y. Liu et al. / Optik 125 (2014) 2479–2483

Fig. 1. The fish-eye panoramic image captured by the master camera, and zoom-in image captured by slave camera.

a practical calibration method is required for master-slave camera systems. Hampapur et al. [1] used two or more calibrated cameras to triangulate a target’s position and determine the slave parameters for a third, PTZ dome camera that is calibrated to the same coordinate system. The master camera is manually calibrated by designating a number of target areas, or regions of interest (such as entrances and high traffic areas), in the master camera views, which are covered by the slave camera. Zhou et al. [2] selected a series of sample master pixel locations in the master camera. Operators manually move the slave camera to center the slave image at the corresponding real-world point and record the rotation angles of PTZ dome camera. Next they determine a mapping for the non-sampled points using a linear interpolation. This approach has two main defects: (a) a large number of sample points are required to achieve the accurate results, which is time consuming and inconvenient. (b) Linear interpolation is not a suitable image warping approach as it leads to many errors. Badri et al. [3] introduced a method which can automatically sample grids in the master camera image, find proper Pan–Tilt parameters of slave camera to observe every grid, and extend lookup table by interpolation. Nevertheless, the lookup table is still inaccurate for large image deformations. Li et al. [4] used a mosaic image created by snapshots from a slave camera to estimate the relationship between a static master camera plane and the Pan–Tilt controls of a slave camera. Though this is an efficient and automatic method, the mapping determined by a liner interpolation is inaccurate. The main drawback of these systems is the inaccuracy cause by the calibration algorithms, which becomes

Fig. 2. (a) The master-slave visual surveillance system, (b) most existing surveillance system can be upgraded to the master-slave system by adding a fish-eye camera above the traditional PTZ camera, or adding a PTZ dome camera underneath the panoramic camera.

increasingly serious when the PTZ dome camera zoom-in for high resolution. In this paper, a novel method is proposed to calibrate a masterslave camera. The solution is high-precision and time saving. This paper is organized as follows: The principle and details of this method are described in Section 3. In Section 4, the experimental result is demonstrated. The conclusion is conducted in Section 5. 3. Calibration The purpose of master-slave calibration is to determine the geometric relationship between the master camera image pixel coordinates M(X, Y) and the Pan–Tilt angles of the slave camera S(P, T). 3.1. Three coordinate systems establishing It is the basis of calibration to establish three coordinate systems: a panoramic coordinate system, a PTZ coordinate system and a spherical coordinate system. As a master-slave camera is composed of a fish-eye panoramic camera and a PTZ dome camera, the panoramic coordinate system is a fisheye coordinate system based on the fish-eye panoramic images, the PTZ coordinate system is a coordinate system based on PTZ camera taking pan angle and tilt angle as parameter, spherical coordinate system is an auxiliary coordinate system transforming from panoramic coordinate system to PTZ coordinate system. Fig. 3(a) illustrates the panoramic coordinate system. The x axis and the y axis represent horizontal and vertical direction, respectively. The optical center is O, and the OO is the optical axis. Let

Fig. 3. (a) Panoramic coordinate system; (b) PTZ coordinate system; (c) PTZ image coordinate system.

Y. Liu et al. / Optik 125 (2014) 2479–2483

2481

W and H be the panoramic image’s width and height, respectively, and let AngleW and AngleH be the panoramic image’s horizontal and vertical angle of view, respectively. The focal length of fish-eye panoramic camera is denoted as Cf =

H W = AngleW AngleH

(1)

(xf , yf ) denotes pixel coordinate in the panoramic image. As shown in Fig. 3(b), the PTZ coordinate system contains two parameters. p is a point on the surface of sphere. Let ˛ be the pan angle between Op and positive x-axis. It increases in the anticlockwise direction viewing from positive z-axis, which ranges from 0◦ to 359◦ . Let ˇ be the tilt angle which ranges from 0◦ to 89◦ . It is the angle between op and XOY plane and increases in the clockwise direction viewing from positive x-axis. The optical center of PTZ dome camera as the starting point, the section of hemisphere as XOY plane, the direction downward perpendicular to the XOY plane as z axis construct the PTZ coordinate system together. Fig. 3(c) illustrates the unit spherical coordinate system. Starting point O and every axis correspond to the PTZ dome coordinate system. The coordinate of point p is denoted as (xs , ys , zs ). ϕ is the angle between Op and positive x-axis which increases in the anticlockwise direction viewing from positive z-axis and ranges from 0◦ to 359◦ .  is the angle between Op and positive z-axis which increases in the anticlockwise direction viewing from positive xaxis and ranges from 0◦ to 89◦ . 3.2. Transformation from panoramic coordinate system to PTZ coordinate system

Fig. 4. (a) PTZ image coordinate system imaging; (b) PTZ camera’s pinhole.

As shown in Fig. 4(a), we can obtain the distance between point p and the center: Rf =



xf2 + yf2

Based on the fish-eye imaging model, the radical angle of point p off center is: =

Rf

(4)

Cf

According to the Rf and , the z axis of point p can be calculated. So the transformation is: xPs = Sxf

(5)

yPs = Syf After implementing the transformation from the panoramic coordinate system to the PTZ coordinate system, we can determine the mapping between each of pixels in the image captured by the fish-eye panoramic image and image captured by the Pan–Tilt angles of PTZ dome camera. However, the transformation cannot be calculated in a single step directly. We need to use spherical coordinate system to link two coordinate systems. The process is completed in three steps. Assume (xf , yf ) to be any pixel with homogeneous coordinates in the panoramic coordinate system with a corresponding spherical homogeneous coordinate (xPs , yPs , zPs ). Let the mapped homogeneous coordinate be (xp , yp , zp ) in the PTZ coordinate system and its corresponding spherical homogeneous coordinate be (xHs , yHs , zHs ). Establishing a mapping TPs→Hs between (xPs , yPs , zPs ) and (xHs , yHs , zHs ): (xHs , yHs , zHs ) = (xPs , yPs , zPs ) × TPs→Hs

(3)

(2)

TPs→Hs is a 3 × 3 matrix which denotes the transformation from panoramic spherical coordinate to PTZ dome spherical coordinate. Before the transformation, we need to make sure the two conditions of restriction that our master-slave camera must meet the following requirements: a. The camera’s optical axis is perpendicular to the image plane and point of intersection is at the center of the image plane. b. The fish-eye panoramic camera’s optical center coincides with the PTZ dome camera’s optical center. Beyond the conditions above, large errors may be produced in results. The TPs→Hs is obtained through a process of transformation among coordinates: a. Transformation from panoramic coordinate system to spherical coordinate system:

(6)



xf2 + yf2



zPs = S tan

xf2 + yf2 /cf



(7)



2 + y2 + z 2 = 1. xPs S is the normalization constant to make Ps Ps b. Transformation from PTZ coordinate system to spherical coordinate system As PTZ dome camera model is a hemisphere model, the PTZ coordinate system is identical with the spherical coordinate system essentially. The relation between the two parameters is:

˛=ϕ

(8) ◦

ˇ = 90 − 

(9)

So the transformation is: xHs = cos ˛ cos ˇ = cos ϕ sin 

(10)

yHs = sin ˛ cos ˇ = sin ϕ sin 

(11)

zHs = sin ˇ = cos 

(12)

c. Transformation from panoramic spherical coordinate system to PTZ dome spherical coordinate system According to the transformation by two steps as described above, the final transformation can be obtained: (xHs , yHs , zHs ) = (xPs , yPs , zPs ) × TPs→Hs

(13)

3.3. Theory between master camera and slave camera (xf , yf ) is the coordinate of pixel point p in panoramic coordinate system. (xp , yp , zp ) is the coordinate of corresponding point in PTZ coordinate system. The mapping relationships T can be determined by finding the matrix that can make two cameras’ optical centers coincide with each other. Namely, panoramic image can be

2482

Y. Liu et al. / Optik 125 (2014) 2479–2483

considered as a large PTZ dome image. PTZ dome camera satisfies the theory of pinhole imaging. According to Tsai [5], the theory of pinhole camera imaging is illustrated in Fig. 4(b). ˜ = PXc sm

(14)

Panoramic camera is placed on top of PTZ dome camera and their optical centers are in the same vertical plane. Op is the optical center of panoramic camera and Op is its principal point. Op is optical center of PTZ dome camera and O is its principle point. Now p is image point in the panoramic coordinate system and pw is the corresponding object point in the world coordinate system. We project PTZ dome image on the panoramic image (PTZ dome image plane is not always parallel to panoramic image plane). According to pinhole imaging, if PTZ dome camera shoots at pw , O will coincide with p. That is the result we need. In our experiment, the mapping TPs→Hs is calculated through a process of selecting sampling points. In order to solve the 3 × 3 matrix TPs→Hs , we select n (n ≥ 3) PTZ dome image, each of which provides a set of points. 3.4. Feature point matching and matrix solution This procedure requires a method of detecting and matching visual feature robust to scale, rotation, view-point, and lightning. The Scale Invariant Feature Transform (SIFT) [6] exhibits great performance under these constraints. So we employ SIFT method to detect feature point in both panoramic image and PTZ dome image. Panoramic image and PTZ dome image are two input parameters in j

j

j

j

j

j

T

j

SIFT method. Let {PHi } = {(XHi , YHi , ZHi )} (i = 1, . . ., n, n ≥ 3; j = 1, . . ., m, m > 1) be the jth feature point’s spherical coordinate of the ith PTZ j

T

dome image. Let {PPi } = {(XPi , YPi , ZPi )} (i = 1, . . ., n, n ≥ 3; j = 1, . . ., m, m > 1) be the panoramic spherical coordinate that corresponds j to {PHi }. Let AHi be feature point matrix in the ith PTZ dome image and APi be matching feature point matrix in the panoramic image. Ti is transformation matrix between the ith PTZ dome image and panoramic image and we solve for it using least square method. 1 2 m AHi = [PHi , PHi , . . ., PHi ]

APi =

(15)

1 2 m [PPi , PPi , . . ., PPi ]

Ti = APi × (ATHi AHi )

(16)

−1 T AHi

(17)

0 be the center point spherical coordinate of the ith PTZ Let PHi 0 be the panoramic spherical coordinate that dome image and PPi 0 . corresponds to PHi 0 PPi

=

0 Ti × PHi (i

= 1, . . ., n)

(18)

Let AH be center point spherical coordinate matrix in the PTZ dome image and Ap be matching point matrix in the panoramic image. 0 0 0 AH = [PH1 , PH2 , . . ., PHn ]

(19)

0 0 0 AP = [PP1 , PP2 , . . ., PPn ]

(20)

TPs→Hs = AP × (ATH AH )

−1

AH

(21)

The aim of the whole calibration process is to solve for the matrix TPs→Hs . When given any pixel p(xf , yf ), according to Eqs. (3)–(12), we can obtain corresponding rotation angle (˛, ˇ). 4. Experiments and results The proposed method has been tested by computer program. Fig. 5(a) shows the monitoring results of master-slave camera cooperates together. The top one is panoramic image and we have selected four areas which have the distinguished feature. The other

Fig. 5. (a) Experimental result after the calibration. Four areas are selected in the panoramic image as focus of interest, and corresponding multi-scaled image is captured by PTZ dome camera; (b) relation between requested and real pan angle; (c) mean error between requested and real pixel.

images below are PTZ dome images with zooming. Fig. 5(b) shows the calibration results and the best linear fit. We select 170 sampling points from panoramic image to analyze errors. As can be seen, the real pan angles match well with the requested angles from 0◦ to 360◦ and there is no obvious error. Similar good results are obtained for tilt angles (not shown here). Fig. 5(c) shows the mean error between requested pixel (px , py ) and real pixel (px , py ) to analyze the distribution of error. The error calculation formula is:



centering error =

(px − px )2 + (py − py )2

(22)

The dark blue area represents 0–35 pixels, the green area represents 35–70 pixels and the red area represents 70–105 pixels. As the PTZ dome image has 1920 × 1080 resolution and its horizontal coverage is 55, we can find that about 35 pixels are equivalent to 1◦ . As can be seen, error can be controlled within 1◦ over 95% of the area. In the calibration process, there are three main factors to influence the accuracy of calibration. The first factor is PTZ dome camera we have used cannot accurate to 0.1◦ . If so, 3.5 pixels will be equivalent to 0.1◦ . Centering error can reduce greatly. The second factor is selection of the PTZ dome images’ position. In our experiment we

Y. Liu et al. / Optik 125 (2014) 2479–2483

have selected 6 PTZ dome image. If the 6 positions are completely uniform distribution in the panoramic image, the red area cannot appear in Fig. 5(c). The third factor is feature point matching, the number and accuracy of measurement of match points influence the result of calibration directly. In order to show the accuracy we mark all pairs of matched points, which display with different colors. If too many errors are displayed, PTZ dome camera will capture another image. The method’s another advantage is time savings. The core process contains subtracting images and image matching. Subtracting one image takes about 2 s. For one pair of images, image matching takes about 5 s. So the whole process (6 PTZ dome images) takes less than 1 min. 5. Conclusion A master-slave camera system that is composed of panoramic and PTZ dome cameras is used for stationary and dynamic visual surveillance. A panoramic camera observes a scene with a large field of view, and PTZ dome cameras simultaneously capture highresolution images with multi-scale information. In this paper, we present a method for calibrating the cameras that obtains the mapping between master camera and slave camera. The availability and accuracy of the method is validated by the experiments shown in this paper. Since the proposed work requires two cameras as close as possible to ensure that they have the same optical center,

2483

future work would consider discrete master-slave camera system which permits the panoramic and PTZ cameras be mounted with distance. Acknowledgement This work was supported by Scientific Research Foundation for Returned Overseas Chinese Scholars, Doctoral Fund of Ministry of Education of China under project (20134307120040) and National University of Defense Technology (JC13-05-02). References [1] A. Hampapur, S. Pankanti, A.W. Senior, Y.L. Tian, L. Brown, R. Bolle, Multi-scale imaging for relating identity to location, in: IEEE Conference on Advanced Video and Signal Based Surveillance, Miami, July 2003, 2003, pp. 13–20. [2] X. Zhou, R. Collins, T. Kanade, P. Metes, A master-slave system to acquire biometric imagery of humans at distance, in: First ACM Proc. Workshop on Video Surveillance, November 2003, 2003, pp. 113–120. [3] J. Badri, C. Tilmant, J. Lavest, Q. Pham, P. Sayd, Camera-to-camera mapping for hybrid Pan–Tilt–zoom sensors calibration, in: 15th Scandinavian Conference, SVIA, July 2007, 2007, pp. 132–141. [4] L. You, S. Li, W. Jia, Automatic weak calibration of master-slave surveillance system based on mosaic image, in: 2010 International Conference on Pattern Recognition, August 2010, 2010, pp. 1824–1827. [5] Tsai, Y. Roger, A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses, IEEE J. Rob. Autom. RA-3 (August (4)) (1987) 323–344. [6] D.G. Lowe, Distinctive image features from scale-invariant keypoints, Int. J. Comput. Vision 60 (2) (2004) 91–110.