An accurate 3-D fire location method based on sub-pixel edge detection and non-parametric stereo matching

Measurement 50 (2014) 160–171

Contents lists available at ScienceDirect

Measurement journal homepage: www.elsevier.com/locate/measurement

An accurate 3-D fire location method based on sub-pixel edge detection and non-parametric stereo matching Tao Song b, Baoping Tang a,b,⇑, Minghang Zhao b, Lei Deng b a b

School of Mechanical & Electrical Engineering, Henan University of Technology, Zhengzhou 450007, PR China The State Key Laboratory of Mechanical Transmission, Chongqing University, Chongqing 400030, PR China

a r t i c l e

i n f o

Article history: Received 28 April 2013 Received in revised form 13 December 2013 Accepted 23 December 2013 Available online 7 January 2014 Key words: Three-dimensional (3-D) fire location Sub-pixel edge detection Non-parametric stereo matching Zernike moments operator Census transform

a b s t r a c t Fire disasters will cause severe damage to human properties and bring terrible mental and physical injury if they cannot be detected and extinguished in time. As traditional fire detectors, usually acting as alarms, fail to automatically locate fires and extinguish fires, fire detection and location based on binocular stereo vision has attracted much attention recently. But current three-dimensional (3-D) location methods based on binocular stereo vision have less accuracy with regards fire location due to imprecise camera calibration and unstable stereo matching, a novel 3-D location method based on sub-pixel edge detection and non-parametric stereo matching technology was explored. Firstly, to improve the camera calibration accuracy, Zernike moments operator was applied to relocate the sub-pixel edges from the feature points detected by Canny operator. Secondly, regional matching combined with epipolar constraint was proposed for fire stereo matching. Epipolar constraint was used to reduce the search area from two dimensions to one. And Census transform based on non-parametric transform was employed for exact regional matching. The experimental results indicated that the proposed method had high performance in 3-D fire location with high accuracy and strong robustness. An automatic fire-fighting system based on the proposed fire location method was developed and successfully applied to a coal to chemical plant in Yunnan province, China. Ó 2014 Elsevier Ltd. All rights reserved.

1. Introduction Fire is often sudden, disastrous, and damaging unless detected, located, and extinguished early. With the rapid development of new building such as large theatres, warehouses, and chemical plants, there is an increasing need for improved early-warning and fire fighting systems [1]. Therefore it lends significance to study the methods of 3D fire location and early fire extinguishing. Current sensor-based fire alarm systems based on heat, smoke, or composite detectors cannot be alerted until the particles or heat actually reaches the sensors and they are just alarms without fire location capability. With the development of ma⇑ Corresponding author at: School of Mechanical & Electrical Engineering, Henan University of Technology, Zhengzhou 450007, PR China. Tel.: +86 13658319901; fax: +86 023 65415883. E-mail address: [email protected] (B. Tang). 0263-2241/$ - see front matter Ó 2014 Elsevier Ltd. All rights reserved. http://dx.doi.org/10.1016/j.measurement.2013.12.022

chine vision technology, vision-based fire detection systems become feasible [2,3]. Not only is the response time for fire and smoke detection faster, but also they can monitor a larger area compared with traditional point sensors. Binocular stereo vision simulates the manner of human eyes observing the depth information of an object [4]. At present, binocular vision is mainly applied to size measurement or robot vision [5–7], whereas there are few studies of 3-D fire location based on binocular stereo vision which, in part, justified its selection for this fire location system research. There are two key and difficult issues which affect the accuracy of 3-D location based on binocular vision: camera calibration [8], and stereo matching [9]. Camera calibration is to obtain the parameters of camera imaging model. The common camera calibration methods include: traditional calibration, active vision calibration and self-calibration. While active vision calibration methods require known parameters of camera motion [10], self-cali-

161

T. Song et al. / Measurement 50 (2014) 160–171

bration is a non-linear calibration method with strong flexibility but weak robustness [11]. The traditional calibration methods are widely used with high-precision and broad applicability which was utilized in the paper. Feature points extracting is the key factor that influences camera calibration accuracy. In order to extract feature points exactly, first a plane calibration board with circular feature points was designed which has stronger anti-interference capability than Harris corner points [12]. Canny presented an optimal edge detector, especially for two dimensional images. Canny operator can give the edge information of both intensity and direction. Then all edges of the feature points were detected by Canny operator, but it is of pixel-level edge detection with low precision. Due to Zernike moments operator has superior performance over other moments in many image processing applications [13], it was utilized for relocating the sub-pixel edges of the points detected by Canny operator. Then exact image coordinate of feature points were obtain by ellipse fitting with the sub-pixel edges. Stereo matching is another important factor that affects the accuracy of 3-D fire location. It is the process of finding corresponding points in multi images. In general, matching algorithms can be classified into region matching and feature matching [14,15]. In this paper, region matching combined with epipolar constraint was proposed for searching the corresponding point. Firstly, the epipolar constraint was used to reduce the search for corresponding points in the stereo images to a linear search along an epipolar line. This reduction of the search space from two dimensions to one not only speeded up the algorithm, but it also greatly reduced the number of opportunities to select false matches scanline [16,17]. Following, region matching was employed for exact stereo matching. The classical region matching algorithms are based on the absolute intensity such as Sum of Absolute Differences (SAD) and Sum of Squared Differences (SSD) [18]. It assumes that the same feature points in the two images have the same intensity. But this assumption is usually not hold in the application environment because of different installation angles of cameras and so on. In this paper, the non-parametric stereo matching algorithm Census transform [19,20] was utilized for regional matching. It has a good inhibiting ability of amplitude distortion as it take relative gray value as the similarity measure function. Compared with some other regional algorithms, it not only keeps the precision but also have better robustness. In the paper, the 3-D fire location method was improved from two aspects: camera calibration and stereo matching. The experimental results indicated that it was an effective method for improving the fire location accuracy and it had strong robustness. Finally, a fire-fighting system based on the proposed method, was developed and successfully applied in a coal to chemical plant. Subsequent material covers: Section 2 – an introduction to the principle of 3-D fire location, Section 3 – the calibration method with sub-pixel edge detection, Section 4 – details of stereo matching based on non-parametric regional algorithms combined with epipolar constraint, Section 5 – description to the 3-D fire location procedure based on the proposed method, Section 6 – 3-D fire location test and system application, and Section 7 – conclusions.

2. The principle of 3-D fire location 3-D fire location is to get the fire position in the 3-D space with a pair of camera images from different viewing angles. Serval processes are required to achieve this goal. Firstly, the camera imaging model parameters are need to know by camera calibration. Secondly, the image coordinates of fire from a pair of cameras should be extracted by image recognition and stereo matching. Finally, the 3-D fire coordinate is obtained by 3-D restruction with the corresponding image coordinates. 2.1. Camera imaging model The camera imaging model is a mathematical formulation for a 3-D object projected onto the imaging plane through the camera lens. The principle of an ideal camera imaging model can be approximately represented by the pinhole model. Imaging point p of any space point P lies at the intersection of POc and the imaging plane where Oc is the optical centre of camera lens. Let the homogeneous image coordinate of p be (u, v, 1), the homogeneous 3-D coordinate of P be (xw, yw, zw, 1). The ideal conversion between the image coordinate and 3-D world coordinate is given by:

zc ðu; v ; 1ÞT ¼ Mðxw ; yw ; zw ; 1ÞT

ð1Þ

where M is designated the homography matrix, M e R34, zc is an unknown coefficient. In fact, the camera lens are not an ideal pinhole imaging model, but suffered some distortions. It results in that the image formed of the space point not being at the position described by the ideal linear model [21,22]. It will seriously affect the accuracy of stereo matching and 3-D reconstruction if without distortion correction. According to the result of a series of tests, a distortion model suitable for the camera used was established. It comprehensively considered the radial and tangential distortions and the image aspect ratio. Set the ideal image coordinate of the imaging point be pu(uu, vu), and the actual coordinate with distortion be pd(ud, vd). The distortion model can be written as:

(

uu ¼ ud  s þ ud ðk1 r 2d þ k2 r 4d Þ þ p1 ð3u2d þ v 2d Þ þ 2p2 ud v d

v u ¼ v d þ v d ðk1 r2d þ k2 r4d Þ þ p2 ð3v 2d þ u2d Þ þ 2p1 ud v d ð2Þ

v

r2d

where ¼ u2d þ 2d , tors, ud ðk1 r2d þ k2 r 4d Þ

k1, k2 are the radial deformation facand v d ðk1 r 2d þ k2 r4d Þ are the radial distortions; p1, p2 are the centrifugal distortion factor, p1 ð3u2d þ v 2d Þ þ 2p2 ud v d and p2 ð3u2d þ v 2d Þ þ 2p1 ud v d are the tangential distortions; s is the image aspect ratio. T By setting distortion coefficient K ¼ ½s k1 k2 p1 p2  , Eq. (3) is obtained from Eq. (2).

"

ud

ud r 2d

ud r 4d

3u2d þ v 2d

2ud v d

0

v

v

3v þ

2ud v d

2 d rd

4 d rd

2 d

u2d

# K¼



uu

vu  vd

 ð3Þ

Camera calibration is to provide the conversion between points in the image and points in space, namely to solve the homography matrix M and distortion coefficient K. With the image coordinates and the corresponding 3-D

162

T. Song et al. / Measurement 50 (2014) 160–171

coordinates of feature points, M and K can be solved by linear least squares and iteration methods according to Eqs. (1) and (3) [23]. The 3-D coordinates can be directly measured, therefore, the focus of camera calibration is to extract the image coordinates of feature points. 2.2. The principle of 3-D reconstruction 3-D reconstruction is the process that recovers the spatial position of fire from the corresponding points of the left and right cameras with the camera imaging model parameters M and K. It means that if the fire image coordinates are known, the fire spatial position can be calculated by 3-D reconstruction. The image coordinates can be got by Image recognition and stereo matching which is discussed in Section 4. Assume that the image coordinates of a pair of corresponding points in two cameras are p01 ðul ; v l Þ and p02 ður ; v r Þ, the homography matrices of the two cameras are Ml and Mr, the distortion coefficients are Kl and Kr and the 3-D coordinate of the pair of corresponding points is P(x, y, z). Firstly, correct the image coordinates according to Eq. (2) and obtain the theoretical coordinates p1(ul, vl) and p2(ur, vr). Secondly, establish the mapping between image coordinates and space coordinates according to the ideal imaging model. Then the following equation can be derived according to Eq. (1).

zl ½ul

vl

zr ½ur

vr

1T ¼ M l ½x y z 1T 1T ¼ M r ½x y z 1T

ð4Þ

Amn ¼

XX f ðu; v ÞV mn ðq; hÞ u2 þ v 2 6 1

ð7Þ

v

u

where f(u, v) is the gray value of coordinate (u, v), complex polynomial V mn ðq; hÞ; as a coordinate, is:

V mn ðq; hÞ ¼

ðnjmjÞ=2 X s¼0

ð1Þs ðn  sÞ!qn2s ejmh s!½ðn þ jmjÞ=2  s!½ðn  jmjÞ=2  s! ð8Þ

ð5Þ

The 3-D coordinate P(x, y, z) can be described by:

P ¼ A b

ular distribution of light. While the circular targets extraction can take full advantage of the rich gradient information surrounding the circle and obtain the circle centre more stably and accurately. Fig. 1 shows the calibration board with its array of solid circle feature points as designed for this research. Its row number, column number, and dimension were each set to suit the ranges and distances that monitored. Then the circular feature points can be extracted by edge detection. Among these edge detection operators, Sobel and Canny are typical. However, the accuracy of both operators can only reach pixel level and fails to satisfy accuracy requirements of 3-D location. This research combines edge detection methods at pixel, and sub-pixel levels. Firstly, the Canny operator was used to extract pixel level edges of feature points. Then, to improve the recognition accuracy, Zernike moments operator was used for subpixel edge relocation. Zernike orthogonal moments of a point denoted as f(u, v) in discrete images are expressed as [24]:

ð6Þ

3 ml31 ul  ml11 ml32 ul  ml12 ml33 ul  ml13 6 ml31 v l  ml21 ml32 v l  ml22 ml33 v l  ml23 7 7 where A¼6 4 mr31 ur  mr11 mr32 ur  mr12 mr33 ul  mr13 5; mr31 v r  mr21 mr32 v r  mr22 mr33 v r  mr23 2 3 ml14  ml34 ul 6 ml24  ml34 v l 7  7 b¼6 4 mr14  mr34 ur 5; A is the generalised inverse matrix mr24  mr34 v r of A, mlij is the element of matrix Ml and mrij is the element of Mr. According to the principle of 3-D fire location, the location accuracy depends on the camera imaging model parameters M, K and 3-D reconstruction. The camera imaging model parameters are got by camera calibration and stereo matching determines the effect of three-dimensional reconstruction. Camera imaging model parameters and stereo matching are the key factors of fire location accuracy though there are some uncontrollable objective factors just as the distance or the light of the monitoring site. So camera calibration and stereo matching are studied in this paper to improve the fire location accuracy.

Calculating A11 and A20 of each point using the Zernike moments operator [13], the rotation angle is:

/ ¼ arctanðImðA11 Þ=ReðA11 ÞÞ

2

ð9Þ

The distance from the central point to the edge is:

I ¼ A20 =ððReðA11 Þ cos / þ ImðA11 Þ sin /ÞÞ

ð10Þ

Assuming those edge pixel coordinates extracted by Canny operator are (u0, v0), while, sub-pixel coordinates derived by Zernike moments are:



u1

v1



 ¼

u0

v0



þI



cos / sin /

 ð11Þ

3. Camera calibration with sub-pixel edge detection In most of the calibration methods, Harris method is used to extract the feature points. But it will reduce the information around the feature points in the case of irreg-

Fig. 1. The plane calibration board with circular array feature points.

163

T. Song et al. / Measurement 50 (2014) 160–171



Those elliptical edges extracted were subjected to least squares fitting to determine the sub-pixel image coordinates (u, v) of the centre of the feature points. The model parameters of a pair of cameras include the homography matrices and distortion coefficients can be exactly obtained with sub-pixel image coordinates of feature points extracting.

z1 U 1 ¼ M 11 X þ m1

ð12Þ

z2 U 2 ¼ M 21 X þ m2

where M i ¼ ½M i1 mi , Mi1 e R33, mi e R31, i = 1, 2, eliminate the parameter X, the following equation is obtained. 1 z2 U 2  z1 M 21 M1 11 U 1 ¼ m2  M 21 M 11 m1

ð13Þ

Eliminate z1, z2, this becomes:

U T2 FU 1

4. Stereo matching of fire with non-parametric algorithm

¼0

ð14Þ

where F ¼ ½M M 21 M 1 11 , [M] is the anti-symmetric matrix of m2  M 21 M 1 11 m1 . Geometrically, FU1 defines the epipolar line of point p1 in the second image. Transposing Eq. (14) yields the symmetric relation from the second image to the first image. With the image coordinate U1 known, a linear equation about U2 can be obtained, and vice versa.

The main challenge of 3-D location is finding the exact corresponding points of a target with multi images from different viewing angles which is called stereo matching. For fire location, only a pair of corresponding points of the fire are necessary. Though the shape of fire is constantly changing which causes difficulties for stereo matching, the bottommost boundary of fire is usually stable in the early time, and it is an important feature for stereo matching. Take the midpoint of the bottommost boundary of one image as the reference point. Then the task is to find the matching point in the other image.

4.2. Non-parametric regional matching based on Census transformation Census transformation is based on a non-parametric similarity measure function. It depends solely on the set of pixel comparisons and produces a bit chain which represents the signs of the comparisons. The entire transformation is just the simple process of data comparison, accumulation and XOR operation, and no complex calculations such as multiplication and prescribing are required. Census transformation first take a square window with the checking pixel as the centre. Then compare the gray value of each neighbor pixel in the square window with the centre pixel. If the intensity of the pixel is lower than the value of the centre pixel, the bit chain of the corresponding position value is defined to be 1, otherwise defined to be 0. Then take hamming distance as the matching cost for each pixel in disparity range. The steps of Census transformation are as follows: assume that the pixel P(i, j) is the centre pixel of the matching window, let I(P) be the gray value, N(P) be the set of pixels of the square neighborhood of diameter n surrounding P(i, j), then the corresponding bit chain of Census transform can be expressed as:

4.1. Epipolar constraint The epipolar constraint is the intrinsic projective geometry between two views. It is based on theoretical models without distortion. It gives the geometry constraint as shown in Fig. 2. Let I1 and I2 be the imaging planes of left and right cameras, OL and OR be the optical centre of the two cameras, p1 and p2 be the images of a space point P in the two image planes respectively. Then P, p1, p2, OL, OR will be coplanar in the plane S. Let e1 and e2 be its projections to imaging planes I1 and I2 respectively. The intersections of the plane S with the images I1 and I2 are termed the epipolar lines and are denoted by l1 and l2. Let U1 and U2 are the homogeneous image coordinates of p1 and p2 respectively, X be the 3-D coordinate of P, M1, M2 be the homography matrices of two cameras respectively, according to Eq. (1), the following equation is obtained:

CTðPÞ ¼ bðn2  1Þ    bðkÞ    bð2Þbð1Þbð0Þ P

S P2

·

P1 I1

· l2

l1 OL

·e

1

YL

I2

XL

e2

·

OR

YR

Fig. 2. The epipolar geometry of binocular stereo vision.

XR

ð15Þ

164

T. Song et al. / Measurement 50 (2014) 160–171

where k e [0, . . ., n2  1], the value of bk can be expressed as:

 bðkÞ ¼

0; IðP 0 Þ P IðPÞ

ð16Þ

1; IðP 0 Þ < IðPÞ

where P0 e N(P). Assume that the bit chain of the reference point is bref and bmat of the matching point. The cost calculation for Census-transformed pixels is the calculation of the Hamming distance between the two bit chain as can be seen in:

Dh ðbref ; bmat Þ ¼

2 1 nX

bref ðmÞ  bmat ðmÞ

ð17Þ

the bit chain of p1 and the bit chain of each point in the search range in the corresponding image. The point p2 with the minimum Hamming distance is the corresponding point of p1.

5. The procedure to 3-D fire location Based on these theories, a complete procedure for fire location with the proposed method is shown in Fig. 4. It can be summarized as follows: Step 1: Calibration for a pair of cameras

m¼1

Calculate the Hamming distance between the reference point and each pixel in disparity range in the matching image. The target point is the pixel index which has the minimum Hamming distance. 4.3. Implementation of the proposed stereo matching algorithm From the discussion above, the implementation of stereo matching can be built up as follows: first, distortion correction is applied for preprocessing because epipolar constraint is based on the ideal linear imaging model. Second, extract the bottommost boundaries of fire of a pair of images. Assume that the images after distortion correction are I1 and I2, the bottommost boundaries are c1 and c2 respectively as shown in Fig. 3. Following, take the midpoint p1 of boundary c1 as the reference point, according to Eq. (14), the corresponding point of p1 is constrained to lie on the epipolar line l2. This greatly reduce the regional search area. In addition, as p1 is on the bottommost boundary c1 of the left view, the corresponding point should be on or near c2. The intersection of l2 and c2 is the expectation of the target point and let it be p0 . At last, Census non-parametric transform is used for searching the corresponding point along l2 with p0 be the centre of the search range. Calculate the Hamming distance between

Camera calibration forms the necessary preparatory work for obtaining the model parameters of cameras. It includes designing a calibration board, capturing multiple images of the calibration board, extracting sub-pixel image coordinates of feature points based on Canny operator and Zernike moments operator. And finally solve the homography matrixes and distortion coefficients Ml, Kl and Mr, Kr for the left and right cameras. Step 2: Stereo matching of fire During real-time fire monitoring, fire is detected by continuous image analysis. Once fire confirmed, the images are corrected with the distortion coefficients. Then the proposed stereo matching method based on Census non-parametric transformation and epipolar constraint is used to find a pair of corresponding points (the midpoint of the bottommost boundary of fire) according to Section 4.3. Let they be p1(ul, vl) and p2(ur, vr). Step 3: 3-D reconstruction With the pair of corresponding points p1(ul, vl) and p2(ur, vr), the corresponding 3-D coordinates of fire P(x, y, z) can be obtained by 3-D reconstruction according to Section 2.2.

I2

I1

Corresponding point Search range

Square window

l2 p1

p

p2

c1 Reference point Reference image

c2 Expected point Corresponding image

Fig. 3. The implementation of nonparametric stereo matching.

165

T. Song et al. / Measurement 50 (2014) 160–171

Camera calibration

Stereo matching

Plane calibration board design

Real-time video acquisition

Calibration board images capturing for the

Fire detection

left and right cameras

Sub-pixel image coordinates extracting based

Distortion correction of corresponding images

on Canny operator and Zernike moment Stereo matching based on epipolar constraint Distortion coefficients KL, KR for the left and right cameras Homography matrix ML, MR for the left and right cameras

and non-parametric regional matchings

Theory image coordinates of corresponding points of fire

p1 (ul , vl ) p2 (ur , vr )

Three-dimensional reconstruction World coordinate P( x, y, z ) of fire Fig. 4. The procedure to 3-D fire location.

Monitoring software

Manual control panel

DVR

Dual-band camera Fire cannon Fig. 5. The experimental apparatus.

6. Experiment and application 6.1. Experiment to 3-D fire location A laboratory experiment for 3-D fire location was designed to follow the proposed method: the test system

consisted of a digital video recorder (DVR), two dual-band cameras, a calibration board, a fire cannon, a manual control panel, and a computer. The 16 channel, type HIK DS8116HF-S, DVR was effectively an embedded computer system for image storage and processing: it supported image formats with multiple resolutions such as QCIF, CIF,

166

T. Song et al. / Measurement 50 (2014) 160–171

DCIF and D1. Here DCIF (528  384) format was selected. The cameras, type HZ-9080P, with 8 mm focal length had an infrared capability which would be useful at night. The CCD size was 1=3 00 . The apparatus is shown in Fig. 5. According to the process shown in Fig. 4, a plane, 1200 mm  1200 mm calibration board with a 7  7 array feature points was designed. Ten pairs of images of this board were captured for camera calibration. The sub-pixel edges detection of the feature points were extracted by Canny operator and Zernike moments operator as shown in Fig. 6. And the image coordinates were extracted by ellipse fitting. To validate the accuracy of camera calibration, the image coordinates of the 490 feature points were extracting by two methods: pixel level edge detection by Canny operator only and sub-pixel level edge detection by Zernike moments operator combined with Canny operator. The calibration results for the left and right cameras were shown in Table 1. Calibration results of pixel level edge detection and subpixel level detection. Then the 490 feature points were chosen as targets for 3-D location with the results of the two methods. The coordinates of the feature points were known when calculating the camera parameters before. In other words, results were used to back-calculate a known quantity. It means testing the theoretical errors of camera calibration without stereo matching. The reconstruction errors (the Euclidean distances between the actual and calculated 3-D coordinates) of the feature points are shown in Fig. 7 from where it was seen that the location error of each feature point was within ±5 mm. Their average absolute error was 1.449 mm with pixel level edge detection and 1.106 with sub-pixel level edge detection. It proved that the 3-d location using subpixel image coordinates detection was more precise with smaller location errors. Because the continuous signal was transformed into discrete pixels after imaging which have some errors, more exact camera imaging model parameters M and K can be got with sub-pixel image coordinates of feature points. To further test the performance of fire stereo matching of the proposed algorithm, fire location test was attempted as shown in Fig. 8. Fig. 8 (a) and (b) shows the different views from two cameras, and the distance between two cameras was about 2 m and between the fire and the cameras was about 25 m. Fig. 8 (c) and (d) shows the boundaries of fire after distortion correction. It can be seen that

the boundaries from two views varied widely due to dynamic characteristics of fire and the different angles of the cameras, but the bottommost boundaries were relatively stable. According to the implementation of the proposed algorithm detailed in Section 4.3, the color images were converted to gray images for stereo matching as shown in Fig. 8 (e) and (f). The midpoint p1 of the bottommost boundary of the left view was selected as the reference point. Then the corresponding epipolar line can be determined in the right view (the red line shown in Fig. 8 (f)). Took the intersection of the epipolar line and the bottommost boundary as the expected point. Finally the Census transform was employed for exactly stereo matching along the epipolar line with the expected point as the midpoint of the searching range. The window size of Census transform was set to 8  8 and the search range was set to 12. The matching point p2 was determined by calculating the Hamming distance. In this test, another two stereo matching methods SAD and SSD were used for 3-D location besides the proposed method. The performance of the proposed algorithm was compared with the two methods. The fire location results are shown in Table 2. From Table 2, it is clear that 3-D fire location accuracy with non-parametric algorithm was better than the other two. The principle of SAD and SSD is to quantify the similarity between two sets of gray levels, and the similarity between two sets of gray levels is to consider them as two points in RNf and to estimate how distant they are. In other words, it consists in calculating the LP norms of the vector of the gray level differences. When compared with SAD or SSD, Census transform had its advantage because it is a non-parametric algorithm based on relative values between the central gray value and neighborhood gray value. From Fig. 8 (e) and (f), it can be seen that the gray values of the two views are differ greatly. Census transform based on the relative values has stronger robustness in some scenes especially with uneven lighting. While comparing the location errors of fire with feature points, we can notice that the accuracy of fire location was lower than that of the feature points. There are several reasons for this result. One is that there was no matching error for 3-D location of feature points. Another reason is that there was larger measurement error while measuring the 3-D coordinates of the fire. In fire fighting applications, as the water ejected from the fire cannon covered a large area after landing, a certain error in fire location is acceptable:

Table 1 Calibration results of pixel level edge detection and sub-pixel level detection. Methods

Ml

Pixel level edge [0.10059 detection 0.00024 0.000013

Sub-pixel level detection

Mr

Kl

0.000552 0.19053 160.4971 [0.102505 0.003037 0.21007 98.2311 [0.923 0.11107 0.0056 146.5068 0.00783 0.105145 0.01626 110.073 8.156  107 0.000001 0.000139 1.0] 0.000047 0.000015 0.000187 1.0] 5.762  1011 1.271  109 2.640  109] [0.946 8.334  107 [0.11686 0.094287 0.646754 157.134 [0.16734 0.00538 0.035535 95.4069 1.263  1011 0.053917 0.6697 1.2554 142.366 0.00399 0.01184 0.070126 107.473 3.741  109 0.0002291 0.000235 0.003083 1.0] 0.000021 0.000708 0.000514 1.0] 5.315  109]

Kr [0.913 6.931  107 1.791  1011 4.512  109 2.511  109] [0.935 6.658  107 2.365  1011 8.635  109 5.754  109]

T. Song et al. / Measurement 50 (2014) 160–171

Fig. 6. Sub-pixel coordinates extraction of feature points. (a) Calibration board; (b) Sub-pixel level edges detection; and (c) image coordinates extraction.

usually, a distance between the water column and the fire source less than 0.5 m is reasonable. From the test result, the location accuracy of the algorithm was in full compliance with application requirements. 6.2. On-site application The fire-fighting system, which was based on the proposed fire location approach, was applied to a coal to

167

chemical plant in Yunnan province, China. The application of the fire-fighting system is shown in Fig. 9. The system was composed of front-end detectors, on-site control equipment, fire linkage equipment, a monitoring and management centre, and an information transmission network. The front-end detectors included detection cameras and on-board cameras. There were seven monitoring sites in total with two detection cameras, an on-board camera and a fire cannon at each site. The detection cameras were used for fire monitoring and binocular location. The onboard cameras, which were attached to the fire cannons, were used for assisting manual fire cannon operation. On-site control equipment consisted of decoders for fire cannons and field controllers. The decoders were used for receiving, executing, and feeding-back the control commands from the monitoring and management centre. The field controllers were used to control the fire cannon and fire valve manually on-site. The fire linkage equipment referred to the fire pumps, fire cannons, fire valves, water flow indicators, sound and light alarms, and so on: they formed the terminal equipment of the fire system. The monitoring and management centre – the most critical part of the system – consisted of DVRs, display devices, a fire detecting server, a manual control panel, and so on. It was installed in the central control room, far from the monitoring sites and used for remote monitoring, video capture, fire detection, alarm, fire location, and fire linkage equipments control. The network for information transmission was composed of the optical transmitter and receiver, Ethernet switch, video splitters, fibre optic transmission cable, netting twine, video cable, and serial line. The Ethernet switch was used for network connections between DVRs and servers. RS485 and RS232 serial communications sufficed for transmission with short-distance (from the field controller to the fire cannon decoder and from the fire detecting server to the manual control panel). To improve the anti-jamming capability of serial communication, optical fibre was necessary for long-distance transmission (from the monitoring scene to the central control room). Video cable and netting twine were used as the communication cables for video signal transmission over short-distances (from the camera to the field controller and from the DVR to the fire detecting server). Optical fibre was used for long-distance transmission (from the scene to control room) and reduced losses in the video signal. Different camera focal lengths ranging from 6 mm to 12 mm according to the different areas monitored and different distances between the cameras and the monitoring regions were selected. The size of the calibration board was 2 m  2m with its feature points in a 3  3 array of solid circles. Table 3 shows the equipment list for this fire-fighting system. The procedure for the 3-D fire location test was as follows: firstly, selected an appropriate point (such as the corner) as the origin of the world coordinate system at each monitoring site, then captured 15 pairs of images of the calibration board for each monitoring site and extracted the sub-pixel image coordinates of the feature points (145 feature points in total from each monitoring site), then calculated the camera homography matrices and distortion coefficients with these coordinates. For automated

168

T. Song et al. / Measurement 50 (2014) 160–171

Fig. 7. 3-D location error of feature points. (a) Location error with pixel level edge detection; and (b) location error with sub-pixel level edge detection.

fire cannon location, the initial position and angle of the fire cannon were also pre-measured. To validate the calibration accuracy, the 3-D location errors of feature points were tested and the results at each monitoring site are shown in Fig. 10. Given the live status of this chemical plant and the proximity of its production equipment to the test area, a real fire could not be ignited: other objects instead of fire were used to verify the reliability of 3-D location. And the test repeated eight times at each site.

The 3-D location errors from each monitoring site are shown in Table 4. Compared to the former experiment, the actual, in situ, 3-D location errors were much larger. There were three main reasons for this: firstly, it was difficult to measure the 3-D coordinates of the calibration board due to the complex environment of the monitoring sites. It thus caused 3-D coordinate errors in the feature points. Secondly, the distances between the cameras and the moni-

p1

(a) The left view of fire

(c) The boundary of fire of left view

(e) The gray image of left view

p2

(b) The right view of fire

(d) The boundary of fire of right view Fig. 8. Stereo matching of fire.

(f) The gray image of right view

169

T. Song et al. / Measurement 50 (2014) 160–171

toring area were farther than the former experiment. Thirdly, there were fewer feature points in this test than in the laboratory experiment: this also exerted an influence on the 3-D location accuracy. In any case, the location error of each feature point was not greater than 30 mm, and the location errors of the tested objects were below 0.2 m which indicated that the location accuracy met the requirements for adoption as an automatic fire protection system.

Table 2 Fire location result. Serial number

Distance error of the proposed method (mm)

Distance error of SAD (mm)

Distance error of SSD (mm)

1 2 3 4 5

77 63 59 65 54

85 91 67 76 61

79 67 55 78 62

RS485

Decoder

Fire cannon

Field controller

… Optical fiber

Optical

Left camera

Camera calibration

Right camera

Camera calibration

fiber

Manual control panel

Chemical plant Sound and light alarm

RS232

DVR

Ethernet

Software of the system Monitoring and management centre Fig. 9. The fire-fighting system based on binocular stereo vision.

170

T. Song et al. / Measurement 50 (2014) 160–171

was employed for extracting image coordinates of feature points. It improved the accuracy of camera calibration. Census transform based on a non-parametric similarity measure function combined with epipolar constraint was proposed for stereo matching of fire. It improved the precision of stereo matching with strong robustness. The experiment results indicated that this proposed approach was an effective method for 3-D fire location and the location accuracy was sufficient to meet engineering requirements. The fire-fighting system based on the proposed method was developed and successfully applied in a coal to chemical plant.

Table 3 Equipment list of the fire monitoring system. Equipment name

Device type

Number of devices

Fire cannon Detection camera On-board camera DVR Calibration board Field controller Decoder Manual control panel Computer

XFP 1EX HZ-9120P HZ-9120P HIKDS-8116HF-S – – – – Server with Xeon 5600 HoneywellP2475RLZ

7 14 8 2 1 8 8 1 2

Sound and light alarm Ethernet switch

1

TL-SG1008

Acknowledgments

1

7. Conclusions An accurate 3-D fire location method for automatic firefighting was proposed in this paper. Zernike moments operator with sub-pixel level edges detection capability

This research was supported by the National Natural Science Foundation of China (51375514, 51305471), Specialized Research Fund for the Doctoral Program of Higher Education (20130191130001), and Fundamental Research Funds for the State Key Laboratory Of Mechanical Transmission in Chongqing University (SKLMT-ZZKT-2012 MS 09). Finally, the authors are grateful to the anonymous

Fig. 10. 3-D location errors of feature points of each monitoring site.

Table 4 3-D location errors of each site. Site

Error1 (mm)

Error2 (mm)

Error3 (mm)

Error4 (mm)

Error5 (mm)

Error6 (mm)

Error7 (mm)

Error8 (mm)

1 2 3 4 5 6 7

74 101 104 130 76 28 117

131 44 179 158 65 47 26

139 141 65 127 64 30 96

201 95 42 165 77 34 94

162 130 118 135 97 27 72

104 72 125 171 79 22 125

171 90 89 169 51 26 101

145 129 134 194 63 39 126

T. Song et al. / Measurement 50 (2014) 160–171

reviewers for their helpful comments and constructive suggestions. References [1] T. Chen, H.Y. Yuan, G.F. Su, Automatic fire location by CCD images in big space, Opt. Tech. 29 (2003) 292–294. [2] P.V.K. Borges, E. Izquierdoet, A probabilistic approach for visionbased fire detection in videos, Circ. Syst. Video Technol. 20 (2010) 721–731. [3] X.L. Zhou, F.X. Yu, Y.C. Wen, et al., Early fire detection based on flame contours in video, Inf. Technol. J. 9 (2010) 899–908. [4] R. Blake, H. Wilson, Binocular vision, Vision. Res. 51 (2011) 754–770. [5] H. Du, M.G. Li, The study for particle image velocimetry system based on binocular vision, Measurement 42 (2009) 619–627. [6] M. Lauer, M. Schönbein, S. Lange, et al., 3D-objecttracking with a mixed omnidirectional stereo camera system, Mechatronics 21 (2011) 390–398. [7] P. Zhao, N.H. Wang, Precise perimeter measurement for 3D object with a binocular stereo vision measurement system[J], Optik 121 (2010) 953–957. [8] Z.Z. Xiao, L. Jin, D.H. Yu, et al., A cross-target-based accurate calibration method of binocular stereo systems with large-scale field-of-view, Measurement 43 (2010) 747–754. [9] H. Saito, M. Mori, Application of genetic algorithms to stereo matching of images, Pattern Recogn. Lett. 16 (1995) 815–821. [10] X. Wan, G.Y. Xu, Camera parameters estimation and evaluation in active vision system, Pattern Recogn. 29 (1996) 439–447. [11] T. Dang, C. Hoffmann, C. Stiller, Continuous stereo self-calibration by camera parameter tracking, IEEE Trans. Image Process. 18 (2009) 1536–1550. [12] X.Q. Meng, Z. Hu, A new easy camera calibration technique based on circular points, Pattern Recogn. 36 (2003) 1155–1164.

171

[13] Y.D. Qu, C.S. Cui, S.B. Chen, et al., A fast subpixel edge detection method using Sobel–Zernike moments operator, Image Vis. Comput. 23 (2005) 11–17. [14] M. Ansari, L. Masmoudi, A. Bensrhair, A new regions matching for color stereo images, Pattern Recogn. Lett. 28 (2007) 1679–1687. [15] M.S. Wu, J.J. Leou, A bipartite matching approach to feature correspondence in stereo vision, Pattern Recogn. Lett. 16 (1995) 23–31. [16] J.D. Tian, Y.B. Ding, X. Peng, Self-calibration of a fringe projection system using epipolar constraint, Opt. Laser Technol. 40 (2008) 538– 544. [17] Z.P. Cui, G. Jiang, S. Yang, C. Wu, A new fast motion estimation algorithm based on the loop–epipolar constraint for multiview video coding, Signal Process.: Image Commun. 27 (2012) 172–179. [18] S. Chambon, A. Crouzil, Similarity measures for image matching despite occlusions in stereo vision, Pattern Recogn. 44 (2011) 2063– 2075. [19] R. Zabih, J. Woodill, Non-parametric local transforms for computing visual correspondence, in: Proceedings of European Conference on Computer Vision, Stockholm, Sweden, May 1994. [20] M. Humenberger, C. Zinner, Michael Weber, Wilfried Kubinger, Markus Vincze, A fast stereo matching algorithm suitable for embedded real-time systems, Comput. Vis. Image Und. 114 (2010) 1180–1202. [21] C. Ricolfe-Viala, A.J. Sánchez-Salmerón, Robust metric calibration of non-linear camera lens distortion, Pattern Recogn. 43 (2010) 1688– 1699. [22] J. Park, S.C. Byun, B. Lee, Lens distortion correction using ideal image coordinates, Consumer Electron. 55 (2009) 987–991. [23] T. Song, B.P. Tang, J.M. Xi, Improvement of the method for space fire positioning based on binocular stereo vision, J. Chongqing Univ. 35 (2012) 16–21. [24] S. Ghosal, R. Mehrotra, Orthogonal moment operators for subpixel edge detection, Pattern Recogn. 26 (1993) 295–306.