- Email: [email protected]
Using cross-ratios to model curve data for aircraft recognition
Pattern Recognition Letters 24 (2003) 2047–2060 www.elsevier.com/locate/patrec
Using cross-ratios to model curve data for aircraft recognition Shen-Chi Tien a, Tsorng-Lin Chia
b,*
, Yibin Lu
a
a
b
Department of Electrical Engineering, Chung Cheng Institute of Technology, National Defense University, Tahsi, Taoyuan 335, Taiwan, ROC Department of Computer and Communication Engineering, Ming Chuan University, 5 Teh-Ming Rd., Gwei Shan District, Taoyuan 333, Taiwan, ROC Received 20 September 2002; received in revised form 25 February 2003
Abstract This study proposes a novel method that utilizes the projective invariance of NURBS (nonuniform rational B-splines) and cross-ratios to recognize aircraft in images. An aircraft contour extracted from an image can be fitted by a NURBS curve in approximation, and a skew-symmetry detection method is employed to determine the start point of the contour for the fitting curve. The control points on the fitting curve are then utilized to calculate a set of cross-ratios to represent the aircraft contour, and a small database is established for aircraft recognition. Moreover, all real crossratios are converted to integral cross-ratios to increase the error tolerance and matching facilitation. The experimental results demonstrate that the proposed method is robust and effective in aircraft recognition. Ó 2003 Elsevier Science B.V. All rights reserved. Keywords: NURBS curve; Cross-ratio; Projective invariant
1. Introduction Object contour is important in model-based vision systems that use geometric primitives or models to recover and recognize objects. A direct approach to object recognition is to match object boundary curves extracted from images, where the curves may undergo various transformations such as translation, rotation, shearing, and scaling. Since object contour can be altered through various projective transformations or viewpoints,
*
Corresponding author. Tel.: +886-33-507001; fax: +886-33294451. E-mail address: [email protected] (T.-L. Chia).
directly matching the curve data for object recognition is difficult. Consequently, numerous techniques have been developed for representing an object curve by a set of features or by models, such as Fourier descriptors (Wu and Sheu, 1997; Arbter et al., 1990), moment invariants (Bamieh and De Figueiredo, 1986; Reeves et al., 1988), curvatures (Mokhtarian, 1997; Dudek and Tsotsos, 1997), wavelets (Lee, 2000), and B-splines or NURBS (Cohen et al., 1995; Huang and Cohen, 1996; Alferez and Wang, 1999). However, Fourier descriptors and moment invariants, which require shape knowledge, suffer from the occlusion problem when some parts of the object data are unavailable. On the other hand, curvatures or arc length, which are based on local properties, have
0167-8655/03/$ - see front matter Ó 2003 Elsevier Science B.V. All rights reserved. doi:10.1016/S0167-8655(03)00042-4
2048
S.-C. Tien et al. / Pattern Recognition Letters 24 (2003) 2047–2060
difficulty in noise toleration, since the calculation of such techniques generally involves high-order derivatives. Wavelets and B-splines, which are based on the basis functions, have the correspondence problem of the start point of the curve data. Among these methods, B-splines have some attractive properties, including spatial uniqueness, continuity, local shape controllability, and invariance to affine transformation, but the B-spline parameters of the curve are not unique. Nevertheless, NURBS are generalizations of nonrational B-spline forms as well as rational and nonrational Bezier curves and surfaces. NURBS are invariant under translation, rotation, scaling, and shearing, as well as under parallel and perspective projections (Piegl, 1991). Accordingly, Alferez and Wang (1999) proposed a framework that applied the projective invariance of NURBS curves and illumination invariants for object recognition. The problem of identifying perspective invariants is formulated as a curve-fitting problem, and the fitting curve is used to recognize objects. However, the database must store the curve data of all aircraft models for matching, and the storage requirements are augmented rapidly when the number of aircraft to be recognized is increased significantly. On the other hand, invariant features can form a compact, intrinsic description of an object and be used to design recognition algorithms that are potentially more efficient than aspect-based approaches (Eggert and Bowyer, 1993). Some previous investigations (Lei, 1990; Rao et al., 1992) used cross-ratios to recognize planar objects in a three-dimensional space, and also formulated perspective invariants. However, objects were restricted to polygons and required accurate positions of vertex. This study presents a hybrid curve matching method which applies the projective invariance of both NURBS curves and cross-ratios for aircraft recognition. The recognition method comprises two stages, learning and recognition, with the flowchart as shown in Fig. 1. The proposed method applies primarily to the aircraft detected at a long distance from different viewpoints, and the aircraft image size is smaller than the distance between the aircraft and the viewpoint. For example, an optical tracking system tracks an air-
Fig. 1. Flowchart of aircraft recognition.
craft fly in the sky or a surveillance airplane (or satellite) detects the aircraft stay on the ground. The aircraft are viewed as planar objects approximately in the three-dimensional space, and the aircraft contours are regarded as coplanar points. Therefore, a three-dimensional aircraft recognition is translated into a matching contours problem in an image plane. To facilitate recognition, this study applies a fixed number of control points to fit the contour of the aircraft model. First, the aircraft contour is fitted using NURBS curves in approximation, and the fitting error of the curve is adjusted to an acceptable range. Since the control points of the NURBS curve are projective invariants (Alferez and Wang, 1999), the geometrical relationships of the control points are maintained in a projective plane. Nevertheless, the aircraft contour can be altered via different projective transformations or by viewing from various viewpoints, and the locations of control points can be adjusted according to the changed contour. The representation features must be invariant for recognition and must solve the problems of projective transformations. Consequently, this study employs
S.-C. Tien et al. / Pattern Recognition Letters 24 (2003) 2047–2060
cross-ratios, which form a cross-ratio vector, to represent the control points determined by the aircraft contour. All cross-ratio vectors of aircraft models can be combined to establish a database for aircraft recognition. To obtain a robust and reliable cross-ratio vector, this study analyzes the stability of the calculated cross-ratios, and a geometrical relationship table that is used to calculate a cross-ratio vector during the recognition stage is established for each aircraft model. Section 3 describes the stability analysis of cross-ratios. The cross-ratio vectors with fixed geometrical relationship tables in the database are insensitive to the viewpoint-object orientation and noise contamination, and these vectors maintain adequate separability between aircraft models. Additionally, the dimension of the extracted cross-ratio vectors should be small to reduce the computational burden in the recognition stage. The recognition method based on geometric invariants employs only a set of numbers to represent an aircraft in an image, and these numbers remain unchanged when the viewpoint orientation is modified or the parameters of projective transformations are altered. However, cross-ratios calculated from these control points are real numbers, and the variational range of cross-ratios is a nonuniform distribution. Though the recognition method can employ these real numbers as aircraft features, defining a fixed and reasonable threshold to restrict matching errors is difficult. Consequently, this study proposes an approach that converts real cross-ratios to integral cross-ratios. This approach generates a uniform distribution of integral cross-ratios and utilizes integral cross-ratios to represent aircraft contours. Using integral cross-ratios to describe aircraft contours enhances the error tolerance of calculated cross-ratios and simplifies matching features. Moreover, the aircraft contour formed by the integral cross-ratios is simply a string of integral numbers, markedly reducing the complexity of searching the database and comparing model data. This paper is arranged as follows. Section 2 elucidates the curve fitting method and the definition and conversion of cross-ratios. Section 3 then examines the cross-ratio stability. The procedure for representing aircraft contours and establishing
2049
the model database, as well as the algorithm of matching the cross-ratio vector are then described in Section 4. Meanwhile, Section 5 presents the experimental results. Finally, Section 6 presents conclusions.
2. Control points and cross-ratios Many contours of manmade objects are symmetrical and smooth curves such as aircraft, and their perspective projections change significantly at different viewpoints. Using a simple model to represent an object curve under different viewpoints is difficult. Hence, various works use Bsplines to interpolate curves and recognize objects. Although B-splines possess some properties such as spatial uniqueness, continuity, local shape controllability, and invariance to affine transformation, the B-spline parameters of the curve are not invariant in the projective transformation. Therefore, NURBS, a popular CAD/CAM representation, is used to describe curve objects, and NURBS curves preserve the properties of B-spline curves and maintain invariant under perspective projection. 2.1. Control points A curve cðuÞ can be expressed in terms of the associated rational basis functions Ri;k ðuÞ and n þ 1 control points: n n X 1 X cðuÞ ¼ wi pi Ni;k ðuÞ ¼ pi Ri;k ðuÞ; ð1Þ wðuÞ i¼0 i¼0 Ri;k ðuÞ ¼
wðuÞ ¼
wi Ni;k ðuÞ ; wðuÞ
n X
wi Ni;k ðuÞ;
ð2Þ
ð3Þ
i¼0
where pi ¼ ðxi ; yi Þ, i ¼ 0; 1; . . . ; n denotes the locations of the control points, while wi represents the weight assigned to each control point, and P R ðuÞ ¼ 1. Ni;k ðuÞ denotes the normalized Bi;k i spline functions of order k (degree k 1) defined over a certain knot sequence (or knot vector)
2050
S.-C. Tien et al. / Pattern Recognition Letters 24 (2003) 2047–2060
u ¼ fu0 ; u1 ; . . . ; unþk g, which is assumed to be nonperiodic. The normalized B-spline functions of order k are also generated using the Cox-deBoor recursion formulas (Rogers and Adams, 1990): 1 if ui 6 u 6 uiþ1 ; ð4Þ Ni;1 ðuÞ ¼ 0 otherwise; and Ni;k ðuÞ ¼
u ui Ni;k1 ðuÞ uiþk1 ui uiþk u þ Niþ1;k1 ðuÞ: uiþk uiþ1
CRL ðp1 ; p2 ; p3 ; p4 Þ ¼
jp1 p3 j jp2 p4 j : jp2 p3 j jp1 p4 j
ð6Þ
Taking vertex a of the polygonal object as a focal point, the angular definition of the cross-ratio is CRa ðh1 ; h2 ; h3 Þ ¼
sinðh1 þ h2 Þ sinðh2 þ h3 Þ ; sinðh2 Þ sinðh1 þ h2 þ h3 Þ
ð7Þ
and the following relationship is obtained and proven in (Lei, 1990), ð5Þ
A planar rational curve can be interpreted as the perspective projection of a nonrational Bspline curve onto the plane z ¼ 1 in three-dimensional space. To avoid the nonlinear problem, all weights are set to one, and the parameters and knots of the curve data are pre-computed. When the number of control points (n) and the order (k) of B-spline functions are given, the positions of the control points can be determined by least squares approximation (Piegl and Tiller, 1995). Section 4 delineates the computational method for locating the control points. 2.2. Cross-ratios A polygonal object in a three-dimensional space forms a projection in a two-dimensional image, and the extracted features of the object in the image are considered to be the vertices of the object, abcde, as illustrated in Fig. 2. Meanwhile, the lines that connect vertex a to other four vertices, b, c, d and e, intersect line L at four points, p1 , p2 , p3 , p4 , respectively. The cross-ratio formed by the four points, p1 , p2 , p3 , p4 , on line L is defined as
Fig. 2. Sketch map of the geometric relationship of crossratios.
CRL ðp1 ; p2 ; p3 ; p4 Þ ¼ CRa ðh1 ; h2 ; h3 Þ:
ð8Þ
Since the cross-ratio has the invariant property in projective geometry, its value is invariant under translation, rotation, and scaling transformations or various viewpoints. Consequently, cross-ratios are employed as features of extracted points in the projective images. These cross-ratios of an object form a feature vector, namely a cross-ratio vector, that acts as a model for recognizing the object in images. Since the control points of the NURBS fitting curve is projective invariants, they serve as the feature points of the curve to calculate a set of cross-ratios for representing the fitting curve. The cross-ratios calculated to represent control points are real numbers (2 R) that are influenced by noise, and the cross-ratio range is a nonuniform distribution of real numbers. Though the recognition method can use these real numbers to match features, defining a fixed and reasonable threshold for restricting matching errors is difficult. Consequently, the real cross-ratios are converted to integral cross-ratios (2 N ) using a statistical method, thus creating a conversion table. This approach increases the error tolerance and facilitates the matching of the integral cross-ratios. The conversion of real cross-ratios is performed as follows. First, the conversion uses Eq. (7) to calculate all cross-ratios in the constrained ranges, 0 6 h1 ; h2 ; h3 6 p and 0 6 h1 þ h2 þ h3 6 p, by increasing each included angle by a small increment Dhi , i ¼ 1; 2; 3, and analyzing the numerical distribution of the calculated results. Next, all calculated cross-ratios are divided into M parts, while the probability of a cross-ratio occurring in the Mth portion is considered to be equal. Finally, the range of integral cross-ratios is set to 1 M, and
S.-C. Tien et al. / Pattern Recognition Letters 24 (2003) 2047–2060
all real cross-ratios are converted to integral crossratios according to the conversion table. The cross-ratios presented in the following sections are integral cross-ratios.
3. Stability analysis The displacements of the control points of the fitting curve, influenced by noise or the extracted thresholds, may vary cross-ratios since the calculated cross-ratio involves five control points. Therefore, the real cross-ratios are converted into integral cross-ratios to increase the error tolerance in matching cross-ratios. However, cross-ratios may be strongly influenced by noise variance. Consequently, a fixed geometrical relationship among control points is utilized to produce a stable cross-ratio, and this approach can reduce the influence of noise. This work follows the analytical method of Song et al. (2000) to derive a criterion for selecting a stable cross-ratio that is suitable for the proposed method. Given that the cross-ratio of five points is s, Eq. (7), CRa ðh1 ; h2 ; h3 Þ ¼ s, can be represented by the following formula: 1
s ¼ 21
jabjjadj sinðh1 þ h2 Þ 12 jacjjaej sinðh2 þ h3 Þ
S1 S2 ; S3 S4
DS3 DS4 DS1 DS2 : þ ER ffi S3 S4 S1 S2
ð12Þ
This equation shows that the error rate of the cross-ratio is a function of the variation of the measured area of the triangle. Differentiating Si yields, oSi ¼ 12ðBi sin ai oAi þ Ai sin ai oBi þ Ai Bi cos ai oai Þ;
ð13Þ
and DSi DAi DBi Dai ffi þ þ ; Si Ai Bi tan ai
ð14Þ
where the variation of Dai is also affected by the vertex displacements of the two adjacent edges, and the maximum variation of Dai is Dai ffi
DAi DBi þ ; Ai Bi
ð15Þ
such that, DSi 1 DAi DBi ffi 1þ þ : tan ai Si Ai Bi Finally, let 1 1 1 Qi ¼ 1 þ þ ; tan ai Ai Bi
ð16Þ
i ¼ 1; 2; 3; 4
and the criterion utilized to select a stable crossratio is specified as
jacjjadj sinðh2 Þ 12 jabjjaej sinðh1 þ h2 þ h3 Þ 2
¼
2051
ð9Þ
where Si ¼ 12 Ai Bi sinðai Þ, i ¼ 1; 2; 3; 4 is the area of the triangle, and ai , bi are the lengths of the two edges adjacent to the including angle ai . The error rate of the cross-ratio influenced by noise or displacement is defined as Ds ð10Þ ER ¼ ; s
SK ¼ jQ3 þ Q4 Q1 Q2 j:
ð17Þ
A smaller SK corresponds to a more stable cross-ratio and indicates that the cross-ratio variation is little influenced by noise or control point displacements. Consequently, a stable cross-ratio is selected for the control point, and the geometrical relationship, which is used to constitute the stable cross-ratio, is recorded to form a mapping table for calculating cross-ratios during the recognition stage.
where Ds denotes the variation of the cross-ratio. Differentiating Eq. (10) then yields os ¼
4. Recognition method
S2 S1 S1 S2 S1 S2 oS1 þ oS2 2 oS3 oS4 ; S3 S4 S3 S4 S3 S4 S3 S43 ð11Þ
and substituting Eqs. (9) and (11) into Eq. (10) yields
4.1. Aircraft contour representation Suppose that an aircraft contour is extracted from an image employing appropriate
2052
S.-C. Tien et al. / Pattern Recognition Letters 24 (2003) 2047–2060
Fig. 3. The processing steps for representing aircraft contour. (a) The projective contour of aircraft F-14 is depicted with a detected skew symmetry axis and two intersected points. (b) The NURBS fitting curve and control points are depicted by black line and asterisks (), respectively. (c) An aircraft contour is represented by a cross-ratio vector and a geometrical relationship table.
preprocesses, and the curve data are stored in an ordered sequence (p points curve data c ¼ ½x; yT , x ¼ ½x1 ; x2 ; . . . ; xp T , y ¼ ½y1 ; y2 ; . . . ; yp T , where T represents the matrix transposition). An aircraft contour can then be represented by a cross-ratio vector which is generated according to the following steps (refer Fig. 3): (a) The aircraft contour is generally formed as a symmetrical closed curve against the medial axis of the contour. This study employs a Hough-based symmetry detection method (Lei and Wong, 1999) to determine the skew symmetry axis of the contour. The skew symmetry axis intersects with the contour at two intersection points, which are assumed to be the start points for the fitting curve. The cross-ratio vector of a selected start point can be obtained through processing via steps (b)–(d). (b) The curve data are stored in an ordered sequence from a selected start point, as determined in step (a), and the curve data parameterization is performed using the inverse chard length method (Huang and Cohen, 1996) and knots are calculated using the average knots method (Ma and Kruth, 1995). Finally, the nonrational B-spline basis functions of order k, N, are calculated using Eqs. (4) and (5), 2 3 N0;k ðu1 Þ N1;k ðu1 Þ Nn;k ðu1 Þ 6 N ðu Þ N ðu Þ N ðu Þ 7 1;k 2 n;k 2 7 6 0;k 2 7: N ¼6 ð18Þ .. .. .. 6 7 4 5 . . . N0;k ðup Þ N1;k ðup Þ Nn;k ðup Þ
(c) The curve data are then fitted by nonrational B-spline basis functions of order k, while the number of control points is given as n þ 1, using the approximation method. The locations of the control points of the fitting curve are solved and formed an ordered sequence. Let p ¼ ½p0 ; p1 ; . . . ; pn T denote the locations of control points. The locations of the control points can then be determined by minimizing the fitting error using the least squares method, N T Np ¼ N T c:
ð19Þ
Since the aircraft contour is a closed curve, the first and last control points are assumed to locate at the same location. Consequently, the number of control points used to constitute the cross-ratio vector is n, i.e., pi , i ¼ 1; 2; . . . ; n. (d) According to the definition of cross-ratios, a selected control point, which served as a focal point, is combined with another four control points, and a cross-ratio can be calculated to represent the control point. For example, a selected control point, pi , which is combined with another four control points, pj1 , pj2 , pj3 , pj4 , is applied to determine a cross-ratio using Eq. (7), CRpi ð\pj1 pi pj2 ; \pj2 pi pj3 ; \pj3 pi pj4 Þ, i 6¼ j1 6¼ j2 6¼ j3 6¼ j4 . Since the selected control point constitutes a cross-ratio when combined with any four control points on the fitting curve, the stability analysis of cross-ratios, described in Section 3, is employed to determine a stable cross-ratio. Notably, the stability of the cross-ratio, CRpi , is determined by Eq. (17), SKpi .
S.-C. Tien et al. / Pattern Recognition Letters 24 (2003) 2047–2060
Comparing all SKpi calculated from various geometrical relationships for the selected control point pi , the cross-ratio and the geometrical relationship with the smallest SKpi are recorded to represent the control point. The control point record is a crossratio CRpi and a row vector ½i; j1 ; j2 ; j3 ; j4 . All determined cross-ratios, which represent the control points, form a cross-ratio vector, CR, while the associated row vectors form a geometrical relationship table, GRT, which is a n 5 matrix. (e) Finally, cross-ratio vectors with different start points are compared with the model database, and the curve data recognition is determined based on the matching results. 4.2. Matching method Suppose that the database contains m records of different aircraft models, and the cross-ratio vector of the jth model is expressed as Dj ¼ ðDj1 ; Dj2 ; . . . ; Djn Þ, j ¼ 1; 2; . . . ; m. The cross-ratio vector of the tested aircraft model is T ¼ ðT1 ; T2 ; . . . ; Tn Þ. Data matching is performed by direct comparison, and the matching formula is n X Vj ¼ Sji ; j ¼ 1; 2; . . . ; m; ð20Þ i¼1
where 1 jTi Dji j 6 ki ; Sji ¼ 0 otherwise;
2053
ki represents the threshold of the ith cross-ratio description and is deduced from the standard deviation of the ith column cross-ratios in the database, and all thresholds form a threshold vector, k. Since two start points are determined by detecting the skew symmetry axis of the contour, two matching results (Vj1 and Vj2 ) are used to identify the correct aircraft model. The determined model number, Mnum , is the argument of the maximum of two matching results, where one of the matching results must exceed n=2. The determining formula is 8 argj maxðVj1 ; Vj2 Þ > > < if Vj1 P n=2 or Vj2 P n=2; ð21Þ Mnum ¼ > 0 > : otherwise: If the number of the recognized aircraft model, Mnum , equals zero, the tested curve data are classified as an uncertain aircraft model.
5. Experimental results Suppose that all aircraft contours are extracted via appropriate preprocesses from images which are normalized to 100 100 pixels, and the positions of curve data (contours) are stored in ordered sequences for all experiments. Fig. 4 displays the contours of all aircraft models used during the learning and recognition stages.
Fig. 4. Aircraft contours in the database.
2054
S.-C. Tien et al. / Pattern Recognition Letters 24 (2003) 2047–2060
5.1. The learning stage Based on the representation method elucidated in Section 4.1, this study constitutes a database based on the control points of the fitting curve in the learning stage. First, an aircraft contour is fitted approximately by NURBS curves, and the order of NURBS curves is set to k ¼ 4, a cubic B-spline curve. Since the aircraft contour is assumed to be located on the same plane in a three-dimensional space, all weights are set to one for the fitting curve. Additionally, the number of control points required to obtain an optimal fitting curve must be determined. Three factors are considered to determine the number of control points: (a) correct recognition rate; (b) decreased fitting error; and (c) computational time of curve fitting and matching. Consequently, this study utilizes a decision function to determine the number of control points for the fitting curve. Assume that a tested set of the number of control point is M ¼ f2j þ d j j ¼ 0; 1; . . . ; 8; d ¼ 13g, where d denotes the minimum of the number of control points in the tested set. Moreover, let c represent the curve data of a tested aircraft, which has p point curve data, and let f i , i 2 M, be the fitting curve data given i control points. The fitting error is defined as
P Fi ¼ p kc f i k, i 2 M, and the computational time, Ti , is defined as the elapsed time for recognizing a tested contour, which is fitted by i control points, executed on a PC (Pentium III, 1 GHz). The decision function is then defined as Dfq ¼
Rq ; Tq Fq
q 2 M;
ð22Þ
where q denotes the number of control points in the tested set. Rq represents the correct recognition rate of an aircraft model recognized by using q control points to fit the curve data. Fq is the fitting error rate, which is normalized by Fd¼13 , Fq ¼ Fi =Fd¼13 , i 2 M, and Tq denotes the rate of the computational time of curve fitting and matching, which is normalized by Td¼13 , Tq ¼ Ti =Td¼13 , i 2 M. The number of control points with the maximum value of the decision function is used to fit curve in all experiments. For example, an aircraft contour, F-15, is used to investigate the determination of the number of control points, and the tested results are depicted in Fig. 5. Fig. 5(a) shows the rates of correct recognition, fitting error and computational time, which utilize different numbers of control points. Meanwhile, Fig. 5(b) plots the decision function given different numbers of control points. Examining the decision function given different numbers of control points, it obtains the
Fig. 5. Experimental results for the number of control points. (a) The rate of correct recognition, Rq , fitting error, Fq , and computational time, Tq , are denoted by circle, square, and -mark lines, respectively. (b) The decision function given different numbers of control points.
S.-C. Tien et al. / Pattern Recognition Letters 24 (2003) 2047–2060
maximum value when the number of control points equals 21. Therefore, the number of control points is set to n ¼ 21 for all experiments in this study. Additionally, the computational time for the stability analysis and the database establishment is 31.8 min/model and 108.8 s, respectively. This work spends long time during the learning stage. Nevertheless, the stability analysis and the database establishment are run in off-line process, and a new aircraft model may not be added into the database in real-time operation. Consequently, the computational time of the learning stage is accepted reasonably, and it can be improved using a high-end computer. Second, the control points of the optimal fitting curve employ the stability analysis (Eq. (7)) to determine the geometrical relationship of the stable cross-ratios. Both stable cross-ratios and geometrical relationships are recorded in a database for aircraft recognition. The number of cross-ratios for each aircraft contour is 20, since the positions of both the first and last control points are set to identical to form a closed curve. An aircraft contour, for example F-14, takes the form of a cross-ratio vector and a geometrical relationship table, as presented in Table 1. In Table 1, the first column lists the cross-ratio vector, while the second column presents the position number of the five control points, which are used to calculate the stable cross-ratio in each row. The first control point serves as the focal point, and is connected with other four control points to calculate the cross-ratio. Additionally, the GRT method compares to the method developed by Lei (1990) that the crossratio was calculated using five consecutive control points, and the constitutions of the control points are shown in Fig. 6. Assume that the position of a control point is ðx; yÞ and is altered within a test range of 2 pixels, ðx 2; y 2Þ. Fig. 6(a) and (b) utilize the GRT of aircraft F-14 to construct five control points for calculating a cross-ratio, where the positions of the first control points are numbered as 3 and 10, respectively, and Fig. 6(c) and (d) utilize five consecutive control points to calculate the cross-ratio of the first control point. Moreover, the stable/unstable ratio is defined as the number of cross-ratios within/beyond the
2055
Table 1 Cross-ratio vector and geometrical relationship table of aircraft contour, F-14, in the database CR
GRT
18 45 29 21 7 6 6 1 13 44 10 16 14 1 6 40 13 11 13 11
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
8 5 11 15 20 5 16 10 16 6 10 6 1 19 20 10 11 1 5 6
3 18 10 1 2 2 2 17 3 1 1 2 2 9 2 6 3 2 6 5
17 3 15 8 14 8 13 19 12 5 3 5 5 12 6 13 5 5 14 14
18 20 19 7 17 15 20 7 19 16 6 18 12 16 9 18 10 8 18 16
threshold divides by the number of total test points in the test range. Experimental results as displayed in Fig. 6 and Table 2 show the cross-ratios calculated from GRT are more stable than those using five consecutive control points. Although the real cross-ratio is changed via varying the positions of the control points, the integral cross-ratio is altered within the threshold. More correct cross-ratios obtained in matching is better for deciding the test contour during the recognition stage. Therefore, the stable cross-ratios improve the reliability and robustness of recognition when the aircraft contour is altered through the noise interference or perspective distortion. Third, the matching threshold used in Eq. (20) is set to k ¼ b0:5 stdc, where std denotes the standard deviation vector of the cross-ratio descriptions in the database. Notably, the threshold vector, k, is set to (4, 6, 8, 6, 9, 9, 8, 11, 9, 9, 6, 6, 11, 15, 9, 6, 7, 8, 10, 7) for all experiments. The recognition results are given as the percentages of correct, uncertain, and inaccurate recognition. The proportion of uncertain results derived from the tested model may be influenced by noise with a large variance, and it is not classified as part of the proportion of inaccurate recognition.
2056
S.-C. Tien et al. / Pattern Recognition Letters 24 (2003) 2047–2060
Fig. 6. Geometrical relationships for calculating the cross-ratio vector. (a,b) Utilize GRT to construct five control points for calculating a cross-ratio, where the positions of control points are numbered as 3 and 10, respectively, and (c,d) utilize five consecutive control points to calculate the cross-ratio of the first control point. Control points are depicted by asterisks (). Dot-signs/-marks represent as cross-ratios within/beyond the threshold generated by the standard deviation method when control points vary within the test range.
Table 2 The stable/unstable ratio of the control points within the test range Method
Stable ratio
Unstable ratio
Control point: 3 (threshold ¼ 8)
GRT Lei
1 0.92
0 0.08
Control point: 10 (threshold ¼ 9)
GRT Lei
1 0.704
0 0.296
The test range contains 125 points in total.
5.2. The recognition stage During the recognition stage, a tested aircraft contour is also fitted by NURBS curves, and the
control points of the optimal curve are used to calculate a set of cross-ratios that employ the geometrical relationship table of each aircraft model in the database. Next, the calculated cross-
S.-C. Tien et al. / Pattern Recognition Letters 24 (2003) 2047–2060
ratios are compared with the cross-ratio vectors in the database, using Eq. (20), and the aircraft model is determined by Eq. (21). In the recognition experiments, the curve data of the tested aircraft are varied with different variance noise, and aircraft recognition is then performed to investigate the robustness and effectiveness of the method. Fig. 7 displays the percentages of the correct, uncertain, and inaccurate recognition, where each model is tested 100 times under various noise conditions. The percentages of correct, uncertain, and inaccurate model identification of each tested model are indicated by the circle, triangle, and mark lines, respectively. Moreover, the fitting curve also changes significantly with increasing noise variance. Fig. 8(a) illustrates the projective contours of aircraft F-14 under various noise conditions and the fitting curve, and Fig. 8(b) shows the recognition results. To examine the noise variation, the specifications of the aircraft F14 are used to calculate the pixel resolution in the
2057
model. The length and unwept wingspan of the aircraft are 18.6 and 19 m, respectively, and are depicted by 84 and 90 pixels in the aircraft model. The pixel resolution of the model is approximately 0.22 m/pixel. Consequently, the variation of the maximum variance of added noise is around 1 m in the tested model, and this large variation increases the percentage of uncertain recognition. The proposed method can achieve a reasonable recognition rate for recognizing the test contours of different noise variances. With the exception of the simulated models, a real image, aircraft F-16, as shown in Fig. 9(a), was used to investigate the feasibility of the proposed method. The extracted contour and the fitting curve are depicted as the gray dotted contour and the black line contour, respectively, and the control points are denoted as asterisks (). Fig. 9(b) presents the matching results. The matching results of model numbers 3 and 4 are all exceed the threshold (n=2), since both aircraft models have
Fig. 7. Recognition results for each aircraft model under various noise conditions, where the noise variance (r2 ) is (a) 0, (b) 1, (c) 2, (d) 3, (e) 4, and (f) 5.
2058
S.-C. Tien et al. / Pattern Recognition Letters 24 (2003) 2047–2060
Fig. 8. Projective contours with different noise variances and the fitting curves are shown in (a)–(f). The noise variance (r2 ) is (a) 0, (b) 1, (c) 2, (d) 3, (e) 4, and (f) 5. Meanwhile, recognition results are illustrated in (g), where circle, triangle, and -mark signs represent the rates of correct, uncertain, and inaccurate recognition, respectively.
Fig. 9. (a) A real image of an F-16 aircraft, and the fitting curve (black contour); and (b) the matching result is Mnum ¼ 3 i.e., F-16 aircraft, where the dotted line indicates the decision threshold in Eq. (21).
similar contours. Nevertheless, the matching result is determined by Eq. (21), and the recognition result reveals that the extracted contour matches the aircraft model F-16, i.e., Mnum ¼ 3, in the database. Notably, the experimental results demonstrate that the proposed method is both robust and effective for recognizing aircraft contours in images.
Finally, the proposed method has some advantages compared to the curve matching method developed by Alferez and Wang (1999), where both methods use cubic B-splines and 21 control points for curve fitting. The projective contour of aircraft F-14 is used as a test contour, as shown in Fig. 8(a)–(f), and the comparative results are listed in Table 3. From the table, both the memory re-
S.-C. Tien et al. / Pattern Recognition Letters 24 (2003) 2047–2060
2059
Table 3 Comparison of the proposed method and the curve matching method Recognition method
The proposed method The curve matching method
Memory requirement (bytes/model) 120 704
Recognition time (s/model) 6.81 7.28
quirements and computational time of the proposed method are less than for the curve matching method. Moreover, the proposed method also obtains better recognition rates than the recognition results of the curve matching method under increased noise variance. The correct recognition rate of the curve matching method reduces significantly when the noise variance exceeds three pixels.
6. Conclusion This study presents a novel method for recognizing aircraft based on NURBS curves fitting and cross-ratios. This method depends on the characteristics of the projective invariance of NURBS curves and cross-ratios. Stable cross-ratios form the feature vectors and are then used to establish a small database for recognition. Moreover, dynamic thresholds are applied to enhance the accuracy of feature comparisons. The experimental results show that the proposed method is robust and effective for aircraft recognition. However, the aircraft contours may be occluded to create different fitting curves, producing various cross-ratios as well as influencing the recognition results. Future investigations should consider ways of overcoming the occlusion problem.
References Alferez, R., Wang, Y.F., 1999. Geometric and illumination invariants for object recognition. IEEE Trans. Pattern Anal. Machine Intell. 21 (6), 505–535. Arbter, K., Snyder, W.E., Burkhardt, H., Hirzinger, G., 1990. Application of affine-invariant fourier descriptors to recog-
Correct recognition rate (%) Various noise (r2 ) 0
1
2
3
4
5
100 100
100 97
99 75
96 5
94 5
85 7
nition of 3-D objects. IEEE Trans. Pattern Anal. Machine Intell. 12 (7), 640–647. Bamieh, B., De Figueiredo, R.J.P., 1986. A general momentinvariants/attributed graph method for three-dimensional object recognition from single image. IEEE Trans. Robot. Automat. RA-2, 31–41. Cohen, F.S., Huang, Z., Yang, Z., 1995. Invariant matching and identification of curves using B-splines curve representation. IEEE Trans. Image Process. 4 (1), 1–10. Dudek, G., Tsotsos, J.K., 1997. Shape representation and recognition from multiscale curvature. Computer Vision and Image Understanding 68 (2), 170–189. Eggert, D., Bowyer, K., 1993. Computing the perspective projection aspect graph of solids of revolution. IEEE Trans. Pattern Anal. Machine Intell. 15 (2), 109–128. Huang, Z., Cohen, F.S., 1996. Affine-invariant B-spline moments for curve matching. IEEE Trans. Image Process. 5 (10), 1473–1480. Mokhtarian, F., 1997. Silhouette-based occluded object recognition through curvature scale space. Mach. Vision Appl. 10, 87–97. Lee, J.D., 2000. A new scheme for planar shape recognition using wavelets. Comput. Math. Appl. 39, 201–216. Lei, G., 1990. Recognition of planar objects in 3-D space from single perspective views using cross ratio. IEEE Trans. Robot. Automat. 6 (4), 432–437. Lei, Y., Wong, K.C., 1999. Detection and localisation of reflectional and rotational symmetry under weak perspective projection. Pattern Recognition 32, 167–180. Ma, W., Kruth, J.P., 1995. Parameterization of randomly measured points for least squares fitting of B-spline curves and surfaces. Computer-Aided Design 27 (9), 663–675. Piegl, L., 1991. On NURBS: A survey. IEEE Comput. Graphics Appl. 11 (1), 55–71. Piegl, L., Tiller, W., 1995. The NURBS Book. Springer-Verlag, Berlin. Rao, N.S.V., Wu, W., Glover, C.W., 1992. Algorithms for recognizing planar polygonal configurations using perspective image. IEEE Trans. Robot. Automat. 8 (4), 480–486. Reeves, A.P., Prokop, R.J., Andrew, S.E., Kuhl, F.P., 1988. Three-dimensional shape analysis using moments and Fourier descriptors. IEEE Trans. Pattern Anal. Machine Intell. 10 (6), 937–943. Rogers, D.F., Adams, J.A., 1990. Mathematical Elements for Computer Graphics, second ed. McGraw-Hill, New York.
2060
S.-C. Tien et al. / Pattern Recognition Letters 24 (2003) 2047–2060
Song, B.S., Yun, I.D., Lee, S.U., 2000. A target recognition technique employing geometric invariants. Pattern Recognition 33, 413–425.
Wu, M.-F., Sheu, H.-T., 1997. Contour-based correspondence using Fourier descriptors. IEE Proc.––Vision Image Signal Process. 144 (3), 150–160.
Using cross-ratios to model curve data for aircraft recognition Shen-Chi Tien a, Tsorng-Lin Chia
b,*
, Yibin Lu
a
a
b
Department of Electrical Engineering, Chung Cheng Institute of Technology, National Defense University, Tahsi, Taoyuan 335, Taiwan, ROC Department of Computer and Communication Engineering, Ming Chuan University, 5 Teh-Ming Rd., Gwei Shan District, Taoyuan 333, Taiwan, ROC Received 20 September 2002; received in revised form 25 February 2003
Abstract This study proposes a novel method that utilizes the projective invariance of NURBS (nonuniform rational B-splines) and cross-ratios to recognize aircraft in images. An aircraft contour extracted from an image can be fitted by a NURBS curve in approximation, and a skew-symmetry detection method is employed to determine the start point of the contour for the fitting curve. The control points on the fitting curve are then utilized to calculate a set of cross-ratios to represent the aircraft contour, and a small database is established for aircraft recognition. Moreover, all real crossratios are converted to integral cross-ratios to increase the error tolerance and matching facilitation. The experimental results demonstrate that the proposed method is robust and effective in aircraft recognition. Ó 2003 Elsevier Science B.V. All rights reserved. Keywords: NURBS curve; Cross-ratio; Projective invariant
1. Introduction Object contour is important in model-based vision systems that use geometric primitives or models to recover and recognize objects. A direct approach to object recognition is to match object boundary curves extracted from images, where the curves may undergo various transformations such as translation, rotation, shearing, and scaling. Since object contour can be altered through various projective transformations or viewpoints,
*
Corresponding author. Tel.: +886-33-507001; fax: +886-33294451. E-mail address: [email protected] (T.-L. Chia).
directly matching the curve data for object recognition is difficult. Consequently, numerous techniques have been developed for representing an object curve by a set of features or by models, such as Fourier descriptors (Wu and Sheu, 1997; Arbter et al., 1990), moment invariants (Bamieh and De Figueiredo, 1986; Reeves et al., 1988), curvatures (Mokhtarian, 1997; Dudek and Tsotsos, 1997), wavelets (Lee, 2000), and B-splines or NURBS (Cohen et al., 1995; Huang and Cohen, 1996; Alferez and Wang, 1999). However, Fourier descriptors and moment invariants, which require shape knowledge, suffer from the occlusion problem when some parts of the object data are unavailable. On the other hand, curvatures or arc length, which are based on local properties, have
0167-8655/03/$ - see front matter Ó 2003 Elsevier Science B.V. All rights reserved. doi:10.1016/S0167-8655(03)00042-4
2048
S.-C. Tien et al. / Pattern Recognition Letters 24 (2003) 2047–2060
difficulty in noise toleration, since the calculation of such techniques generally involves high-order derivatives. Wavelets and B-splines, which are based on the basis functions, have the correspondence problem of the start point of the curve data. Among these methods, B-splines have some attractive properties, including spatial uniqueness, continuity, local shape controllability, and invariance to affine transformation, but the B-spline parameters of the curve are not unique. Nevertheless, NURBS are generalizations of nonrational B-spline forms as well as rational and nonrational Bezier curves and surfaces. NURBS are invariant under translation, rotation, scaling, and shearing, as well as under parallel and perspective projections (Piegl, 1991). Accordingly, Alferez and Wang (1999) proposed a framework that applied the projective invariance of NURBS curves and illumination invariants for object recognition. The problem of identifying perspective invariants is formulated as a curve-fitting problem, and the fitting curve is used to recognize objects. However, the database must store the curve data of all aircraft models for matching, and the storage requirements are augmented rapidly when the number of aircraft to be recognized is increased significantly. On the other hand, invariant features can form a compact, intrinsic description of an object and be used to design recognition algorithms that are potentially more efficient than aspect-based approaches (Eggert and Bowyer, 1993). Some previous investigations (Lei, 1990; Rao et al., 1992) used cross-ratios to recognize planar objects in a three-dimensional space, and also formulated perspective invariants. However, objects were restricted to polygons and required accurate positions of vertex. This study presents a hybrid curve matching method which applies the projective invariance of both NURBS curves and cross-ratios for aircraft recognition. The recognition method comprises two stages, learning and recognition, with the flowchart as shown in Fig. 1. The proposed method applies primarily to the aircraft detected at a long distance from different viewpoints, and the aircraft image size is smaller than the distance between the aircraft and the viewpoint. For example, an optical tracking system tracks an air-
Fig. 1. Flowchart of aircraft recognition.
craft fly in the sky or a surveillance airplane (or satellite) detects the aircraft stay on the ground. The aircraft are viewed as planar objects approximately in the three-dimensional space, and the aircraft contours are regarded as coplanar points. Therefore, a three-dimensional aircraft recognition is translated into a matching contours problem in an image plane. To facilitate recognition, this study applies a fixed number of control points to fit the contour of the aircraft model. First, the aircraft contour is fitted using NURBS curves in approximation, and the fitting error of the curve is adjusted to an acceptable range. Since the control points of the NURBS curve are projective invariants (Alferez and Wang, 1999), the geometrical relationships of the control points are maintained in a projective plane. Nevertheless, the aircraft contour can be altered via different projective transformations or by viewing from various viewpoints, and the locations of control points can be adjusted according to the changed contour. The representation features must be invariant for recognition and must solve the problems of projective transformations. Consequently, this study employs
S.-C. Tien et al. / Pattern Recognition Letters 24 (2003) 2047–2060
cross-ratios, which form a cross-ratio vector, to represent the control points determined by the aircraft contour. All cross-ratio vectors of aircraft models can be combined to establish a database for aircraft recognition. To obtain a robust and reliable cross-ratio vector, this study analyzes the stability of the calculated cross-ratios, and a geometrical relationship table that is used to calculate a cross-ratio vector during the recognition stage is established for each aircraft model. Section 3 describes the stability analysis of cross-ratios. The cross-ratio vectors with fixed geometrical relationship tables in the database are insensitive to the viewpoint-object orientation and noise contamination, and these vectors maintain adequate separability between aircraft models. Additionally, the dimension of the extracted cross-ratio vectors should be small to reduce the computational burden in the recognition stage. The recognition method based on geometric invariants employs only a set of numbers to represent an aircraft in an image, and these numbers remain unchanged when the viewpoint orientation is modified or the parameters of projective transformations are altered. However, cross-ratios calculated from these control points are real numbers, and the variational range of cross-ratios is a nonuniform distribution. Though the recognition method can employ these real numbers as aircraft features, defining a fixed and reasonable threshold to restrict matching errors is difficult. Consequently, this study proposes an approach that converts real cross-ratios to integral cross-ratios. This approach generates a uniform distribution of integral cross-ratios and utilizes integral cross-ratios to represent aircraft contours. Using integral cross-ratios to describe aircraft contours enhances the error tolerance of calculated cross-ratios and simplifies matching features. Moreover, the aircraft contour formed by the integral cross-ratios is simply a string of integral numbers, markedly reducing the complexity of searching the database and comparing model data. This paper is arranged as follows. Section 2 elucidates the curve fitting method and the definition and conversion of cross-ratios. Section 3 then examines the cross-ratio stability. The procedure for representing aircraft contours and establishing
2049
the model database, as well as the algorithm of matching the cross-ratio vector are then described in Section 4. Meanwhile, Section 5 presents the experimental results. Finally, Section 6 presents conclusions.
2. Control points and cross-ratios Many contours of manmade objects are symmetrical and smooth curves such as aircraft, and their perspective projections change significantly at different viewpoints. Using a simple model to represent an object curve under different viewpoints is difficult. Hence, various works use Bsplines to interpolate curves and recognize objects. Although B-splines possess some properties such as spatial uniqueness, continuity, local shape controllability, and invariance to affine transformation, the B-spline parameters of the curve are not invariant in the projective transformation. Therefore, NURBS, a popular CAD/CAM representation, is used to describe curve objects, and NURBS curves preserve the properties of B-spline curves and maintain invariant under perspective projection. 2.1. Control points A curve cðuÞ can be expressed in terms of the associated rational basis functions Ri;k ðuÞ and n þ 1 control points: n n X 1 X cðuÞ ¼ wi pi Ni;k ðuÞ ¼ pi Ri;k ðuÞ; ð1Þ wðuÞ i¼0 i¼0 Ri;k ðuÞ ¼
wðuÞ ¼
wi Ni;k ðuÞ ; wðuÞ
n X
wi Ni;k ðuÞ;
ð2Þ
ð3Þ
i¼0
where pi ¼ ðxi ; yi Þ, i ¼ 0; 1; . . . ; n denotes the locations of the control points, while wi represents the weight assigned to each control point, and P R ðuÞ ¼ 1. Ni;k ðuÞ denotes the normalized Bi;k i spline functions of order k (degree k 1) defined over a certain knot sequence (or knot vector)
2050
S.-C. Tien et al. / Pattern Recognition Letters 24 (2003) 2047–2060
u ¼ fu0 ; u1 ; . . . ; unþk g, which is assumed to be nonperiodic. The normalized B-spline functions of order k are also generated using the Cox-deBoor recursion formulas (Rogers and Adams, 1990): 1 if ui 6 u 6 uiþ1 ; ð4Þ Ni;1 ðuÞ ¼ 0 otherwise; and Ni;k ðuÞ ¼
u ui Ni;k1 ðuÞ uiþk1 ui uiþk u þ Niþ1;k1 ðuÞ: uiþk uiþ1
CRL ðp1 ; p2 ; p3 ; p4 Þ ¼
jp1 p3 j jp2 p4 j : jp2 p3 j jp1 p4 j
ð6Þ
Taking vertex a of the polygonal object as a focal point, the angular definition of the cross-ratio is CRa ðh1 ; h2 ; h3 Þ ¼
sinðh1 þ h2 Þ sinðh2 þ h3 Þ ; sinðh2 Þ sinðh1 þ h2 þ h3 Þ
ð7Þ
and the following relationship is obtained and proven in (Lei, 1990), ð5Þ
A planar rational curve can be interpreted as the perspective projection of a nonrational Bspline curve onto the plane z ¼ 1 in three-dimensional space. To avoid the nonlinear problem, all weights are set to one, and the parameters and knots of the curve data are pre-computed. When the number of control points (n) and the order (k) of B-spline functions are given, the positions of the control points can be determined by least squares approximation (Piegl and Tiller, 1995). Section 4 delineates the computational method for locating the control points. 2.2. Cross-ratios A polygonal object in a three-dimensional space forms a projection in a two-dimensional image, and the extracted features of the object in the image are considered to be the vertices of the object, abcde, as illustrated in Fig. 2. Meanwhile, the lines that connect vertex a to other four vertices, b, c, d and e, intersect line L at four points, p1 , p2 , p3 , p4 , respectively. The cross-ratio formed by the four points, p1 , p2 , p3 , p4 , on line L is defined as
Fig. 2. Sketch map of the geometric relationship of crossratios.
CRL ðp1 ; p2 ; p3 ; p4 Þ ¼ CRa ðh1 ; h2 ; h3 Þ:
ð8Þ
Since the cross-ratio has the invariant property in projective geometry, its value is invariant under translation, rotation, and scaling transformations or various viewpoints. Consequently, cross-ratios are employed as features of extracted points in the projective images. These cross-ratios of an object form a feature vector, namely a cross-ratio vector, that acts as a model for recognizing the object in images. Since the control points of the NURBS fitting curve is projective invariants, they serve as the feature points of the curve to calculate a set of cross-ratios for representing the fitting curve. The cross-ratios calculated to represent control points are real numbers (2 R) that are influenced by noise, and the cross-ratio range is a nonuniform distribution of real numbers. Though the recognition method can use these real numbers to match features, defining a fixed and reasonable threshold for restricting matching errors is difficult. Consequently, the real cross-ratios are converted to integral cross-ratios (2 N ) using a statistical method, thus creating a conversion table. This approach increases the error tolerance and facilitates the matching of the integral cross-ratios. The conversion of real cross-ratios is performed as follows. First, the conversion uses Eq. (7) to calculate all cross-ratios in the constrained ranges, 0 6 h1 ; h2 ; h3 6 p and 0 6 h1 þ h2 þ h3 6 p, by increasing each included angle by a small increment Dhi , i ¼ 1; 2; 3, and analyzing the numerical distribution of the calculated results. Next, all calculated cross-ratios are divided into M parts, while the probability of a cross-ratio occurring in the Mth portion is considered to be equal. Finally, the range of integral cross-ratios is set to 1 M, and
S.-C. Tien et al. / Pattern Recognition Letters 24 (2003) 2047–2060
all real cross-ratios are converted to integral crossratios according to the conversion table. The cross-ratios presented in the following sections are integral cross-ratios.
3. Stability analysis The displacements of the control points of the fitting curve, influenced by noise or the extracted thresholds, may vary cross-ratios since the calculated cross-ratio involves five control points. Therefore, the real cross-ratios are converted into integral cross-ratios to increase the error tolerance in matching cross-ratios. However, cross-ratios may be strongly influenced by noise variance. Consequently, a fixed geometrical relationship among control points is utilized to produce a stable cross-ratio, and this approach can reduce the influence of noise. This work follows the analytical method of Song et al. (2000) to derive a criterion for selecting a stable cross-ratio that is suitable for the proposed method. Given that the cross-ratio of five points is s, Eq. (7), CRa ðh1 ; h2 ; h3 Þ ¼ s, can be represented by the following formula: 1
s ¼ 21
jabjjadj sinðh1 þ h2 Þ 12 jacjjaej sinðh2 þ h3 Þ
S1 S2 ; S3 S4
DS3 DS4 DS1 DS2 : þ ER ffi S3 S4 S1 S2
ð12Þ
This equation shows that the error rate of the cross-ratio is a function of the variation of the measured area of the triangle. Differentiating Si yields, oSi ¼ 12ðBi sin ai oAi þ Ai sin ai oBi þ Ai Bi cos ai oai Þ;
ð13Þ
and DSi DAi DBi Dai ffi þ þ ; Si Ai Bi tan ai
ð14Þ
where the variation of Dai is also affected by the vertex displacements of the two adjacent edges, and the maximum variation of Dai is Dai ffi
DAi DBi þ ; Ai Bi
ð15Þ
such that, DSi 1 DAi DBi ffi 1þ þ : tan ai Si Ai Bi Finally, let 1 1 1 Qi ¼ 1 þ þ ; tan ai Ai Bi
ð16Þ
i ¼ 1; 2; 3; 4
and the criterion utilized to select a stable crossratio is specified as
jacjjadj sinðh2 Þ 12 jabjjaej sinðh1 þ h2 þ h3 Þ 2
¼
2051
ð9Þ
where Si ¼ 12 Ai Bi sinðai Þ, i ¼ 1; 2; 3; 4 is the area of the triangle, and ai , bi are the lengths of the two edges adjacent to the including angle ai . The error rate of the cross-ratio influenced by noise or displacement is defined as Ds ð10Þ ER ¼ ; s
SK ¼ jQ3 þ Q4 Q1 Q2 j:
ð17Þ
A smaller SK corresponds to a more stable cross-ratio and indicates that the cross-ratio variation is little influenced by noise or control point displacements. Consequently, a stable cross-ratio is selected for the control point, and the geometrical relationship, which is used to constitute the stable cross-ratio, is recorded to form a mapping table for calculating cross-ratios during the recognition stage.
where Ds denotes the variation of the cross-ratio. Differentiating Eq. (10) then yields os ¼
4. Recognition method
S2 S1 S1 S2 S1 S2 oS1 þ oS2 2 oS3 oS4 ; S3 S4 S3 S4 S3 S4 S3 S43 ð11Þ
and substituting Eqs. (9) and (11) into Eq. (10) yields
4.1. Aircraft contour representation Suppose that an aircraft contour is extracted from an image employing appropriate
2052
S.-C. Tien et al. / Pattern Recognition Letters 24 (2003) 2047–2060
Fig. 3. The processing steps for representing aircraft contour. (a) The projective contour of aircraft F-14 is depicted with a detected skew symmetry axis and two intersected points. (b) The NURBS fitting curve and control points are depicted by black line and asterisks (), respectively. (c) An aircraft contour is represented by a cross-ratio vector and a geometrical relationship table.
preprocesses, and the curve data are stored in an ordered sequence (p points curve data c ¼ ½x; yT , x ¼ ½x1 ; x2 ; . . . ; xp T , y ¼ ½y1 ; y2 ; . . . ; yp T , where T represents the matrix transposition). An aircraft contour can then be represented by a cross-ratio vector which is generated according to the following steps (refer Fig. 3): (a) The aircraft contour is generally formed as a symmetrical closed curve against the medial axis of the contour. This study employs a Hough-based symmetry detection method (Lei and Wong, 1999) to determine the skew symmetry axis of the contour. The skew symmetry axis intersects with the contour at two intersection points, which are assumed to be the start points for the fitting curve. The cross-ratio vector of a selected start point can be obtained through processing via steps (b)–(d). (b) The curve data are stored in an ordered sequence from a selected start point, as determined in step (a), and the curve data parameterization is performed using the inverse chard length method (Huang and Cohen, 1996) and knots are calculated using the average knots method (Ma and Kruth, 1995). Finally, the nonrational B-spline basis functions of order k, N, are calculated using Eqs. (4) and (5), 2 3 N0;k ðu1 Þ N1;k ðu1 Þ Nn;k ðu1 Þ 6 N ðu Þ N ðu Þ N ðu Þ 7 1;k 2 n;k 2 7 6 0;k 2 7: N ¼6 ð18Þ .. .. .. 6 7 4 5 . . . N0;k ðup Þ N1;k ðup Þ Nn;k ðup Þ
(c) The curve data are then fitted by nonrational B-spline basis functions of order k, while the number of control points is given as n þ 1, using the approximation method. The locations of the control points of the fitting curve are solved and formed an ordered sequence. Let p ¼ ½p0 ; p1 ; . . . ; pn T denote the locations of control points. The locations of the control points can then be determined by minimizing the fitting error using the least squares method, N T Np ¼ N T c:
ð19Þ
Since the aircraft contour is a closed curve, the first and last control points are assumed to locate at the same location. Consequently, the number of control points used to constitute the cross-ratio vector is n, i.e., pi , i ¼ 1; 2; . . . ; n. (d) According to the definition of cross-ratios, a selected control point, which served as a focal point, is combined with another four control points, and a cross-ratio can be calculated to represent the control point. For example, a selected control point, pi , which is combined with another four control points, pj1 , pj2 , pj3 , pj4 , is applied to determine a cross-ratio using Eq. (7), CRpi ð\pj1 pi pj2 ; \pj2 pi pj3 ; \pj3 pi pj4 Þ, i 6¼ j1 6¼ j2 6¼ j3 6¼ j4 . Since the selected control point constitutes a cross-ratio when combined with any four control points on the fitting curve, the stability analysis of cross-ratios, described in Section 3, is employed to determine a stable cross-ratio. Notably, the stability of the cross-ratio, CRpi , is determined by Eq. (17), SKpi .
S.-C. Tien et al. / Pattern Recognition Letters 24 (2003) 2047–2060
Comparing all SKpi calculated from various geometrical relationships for the selected control point pi , the cross-ratio and the geometrical relationship with the smallest SKpi are recorded to represent the control point. The control point record is a crossratio CRpi and a row vector ½i; j1 ; j2 ; j3 ; j4 . All determined cross-ratios, which represent the control points, form a cross-ratio vector, CR, while the associated row vectors form a geometrical relationship table, GRT, which is a n 5 matrix. (e) Finally, cross-ratio vectors with different start points are compared with the model database, and the curve data recognition is determined based on the matching results. 4.2. Matching method Suppose that the database contains m records of different aircraft models, and the cross-ratio vector of the jth model is expressed as Dj ¼ ðDj1 ; Dj2 ; . . . ; Djn Þ, j ¼ 1; 2; . . . ; m. The cross-ratio vector of the tested aircraft model is T ¼ ðT1 ; T2 ; . . . ; Tn Þ. Data matching is performed by direct comparison, and the matching formula is n X Vj ¼ Sji ; j ¼ 1; 2; . . . ; m; ð20Þ i¼1
where 1 jTi Dji j 6 ki ; Sji ¼ 0 otherwise;
2053
ki represents the threshold of the ith cross-ratio description and is deduced from the standard deviation of the ith column cross-ratios in the database, and all thresholds form a threshold vector, k. Since two start points are determined by detecting the skew symmetry axis of the contour, two matching results (Vj1 and Vj2 ) are used to identify the correct aircraft model. The determined model number, Mnum , is the argument of the maximum of two matching results, where one of the matching results must exceed n=2. The determining formula is 8 argj maxðVj1 ; Vj2 Þ > > < if Vj1 P n=2 or Vj2 P n=2; ð21Þ Mnum ¼ > 0 > : otherwise: If the number of the recognized aircraft model, Mnum , equals zero, the tested curve data are classified as an uncertain aircraft model.
5. Experimental results Suppose that all aircraft contours are extracted via appropriate preprocesses from images which are normalized to 100 100 pixels, and the positions of curve data (contours) are stored in ordered sequences for all experiments. Fig. 4 displays the contours of all aircraft models used during the learning and recognition stages.
Fig. 4. Aircraft contours in the database.
2054
S.-C. Tien et al. / Pattern Recognition Letters 24 (2003) 2047–2060
5.1. The learning stage Based on the representation method elucidated in Section 4.1, this study constitutes a database based on the control points of the fitting curve in the learning stage. First, an aircraft contour is fitted approximately by NURBS curves, and the order of NURBS curves is set to k ¼ 4, a cubic B-spline curve. Since the aircraft contour is assumed to be located on the same plane in a three-dimensional space, all weights are set to one for the fitting curve. Additionally, the number of control points required to obtain an optimal fitting curve must be determined. Three factors are considered to determine the number of control points: (a) correct recognition rate; (b) decreased fitting error; and (c) computational time of curve fitting and matching. Consequently, this study utilizes a decision function to determine the number of control points for the fitting curve. Assume that a tested set of the number of control point is M ¼ f2j þ d j j ¼ 0; 1; . . . ; 8; d ¼ 13g, where d denotes the minimum of the number of control points in the tested set. Moreover, let c represent the curve data of a tested aircraft, which has p point curve data, and let f i , i 2 M, be the fitting curve data given i control points. The fitting error is defined as
P Fi ¼ p kc f i k, i 2 M, and the computational time, Ti , is defined as the elapsed time for recognizing a tested contour, which is fitted by i control points, executed on a PC (Pentium III, 1 GHz). The decision function is then defined as Dfq ¼
Rq ; Tq Fq
q 2 M;
ð22Þ
where q denotes the number of control points in the tested set. Rq represents the correct recognition rate of an aircraft model recognized by using q control points to fit the curve data. Fq is the fitting error rate, which is normalized by Fd¼13 , Fq ¼ Fi =Fd¼13 , i 2 M, and Tq denotes the rate of the computational time of curve fitting and matching, which is normalized by Td¼13 , Tq ¼ Ti =Td¼13 , i 2 M. The number of control points with the maximum value of the decision function is used to fit curve in all experiments. For example, an aircraft contour, F-15, is used to investigate the determination of the number of control points, and the tested results are depicted in Fig. 5. Fig. 5(a) shows the rates of correct recognition, fitting error and computational time, which utilize different numbers of control points. Meanwhile, Fig. 5(b) plots the decision function given different numbers of control points. Examining the decision function given different numbers of control points, it obtains the
Fig. 5. Experimental results for the number of control points. (a) The rate of correct recognition, Rq , fitting error, Fq , and computational time, Tq , are denoted by circle, square, and -mark lines, respectively. (b) The decision function given different numbers of control points.
S.-C. Tien et al. / Pattern Recognition Letters 24 (2003) 2047–2060
maximum value when the number of control points equals 21. Therefore, the number of control points is set to n ¼ 21 for all experiments in this study. Additionally, the computational time for the stability analysis and the database establishment is 31.8 min/model and 108.8 s, respectively. This work spends long time during the learning stage. Nevertheless, the stability analysis and the database establishment are run in off-line process, and a new aircraft model may not be added into the database in real-time operation. Consequently, the computational time of the learning stage is accepted reasonably, and it can be improved using a high-end computer. Second, the control points of the optimal fitting curve employ the stability analysis (Eq. (7)) to determine the geometrical relationship of the stable cross-ratios. Both stable cross-ratios and geometrical relationships are recorded in a database for aircraft recognition. The number of cross-ratios for each aircraft contour is 20, since the positions of both the first and last control points are set to identical to form a closed curve. An aircraft contour, for example F-14, takes the form of a cross-ratio vector and a geometrical relationship table, as presented in Table 1. In Table 1, the first column lists the cross-ratio vector, while the second column presents the position number of the five control points, which are used to calculate the stable cross-ratio in each row. The first control point serves as the focal point, and is connected with other four control points to calculate the cross-ratio. Additionally, the GRT method compares to the method developed by Lei (1990) that the crossratio was calculated using five consecutive control points, and the constitutions of the control points are shown in Fig. 6. Assume that the position of a control point is ðx; yÞ and is altered within a test range of 2 pixels, ðx 2; y 2Þ. Fig. 6(a) and (b) utilize the GRT of aircraft F-14 to construct five control points for calculating a cross-ratio, where the positions of the first control points are numbered as 3 and 10, respectively, and Fig. 6(c) and (d) utilize five consecutive control points to calculate the cross-ratio of the first control point. Moreover, the stable/unstable ratio is defined as the number of cross-ratios within/beyond the
2055
Table 1 Cross-ratio vector and geometrical relationship table of aircraft contour, F-14, in the database CR
GRT
18 45 29 21 7 6 6 1 13 44 10 16 14 1 6 40 13 11 13 11
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
8 5 11 15 20 5 16 10 16 6 10 6 1 19 20 10 11 1 5 6
3 18 10 1 2 2 2 17 3 1 1 2 2 9 2 6 3 2 6 5
17 3 15 8 14 8 13 19 12 5 3 5 5 12 6 13 5 5 14 14
18 20 19 7 17 15 20 7 19 16 6 18 12 16 9 18 10 8 18 16
threshold divides by the number of total test points in the test range. Experimental results as displayed in Fig. 6 and Table 2 show the cross-ratios calculated from GRT are more stable than those using five consecutive control points. Although the real cross-ratio is changed via varying the positions of the control points, the integral cross-ratio is altered within the threshold. More correct cross-ratios obtained in matching is better for deciding the test contour during the recognition stage. Therefore, the stable cross-ratios improve the reliability and robustness of recognition when the aircraft contour is altered through the noise interference or perspective distortion. Third, the matching threshold used in Eq. (20) is set to k ¼ b0:5 stdc, where std denotes the standard deviation vector of the cross-ratio descriptions in the database. Notably, the threshold vector, k, is set to (4, 6, 8, 6, 9, 9, 8, 11, 9, 9, 6, 6, 11, 15, 9, 6, 7, 8, 10, 7) for all experiments. The recognition results are given as the percentages of correct, uncertain, and inaccurate recognition. The proportion of uncertain results derived from the tested model may be influenced by noise with a large variance, and it is not classified as part of the proportion of inaccurate recognition.
2056
S.-C. Tien et al. / Pattern Recognition Letters 24 (2003) 2047–2060
Fig. 6. Geometrical relationships for calculating the cross-ratio vector. (a,b) Utilize GRT to construct five control points for calculating a cross-ratio, where the positions of control points are numbered as 3 and 10, respectively, and (c,d) utilize five consecutive control points to calculate the cross-ratio of the first control point. Control points are depicted by asterisks (). Dot-signs/-marks represent as cross-ratios within/beyond the threshold generated by the standard deviation method when control points vary within the test range.
Table 2 The stable/unstable ratio of the control points within the test range Method
Stable ratio
Unstable ratio
Control point: 3 (threshold ¼ 8)
GRT Lei
1 0.92
0 0.08
Control point: 10 (threshold ¼ 9)
GRT Lei
1 0.704
0 0.296
The test range contains 125 points in total.
5.2. The recognition stage During the recognition stage, a tested aircraft contour is also fitted by NURBS curves, and the
control points of the optimal curve are used to calculate a set of cross-ratios that employ the geometrical relationship table of each aircraft model in the database. Next, the calculated cross-
S.-C. Tien et al. / Pattern Recognition Letters 24 (2003) 2047–2060
ratios are compared with the cross-ratio vectors in the database, using Eq. (20), and the aircraft model is determined by Eq. (21). In the recognition experiments, the curve data of the tested aircraft are varied with different variance noise, and aircraft recognition is then performed to investigate the robustness and effectiveness of the method. Fig. 7 displays the percentages of the correct, uncertain, and inaccurate recognition, where each model is tested 100 times under various noise conditions. The percentages of correct, uncertain, and inaccurate model identification of each tested model are indicated by the circle, triangle, and mark lines, respectively. Moreover, the fitting curve also changes significantly with increasing noise variance. Fig. 8(a) illustrates the projective contours of aircraft F-14 under various noise conditions and the fitting curve, and Fig. 8(b) shows the recognition results. To examine the noise variation, the specifications of the aircraft F14 are used to calculate the pixel resolution in the
2057
model. The length and unwept wingspan of the aircraft are 18.6 and 19 m, respectively, and are depicted by 84 and 90 pixels in the aircraft model. The pixel resolution of the model is approximately 0.22 m/pixel. Consequently, the variation of the maximum variance of added noise is around 1 m in the tested model, and this large variation increases the percentage of uncertain recognition. The proposed method can achieve a reasonable recognition rate for recognizing the test contours of different noise variances. With the exception of the simulated models, a real image, aircraft F-16, as shown in Fig. 9(a), was used to investigate the feasibility of the proposed method. The extracted contour and the fitting curve are depicted as the gray dotted contour and the black line contour, respectively, and the control points are denoted as asterisks (). Fig. 9(b) presents the matching results. The matching results of model numbers 3 and 4 are all exceed the threshold (n=2), since both aircraft models have
Fig. 7. Recognition results for each aircraft model under various noise conditions, where the noise variance (r2 ) is (a) 0, (b) 1, (c) 2, (d) 3, (e) 4, and (f) 5.
2058
S.-C. Tien et al. / Pattern Recognition Letters 24 (2003) 2047–2060
Fig. 8. Projective contours with different noise variances and the fitting curves are shown in (a)–(f). The noise variance (r2 ) is (a) 0, (b) 1, (c) 2, (d) 3, (e) 4, and (f) 5. Meanwhile, recognition results are illustrated in (g), where circle, triangle, and -mark signs represent the rates of correct, uncertain, and inaccurate recognition, respectively.
Fig. 9. (a) A real image of an F-16 aircraft, and the fitting curve (black contour); and (b) the matching result is Mnum ¼ 3 i.e., F-16 aircraft, where the dotted line indicates the decision threshold in Eq. (21).
similar contours. Nevertheless, the matching result is determined by Eq. (21), and the recognition result reveals that the extracted contour matches the aircraft model F-16, i.e., Mnum ¼ 3, in the database. Notably, the experimental results demonstrate that the proposed method is both robust and effective for recognizing aircraft contours in images.
Finally, the proposed method has some advantages compared to the curve matching method developed by Alferez and Wang (1999), where both methods use cubic B-splines and 21 control points for curve fitting. The projective contour of aircraft F-14 is used as a test contour, as shown in Fig. 8(a)–(f), and the comparative results are listed in Table 3. From the table, both the memory re-
S.-C. Tien et al. / Pattern Recognition Letters 24 (2003) 2047–2060
2059
Table 3 Comparison of the proposed method and the curve matching method Recognition method
The proposed method The curve matching method
Memory requirement (bytes/model) 120 704
Recognition time (s/model) 6.81 7.28
quirements and computational time of the proposed method are less than for the curve matching method. Moreover, the proposed method also obtains better recognition rates than the recognition results of the curve matching method under increased noise variance. The correct recognition rate of the curve matching method reduces significantly when the noise variance exceeds three pixels.
6. Conclusion This study presents a novel method for recognizing aircraft based on NURBS curves fitting and cross-ratios. This method depends on the characteristics of the projective invariance of NURBS curves and cross-ratios. Stable cross-ratios form the feature vectors and are then used to establish a small database for recognition. Moreover, dynamic thresholds are applied to enhance the accuracy of feature comparisons. The experimental results show that the proposed method is robust and effective for aircraft recognition. However, the aircraft contours may be occluded to create different fitting curves, producing various cross-ratios as well as influencing the recognition results. Future investigations should consider ways of overcoming the occlusion problem.
References Alferez, R., Wang, Y.F., 1999. Geometric and illumination invariants for object recognition. IEEE Trans. Pattern Anal. Machine Intell. 21 (6), 505–535. Arbter, K., Snyder, W.E., Burkhardt, H., Hirzinger, G., 1990. Application of affine-invariant fourier descriptors to recog-
Correct recognition rate (%) Various noise (r2 ) 0
1
2
3
4
5
100 100
100 97
99 75
96 5
94 5
85 7
nition of 3-D objects. IEEE Trans. Pattern Anal. Machine Intell. 12 (7), 640–647. Bamieh, B., De Figueiredo, R.J.P., 1986. A general momentinvariants/attributed graph method for three-dimensional object recognition from single image. IEEE Trans. Robot. Automat. RA-2, 31–41. Cohen, F.S., Huang, Z., Yang, Z., 1995. Invariant matching and identification of curves using B-splines curve representation. IEEE Trans. Image Process. 4 (1), 1–10. Dudek, G., Tsotsos, J.K., 1997. Shape representation and recognition from multiscale curvature. Computer Vision and Image Understanding 68 (2), 170–189. Eggert, D., Bowyer, K., 1993. Computing the perspective projection aspect graph of solids of revolution. IEEE Trans. Pattern Anal. Machine Intell. 15 (2), 109–128. Huang, Z., Cohen, F.S., 1996. Affine-invariant B-spline moments for curve matching. IEEE Trans. Image Process. 5 (10), 1473–1480. Mokhtarian, F., 1997. Silhouette-based occluded object recognition through curvature scale space. Mach. Vision Appl. 10, 87–97. Lee, J.D., 2000. A new scheme for planar shape recognition using wavelets. Comput. Math. Appl. 39, 201–216. Lei, G., 1990. Recognition of planar objects in 3-D space from single perspective views using cross ratio. IEEE Trans. Robot. Automat. 6 (4), 432–437. Lei, Y., Wong, K.C., 1999. Detection and localisation of reflectional and rotational symmetry under weak perspective projection. Pattern Recognition 32, 167–180. Ma, W., Kruth, J.P., 1995. Parameterization of randomly measured points for least squares fitting of B-spline curves and surfaces. Computer-Aided Design 27 (9), 663–675. Piegl, L., 1991. On NURBS: A survey. IEEE Comput. Graphics Appl. 11 (1), 55–71. Piegl, L., Tiller, W., 1995. The NURBS Book. Springer-Verlag, Berlin. Rao, N.S.V., Wu, W., Glover, C.W., 1992. Algorithms for recognizing planar polygonal configurations using perspective image. IEEE Trans. Robot. Automat. 8 (4), 480–486. Reeves, A.P., Prokop, R.J., Andrew, S.E., Kuhl, F.P., 1988. Three-dimensional shape analysis using moments and Fourier descriptors. IEEE Trans. Pattern Anal. Machine Intell. 10 (6), 937–943. Rogers, D.F., Adams, J.A., 1990. Mathematical Elements for Computer Graphics, second ed. McGraw-Hill, New York.
2060
S.-C. Tien et al. / Pattern Recognition Letters 24 (2003) 2047–2060
Song, B.S., Yun, I.D., Lee, S.U., 2000. A target recognition technique employing geometric invariants. Pattern Recognition 33, 413–425.
Wu, M.-F., Sheu, H.-T., 1997. Contour-based correspondence using Fourier descriptors. IEE Proc.––Vision Image Signal Process. 144 (3), 150–160.