- Email: [email protected]
On the choice of consistent canonical form during moment normalization
Pattern Recognition Letters 24 (2003) 3205–3215 www.elsevier.com/locate/patrec
On the choice of consistent canonical form during moment normalization Yani Zhang a, Changyun Wen a
a,*
, Ying Zhang b, Yeng Chai Soh
a
School of Electrical and Electronic Engineering, Nanyang Technological University, Nanyang Avenue, Singapore 639798, Singapore b Singapore Institute of Manufacturing Technology, 71 Nanyang Drive, Singapore 638075, Singapore Received 31 October 2002; received in revised form 13 August 2003
Abstract The key issue in normalization with respect to geometric transformation is how to obtain a consistent canonical form which remains unchanged for different geometric transformation related images. Due to ambiguities inherent in the specified normalization methods, more than one canonical form may occur during the normalization procedure. This causes difficulties in obtaining the expected invariant features or the transformation parameters through normalization. This paper aims to provide general schemes to analyze the ambiguity characteristics and to derive the ambiguity matrices through which ambiguities can be eliminated. Three kinds of ambiguities caused by the multi-roots of highorder polynomials, the symmetrical normalization constraints, and the reflection are addressed and solutions are provided to obtain a consistent canonical form for each kind of ambiguities. 2003 Elsevier B.V. All rights reserved. Keywords: Canonical form; Moment normalization; Affine transformation; Ambiguity
1. Introduction In pattern recognition, some function parameters such as moments and Fourier coefficients are usually used to describe the image features to be recognized. In order to uniquely identify the features, invariants are constructed using the function parameters to eliminate the effects which the geometric transformation imposes on the function
*
Corresponding author. Tel.: +65-6790-4947/790-4947; fax: +65-6792-0425/792-0415. E-mail address: [email protected] (C. Wen).
parameters. Normalization is an effective and systematic approach for this purpose (Reeves et al., 1988; Leu, 1989; Sinclair and Blake, 1994; Adler and Krishnan, 1998). The basic idea of image normalization is to transform the original image into its canonical form so that the function parameters of the canonical form are independent of the geometric transformations and at the same time they retain all the relevant information of the original image. The normalization is usually achieved by imposing the so-called normalization constraints on some chosen function parameters so that all the remaining function parameters are independent of the geometric transformations.
0167-8655/$ - see front matter 2003 Elsevier B.V. All rights reserved. doi:10.1016/j.patrec.2003.08.006
3206
Y. Zhang et al. / Pattern Recognition Letters 24 (2003) 3205–3215
For two dimensional planar images, normalization methods based on affine transformation were studied extensively in the literature. For example, it has been shown that the affine transformation parameters can be estimated through the normalization matrix which transforms the original image to its canonical form (Wohn and Wu, 1990; Reiss, 1993; Tzanetakis and Tziritas, 1998). In (Shen and Ip, 1997), a generalized image normalization method was proposed to handle shaped planar images. The normalization of curve, shape, or gray-level planar images are discussed in references (Pei and Lin, 1995; Rothe et al., 1996; Voss and Suesse, 1997). Recently, Zhang et al. proposed a normalization method for constructing moment invariants against both blur and affine transforms (Zhang et al., 2002). However, all these works suffer from one common problem in that the expected invariant features and transformation parameters may not be obtained if the canonical form is non-unique. In some moment based normalization methods (Rothe et al., 1996; Voss and Suesse, 1997; Zhang et al., 2002), there do exist more than one canonical form due to inherent ambiguities. This gives rise to the non-consistence of the canonical form. The problem was initially but briefly addressed in (Zhang et al., 2002). With a focus on planar images with two dimensional affine transformation, this paper will systematically analyze the causes of ambiguities and provide ways to obtain a consistent canonical form that is free of ambiguities. In this paper, three kinds of ambiguities are identified as the sources which cause inconsistence of the canonical form. The first kind of ambiguity is caused by the symmetrical characteristic of the normalization constraints. The second kind of ambiguity is due to the roots of a high-order polynomial. For example, in the polynomial normalization method (Rothe et al., 1996), the third order central moment l30 ¼ 0 is used as the x-shearing normalization constraint. There are three different solutions which satisfy l30 ¼ 0 and these solutions will yield three canonical forms. The last kind of ambiguity is due to the reflection ambiguity, which was only briefly addressed in (Pei and Lin, 1995; Avrithis et al., 2000). In the study presented here, it is noted that reflection transformation does occur not only during the
imaging process but also during the normalization procedure. In the following sections, the above-mentioned three kinds of ambiguities are first analyzed, each illustrated with examples. Then a case study is conducted in the presence of combined ambiguities. For each kind of ambiguity, general schemes are proposed to obtain the unique canonical form. Although the study in this paper is developed using some common normalization constraints used in previous studies, the proposed ideas can also be used in other moment normalization methods. The case study shows that in the presence of ambiguities, the schemes proposed in this paper can effectively obtain a consistent canonical form. Analysis and experimental results verify the effectiveness.
2. Moment normalization under affine transformation A general affine transformation is a linear transformation plus a translation. In the image coordinates, it can be represented by 0 x x b ¼A þ 1 ; ð1Þ y0 b2 y a12 a where A ¼ 11 is called the homogeneous a21 a22 affine transformation matrix (Rothe et al., 1996; Voss and Suesse, 1997). Since the translation can be overcome by using the central moments, only the homogeneous affine transform is considered in the studies of moment normalization. In order to avoid solving complex non-linear systems of equations during normalization, the homogeneous affine transformation matrix is decomposed. In the literature (Rothe et al., 1996, Voss and Suesse, 1997; Zhang et al., 2000), two kinds of decomposition called XSR and XYS decompositions are used. 2.1. XSR decomposition The widely used affine decomposition is to decompose the affine matrix A into a shearing, an
Y. Zhang et al. / Pattern Recognition Letters 24 (2003) 3205–3215
anisotrope scaling matrix, and a rotation matrix, i.e. a11 a12 cosðhÞ sinðhÞ a 0 1 b ¼ ; a21 a22 sinðhÞ cosðhÞ 0 c 0 1 ð2Þ where a; c; b 2 R, h 2 ½0; 2p. The condition that detðAÞ 6¼ 0 is required to ensure the uniqueness of the decomposition. Based on this decomposition, the normalization constraints l11 ¼ 0;
l20 ¼ l02 ¼ 1;
l30 þ l12 ¼ 0
ð3Þ
are used in (Pei and Lin, 1995; Rothe et al., 1996; Reiss, 1993) to normalize the shear, scale and rotation, respectively. In (3), lpq denote the ðp þ qÞth central moments (Rothe et al., 1996; Zhang et al., 2002). 2.2. XYS decomposition The homogenous affine transformation matrix A can also be decomposed as an x-shearing, a yshearing and an anisotrope scaling matrix (Rothe et al., 1996) a11 a12 a 0 1 0 1 b ¼ ð4Þ a21 a22 0 d c 1 0 1 with a; d; c; b 2 C. If detðAÞ 6¼ 0 and a11 6¼ 0, the uniqueness of the decomposition can be guaranteed. Under this decomposition, Rothe et al. (1996) used the constraints l30 ¼ 0;
l11 ¼ 0;
l20 ¼ l02 ¼ 1
ð5Þ
3207
normalization constraints exists in the normalization methods based on XYS decomposition.
3. Ambiguity due to symmetrical normalization constraints 3.1. Cause of the ambiguity Since the homogeneous affine transformation matrix has four independent parameters, four constraints should be used to implement the normalization. If the constraints are selected as lp1 q1 ¼ lq1 p1 ¼ c1 and lp2 q2 ¼ lq2 p2 ¼ c2 , they are symmetrical about x and y directions and they cannot discriminate the x and y directions. We assume that Iðx; yÞ has two canonical forms N ðxn ; yn Þ and N 0 ðx0n ; yn0 Þ after normalization and they are related by 0 xn x ¼ Am n ; ð8Þ yn0 yn where Am is called the ambiguity matrix. Through Am we can determine if there exists ambiguity of canonical forms. If there is no ambiguity, Am must be a unit matrix. Otherwise, there exists ambiguity. Now we use the iterative normalization method (Voss and Suesse, 1997) as an example to illustrate this problem. This normalization procedure is based on the decomposition in Eq. (4) and uses constraints that l13 ¼ 0 and l31 ¼ 0 to implement the x-shearing and y-shearing normalization simultaneously and l20 ¼ l02 ¼ 1 to normalize anisotrope scaling. For the iterative normalization method we have the following proposition.
while Voss and Suesse (1997) used the constraints l31 ¼ l13 ¼ 0;
l20 ¼ l02 ¼ 1
ð6Þ
to process normalization. Recently, Zhang et al. (2002) selected the constraints l30 ¼ l03 ¼ 0;
l21 ¼ l12 ¼ 1
ð7Þ
to derive blur and affine combined moment invariants. As shown in the following sections, reflection ambiguity exists in both XSR and XYS decomposition based normalization methods. The ambiguity caused by multi-roots and symmetrical
Proposition 1. Under normalization constraints that l31 ¼ 0, l13 ¼ 0, l20 ¼ 1, and l02 ¼ 1, it is possible that 0 1 Am ¼ : ð9Þ 1 0
a b cf 0 , and lcf pq and lpq denote c d the moment of N ðxn ; yn Þ and N 0 ðx0n ; yn0 Þ respectively. Applying the normalization constraints and using Eq. (8), we have
Proof. Let Am ¼
3208
8 0 > > lcf > 20 > > 0 > > lcf > 02 > > < cf 0 l31 > > > > 0 > > > lcf > 13 > > :
Y. Zhang et al. / Pattern Recognition Letters 24 (2003) 3205–3215 2 ¼ a2 þ 2ablcf 11 þ b ¼ 1; 2 ¼ c2 þ 2cdlcf 11 þ d ¼ 1; cf 2 2 ¼ a3 clcf 40 þ ð3ab c þ 2a bdÞl22
þ b3 dlcf 04 ¼ 0;
ð10Þ
cf 2 2 ¼ ac3 lcf 40 þ ð3acd þ 2bc dÞl22
þ bd 3 lcf 04 ¼ 0:
cf cf cf Eq. (10) holds independently of lcf 11 , l04 , l40 , l22 . Thus we have 2 a þ b2 ¼ c2 þ d 2 ¼ 1; ð11Þ ab ¼ cd ¼ ac ¼ bd ¼ 0:
Solutions of (11) are 1 0 0 or 0 1 1
1 0
ð12Þ
‘‘)1’’ in the solutions is caused by reflection transformation which will be discussed later in Section 5. Excluding the reflection transformation, the solutions become 1 0 0 1 or : ð13Þ 0 1 1 0 The first solution is a unit matrix, but from the second solution we can note that there may exist two canonical forms which are related by Am ¼ 0 1 , i.e. the two canonical forms are sym1 0 metrical about the x ¼ y direction. h The procedure of the above example can be generalized to derive the ambiguity matrix Am as summarized below. a b (1) Assume that Am ¼ . c d (2) Use the normalization constraints to derive the moment relationship between canonical forms, for example, Eq. (10). (3) Considering that the relationship holds independently of the concrete moment values, a group of equations only in terms of parameters of Am can be obtained. (4) Obtain the solutions of the equations in Step (3) to get Am . Here we eliminate the possible reflection factors to simplify Am . For example in
Eq. (12) there are only two non-zero values in the resulting matrix, and Ô)1Õ implies the reflection. A simpler form as in Eq. (13) is obtained by eliminating Ô)1Õ in Eq. (12). Reflection always exists in the imaging process and/or normalization procedures. Normalization with respect to reflection can be a further step of normalization procedure as discussed in Section 5. 3.2. Criterion for selecting canonical form In order to eliminate the ambiguity caused by symmetrical normalization constraints, for each 0 1 to preimage after normalization we use 1 0 multiply the normalization matrix Tn obtained during the normalization procedure to get a new normalization matrix Tn0 . Thus we get two canonical forms N ðx; yÞ and N 0 ðx0 ; y 0 Þ by Tn and Tn0 for 0 each original image. Let lNpq and lNpq be the mo0 ments of N ðx; yÞ and N 0 ðx0 ; y 0 Þ. Then if lNpc qc 6¼ lNpc qc with minðpc þ qc Þ, ldif ¼ lpc qc . We then choose the canonical form with the minðldif Þ or maxðldif Þ as the final one. 3.3. An example of symmetrical normalization constraints For illustration, we give an example using the iterative normalization procedure (Voss and Suesse, 1997). An original image (a) and two of the affine transformed versions (b) and (c) are given in the first row of Table 1. Images (a0 ), (b0 ) and (c0 ) in the second row of Table 1 show the corresponding canonical forms obtained by the iterative method respectively. We can see that (a0 ) and (b0 ) are the 0 same, but (c ) is different due to the ambiguity 0 1 matrix . Clearly the normalization results 1 0 do not satisfy the consistence requirement of normalization. The third row of Table 1 shows some moments of the canonical forms. From the values of l12 , l21 , l30 and l03 we can also see the relationship between canonical forms (a0 ) and (c0 ). Using the canonical form selection criterion proposed in Section 3.2, when minðl21 Þ is chosen as the criterion, the final consistent canonical form
Y. Zhang et al. / Pattern Recognition Letters 24 (2003) 3205–3215
3209
Table 1 Examples of ambiguity arising from symmetrical normalization constraints Images
Canonical forms
2
l11 Normalized moments 4 l21 l30
3 l12 l22 5 l03
2
3 0:0403 0:0104 4 0:0398 0:0155 5 0:0341 0:0223
will be (c0), and (a0 ) and (b0 ) should be multiplied 0 1 by to reach the final results. If maxðl21 Þ is 1 0 0 chosen, (a ) and (b0 ) will be the final canonical forms and (c0 ) should be modified accordingly.
2
3 0:0403 0:0104 4 0:0398 0:0155 5 0:0341 0:0223
2
3 0:0403 0:0398 4 0:0104 0:0155 5 0:0223 0:0341
(Rothe et al., 1996) which is based on the XYS decomposition (Eq. (4)). In this method lx30 ¼ 0 is used as the x-shearing normalization constraint. This constraint gives P ðbÞ ¼ lx30 ¼ l30 þ 3bl21 þ 3b2 l12 þ b3 l03 ¼ 0: ð16Þ
4. Multi-roots ambiguity 4.1. Cause of the ambiguity The multi-roots ambiguity happens during the x or y-shearing normalization. We take x-shearing as an example. Let lpq and lxpq denote the moments of the original image Iðx; yÞ and the image N x ðx0 ; y 0 Þ after x-shearing normalization respectively. Then we have 0 x 1 b x ð14Þ ¼ y0 0 1 y and lxpq ¼
Z Z
ðx þ byÞp y q Iðx; yÞ dx dy:
ð15Þ
If the condition lxp0 q0 ¼ c, where c is a constant and selected as 0 normally, is chosen as the x-shearing normalization constraint, this will give a highorder polynomial of b if p0 > 1. For example, we consider the polynomial normalization method
Each root of Eq. (16) creates a canonical form and thus one image will have three canonical forms after normalization. So we must give a rule for root selection to guarantee the consistency of the canonical form. 4.2. Criterion for selecting canonical form The analysis in this subsection is based on the polynomial normalization method (Rothe et al., 1996) where the constraints l30 ¼ 0, l11 ¼ 0, l20 ¼ l02 ¼ 1 are used to normalize the x-shearing, y-shearing and anisotrope scaling respectively. Let xys cf lxpq , lxy pq , lpq and lpq denote the moments of the x-shearing, the y-shearing, anisotrope scaling and the final normalization results respectively. Assume that I org ðx; yÞ and I aff ðx; yÞ are images before and after affine transformation. After x-shearing normalization we obtain Eq. (16) in which P ðbÞ may have three real roots or a pair of conjugate complex roots and a real root.
3210
Y. Zhang et al. / Pattern Recognition Letters 24 (2003) 3205–3215
Table 2 Affine transformation related images and their canonical forms b, lcf 31
Images
Canonical form 1
b ¼ 1:986, lcf 31 ¼ 0:0504
b ¼ 0:876, lcf 31 ¼ 0:062
b ¼ 4:250, lcf 31 ¼ 0:0504
b ¼ 0:323, lcf 31 ¼ 0:062
b ¼ 8:760, lcf 31 ¼ 0:0504
b ¼ 0:420, lcf 31 ¼ 0:062
Analysis is divided into two parts corresponding to the two cases. 4.2.1. Polynomial P (b) has three real roots In this case, we will have three canonical forms. A suitable b can be obtained based on the following criterion: b ¼ arg minðldif ðbi ÞÞ: bi
ð17Þ
That is, the b that results in the minimum of ldif is chosen. Similarly, b ¼ arg maxðldif ðbi ÞÞ bi
b; lcf 31
ð18Þ
can also be used. To show this, now we consider the following example. Three affine transformation related images are shown in the first column of Table 2. Moment ldif is selected as lcf 31 . The selected minimum lcf and b are shown in the second column of 31 Table 2 and the three selected canonical forms are shown in column 3. It is obvious that the selected canonical forms are consistent for the three affine transformation related images. By selecting the maximum of lcf 31 we obtain another set of canonical form shown in column 5 of Table 2. 4.2.2. P (b) has a pair of conjugate complex roots In this case the real root stands for an independent position which corresponds to a consistent canonical form. We now show this claim by contradiction. In the following, notations with the
Canonical form 2
subscript org and aff denote the parameters of I org ðx; yÞ and I aff ðx; yÞ respectively. Assume that the real root is chosen for the original image, i.e. borg is real, and one of the complex roots is chosen for the affine transformed image, i.e. baff is complex. From the normalization constraints l30 ¼ 0, l11 ¼ 0, l20 ¼ l02 ¼ 1 we can get the parameters in the XYS decomposition (Eq. (4)) as sffiffiffiffiffiffiffiffiffiffiffiffi sffiffiffiffiffiffiffiffiffiffiffiffi lx11 lxy lxy 8 8 02 20 c¼ x ; a¼ ; d ¼ : ð19Þ 3 xy 3 l20 ðlxy Þ ð l 20 02 Þ Since lx11 ¼ l11 þ bl02 , lx20 ¼ l20 þ 2bl11 þ b2 l02 , xy 2 x x x x lx02 ¼ l02 , lxy 02 ¼ c l20 þ 2cl11 þ l02 , and l20 ¼ l20 , we get c¼
l11 þ bl02 ; l20 þ 2bl11 þ b2 l02
ð20Þ
sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi x x 2 x 8 ðc l20 þ 2cl11 þ l02 Þ a¼ ; 3 ðlx20 Þ
ð21Þ
sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ðlx20 Þ 8 : d¼ 3 x ðc2 l20 þ 2clx11 þ lx02 Þ
ð22Þ
From Eqs. (19)–(22), we can note that if borg is real, corg , dorg and aorg are all real and if baff is complex, caff , daff and aaff will be complex. Let the normalization matrices which transform the two images to their canonical forms be denoted as Torg and Taff respectively. Then we have
Y. Zhang et al. / Pattern Recognition Letters 24 (2003) 3205–3215
" Torg ¼ Taff ¼
aorg dorg corg
aaff daff caff
# aorg borg ; dorg ðcorg borg þ 1Þ aaff baff : daff ðcaff baff þ 1Þ
Under the above assumption, Torg is real but Taff is complex. On the other hand, Torg ¼ ATaff ;
ð23Þ
where A is real. Thus the above assumption does not hold and therefore the canonical form which corresponds to the real value of b is the only consistent one. Remark 1. Although we only consider the problem using polynomial normalization method, the proposed idea can be applied to other moment based normalization methods where multi-roots ambiguity exists.
5. Reflection ambiguity In the existing normalization methods, reflection has not received much attention. During our study we found that reflection transformation does occur not only during the imaging process but also during the normalization procedure itself. In this section, we will address the normalization with respect to reflection transformation systematically. The x and y reflections are expressed by diagf1; 1g and diagf1; 1g respectively. For images related only by reflection transformation, the normalization constraints given by Eqs. (3), (5) and (6) yield the same values. That means these constraints cannot discriminate reflection. Although the constraints in Eq. (7) are reflection sensitive, the x and y directions of the reflection are swapped as described in Section 5.2. From the decomposition of the affine matrix as shown in Eqs. (2) and (4), we note that the reflection can only be detected from the scaling matrix. The analysis of this problem begins with the anisotrope scaling normalization and the resulting lspq is expressed as lspq ¼ apþ1 dqþ1 lps pq ;
ð24Þ
3211
where a and d are the anisotrope scaling parameters, and lps pq is the result prior to the scaling normalization. Generally, conditions of lsp0 q0 ¼ 1 and lsq0 p0 ¼ 1 are used as the anisotrope scaling normalization constraints. We can then get vffiffiffiffiffiffiffiffiffiffiffiffi u psp0 þ1 u 2 ðq0 þ1Þ ðp0 þ1Þ2 lp q t 00 ; a¼ q0 þ1 lps q0 p0 vffiffiffiffiffiffiffiffiffiffiffiffi u psp0 þ1 u 2 ðq0 þ1Þ ðp0 þ1Þ2 lq0 p0 t ð25Þ d¼ q0 þ1 : lps p0 q 0 According to the odevity of p0 and q0 , the analysis can be divided into two cases, which is also illustrated by the flowchart in Fig. 1. In both cases, conditions of lsp0 q0 ¼ 1 and lsq0 p0 ¼ 1 are used as the anisotropic scaling normalization constraints. 5.1. Case 1: p0 and q0 are even numbers In this case we can note that the values of a and d calculated from Eq. (25) are always positive. So this kind of scaling normalization constraints cannot discriminate reflection. We must impose additional constraints to the canonical form. The scheme to detect and constrain the reflection effect can be divided in the following three situations: (1) Using lcf p1 q1 > 0, with p1 odd and q1 even, we can constrain x direction reflection since in this situation the difference between the moments with p and without reflection are ð1Þ 1 . If lsp1 q1 > 0, no further manipulation is needed. If lsp1 q1 < 0, the following equation should be used to get lcf pq p
s lcf pq ¼ ð1Þ lpq
ð26Þ
and the normalization matrix should be modified by pre-multiplying diagf1; 1g. (2) Similarly the condition that lcf q1 p1 > 0 can constrain y direction reflection since in this situation the difference between the moments with p and without reflection are ð1Þ 1 . If lsq1 p1 > 0, no further manipulation is needed. If lsq1 p1 < 0, we should modify the final result according to q
s lcf pq ¼ ð1Þ lpq
ð27Þ
and the transformation parameters should be modified by pre-multiplying diagf1; 1g.
3212
Y. Zhang et al. / Pattern Recognition Letters 24 (2003) 3205–3215
START
p 0 , q 0 even
Yes
Yes
u qs , p > 0 1
1
No
u sp , q > 0 1
u ps
2
u qs
Y normalization
Yes
1
Yes
No
p 0 odd , q 0 even
Odevity of p 0 , q 0
1,
X normalization
p
1
,q 2
>0
No
>0 No
XY normalization
X or Y normalization
END
Fig. 1. A general reflection normalization procedure.
(3) If lsp1 q1 < 0 and lsq1 p1 < 0, then both x and y direction reflections exist. Then we should modify the final result according to pþq s lpq
lcf pq ¼ ð1Þ
ð28Þ
and the transformation parameters should be modified by pre-multiplying diagf1; 1g. Remark 2. The effect of x and y direction reflections equals to a rotation of 180. For XSR decomposition, rotation is included in the rotation matrix. But for XYS decomposition, a rotation of 180 can only be expressed by the scaling matrix. We still need to normalize x and y direction reflections as shown above. 5.2. Case 2: p0 is odd and q0 is even As an example, the constraints can be set as ls12 ¼ 1 and ls21 ¼ 1. Then we have sffiffiffiffiffiffiffiffiffiffiffiffi sffiffiffiffiffiffiffiffiffiffiffiffi ps 2 ps 2 5 ðl12 Þ 5 ðl21 Þ ; d¼ : ð29Þ a¼ 3 ps 3 ðl21 Þ ðlps 12 Þ In this case we use that lcf p2 q2 > 0, with p2 and q2 even, as a normalization constrain. If lsp2 q2 > 0, no further manipulation is required. Otherwise, we
should modify the final result using Eq. (30) below according to the following analysis. Normally reflections can be detected through ls12 and ls21 , which have different signs when x or y reflection exists. But in this case the constraints swap the x and y reflection directions, which can be noted from Eq. (29). When x reflection exists, we p should use ð1Þ to modify the scaling normalization results. But in this kind of normalization constraints, the scaling normalization results have ðqþ1Þ an additional factor ð1Þ in d. So the following equation is used to get the final results: lcf pq ¼ ð1Þ
pþqþ1 s lpq
ð30Þ
and the normalization matrix should be modified by pre-multiplying diagf1; 1g. The analysis of y reflection is similar. Note that in this case if there exist both x and y reflections, no further normalization is needed. 5.3. Reflection examples This section presents two examples of using the proposed reflection normalization scheme to remove the reflection ambiguities in the existing normalization methods.
Y. Zhang et al. / Pattern Recognition Letters 24 (2003) 3205–3215
3213
Table 3 Examples of reflection normalization Images
Normalized by l20 ¼ l02 ¼ 1
ls12 ls21
0.0297 0.0207
)0.0297 0.0207
0.0297 )0.0207
)0.0297 )0.0207
1.723
)1.723
)1.723
1.723
Final results with additional constraints: ls12 > 0 and ls21 > 0 Normalized by l21 ¼ l12 ¼ 1 ls22 Final results with additional constraint: ls22 > 0
5.3.1. Example 1 The original image, the images with x reflection, y reflection, x and y reflection are shown as (a)–(d) in the first row of Table 3 respectively. After the scaling normalization under constraints l20 ¼ l02 ¼ 1 corresponding to Section 5.1, we get the normalized images which are shown in the second row and the resulting ls12 and ls21 are shown in the third row. Clearly, reflection still exists from the values of ls12 and ls21 . The conditions that lcf 12 > 0 and lcf > 0 are chosen as the x and y reflection 21 normalization constraints, respectively. Following our proposed method the final canonical forms
which are the same for all the four images are shown in the fourth row. To consider the situation discussed in Section 5.2, we show scaling normalization results under constraints l21 ¼ l12 ¼ 1 in row 5 of Table 3. The resulting ls22 are shown in the sixth row. Using lcf 22 > 0 as reflection normalization constraints, the obtained canonical forms are shown in the last row of Table 3. Again, these forms are the same. 5.3.2. Example 2 In (Shen and Ip, 1997), a normalization method was proposed to produce a unique normalization
Table 4 Example of removing reflection ambiguity in normalization of RSIs Images
Normalized by Shen and Ip (1997)
ls12 ls21 Final results with additional constraints: ls12 > 0, ls21 > 0
)5.308 )1.122
)5.308 1.122
3214
Y. Zhang et al. / Pattern Recognition Letters 24 (2003) 3205–3215
representation particularly for the rotationally symmetric images (RSI). However, reflection ambiguity exists in some of the RSIs. To see this, we use the same image as in (Shen and Ip, 1997). The original image and the image with x reflection are shown respectively in (a) and (b) in the first row of Table 4. The normalized images using the method in (Shen and Ip, 1997) are shown in the second row and the corresponding moments ls12 and ls21 are listed in the third row. Clearly, reflection still exists. After further normalization by our proposed method, the final canonical forms are the same for the two test images, as shown in the fourth row.
6. A case study with combined ambiguities In some normalization procedures, the above discussed ambiguities may occur simultaneously. In this section we apply the schemes proposed in this paper to the normalization method proposed in (Zhang et al., 2002) where the constraints l30 ¼ l03 ¼ 0 and l12 ¼ l21 ¼ 1 are used to normalize blur and affine combined deformed images to derive blur and affine combined moment invariants. In this interesting normalization method, the above mentioned three kinds of ambiguities all exist. In the following, we will analyze the choice of a consistent canonical form arising from the multi-roots and symmetric normalization constraints. 6.1. Analysis in case of the multi-roots ambiguity With the normalization method in (Zhang et al., 2002), the x-shearing normalization constraint requires P ðbÞ ¼ l30 þ 3bl21 þ 3b2 l12 þ b3 l03 ¼ 0
ð31Þ
and the y-shearing normalization constraint requires P ðcÞ ¼ 3c2 lx21 þ 3clx12 þ lx03 ¼ 0:
ð32Þ
Based on Section 4, there are six canonical forms. We now briefly describe the root selection method in two cases as follows. For details, please refer to (Zhang et al., 2002).
• Case 1: P ðbÞ has three real roots. In this case it can be shown as in (Zhang et al., 2002) that P ðcÞ will result in two real roots and there are six groups of solutions from Eqs. (31) and (32). By considering ldif , a suitable group of parameters can be obtained by using b; c; a; d ¼ arg minldif ðb; c; a; dÞ: fb;cg
ð33Þ
That is, the canonical form which corresponds to the group of ðb; c; a; dÞ resulting in the minimum value of ldif being chosen. • Case 2: P ðbÞ has a pair of conjugate complex roots. In this case P ðcÞ has a pair of conjugate complex roots and only the real root of P ðbÞ can satisfy the normalization constraints. So parameter b is chosen to be the real root of P ðbÞ, c is chosen as the root of P ðcÞ that gives the same sign of the real part and imaginary part of ldif , a and d are then determined by the scaling normalization procedure. After further reflection normalization as shown in Section 5.2, we can obtain a consistent canonical form for affine transformed images as follows. 6.2. Analysis in case of the symmetrical normalization constraints Assume that the image Iðx; yÞ has two canonical forms N 0 ðx0n ; yn0 Þ and N ðxn ; yn Þ after normalization and they are related by 0 xn a b xn ¼ : ð34Þ yn0 c d yn We have 8 0 cf 3 cf 2 > > lcf 30 ¼ a l30 þ 3a bl21 > > > 3 cf > > þ3ab2 lcf > 12 þ b l03 ; > > 0 cf > 3 cf 2 > lcf > 03 ¼ c l30 þ 3c dl21 > > < þ3cd 2 lcf þ d 3 lcf ; 12
03
0 cf cf 2 2 > lcf > 21 ¼ a cl30 þ ða d þ 2abcÞl21 > > > cf > þðb2 c þ 2abdÞl12 þ b2 dlcf > 03 ; > > > cf 0 cf cf > 2 2 > l12 ¼ ac l30 þ ðbc þ 2acdÞl21 > > > : 2 cf þðad 2 þ 2bcdÞlcf 12 þ bd l03 :
ð35Þ
Y. Zhang et al. / Pattern Recognition Letters 24 (2003) 3205–3215
Substituting the normalization constraints into Eq. (35), we have 8 2 a b þ ab2 ¼ 0; > > < 2 c d þ cd 2 ¼ 0; ð36Þ a2 d þ 2abc þ 2abd þ b2 c ¼ 1; > > : 2 bc þ 2acd þ 2bcd þ ad 2 ¼ 1: Solutions 1 A1 ¼ 0 0 A3 ¼ 1 1 A5 ¼ 0
of (36) are 0 0 1 ; A2 ¼ ; 1 1 0 1 1 0 ; A4 ¼ ; 1 1 1 1 1 1 ; A6 ¼ : 1 1 0
These solutions show the relationship between the six possible canonical forms. In this case, once obtaining one of the canonical forms, i.e., one group of ðb; c; a; dÞ, we can get the other canonical forms using A2 –A6 . Then we select ldif from the resulting moment sets as the criterion to choose a consistent canonical form. 7. Conclusions In this paper we consider the problem of obtaining a consistent canonical form under moment based normalization. We analyze three kinds of ambiguities due to multi-roots, symmetrical normalization constraints and reflection. Schemes to derive the ambiguity matrices are presented and solutions are provided to obtain a consistent canonical form in the presence of these ambiguities. A case study is given in which all the three kinds of ambiguities occur simultaneously. The proposed schemes are verified with some illustrative examples. It should be noted that the results presented here can be easily generalized to other moment based normalization methods with different constraints.
3215
Acknowledgements The authors would like to thank Dr. Dinggang Shen for providing some of the test images.
References Adler, S.L., Krishnan, R., 1998. Similarity and affine normalization of partially occluded planar curves using first and second derivatives. Pattern Recognition, 1551–1556. Avrithis, Y., Xirouhakis, Y., Kollias, S., 2000. Affine-invariant curve normalization for shape-based retrieval. In: Proc. 15th Internat. Conf. on Pattern Recognition, pp. 1015–1018. Leu, J., 1989. Shape normalization through compacting. Pattern Recognition Lett., 243–250. Pei, S., Lin, C., 1995. Image normalization for pattern recognition. Image Vision Comput. 13, 711–723. Reeves, A.P., Proko, R.J., Andrew, S.E., Kuhl, F.P., 1988. Three-dimensional shape analysis using moments and fourier descriptors. IEEE Trans. Pattern Anal. Machine Intell., 388–397. Reiss, T.H., 1993. Recognizing Planar Objects Using Invariant Image Features. Springer Verlag. Rothe, I., Susse, H., Voss, K., 1996. The method of normalization to determine invariants. IEEE Trans. Pattern Anal. Machine Intell. 18, 366–375. Shen, D., Ip, H., 1997. Generalized affine invariant image normalization. IEEE Trans. Pattern Anal. Machine Intell. 19, 431–440. Sinclair, D., Blake, A., 1994. Isoperimetric normalization of planar curves. IEEE Trans. Pattern Anal. Machine Intell., 769–777. Tzanetakis, M.T.G., Tziritas, G., 1998. Motion estimation based on affine moment invariants. Signal Process. In: Eusipco-98, pp. 925–928. Voss, K., Suesse, H., 1997. Invariant fitting of planar objects by primitives. IEEE Trans. Pattern Anal. Machine Intell. 19, 80–84. Wohn, K., Wu, J., 1990. Estimating the finite displacement using moments. Pattern Recognition Lett., 371–378. Zhang, Y., Wen, C., Zhang, Y., 2000. Estimation of motion parameters from blurred images. Pattern Recognition Lett., 425–433. Zhang, Y., Wen, C., Zhang, Y., Soh, Y.C., 2002. Determination of blur and affine combined invariants by normalization. Pattern Recognition, 211–221.
On the choice of consistent canonical form during moment normalization Yani Zhang a, Changyun Wen a
a,*
, Ying Zhang b, Yeng Chai Soh
a
School of Electrical and Electronic Engineering, Nanyang Technological University, Nanyang Avenue, Singapore 639798, Singapore b Singapore Institute of Manufacturing Technology, 71 Nanyang Drive, Singapore 638075, Singapore Received 31 October 2002; received in revised form 13 August 2003
Abstract The key issue in normalization with respect to geometric transformation is how to obtain a consistent canonical form which remains unchanged for different geometric transformation related images. Due to ambiguities inherent in the specified normalization methods, more than one canonical form may occur during the normalization procedure. This causes difficulties in obtaining the expected invariant features or the transformation parameters through normalization. This paper aims to provide general schemes to analyze the ambiguity characteristics and to derive the ambiguity matrices through which ambiguities can be eliminated. Three kinds of ambiguities caused by the multi-roots of highorder polynomials, the symmetrical normalization constraints, and the reflection are addressed and solutions are provided to obtain a consistent canonical form for each kind of ambiguities. 2003 Elsevier B.V. All rights reserved. Keywords: Canonical form; Moment normalization; Affine transformation; Ambiguity
1. Introduction In pattern recognition, some function parameters such as moments and Fourier coefficients are usually used to describe the image features to be recognized. In order to uniquely identify the features, invariants are constructed using the function parameters to eliminate the effects which the geometric transformation imposes on the function
*
Corresponding author. Tel.: +65-6790-4947/790-4947; fax: +65-6792-0425/792-0415. E-mail address: [email protected] (C. Wen).
parameters. Normalization is an effective and systematic approach for this purpose (Reeves et al., 1988; Leu, 1989; Sinclair and Blake, 1994; Adler and Krishnan, 1998). The basic idea of image normalization is to transform the original image into its canonical form so that the function parameters of the canonical form are independent of the geometric transformations and at the same time they retain all the relevant information of the original image. The normalization is usually achieved by imposing the so-called normalization constraints on some chosen function parameters so that all the remaining function parameters are independent of the geometric transformations.
0167-8655/$ - see front matter 2003 Elsevier B.V. All rights reserved. doi:10.1016/j.patrec.2003.08.006
3206
Y. Zhang et al. / Pattern Recognition Letters 24 (2003) 3205–3215
For two dimensional planar images, normalization methods based on affine transformation were studied extensively in the literature. For example, it has been shown that the affine transformation parameters can be estimated through the normalization matrix which transforms the original image to its canonical form (Wohn and Wu, 1990; Reiss, 1993; Tzanetakis and Tziritas, 1998). In (Shen and Ip, 1997), a generalized image normalization method was proposed to handle shaped planar images. The normalization of curve, shape, or gray-level planar images are discussed in references (Pei and Lin, 1995; Rothe et al., 1996; Voss and Suesse, 1997). Recently, Zhang et al. proposed a normalization method for constructing moment invariants against both blur and affine transforms (Zhang et al., 2002). However, all these works suffer from one common problem in that the expected invariant features and transformation parameters may not be obtained if the canonical form is non-unique. In some moment based normalization methods (Rothe et al., 1996; Voss and Suesse, 1997; Zhang et al., 2002), there do exist more than one canonical form due to inherent ambiguities. This gives rise to the non-consistence of the canonical form. The problem was initially but briefly addressed in (Zhang et al., 2002). With a focus on planar images with two dimensional affine transformation, this paper will systematically analyze the causes of ambiguities and provide ways to obtain a consistent canonical form that is free of ambiguities. In this paper, three kinds of ambiguities are identified as the sources which cause inconsistence of the canonical form. The first kind of ambiguity is caused by the symmetrical characteristic of the normalization constraints. The second kind of ambiguity is due to the roots of a high-order polynomial. For example, in the polynomial normalization method (Rothe et al., 1996), the third order central moment l30 ¼ 0 is used as the x-shearing normalization constraint. There are three different solutions which satisfy l30 ¼ 0 and these solutions will yield three canonical forms. The last kind of ambiguity is due to the reflection ambiguity, which was only briefly addressed in (Pei and Lin, 1995; Avrithis et al., 2000). In the study presented here, it is noted that reflection transformation does occur not only during the
imaging process but also during the normalization procedure. In the following sections, the above-mentioned three kinds of ambiguities are first analyzed, each illustrated with examples. Then a case study is conducted in the presence of combined ambiguities. For each kind of ambiguity, general schemes are proposed to obtain the unique canonical form. Although the study in this paper is developed using some common normalization constraints used in previous studies, the proposed ideas can also be used in other moment normalization methods. The case study shows that in the presence of ambiguities, the schemes proposed in this paper can effectively obtain a consistent canonical form. Analysis and experimental results verify the effectiveness.
2. Moment normalization under affine transformation A general affine transformation is a linear transformation plus a translation. In the image coordinates, it can be represented by 0 x x b ¼A þ 1 ; ð1Þ y0 b2 y a12 a where A ¼ 11 is called the homogeneous a21 a22 affine transformation matrix (Rothe et al., 1996; Voss and Suesse, 1997). Since the translation can be overcome by using the central moments, only the homogeneous affine transform is considered in the studies of moment normalization. In order to avoid solving complex non-linear systems of equations during normalization, the homogeneous affine transformation matrix is decomposed. In the literature (Rothe et al., 1996, Voss and Suesse, 1997; Zhang et al., 2000), two kinds of decomposition called XSR and XYS decompositions are used. 2.1. XSR decomposition The widely used affine decomposition is to decompose the affine matrix A into a shearing, an
Y. Zhang et al. / Pattern Recognition Letters 24 (2003) 3205–3215
anisotrope scaling matrix, and a rotation matrix, i.e. a11 a12 cosðhÞ sinðhÞ a 0 1 b ¼ ; a21 a22 sinðhÞ cosðhÞ 0 c 0 1 ð2Þ where a; c; b 2 R, h 2 ½0; 2p. The condition that detðAÞ 6¼ 0 is required to ensure the uniqueness of the decomposition. Based on this decomposition, the normalization constraints l11 ¼ 0;
l20 ¼ l02 ¼ 1;
l30 þ l12 ¼ 0
ð3Þ
are used in (Pei and Lin, 1995; Rothe et al., 1996; Reiss, 1993) to normalize the shear, scale and rotation, respectively. In (3), lpq denote the ðp þ qÞth central moments (Rothe et al., 1996; Zhang et al., 2002). 2.2. XYS decomposition The homogenous affine transformation matrix A can also be decomposed as an x-shearing, a yshearing and an anisotrope scaling matrix (Rothe et al., 1996) a11 a12 a 0 1 0 1 b ¼ ð4Þ a21 a22 0 d c 1 0 1 with a; d; c; b 2 C. If detðAÞ 6¼ 0 and a11 6¼ 0, the uniqueness of the decomposition can be guaranteed. Under this decomposition, Rothe et al. (1996) used the constraints l30 ¼ 0;
l11 ¼ 0;
l20 ¼ l02 ¼ 1
ð5Þ
3207
normalization constraints exists in the normalization methods based on XYS decomposition.
3. Ambiguity due to symmetrical normalization constraints 3.1. Cause of the ambiguity Since the homogeneous affine transformation matrix has four independent parameters, four constraints should be used to implement the normalization. If the constraints are selected as lp1 q1 ¼ lq1 p1 ¼ c1 and lp2 q2 ¼ lq2 p2 ¼ c2 , they are symmetrical about x and y directions and they cannot discriminate the x and y directions. We assume that Iðx; yÞ has two canonical forms N ðxn ; yn Þ and N 0 ðx0n ; yn0 Þ after normalization and they are related by 0 xn x ¼ Am n ; ð8Þ yn0 yn where Am is called the ambiguity matrix. Through Am we can determine if there exists ambiguity of canonical forms. If there is no ambiguity, Am must be a unit matrix. Otherwise, there exists ambiguity. Now we use the iterative normalization method (Voss and Suesse, 1997) as an example to illustrate this problem. This normalization procedure is based on the decomposition in Eq. (4) and uses constraints that l13 ¼ 0 and l31 ¼ 0 to implement the x-shearing and y-shearing normalization simultaneously and l20 ¼ l02 ¼ 1 to normalize anisotrope scaling. For the iterative normalization method we have the following proposition.
while Voss and Suesse (1997) used the constraints l31 ¼ l13 ¼ 0;
l20 ¼ l02 ¼ 1
ð6Þ
to process normalization. Recently, Zhang et al. (2002) selected the constraints l30 ¼ l03 ¼ 0;
l21 ¼ l12 ¼ 1
ð7Þ
to derive blur and affine combined moment invariants. As shown in the following sections, reflection ambiguity exists in both XSR and XYS decomposition based normalization methods. The ambiguity caused by multi-roots and symmetrical
Proposition 1. Under normalization constraints that l31 ¼ 0, l13 ¼ 0, l20 ¼ 1, and l02 ¼ 1, it is possible that 0 1 Am ¼ : ð9Þ 1 0
a b cf 0 , and lcf pq and lpq denote c d the moment of N ðxn ; yn Þ and N 0 ðx0n ; yn0 Þ respectively. Applying the normalization constraints and using Eq. (8), we have
Proof. Let Am ¼
3208
8 0 > > lcf > 20 > > 0 > > lcf > 02 > > < cf 0 l31 > > > > 0 > > > lcf > 13 > > :
Y. Zhang et al. / Pattern Recognition Letters 24 (2003) 3205–3215 2 ¼ a2 þ 2ablcf 11 þ b ¼ 1; 2 ¼ c2 þ 2cdlcf 11 þ d ¼ 1; cf 2 2 ¼ a3 clcf 40 þ ð3ab c þ 2a bdÞl22
þ b3 dlcf 04 ¼ 0;
ð10Þ
cf 2 2 ¼ ac3 lcf 40 þ ð3acd þ 2bc dÞl22
þ bd 3 lcf 04 ¼ 0:
cf cf cf Eq. (10) holds independently of lcf 11 , l04 , l40 , l22 . Thus we have 2 a þ b2 ¼ c2 þ d 2 ¼ 1; ð11Þ ab ¼ cd ¼ ac ¼ bd ¼ 0:
Solutions of (11) are 1 0 0 or 0 1 1
1 0
ð12Þ
‘‘)1’’ in the solutions is caused by reflection transformation which will be discussed later in Section 5. Excluding the reflection transformation, the solutions become 1 0 0 1 or : ð13Þ 0 1 1 0 The first solution is a unit matrix, but from the second solution we can note that there may exist two canonical forms which are related by Am ¼ 0 1 , i.e. the two canonical forms are sym1 0 metrical about the x ¼ y direction. h The procedure of the above example can be generalized to derive the ambiguity matrix Am as summarized below. a b (1) Assume that Am ¼ . c d (2) Use the normalization constraints to derive the moment relationship between canonical forms, for example, Eq. (10). (3) Considering that the relationship holds independently of the concrete moment values, a group of equations only in terms of parameters of Am can be obtained. (4) Obtain the solutions of the equations in Step (3) to get Am . Here we eliminate the possible reflection factors to simplify Am . For example in
Eq. (12) there are only two non-zero values in the resulting matrix, and Ô)1Õ implies the reflection. A simpler form as in Eq. (13) is obtained by eliminating Ô)1Õ in Eq. (12). Reflection always exists in the imaging process and/or normalization procedures. Normalization with respect to reflection can be a further step of normalization procedure as discussed in Section 5. 3.2. Criterion for selecting canonical form In order to eliminate the ambiguity caused by symmetrical normalization constraints, for each 0 1 to preimage after normalization we use 1 0 multiply the normalization matrix Tn obtained during the normalization procedure to get a new normalization matrix Tn0 . Thus we get two canonical forms N ðx; yÞ and N 0 ðx0 ; y 0 Þ by Tn and Tn0 for 0 each original image. Let lNpq and lNpq be the mo0 ments of N ðx; yÞ and N 0 ðx0 ; y 0 Þ. Then if lNpc qc 6¼ lNpc qc with minðpc þ qc Þ, ldif ¼ lpc qc . We then choose the canonical form with the minðldif Þ or maxðldif Þ as the final one. 3.3. An example of symmetrical normalization constraints For illustration, we give an example using the iterative normalization procedure (Voss and Suesse, 1997). An original image (a) and two of the affine transformed versions (b) and (c) are given in the first row of Table 1. Images (a0 ), (b0 ) and (c0 ) in the second row of Table 1 show the corresponding canonical forms obtained by the iterative method respectively. We can see that (a0 ) and (b0 ) are the 0 same, but (c ) is different due to the ambiguity 0 1 matrix . Clearly the normalization results 1 0 do not satisfy the consistence requirement of normalization. The third row of Table 1 shows some moments of the canonical forms. From the values of l12 , l21 , l30 and l03 we can also see the relationship between canonical forms (a0 ) and (c0 ). Using the canonical form selection criterion proposed in Section 3.2, when minðl21 Þ is chosen as the criterion, the final consistent canonical form
Y. Zhang et al. / Pattern Recognition Letters 24 (2003) 3205–3215
3209
Table 1 Examples of ambiguity arising from symmetrical normalization constraints Images
Canonical forms
2
l11 Normalized moments 4 l21 l30
3 l12 l22 5 l03
2
3 0:0403 0:0104 4 0:0398 0:0155 5 0:0341 0:0223
will be (c0), and (a0 ) and (b0 ) should be multiplied 0 1 by to reach the final results. If maxðl21 Þ is 1 0 0 chosen, (a ) and (b0 ) will be the final canonical forms and (c0 ) should be modified accordingly.
2
3 0:0403 0:0104 4 0:0398 0:0155 5 0:0341 0:0223
2
3 0:0403 0:0398 4 0:0104 0:0155 5 0:0223 0:0341
(Rothe et al., 1996) which is based on the XYS decomposition (Eq. (4)). In this method lx30 ¼ 0 is used as the x-shearing normalization constraint. This constraint gives P ðbÞ ¼ lx30 ¼ l30 þ 3bl21 þ 3b2 l12 þ b3 l03 ¼ 0: ð16Þ
4. Multi-roots ambiguity 4.1. Cause of the ambiguity The multi-roots ambiguity happens during the x or y-shearing normalization. We take x-shearing as an example. Let lpq and lxpq denote the moments of the original image Iðx; yÞ and the image N x ðx0 ; y 0 Þ after x-shearing normalization respectively. Then we have 0 x 1 b x ð14Þ ¼ y0 0 1 y and lxpq ¼
Z Z
ðx þ byÞp y q Iðx; yÞ dx dy:
ð15Þ
If the condition lxp0 q0 ¼ c, where c is a constant and selected as 0 normally, is chosen as the x-shearing normalization constraint, this will give a highorder polynomial of b if p0 > 1. For example, we consider the polynomial normalization method
Each root of Eq. (16) creates a canonical form and thus one image will have three canonical forms after normalization. So we must give a rule for root selection to guarantee the consistency of the canonical form. 4.2. Criterion for selecting canonical form The analysis in this subsection is based on the polynomial normalization method (Rothe et al., 1996) where the constraints l30 ¼ 0, l11 ¼ 0, l20 ¼ l02 ¼ 1 are used to normalize the x-shearing, y-shearing and anisotrope scaling respectively. Let xys cf lxpq , lxy pq , lpq and lpq denote the moments of the x-shearing, the y-shearing, anisotrope scaling and the final normalization results respectively. Assume that I org ðx; yÞ and I aff ðx; yÞ are images before and after affine transformation. After x-shearing normalization we obtain Eq. (16) in which P ðbÞ may have three real roots or a pair of conjugate complex roots and a real root.
3210
Y. Zhang et al. / Pattern Recognition Letters 24 (2003) 3205–3215
Table 2 Affine transformation related images and their canonical forms b, lcf 31
Images
Canonical form 1
b ¼ 1:986, lcf 31 ¼ 0:0504
b ¼ 0:876, lcf 31 ¼ 0:062
b ¼ 4:250, lcf 31 ¼ 0:0504
b ¼ 0:323, lcf 31 ¼ 0:062
b ¼ 8:760, lcf 31 ¼ 0:0504
b ¼ 0:420, lcf 31 ¼ 0:062
Analysis is divided into two parts corresponding to the two cases. 4.2.1. Polynomial P (b) has three real roots In this case, we will have three canonical forms. A suitable b can be obtained based on the following criterion: b ¼ arg minðldif ðbi ÞÞ: bi
ð17Þ
That is, the b that results in the minimum of ldif is chosen. Similarly, b ¼ arg maxðldif ðbi ÞÞ bi
b; lcf 31
ð18Þ
can also be used. To show this, now we consider the following example. Three affine transformation related images are shown in the first column of Table 2. Moment ldif is selected as lcf 31 . The selected minimum lcf and b are shown in the second column of 31 Table 2 and the three selected canonical forms are shown in column 3. It is obvious that the selected canonical forms are consistent for the three affine transformation related images. By selecting the maximum of lcf 31 we obtain another set of canonical form shown in column 5 of Table 2. 4.2.2. P (b) has a pair of conjugate complex roots In this case the real root stands for an independent position which corresponds to a consistent canonical form. We now show this claim by contradiction. In the following, notations with the
Canonical form 2
subscript org and aff denote the parameters of I org ðx; yÞ and I aff ðx; yÞ respectively. Assume that the real root is chosen for the original image, i.e. borg is real, and one of the complex roots is chosen for the affine transformed image, i.e. baff is complex. From the normalization constraints l30 ¼ 0, l11 ¼ 0, l20 ¼ l02 ¼ 1 we can get the parameters in the XYS decomposition (Eq. (4)) as sffiffiffiffiffiffiffiffiffiffiffiffi sffiffiffiffiffiffiffiffiffiffiffiffi lx11 lxy lxy 8 8 02 20 c¼ x ; a¼ ; d ¼ : ð19Þ 3 xy 3 l20 ðlxy Þ ð l 20 02 Þ Since lx11 ¼ l11 þ bl02 , lx20 ¼ l20 þ 2bl11 þ b2 l02 , xy 2 x x x x lx02 ¼ l02 , lxy 02 ¼ c l20 þ 2cl11 þ l02 , and l20 ¼ l20 , we get c¼
l11 þ bl02 ; l20 þ 2bl11 þ b2 l02
ð20Þ
sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi x x 2 x 8 ðc l20 þ 2cl11 þ l02 Þ a¼ ; 3 ðlx20 Þ
ð21Þ
sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ðlx20 Þ 8 : d¼ 3 x ðc2 l20 þ 2clx11 þ lx02 Þ
ð22Þ
From Eqs. (19)–(22), we can note that if borg is real, corg , dorg and aorg are all real and if baff is complex, caff , daff and aaff will be complex. Let the normalization matrices which transform the two images to their canonical forms be denoted as Torg and Taff respectively. Then we have
Y. Zhang et al. / Pattern Recognition Letters 24 (2003) 3205–3215
" Torg ¼ Taff ¼
aorg dorg corg
aaff daff caff
# aorg borg ; dorg ðcorg borg þ 1Þ aaff baff : daff ðcaff baff þ 1Þ
Under the above assumption, Torg is real but Taff is complex. On the other hand, Torg ¼ ATaff ;
ð23Þ
where A is real. Thus the above assumption does not hold and therefore the canonical form which corresponds to the real value of b is the only consistent one. Remark 1. Although we only consider the problem using polynomial normalization method, the proposed idea can be applied to other moment based normalization methods where multi-roots ambiguity exists.
5. Reflection ambiguity In the existing normalization methods, reflection has not received much attention. During our study we found that reflection transformation does occur not only during the imaging process but also during the normalization procedure itself. In this section, we will address the normalization with respect to reflection transformation systematically. The x and y reflections are expressed by diagf1; 1g and diagf1; 1g respectively. For images related only by reflection transformation, the normalization constraints given by Eqs. (3), (5) and (6) yield the same values. That means these constraints cannot discriminate reflection. Although the constraints in Eq. (7) are reflection sensitive, the x and y directions of the reflection are swapped as described in Section 5.2. From the decomposition of the affine matrix as shown in Eqs. (2) and (4), we note that the reflection can only be detected from the scaling matrix. The analysis of this problem begins with the anisotrope scaling normalization and the resulting lspq is expressed as lspq ¼ apþ1 dqþ1 lps pq ;
ð24Þ
3211
where a and d are the anisotrope scaling parameters, and lps pq is the result prior to the scaling normalization. Generally, conditions of lsp0 q0 ¼ 1 and lsq0 p0 ¼ 1 are used as the anisotrope scaling normalization constraints. We can then get vffiffiffiffiffiffiffiffiffiffiffiffi u psp0 þ1 u 2 ðq0 þ1Þ ðp0 þ1Þ2 lp q t 00 ; a¼ q0 þ1 lps q0 p0 vffiffiffiffiffiffiffiffiffiffiffiffi u psp0 þ1 u 2 ðq0 þ1Þ ðp0 þ1Þ2 lq0 p0 t ð25Þ d¼ q0 þ1 : lps p0 q 0 According to the odevity of p0 and q0 , the analysis can be divided into two cases, which is also illustrated by the flowchart in Fig. 1. In both cases, conditions of lsp0 q0 ¼ 1 and lsq0 p0 ¼ 1 are used as the anisotropic scaling normalization constraints. 5.1. Case 1: p0 and q0 are even numbers In this case we can note that the values of a and d calculated from Eq. (25) are always positive. So this kind of scaling normalization constraints cannot discriminate reflection. We must impose additional constraints to the canonical form. The scheme to detect and constrain the reflection effect can be divided in the following three situations: (1) Using lcf p1 q1 > 0, with p1 odd and q1 even, we can constrain x direction reflection since in this situation the difference between the moments with p and without reflection are ð1Þ 1 . If lsp1 q1 > 0, no further manipulation is needed. If lsp1 q1 < 0, the following equation should be used to get lcf pq p
s lcf pq ¼ ð1Þ lpq
ð26Þ
and the normalization matrix should be modified by pre-multiplying diagf1; 1g. (2) Similarly the condition that lcf q1 p1 > 0 can constrain y direction reflection since in this situation the difference between the moments with p and without reflection are ð1Þ 1 . If lsq1 p1 > 0, no further manipulation is needed. If lsq1 p1 < 0, we should modify the final result according to q
s lcf pq ¼ ð1Þ lpq
ð27Þ
and the transformation parameters should be modified by pre-multiplying diagf1; 1g.
3212
Y. Zhang et al. / Pattern Recognition Letters 24 (2003) 3205–3215
START
p 0 , q 0 even
Yes
Yes
u qs , p > 0 1
1
No
u sp , q > 0 1
u ps
2
u qs
Y normalization
Yes
1
Yes
No
p 0 odd , q 0 even
Odevity of p 0 , q 0
1,
X normalization
p
1
,q 2
>0
No
>0 No
XY normalization
X or Y normalization
END
Fig. 1. A general reflection normalization procedure.
(3) If lsp1 q1 < 0 and lsq1 p1 < 0, then both x and y direction reflections exist. Then we should modify the final result according to pþq s lpq
lcf pq ¼ ð1Þ
ð28Þ
and the transformation parameters should be modified by pre-multiplying diagf1; 1g. Remark 2. The effect of x and y direction reflections equals to a rotation of 180. For XSR decomposition, rotation is included in the rotation matrix. But for XYS decomposition, a rotation of 180 can only be expressed by the scaling matrix. We still need to normalize x and y direction reflections as shown above. 5.2. Case 2: p0 is odd and q0 is even As an example, the constraints can be set as ls12 ¼ 1 and ls21 ¼ 1. Then we have sffiffiffiffiffiffiffiffiffiffiffiffi sffiffiffiffiffiffiffiffiffiffiffiffi ps 2 ps 2 5 ðl12 Þ 5 ðl21 Þ ; d¼ : ð29Þ a¼ 3 ps 3 ðl21 Þ ðlps 12 Þ In this case we use that lcf p2 q2 > 0, with p2 and q2 even, as a normalization constrain. If lsp2 q2 > 0, no further manipulation is required. Otherwise, we
should modify the final result using Eq. (30) below according to the following analysis. Normally reflections can be detected through ls12 and ls21 , which have different signs when x or y reflection exists. But in this case the constraints swap the x and y reflection directions, which can be noted from Eq. (29). When x reflection exists, we p should use ð1Þ to modify the scaling normalization results. But in this kind of normalization constraints, the scaling normalization results have ðqþ1Þ an additional factor ð1Þ in d. So the following equation is used to get the final results: lcf pq ¼ ð1Þ
pþqþ1 s lpq
ð30Þ
and the normalization matrix should be modified by pre-multiplying diagf1; 1g. The analysis of y reflection is similar. Note that in this case if there exist both x and y reflections, no further normalization is needed. 5.3. Reflection examples This section presents two examples of using the proposed reflection normalization scheme to remove the reflection ambiguities in the existing normalization methods.
Y. Zhang et al. / Pattern Recognition Letters 24 (2003) 3205–3215
3213
Table 3 Examples of reflection normalization Images
Normalized by l20 ¼ l02 ¼ 1
ls12 ls21
0.0297 0.0207
)0.0297 0.0207
0.0297 )0.0207
)0.0297 )0.0207
1.723
)1.723
)1.723
1.723
Final results with additional constraints: ls12 > 0 and ls21 > 0 Normalized by l21 ¼ l12 ¼ 1 ls22 Final results with additional constraint: ls22 > 0
5.3.1. Example 1 The original image, the images with x reflection, y reflection, x and y reflection are shown as (a)–(d) in the first row of Table 3 respectively. After the scaling normalization under constraints l20 ¼ l02 ¼ 1 corresponding to Section 5.1, we get the normalized images which are shown in the second row and the resulting ls12 and ls21 are shown in the third row. Clearly, reflection still exists from the values of ls12 and ls21 . The conditions that lcf 12 > 0 and lcf > 0 are chosen as the x and y reflection 21 normalization constraints, respectively. Following our proposed method the final canonical forms
which are the same for all the four images are shown in the fourth row. To consider the situation discussed in Section 5.2, we show scaling normalization results under constraints l21 ¼ l12 ¼ 1 in row 5 of Table 3. The resulting ls22 are shown in the sixth row. Using lcf 22 > 0 as reflection normalization constraints, the obtained canonical forms are shown in the last row of Table 3. Again, these forms are the same. 5.3.2. Example 2 In (Shen and Ip, 1997), a normalization method was proposed to produce a unique normalization
Table 4 Example of removing reflection ambiguity in normalization of RSIs Images
Normalized by Shen and Ip (1997)
ls12 ls21 Final results with additional constraints: ls12 > 0, ls21 > 0
)5.308 )1.122
)5.308 1.122
3214
Y. Zhang et al. / Pattern Recognition Letters 24 (2003) 3205–3215
representation particularly for the rotationally symmetric images (RSI). However, reflection ambiguity exists in some of the RSIs. To see this, we use the same image as in (Shen and Ip, 1997). The original image and the image with x reflection are shown respectively in (a) and (b) in the first row of Table 4. The normalized images using the method in (Shen and Ip, 1997) are shown in the second row and the corresponding moments ls12 and ls21 are listed in the third row. Clearly, reflection still exists. After further normalization by our proposed method, the final canonical forms are the same for the two test images, as shown in the fourth row.
6. A case study with combined ambiguities In some normalization procedures, the above discussed ambiguities may occur simultaneously. In this section we apply the schemes proposed in this paper to the normalization method proposed in (Zhang et al., 2002) where the constraints l30 ¼ l03 ¼ 0 and l12 ¼ l21 ¼ 1 are used to normalize blur and affine combined deformed images to derive blur and affine combined moment invariants. In this interesting normalization method, the above mentioned three kinds of ambiguities all exist. In the following, we will analyze the choice of a consistent canonical form arising from the multi-roots and symmetric normalization constraints. 6.1. Analysis in case of the multi-roots ambiguity With the normalization method in (Zhang et al., 2002), the x-shearing normalization constraint requires P ðbÞ ¼ l30 þ 3bl21 þ 3b2 l12 þ b3 l03 ¼ 0
ð31Þ
and the y-shearing normalization constraint requires P ðcÞ ¼ 3c2 lx21 þ 3clx12 þ lx03 ¼ 0:
ð32Þ
Based on Section 4, there are six canonical forms. We now briefly describe the root selection method in two cases as follows. For details, please refer to (Zhang et al., 2002).
• Case 1: P ðbÞ has three real roots. In this case it can be shown as in (Zhang et al., 2002) that P ðcÞ will result in two real roots and there are six groups of solutions from Eqs. (31) and (32). By considering ldif , a suitable group of parameters can be obtained by using b; c; a; d ¼ arg minldif ðb; c; a; dÞ: fb;cg
ð33Þ
That is, the canonical form which corresponds to the group of ðb; c; a; dÞ resulting in the minimum value of ldif being chosen. • Case 2: P ðbÞ has a pair of conjugate complex roots. In this case P ðcÞ has a pair of conjugate complex roots and only the real root of P ðbÞ can satisfy the normalization constraints. So parameter b is chosen to be the real root of P ðbÞ, c is chosen as the root of P ðcÞ that gives the same sign of the real part and imaginary part of ldif , a and d are then determined by the scaling normalization procedure. After further reflection normalization as shown in Section 5.2, we can obtain a consistent canonical form for affine transformed images as follows. 6.2. Analysis in case of the symmetrical normalization constraints Assume that the image Iðx; yÞ has two canonical forms N 0 ðx0n ; yn0 Þ and N ðxn ; yn Þ after normalization and they are related by 0 xn a b xn ¼ : ð34Þ yn0 c d yn We have 8 0 cf 3 cf 2 > > lcf 30 ¼ a l30 þ 3a bl21 > > > 3 cf > > þ3ab2 lcf > 12 þ b l03 ; > > 0 cf > 3 cf 2 > lcf > 03 ¼ c l30 þ 3c dl21 > > < þ3cd 2 lcf þ d 3 lcf ; 12
03
0 cf cf 2 2 > lcf > 21 ¼ a cl30 þ ða d þ 2abcÞl21 > > > cf > þðb2 c þ 2abdÞl12 þ b2 dlcf > 03 ; > > > cf 0 cf cf > 2 2 > l12 ¼ ac l30 þ ðbc þ 2acdÞl21 > > > : 2 cf þðad 2 þ 2bcdÞlcf 12 þ bd l03 :
ð35Þ
Y. Zhang et al. / Pattern Recognition Letters 24 (2003) 3205–3215
Substituting the normalization constraints into Eq. (35), we have 8 2 a b þ ab2 ¼ 0; > > < 2 c d þ cd 2 ¼ 0; ð36Þ a2 d þ 2abc þ 2abd þ b2 c ¼ 1; > > : 2 bc þ 2acd þ 2bcd þ ad 2 ¼ 1: Solutions 1 A1 ¼ 0 0 A3 ¼ 1 1 A5 ¼ 0
of (36) are 0 0 1 ; A2 ¼ ; 1 1 0 1 1 0 ; A4 ¼ ; 1 1 1 1 1 1 ; A6 ¼ : 1 1 0
These solutions show the relationship between the six possible canonical forms. In this case, once obtaining one of the canonical forms, i.e., one group of ðb; c; a; dÞ, we can get the other canonical forms using A2 –A6 . Then we select ldif from the resulting moment sets as the criterion to choose a consistent canonical form. 7. Conclusions In this paper we consider the problem of obtaining a consistent canonical form under moment based normalization. We analyze three kinds of ambiguities due to multi-roots, symmetrical normalization constraints and reflection. Schemes to derive the ambiguity matrices are presented and solutions are provided to obtain a consistent canonical form in the presence of these ambiguities. A case study is given in which all the three kinds of ambiguities occur simultaneously. The proposed schemes are verified with some illustrative examples. It should be noted that the results presented here can be easily generalized to other moment based normalization methods with different constraints.
3215
Acknowledgements The authors would like to thank Dr. Dinggang Shen for providing some of the test images.
References Adler, S.L., Krishnan, R., 1998. Similarity and affine normalization of partially occluded planar curves using first and second derivatives. Pattern Recognition, 1551–1556. Avrithis, Y., Xirouhakis, Y., Kollias, S., 2000. Affine-invariant curve normalization for shape-based retrieval. In: Proc. 15th Internat. Conf. on Pattern Recognition, pp. 1015–1018. Leu, J., 1989. Shape normalization through compacting. Pattern Recognition Lett., 243–250. Pei, S., Lin, C., 1995. Image normalization for pattern recognition. Image Vision Comput. 13, 711–723. Reeves, A.P., Proko, R.J., Andrew, S.E., Kuhl, F.P., 1988. Three-dimensional shape analysis using moments and fourier descriptors. IEEE Trans. Pattern Anal. Machine Intell., 388–397. Reiss, T.H., 1993. Recognizing Planar Objects Using Invariant Image Features. Springer Verlag. Rothe, I., Susse, H., Voss, K., 1996. The method of normalization to determine invariants. IEEE Trans. Pattern Anal. Machine Intell. 18, 366–375. Shen, D., Ip, H., 1997. Generalized affine invariant image normalization. IEEE Trans. Pattern Anal. Machine Intell. 19, 431–440. Sinclair, D., Blake, A., 1994. Isoperimetric normalization of planar curves. IEEE Trans. Pattern Anal. Machine Intell., 769–777. Tzanetakis, M.T.G., Tziritas, G., 1998. Motion estimation based on affine moment invariants. Signal Process. In: Eusipco-98, pp. 925–928. Voss, K., Suesse, H., 1997. Invariant fitting of planar objects by primitives. IEEE Trans. Pattern Anal. Machine Intell. 19, 80–84. Wohn, K., Wu, J., 1990. Estimating the finite displacement using moments. Pattern Recognition Lett., 371–378. Zhang, Y., Wen, C., Zhang, Y., 2000. Estimation of motion parameters from blurred images. Pattern Recognition Lett., 425–433. Zhang, Y., Wen, C., Zhang, Y., Soh, Y.C., 2002. Determination of blur and affine combined invariants by normalization. Pattern Recognition, 211–221.