- Email: [email protected]
Affine invariant detection of perceptually parallel 3D planar curves
Pattern Recognition 33 (2000) 1909}1918
A$ne invariant detection of perceptually parallel 3D planar curves Dinggang Shen , Horace H.S. Ip *, Eam Khwang Teoh Department of Radiology, School of Medicine, Johns Hopkins University, 601 N. Caroline Street, Baltimore, MD 21287, USA Image Computing Group, Department of Computer Science, City University of Hong Kong, Kowloon, Hong Kong School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore 639798 Received 7 January 1999; received in revised form 9 July 1999; accepted 9 July 1999
Abstract The problem of parallelism detection between two curves has been formulated in this paper as a line detection problem within an azne-invariant local similarity matrix computed for the two curves. Each element of this matrix gives an a$ne invariant measure of local parallelism for any pair of curve segments along the two curves. This approach enables the detection of a pair of parallel 3D planar curves as well as parallel 2D curves under general a$ne transform. Two descriptors were also used here to provide a multi-resolution representation of a curve. Since these two descriptors provide su$cient local and semi-local shape information at every feature point on the curves, the process of detecting parallelism is thus robust against both noise and deformations. Moreover, the proposed technique allows all signi"cant pairs of parallel segments within any two curves in the scene to be detected. Experiments on detecting randomly a$ne-transformed curves, which are obtained from natural images or arti"cially generated images, have demonstrated the e!ectiveness of the technique. 2000 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. Keywords: Parallelism detection; Local invariants; Semi-local invariants; A$ne invariant; Shape matching; Symmetry detection; 3D parallel curves; Hand-written digit recognition
1. Introduction The detection of symmetries is an essential step when inferring shapes from contours [1]. Particularly, skewed symmetries and parallel symmetries have been shown to be useful in recovering the 3D structure of various types of objects from their 2D contours [2,3]. Results from the study of perceptual organization in Ref. [4] also showed that primitive grouping in our vision system is done on proximity, collinearity, curvilinearity, parallelism, sym-
* Corresponding author. Tel.: #852-2788-8641; fax: #8522784-4916. E-mail addresses: [email protected] (D. Shen), [email protected] (H.H.S. Ip).
metry, closure and repetitive patterns. Symmetries (including parallelism) have played important roles in shape perception [5]. The detection of skewed symmetries has been investigated by many researchers in the past [6,7]. Depending on the nature of the features extracted from a shape, existing methods for symmetry detection can be roughly classi"ed into local and global approaches [6]. There are advantages and disadvantages associated, respectively, with these two approaches. To overcome some of the disadvantages relating to both global and local approaches, Sato and Cipolla [8] proposed integral invariant based on group invariant parameterization. They showed that integral invariant did not su!er from occlusion problem and is less sensitive to noise than di!erential invariant. We also have developed techniques in Ref. [9] for the detection of both local and global symmetries
0031-3203/00/$20.00 2000 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. PII: S 0 0 3 1 - 3 2 0 3 ( 9 9 ) 0 0 1 7 2 - 7
1910
D. Shen et al. / Pattern Recognition 33 (2000) 1909}1918
as well as skewed symmetries using the a$ne-invariant representation developed in Refs. [10,11]. Based on this representation, in this paper, the detection of skewed symmetries has been simpli"ed and formulated as a line detection problem with known orientations within a local similarity matrix. Parallelism is an important form of symmetries, and the detection of parallelism between two curves has attracted much attention in the vision community in the past. By representing the various properties of parallelism in terms of several distinctive forces, the second author has developed a force-driven model as a new optimization strategy to compute correspondence between salient points extracted from the matching curves [12]. This force-driven mechanism provides good coupling (or correspondence matching) results, which is the prerequisite for the correct detection of parallelism between curves. For curves to be parallel, certain point-wise correspondence between two curves must exist. Based on this observation, the research issues on parallelism detection are closely related to those for shape matching. Additionally, extra constraints like proximity and similarity in orientation should be added if parallelism detection is to be formulated as a shape matching problem. This is due to the fact that in shape matching the two input shapes may be under very di!erent orientation or even under severe a$ne transformation with respect to each other. Almost all solutions to shape (curve) matching were based on descriptor distance measures. Normally, local descriptors were favored since global descriptors could not handle partial matching, but local descriptors are sensitive to both noise and local deformation. Furthermore, when the correspondence for the two matching curves are known, transformation parameters between these two curves can be estimated. In this paper, we introduced two descriptors, namely, Semi-local shape descriptor and Tree-descriptor. Since these two descriptors extract both local and semi-local shape information from the curves, the process of detecting parallelism is thus robust against both noise and deformations. More importantly, the problem of parallelism detection can then be formulated as a line detection problem with known orientation within an azne-invariant local similarity matrix. Since the technique proposed here allows all signi"cant pairs of parallel segments contained in the two matching curves to be detected, partial matches for parallelism can also be detected. We organize the paper as follows. The formulations of two descriptors are given in Section 2. There, for every feature point, we establish (a) a feature vector which is able to capture both local and semi-local shape features, and (b) a set of Tree-like lines which connect the current feature point and its neighboring feature points for successive scales. Based on the two descriptors de"ned in Sections 2 and 3 describes a technique for detecting all
signi"cant pairs of parallel segments contained in the two matching curves. In fact, the process of parallelism detection is achieved by detecting lines of known orientations within an azne-invariant local similarity matrix, which contains the similarity measures calculated through the two descriptors for every pair of feature points along the two matching curves. A number of experiments on detecting randomly a$ne-transformed curves, which are obtained from natural or hand-drawn images, are given in Section 4. This paper concludes in Section 5.
2. Two new shape descriptors Based on the de"nition of the parallelism given below, we provide two descriptors for detecting parallel segments within two curves. These two descriptors were then combined to de"ne a set of similarity measures in the next section, and then used for parallelism detection for the two curves. 2.1. Dexnition of parallelism For curves to be parallel, certain point-wise correspondence between two curves must exist. Based on this requirement, the de"nition of parallelism from [1] is given below. 2.1.1. A dexnition of parallelism Let rG(t)"[xG(t), yG(t)]2, for i"1, 2, be two curves parameterized by arc length t. Let hG(t)"arctan ((dyG(t)/dt)/(dxG(t)/dt)). Then r(t) and r(t) are said to be parallel symmetric if there exist a point-wise correspondence f (t) between them such that h(t)"h( f (t)) for all values of t for which r and r are de"ned and f (t) is a continuous monotonous function. A useful special case is when f (t) is restricted to be a linear function. To solve the parallelism detection problem for the discrete case, let us assume that +(xG , yG ), (xG , yG ) ,2, (xG G , yG G ),, for i"1, 2, is the set of feature points , , representing the curve rG in a particular scale. Here, NG is the total number of sampling points on the curve rG. These feature points can be extracted, for example, from the sample points, which are evenly distributed along the azne-length shape boundary [13]. To render these feature points robust against noise, we take the same number of points from both sides of a sample point and average them to obtain a feature point. Based on the above de"nition of parallelism, two shape descriptors can be de"ned as follows. 2.2. Semi-local shape descriptor From the point of view of shape matching, the detection of parallelism can be approached as the problem of matching between two shapes. An a$ne-invariant
D. Shen et al. / Pattern Recognition 33 (2000) 1909}1918
feature vector for every feature point on the curve has been developed by the authors and applied to a number of problems: shape matching and indexing [10,14], model-based adaptive image segmentation [11], and detection of the skewed symmetries (both rotational symmetry and re#ectional symmetry) [9]. In the following, we will use our a$ne-invariant feature vector as a semi-local shape descriptor for every feature point on the two curves. This way, corresponding feature points on a pair of parallel segments should have similar feature vectors. The de"nition of the a$ne-invariant feature vector has been presented in Ref. [11] but we will brie#y review it here for completeness. Let r G(t) and rK G(t) be the "rst and second derivatives of the curve rG(t), where i"1, 2. Their de"nitions are
rR G(t)"[xR G(t) y G(t)]2"
dxG(t) dyG(t) 2 , dt dt
rK G(t)"[xK G(t) yK G(t)]2"
dxG(t) dyG(t) 2 . dt dt
vertices (xG(t!2*t), yG(t!2*t)), (xG(t!*t), yG(t!*t)) and (xG(t), yG(t)). In fact, the area of any triangle is relatively a$ne invariant and the ratios of such areas are a$ne invariant:
xG(t ) 1 area " yG(t ) R R R 2 1
xG(t ) yG(t ) 1
xG(t ) yG(t ) , 1
where area is the area of a triangle, whose three R R R vertices are (xG(t ), yG(t )), (xG(t ), yG(t )) and (xG(t ), yG(t )). Notice, t , t and t are the parameters of the curve. Accordingly, an a$ne-invariant feature vector can be designed for every feature point of the given curves. For the kth feature point (xG , yG ), its corresponding feature I I vector is de"ned as FG "[ f G f G .... f G ]2, I I I +I where
Under general a$ne transformations, the following three expressions are invariant up to a scale factor: "(rG(t)!rG(t )) (rG(t)!rG(t ))", "(rG(t)!rG(t )) r G(t)", "r G(t) r( G(t)", where " ) " denotes the determinant of a matrix. That is, their values remain unchanged up to a factor which is equal to the determinant of the a$ne transformation. In this paper, we will use "r G(t) rK G(t)""(x G(t)yK G(t)!xK G(t)y G(t)) to de"ne an a$ne-invariant feature vector for every point along the curve rG(t). The "rst and second derivatives of xG(t) and yG(t) with respect to t can be approximated as follows: dxG(t) xG(t)!xG(t!*t) x G(t)" & dt *t dxG(t) xG(t)!2xG(t!*t)#xG(t!2*t) xK G(t)" & dt *t *t dyG(t) yG(t)!yG(t!*t) y G(t)" & dt *t dyG(t) yG(t)!2yG(t!*t)#yG(t!2*t) yK G(t)" & dt *t *t Using the above approximations, we can obtain
1911
xG(t!2*t) xG(t!*t) xG(t) 1 1 "r G(t) rK G(t)"" xG(t!2*t) yG(t!*t) yG(t) . 2 2 1 1 1 The above expression indicates that the value of "r G(t) rK G(t)" denotes the area of a triangle, formed by three
xG 1 I\H f G " yG HI 2 I\H 1
xG I yG I 1
xG I>H yG I>H 1
is the area of a triangle formed by (xG , yG ), (xG , yG ), I\H I\H I I and (xG , yG ). (Notice that the integer variables k I>H I>H and l are in domains, M#1)k)N!M and M#1)l)N!M.) The size of M in e!ect determines the sampling resolution of the curve for our representation. It should be noted that if j is close to 1, f G denotes HI local feature, i.e. f G (for small j) extracts information of HI local shape centered on the point (xG , yG ), which is sensiI I tive to noises. As j increases, f G gradually captures the HI semi-local features of the curve, i.e. f G extracts low-freHI quency (smooth) information of the shape, which is less sensitive to digitization noise. In essence, f G gives, after HI suitable normalization, in a single feature vector, both local and semi-local a$ne-invariant information of the curve with respect to a single point on the curve. Fig. 1 shows the Semi-local shape descriptor for the kth feature point (xG , yG ) on the ith curve. I I FG can be made exactly a$ne-invariant by the followI ing normalization: FG I FK G " G , I , + " f G " I H HI where FK G "[ fK G fK G .... fK G ]2. I I I +I The curve rG(t) can thus be described by a set of a$neinvariant feature vectors, +FK G , k"M#1, M#2,2, NG!M and i"1, 2,. I In Section 3, we will adopt this a$ne-invariant local shape descriptor and use them to compute a set of similarity measures for every coupled feature points on a pair of curves.
1912
D. Shen et al. / Pattern Recognition 33 (2000) 1909}1918
consider the same number (M) of feature points before and after the feature point
Fig. 1. An illustration of the area-based semi-local shape descriptor.
3.1.1. A similarity measure based on the Semi-local shape descriptor Suppose that two curves r and r are parallel and the feature points < and <, respectively, on the "rst curve I J r and the second curve r are the corresponding points. Then their feature vectors, FK and FK , should be the I J same. More importantly, this property holds even under a$ne-transformation. Mathematically, the matching error S between the feature vectors FK and FK is as I J I J follows: S "(FK !FK )2 G(FK !FK ), I J J I J I where G is an M;M diagonal matrix de"ned by a Gaussian function, i.e. K +\ 1 G"[g ] and g " e\ N . KK +"+ KK (2pp
Fig. 2. (a) An illustration of the Tree descriptor; (b) A di!erence measure between two corresponding vectors.
2.3. Tree-descriptor It is easy to observe that for every feature point (x, y) I I on the curve r, the vector joining this feature point and its neighboring feature point should be parallel to the corresponding vector on the curve r if the two curves were parallel. In Fig. 2, for example, if the two matching curves were parallel, the vector linking two successive feature points < and < on one curve should also be I I\H parallel to the vector linking the corresponding pairs of feature points < and < on the other curve, where J J\H !M)j)1 or 1)j)M and M is a small constant compared to NG. (To satisfy the requirements, 1)k!j)N and 1)l!j)N, here we limit the integer variables k and l in the domains, M#1) k)N!M and M#1)l)N!M.) If we jointly
In fact, G is a weighting matrix used for controlling the contributions from di!erent feature elements. For simpli"cation, we use a normalized similarity measure SK to I J represent the similarity of feature point pair. A de"nition of SK can be obtained from the matching error S as I J I J follows: S I J SK "1! . I J Max S I J I J
(1)
From the above de"nition, it is easy to observe that the value of SK is also invariant to any a$ne transformation. G J 3.1.2. A similarity measure based on the Tree-descriptor From Section 2.3, we observe qualitatively that for two parallel curves, the corresponding vectors, respectively, in the sets ¹ree(<) and ¹ree(<) should also be parallel. I J That is, the vector << should be parallel to the I I\H vector << . To formalize this observation and to J J\H design a similarity measure that is a$ne invariant, an area-based technique is also suggested here. If the vector
D. Shen et al. / Pattern Recognition 33 (2000) 1909}1918
<< is exactly parallel to the vector << , the I I\H J J\H following measure ¹ ( j) should be 0. IJ ¹ ( j)"0.5#<< # ) #<< # ) "sin h ", I J I I\H J J\H IJH where h "angle(<< , << ) and #<< # is the IJH I I\H J J\H I I\H length of vector << . ¹ ( j) is in e!ect the magnitude I I\H IJ of the cross-product of the two vectors. In Fig. 2(b), we align the vectors << and << and made the point J J\H I I\H < localized on the point <. This way, the descriptor J I ¹ ( j) is equal to the area of the triangle with the three IJ vertices <, < and (< ) . (Notice that the de"nition of I I\H J\H ¹ ( j ) is relatively a$ne-invariant.) But in practice, for I J example, for hand-drawn parallel curves or curves extracted by image processing routines, two curves are seldom exactly parallel. Fig. 3 gives examples for such inexact parallelism. We can determine the degree of local parallelism ¹ beI J tween the two curve segments from their corresponding Tree descriptor, ¹ree(<) and ¹ree(<), as follows: I J + ¹ " ¹ ( j) I J I J H\+ H$ + " 0.5#<< # ) #<< # ) "sin h ", I I\H J J\H IJH H\+ H$ where h "angle(<< , << ). IJH I I\H J J\H
1913
Similarly, we can normalize ¹ as follow, and render I J ¹K exactly a$ne invariant. I J ¹ I J ¹K "1! . I J Max ¹ I J I J
(2)
3.1.3. Azne-invariant local similarity matrix These two measures of local similarity, derived, respectively, from our Semi-local shape descriptor and Tree descriptor, can be combined and used to de"ne the azneinvariant local similarity measure as follows: E "(1!g)SK #g¹K , I J I J I J
(3)
where g is a regularization parameter used for controlling the contribution from the two terms. In our experiments, we use g"0.75. Notice the value of E is between 0 I J and 1. We can therefore compute the above similarity measures for every pair of feature points, respectively, on the two curves to form an azne-invariant local similarity matrix. Each element of such a matrix gives a measure of local similarity for a pair of points, respectively, on the two curves. We show in the following that using the azne-invariant local similarity matrix, parallelism detection between two curves can be achieved by detecting
Fig. 3. An example showing that in practice two perceptually parallel curves are not always exactly parallel, but their corresponding Tree descriptors are similar (see right-corner sub"gure, where one curve has been moved to match another curve).
1914
D. Shen et al. / Pattern Recognition 33 (2000) 1909}1918
a line with slope #1 within the matrix. This greatly simpli"es the parallelism detection process. Let us assume that one segment (Seg ) on the "rst curve r is parallel to another segment (Seg ) on the second curve r, and also the kth feature point on the segment Seg of the "rst curve r corresponds to the lth feature point on the segment Seg of the second curve r. Under this ideal condition, the a$ne-invariant local similarity measure should be E "1. This way, if we I J
visualize the local similarity matrix, E"[E ], as an I J image with intensities between 0 and 1 (level 1 means white), then, in the ideal case, we would observe in this image a continuous white line (i.e. line of 1s) passing through the position (k, l) with slope equal to#1 (see Fig. 4(c1) for example). In practice, since noise and deformation on the curves may occur, the a$ne-invariant local similarity measure, E , may not be exactly 1. To cater for noise, as well as I J
Fig. 4. Parallelism detection on two thick curves in (a1). Sub"gures (a1}a4) shows the a$ne-transformed curves and their parallelism detection results by our method, where thin lines link the corresponding points. Sub"gure (b1) represents the shape similarity measures based on Semi-local shape descriptor, while sub"gure (b2) represents the parallel measures based on the Tree-descriptor. The combined a$ne-invariant local similarity matrix is shown in (c1). The binarized a$ne invariant local similarity matrix is given in (c2). Sub"gure (c3) gives the result of line segmentation, while in (c4) the grey line is the "tting line for this segmented region.
D. Shen et al. / Pattern Recognition 33 (2000) 1909}1918
variability in the sample point selection and to make allowance for deformation, we instead look for high intensity lines with slope #1 in the image produced by the a$ne-invariant local similarity matrix (also see Fig. 4. (c1}c4) for example). The reason why we suggest combining the two similarity measures for parallelism detection can be explained by the example described below. For instance, if the shapes of two curves are identical then the similarity measure based on the semi-local shape descriptor alone is su$cient to detect such occurrence. However, in general, two curves having identical shape cannot be considered parallel, e.g. when one curve has undergone a rotational transform with respect to the other. So the second similarity measure based on Tree descriptor is needed in order to constrain that the two curves to have the similar shape as well as orientation for parallelism detection. 3.2. Parallelism detection based on the azne-invariant local similarity matrix An important conclusion of the above discussions is that lines of 1s with slope #1 in the azne-invariant local similarity matrix indicates the location and existence of parallel segments, respectively, on the two curves. It follows that parallelism detection can be formulated as a line detection problem within the similarity matrix, more importantly, the orientation of the line becomes a known a priori under this formulation. Many algorithms exist for line detection from image intensities. Here, we "rst binarize the a$ne-invariant local similarity matrix at a suitable threshold value. In our experiments, we set our threshold at S "0.55. UsG J ing this threshold, we can get a binarized local similarity matrix, where E "0 if E (0.55. (Fig. 4(c2) shows a I J I J binarized local similarity image from Fig. 4(c1).) Given the binarized local similarity matrix, it is easy to detect all line segments with slope #1 within it through suitable spatial "ltering such as morphological opening and closing with a line structuring element (Fig. 4(c3)). For every connected regions, a line "tting procedure is employed to obtain the "tted line. See Fig. 4(c4) for example. Interestingly, there may exist a set of lines within the matrix, which gives the set of potential parallel curve segments in the two curves. In fact, we can easily obtain the transformation relationship between every pair of segments, since we already know their correspondence [10,11]. Such transformation information can be used to determine whether one curve segment has been rotated through a large orientation with respect to another curve segment. We can "lter out the pair of curve segments which are separated by a signi"cant angle of rotation between them if necessary.
1915
Based on the above development, we can de"ne our parallelism detection algorithm as follows: (1) Binarize the a$ne-invariant local similarity matrix. (2) Detect line segments within it with slope #1 using line-"tting procedure. (3) Select "tted lines satisfying the constraint condition that the pairs of curve segments should be related with only a small rotation angle, and with at least a certain length. (4) Display the lines joining the corresponding points, respectively, on the two curves.
4. Experimental results In this section, we will provide a series of experimental results on parallelism detection. The proposed algorithm is tested by (a) curves obtained from natural or handdrawn images (Figs. 4}7), and (b) curves from handwritten digits (Fig. 8). Fig. 4 illustrates the process of parallelism detection. The two curves being studied are the two thick curves in (a1). Sub"gure (b1) represents the Semi-local similarity measures SK calculated by Semi-local shape descriptor, G J while sub"gure (b2) represents the Tree similarity measures ¹K by the Tree descriptor. Observing these I J
Fig. 5. Parallelism detection on two thick curves. Tests are performed under random a$ne-transformations.
For some applications, we may be unable to guarantee that the sampling orders, respectively, for the two studied curves are the same. In these cases, we have to operate the above algorithm on the same curves twice. During the xrst operation, the algorithm is performed on the two curves with the sampling orders provided originally. If there exist any pair of parallel segments on these two curves, they can be directly detected. During the second operation, we reverse one curve's sampling order and then perform the algorithm again.
1916
D. Shen et al. / Pattern Recognition 33 (2000) 1909}1918
two similarity images, it is easy to see that the Semi-local similarity measure SK and the Tree similarity measure G J ¹K in fact compare di!erent shape information localized I J on the two curves. The a$ne-invariant local similarity
matrix is shown in (c1). The binarized similarity matrix is given in (c2). The result of line detection is given in sub"gure (c3), while the "tted line for detected line segments is shown as a grey line in (c4). Sub"gures (a1}a4) shows the a$ne-transformed curves (thick curves) and the results of parallelism detection by our method, where thin lines link the corresponding points. Other two parallelism detection experiments are given in Figs. 5 and 6. There may exist more than one signi"cant pairs of parallel segments on two curves. Fig. 7 shows an example of the detection of two pairs of parallel segments. As an application, we also show experimentally that our algorithm can be applied to the skeletons of
Fig. 6. Another example on parallelism detection, where two thick curves are the matching curves. All the tests are performed under random a$ne-transformations on a pair of curves.
Fig. 7. Two pairs of segments on the two thick curves are, respectively, parallel. The thin lines are the resulted parallelism detected by our method.
Fig. 8. Parallelism detection experiments on handwriting digits under a$ne-transformations.
D. Shen et al. / Pattern Recognition 33 (2000) 1909}1918
handwriting digits, which may be useful for online or o!-line handwriting digits checking. Since the result of parallelism detection and the total matching error (a summation of the combined local similarity measures) on the detected pair of segments can be used for determining the similarity between the input handwriting digit and the models in the database. Fig. 8 shows some preliminary results on hand-written digits.
5. Conclusion In this paper, we present two new shape descriptors, Semi-local shape descriptor and Tree descriptor, which can be used for parallelism detection between two 3D planar curves. The Semi-local shape descriptor captures both local and semi-local features for every feature point on the curves, and the Tree descriptor serves to give measure on the degree of local parallelism for the segments on a curve with respect to another curve. Based on these two descriptors, we develop an a$ne-invariant local similarity matrix and simplify the problem of parallelism detection into a line detection problem within the a$ne-invariant local similarity matrix, computed for the two curves. Since these two shape descriptors provide su$cient local and semi-local shape information at every feature point along a curve, the process of detecting parallelism is thus robust against both noise and deformations. Moreover, the technique allows all signi"cant pairs of parallel segments within two curves to be detected. In our experiments, we applied the proposed technique to detecting randomly a$ne-transformed curves. Some of the testing curves are obtained from natural images, while others are manually drawn. We also applied the proposed technique to the skeletons of handwriting digits, which may be interesting if parallelism detection were used as an approach for recognizing online/o!-line handwriting digits.
1917
References [1] F. Ulupinar, R. Nevatia, Inferring shape from contour for curved surfaces, in: Proceedings of the International Conference on Pattern Recognition, 1990, pp. 147}154. [2] F. Ulupinar, R. Nevatia, Recovering shape from contour for constant cross section generalized cylinders, in: Proceedings of the IEEE Conference Computing Vision Pattern Recognition, Maui, HI, 1991, pp. 674}676. [3] T. Kanade, Recovery of the three-dimensional shape of an object from a single view, Arti"cial Intell. 17 (1981) 409}460. [4] M. Wertheimer, Principles of perceptual organisation (translated), in: D. Beardslee, M. Wertheimer (Eds.), Readings in Perception, Van Nostrand, Princeton, 1958. [5] P. Saintmarc, H. Rom, G. Medioni, B-spline contour representation and symmetry detection, IEEE Trans. Pattern Anal. Mach. Intell. 15 (11) (1993) 1191}1197. [6] A.D. Gross, T.E. Boult, Analyzing skewed symmetries, Int. J. Comput. Vision 13 (1994) 91}111. [7] A.M. Bruckstein, D. Shaked, Skew-symmetry detection via invariant signatures, Pattern Recognition 31 (2) (1998) 181}192. [8] J. Sato, R. Cipolla, A$ne integral invariants for extracting symmetry axes, Image Vision Comput. 15 (8) (1997) 627}635. [9] D. Shen, H.H.S. Ip, E.K. Teoh, Robust detection of skewed symmetries by combining local and semi-local a$ne invariants, IEEE Trans. PAMI, 1998, submitted for publication. [10] D. Shen, W.H. Wong, H.H.S. Ip, A$ne invariant image retrieval by correspondence matching of shapes. Image Vision Comput. 17 (7) (1999) 489}499. [11] H.H.S. Ip, D. Shen, An a$ne-invariant active contour model (AI-snake) for model-based segmentation, Image Vision Comput. 16 (2) (1998) 135}146. [12] H.H.S. Ip, W.H. Wong, Detecting perceptually parallel curves * criteria and force-driven optimization, Comput. Vision Image Understanding 68 (2) (1997) 190}208. [13] L. van Gool, T. Moons, D. Ungureanu, E. Pauwels, Symmetry from shape and shape from symmetry, Int. J. Robotics Res. 14 (5) (1995) 407}424. [14] W.H. Wong, D. Shen, H.H.S. Ip, A$ne invariant shape matching for content-based retrieval, ICARCV'96, Singapore, December 1996.
About the Author*DINGGANG SHEN received his B.S., M.S. and Ph.D. degrees in Electronics Engineering from Shanghai JiaoTong University (SJTU) in 1990, 1992 and 1995, respectively. He worked as Research Assistant in the Department of Computer Science at the Hong Kong University of Science & Technology from December 1994 to June 1995. From September 1995 to February 1996, he was a Lecturer of Communication Engineering at Shanghai JiaoTong University. He was a Research Fellow in the Department of Computer Science at City University of Hong Kong, from February 1996 to August 1997. From June 1997 to February 1999, he worked "rst as Post-Doctoral Fellow and then as Research Fellow in the School of Electrical and Electronic Engineering at Nanyang Technological University, Singapore. Since January 1999, he worked as Postdoctoral Research Fellow in School of Medicine at Johns Hopkins University, doing medical imaging. His research interests are in the areas of computer vision, pattern recognition, image processing, neural network and image indexing and retrieval.
About the Author*HORACE H.S. IP received his B.Sc. (First Class Honours) degree in Applied Physics and Ph.D. degree in Image Processing from University College London, United Kingdom, in 1980 and 1983, respectively. Presently, he is the Professor and Head of the Computer Science Department and also heads the Image Computing Research Group, at City University of Hong Kong. His research interests include image processing and analysis, pattern recognition, hypermedia computing systems and computer graphics. Prof. Ip is a member of the Editorial Boards for Pattern Recognition (Elsevier), the International Journal of Multimedia Tools and
1918
D. Shen et al. / Pattern Recognition 33 (2000) 1909}1918
Applications (Kluwer Academic), and the Chinese Journal of CAD and Computer Graphics (The Chinese Academy of Science) and a guest editor of the international journal of Real-Time Imaging (Academic Press). Prof. Ip serves on the IAPR Governing Board and co-chairs its Technical Committee on Multimedia Systems. He was the Chairman of the IEEE (HK) Computer chapter, and the Founding President of the Hong Kong Society for Multimedia and Image Computing. He has published over 80 papers in international journals and conference proceedings. About the Author*EAM KHWANG TEOH received his B.E. and M.E. degrees in Electrical Engineering from the University of Auckland, New Zealand in 1980 and 1982, respectively, and the Ph.D. degree in Electrical & Computer Engineering from the University of Newcastle, New South Wales in 1986. Presently, he is a Vice-Dean and an Associate Professor in the School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore. He is a co-inventor of an Australian Patent on An Adaptive Thickness Controller for a Rolling Mill. His current research interests are in the "eld of Computer Vision and Pattern Recognition, Intelligent Systems, Robotics and Industrial Automation. He has published more than 150 journal and conference papers in these areas.
A$ne invariant detection of perceptually parallel 3D planar curves Dinggang Shen , Horace H.S. Ip *, Eam Khwang Teoh Department of Radiology, School of Medicine, Johns Hopkins University, 601 N. Caroline Street, Baltimore, MD 21287, USA Image Computing Group, Department of Computer Science, City University of Hong Kong, Kowloon, Hong Kong School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore 639798 Received 7 January 1999; received in revised form 9 July 1999; accepted 9 July 1999
Abstract The problem of parallelism detection between two curves has been formulated in this paper as a line detection problem within an azne-invariant local similarity matrix computed for the two curves. Each element of this matrix gives an a$ne invariant measure of local parallelism for any pair of curve segments along the two curves. This approach enables the detection of a pair of parallel 3D planar curves as well as parallel 2D curves under general a$ne transform. Two descriptors were also used here to provide a multi-resolution representation of a curve. Since these two descriptors provide su$cient local and semi-local shape information at every feature point on the curves, the process of detecting parallelism is thus robust against both noise and deformations. Moreover, the proposed technique allows all signi"cant pairs of parallel segments within any two curves in the scene to be detected. Experiments on detecting randomly a$ne-transformed curves, which are obtained from natural images or arti"cially generated images, have demonstrated the e!ectiveness of the technique. 2000 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. Keywords: Parallelism detection; Local invariants; Semi-local invariants; A$ne invariant; Shape matching; Symmetry detection; 3D parallel curves; Hand-written digit recognition
1. Introduction The detection of symmetries is an essential step when inferring shapes from contours [1]. Particularly, skewed symmetries and parallel symmetries have been shown to be useful in recovering the 3D structure of various types of objects from their 2D contours [2,3]. Results from the study of perceptual organization in Ref. [4] also showed that primitive grouping in our vision system is done on proximity, collinearity, curvilinearity, parallelism, sym-
* Corresponding author. Tel.: #852-2788-8641; fax: #8522784-4916. E-mail addresses: [email protected] (D. Shen), [email protected] (H.H.S. Ip).
metry, closure and repetitive patterns. Symmetries (including parallelism) have played important roles in shape perception [5]. The detection of skewed symmetries has been investigated by many researchers in the past [6,7]. Depending on the nature of the features extracted from a shape, existing methods for symmetry detection can be roughly classi"ed into local and global approaches [6]. There are advantages and disadvantages associated, respectively, with these two approaches. To overcome some of the disadvantages relating to both global and local approaches, Sato and Cipolla [8] proposed integral invariant based on group invariant parameterization. They showed that integral invariant did not su!er from occlusion problem and is less sensitive to noise than di!erential invariant. We also have developed techniques in Ref. [9] for the detection of both local and global symmetries
0031-3203/00/$20.00 2000 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. PII: S 0 0 3 1 - 3 2 0 3 ( 9 9 ) 0 0 1 7 2 - 7
1910
D. Shen et al. / Pattern Recognition 33 (2000) 1909}1918
as well as skewed symmetries using the a$ne-invariant representation developed in Refs. [10,11]. Based on this representation, in this paper, the detection of skewed symmetries has been simpli"ed and formulated as a line detection problem with known orientations within a local similarity matrix. Parallelism is an important form of symmetries, and the detection of parallelism between two curves has attracted much attention in the vision community in the past. By representing the various properties of parallelism in terms of several distinctive forces, the second author has developed a force-driven model as a new optimization strategy to compute correspondence between salient points extracted from the matching curves [12]. This force-driven mechanism provides good coupling (or correspondence matching) results, which is the prerequisite for the correct detection of parallelism between curves. For curves to be parallel, certain point-wise correspondence between two curves must exist. Based on this observation, the research issues on parallelism detection are closely related to those for shape matching. Additionally, extra constraints like proximity and similarity in orientation should be added if parallelism detection is to be formulated as a shape matching problem. This is due to the fact that in shape matching the two input shapes may be under very di!erent orientation or even under severe a$ne transformation with respect to each other. Almost all solutions to shape (curve) matching were based on descriptor distance measures. Normally, local descriptors were favored since global descriptors could not handle partial matching, but local descriptors are sensitive to both noise and local deformation. Furthermore, when the correspondence for the two matching curves are known, transformation parameters between these two curves can be estimated. In this paper, we introduced two descriptors, namely, Semi-local shape descriptor and Tree-descriptor. Since these two descriptors extract both local and semi-local shape information from the curves, the process of detecting parallelism is thus robust against both noise and deformations. More importantly, the problem of parallelism detection can then be formulated as a line detection problem with known orientation within an azne-invariant local similarity matrix. Since the technique proposed here allows all signi"cant pairs of parallel segments contained in the two matching curves to be detected, partial matches for parallelism can also be detected. We organize the paper as follows. The formulations of two descriptors are given in Section 2. There, for every feature point, we establish (a) a feature vector which is able to capture both local and semi-local shape features, and (b) a set of Tree-like lines which connect the current feature point and its neighboring feature points for successive scales. Based on the two descriptors de"ned in Sections 2 and 3 describes a technique for detecting all
signi"cant pairs of parallel segments contained in the two matching curves. In fact, the process of parallelism detection is achieved by detecting lines of known orientations within an azne-invariant local similarity matrix, which contains the similarity measures calculated through the two descriptors for every pair of feature points along the two matching curves. A number of experiments on detecting randomly a$ne-transformed curves, which are obtained from natural or hand-drawn images, are given in Section 4. This paper concludes in Section 5.
2. Two new shape descriptors Based on the de"nition of the parallelism given below, we provide two descriptors for detecting parallel segments within two curves. These two descriptors were then combined to de"ne a set of similarity measures in the next section, and then used for parallelism detection for the two curves. 2.1. Dexnition of parallelism For curves to be parallel, certain point-wise correspondence between two curves must exist. Based on this requirement, the de"nition of parallelism from [1] is given below. 2.1.1. A dexnition of parallelism Let rG(t)"[xG(t), yG(t)]2, for i"1, 2, be two curves parameterized by arc length t. Let hG(t)"arctan ((dyG(t)/dt)/(dxG(t)/dt)). Then r(t) and r(t) are said to be parallel symmetric if there exist a point-wise correspondence f (t) between them such that h(t)"h( f (t)) for all values of t for which r and r are de"ned and f (t) is a continuous monotonous function. A useful special case is when f (t) is restricted to be a linear function. To solve the parallelism detection problem for the discrete case, let us assume that +(xG , yG ), (xG , yG ) ,2, (xG G , yG G ),, for i"1, 2, is the set of feature points , , representing the curve rG in a particular scale. Here, NG is the total number of sampling points on the curve rG. These feature points can be extracted, for example, from the sample points, which are evenly distributed along the azne-length shape boundary [13]. To render these feature points robust against noise, we take the same number of points from both sides of a sample point and average them to obtain a feature point. Based on the above de"nition of parallelism, two shape descriptors can be de"ned as follows. 2.2. Semi-local shape descriptor From the point of view of shape matching, the detection of parallelism can be approached as the problem of matching between two shapes. An a$ne-invariant
D. Shen et al. / Pattern Recognition 33 (2000) 1909}1918
feature vector for every feature point on the curve has been developed by the authors and applied to a number of problems: shape matching and indexing [10,14], model-based adaptive image segmentation [11], and detection of the skewed symmetries (both rotational symmetry and re#ectional symmetry) [9]. In the following, we will use our a$ne-invariant feature vector as a semi-local shape descriptor for every feature point on the two curves. This way, corresponding feature points on a pair of parallel segments should have similar feature vectors. The de"nition of the a$ne-invariant feature vector has been presented in Ref. [11] but we will brie#y review it here for completeness. Let r G(t) and rK G(t) be the "rst and second derivatives of the curve rG(t), where i"1, 2. Their de"nitions are
rR G(t)"[xR G(t) y G(t)]2"
dxG(t) dyG(t) 2 , dt dt
rK G(t)"[xK G(t) yK G(t)]2"
dxG(t) dyG(t) 2 . dt dt
vertices (xG(t!2*t), yG(t!2*t)), (xG(t!*t), yG(t!*t)) and (xG(t), yG(t)). In fact, the area of any triangle is relatively a$ne invariant and the ratios of such areas are a$ne invariant:
xG(t ) 1 area " yG(t ) R R R 2 1
xG(t ) yG(t ) 1
xG(t ) yG(t ) , 1
where area is the area of a triangle, whose three R R R vertices are (xG(t ), yG(t )), (xG(t ), yG(t )) and (xG(t ), yG(t )). Notice, t , t and t are the parameters of the curve. Accordingly, an a$ne-invariant feature vector can be designed for every feature point of the given curves. For the kth feature point (xG , yG ), its corresponding feature I I vector is de"ned as FG "[ f G f G .... f G ]2, I I I +I where
Under general a$ne transformations, the following three expressions are invariant up to a scale factor: "(rG(t)!rG(t )) (rG(t)!rG(t ))", "(rG(t)!rG(t )) r G(t)", "r G(t) r( G(t)", where " ) " denotes the determinant of a matrix. That is, their values remain unchanged up to a factor which is equal to the determinant of the a$ne transformation. In this paper, we will use "r G(t) rK G(t)""(x G(t)yK G(t)!xK G(t)y G(t)) to de"ne an a$ne-invariant feature vector for every point along the curve rG(t). The "rst and second derivatives of xG(t) and yG(t) with respect to t can be approximated as follows: dxG(t) xG(t)!xG(t!*t) x G(t)" & dt *t dxG(t) xG(t)!2xG(t!*t)#xG(t!2*t) xK G(t)" & dt *t *t dyG(t) yG(t)!yG(t!*t) y G(t)" & dt *t dyG(t) yG(t)!2yG(t!*t)#yG(t!2*t) yK G(t)" & dt *t *t Using the above approximations, we can obtain
1911
xG(t!2*t) xG(t!*t) xG(t) 1 1 "r G(t) rK G(t)"" xG(t!2*t) yG(t!*t) yG(t) . 2 2 1 1 1 The above expression indicates that the value of "r G(t) rK G(t)" denotes the area of a triangle, formed by three
xG 1 I\H f G " yG HI 2 I\H 1
xG I yG I 1
xG I>H yG I>H 1
is the area of a triangle formed by (xG , yG ), (xG , yG ), I\H I\H I I and (xG , yG ). (Notice that the integer variables k I>H I>H and l are in domains, M#1)k)N!M and M#1)l)N!M.) The size of M in e!ect determines the sampling resolution of the curve for our representation. It should be noted that if j is close to 1, f G denotes HI local feature, i.e. f G (for small j) extracts information of HI local shape centered on the point (xG , yG ), which is sensiI I tive to noises. As j increases, f G gradually captures the HI semi-local features of the curve, i.e. f G extracts low-freHI quency (smooth) information of the shape, which is less sensitive to digitization noise. In essence, f G gives, after HI suitable normalization, in a single feature vector, both local and semi-local a$ne-invariant information of the curve with respect to a single point on the curve. Fig. 1 shows the Semi-local shape descriptor for the kth feature point (xG , yG ) on the ith curve. I I FG can be made exactly a$ne-invariant by the followI ing normalization: FG I FK G " G , I , + " f G " I H HI where FK G "[ fK G fK G .... fK G ]2. I I I +I The curve rG(t) can thus be described by a set of a$neinvariant feature vectors, +FK G , k"M#1, M#2,2, NG!M and i"1, 2,. I In Section 3, we will adopt this a$ne-invariant local shape descriptor and use them to compute a set of similarity measures for every coupled feature points on a pair of curves.
1912
D. Shen et al. / Pattern Recognition 33 (2000) 1909}1918
consider the same number (M) of feature points before and after the feature point
Fig. 1. An illustration of the area-based semi-local shape descriptor.
3.1.1. A similarity measure based on the Semi-local shape descriptor Suppose that two curves r and r are parallel and the feature points < and <, respectively, on the "rst curve I J r and the second curve r are the corresponding points. Then their feature vectors, FK and FK , should be the I J same. More importantly, this property holds even under a$ne-transformation. Mathematically, the matching error S between the feature vectors FK and FK is as I J I J follows: S "(FK !FK )2 G(FK !FK ), I J J I J I where G is an M;M diagonal matrix de"ned by a Gaussian function, i.e. K +\ 1 G"[g ] and g " e\ N . KK +"+ KK (2pp
Fig. 2. (a) An illustration of the Tree descriptor; (b) A di!erence measure between two corresponding vectors.
2.3. Tree-descriptor It is easy to observe that for every feature point (x, y) I I on the curve r, the vector joining this feature point and its neighboring feature point should be parallel to the corresponding vector on the curve r if the two curves were parallel. In Fig. 2, for example, if the two matching curves were parallel, the vector linking two successive feature points < and < on one curve should also be I I\H parallel to the vector linking the corresponding pairs of feature points < and < on the other curve, where J J\H !M)j)1 or 1)j)M and M is a small constant compared to NG. (To satisfy the requirements, 1)k!j)N and 1)l!j)N, here we limit the integer variables k and l in the domains, M#1) k)N!M and M#1)l)N!M.) If we jointly
In fact, G is a weighting matrix used for controlling the contributions from di!erent feature elements. For simpli"cation, we use a normalized similarity measure SK to I J represent the similarity of feature point pair. A de"nition of SK can be obtained from the matching error S as I J I J follows: S I J SK "1! . I J Max S I J I J
(1)
From the above de"nition, it is easy to observe that the value of SK is also invariant to any a$ne transformation. G J 3.1.2. A similarity measure based on the Tree-descriptor From Section 2.3, we observe qualitatively that for two parallel curves, the corresponding vectors, respectively, in the sets ¹ree(<) and ¹ree(<) should also be parallel. I J That is, the vector << should be parallel to the I I\H vector << . To formalize this observation and to J J\H design a similarity measure that is a$ne invariant, an area-based technique is also suggested here. If the vector
D. Shen et al. / Pattern Recognition 33 (2000) 1909}1918
<< is exactly parallel to the vector << , the I I\H J J\H following measure ¹ ( j) should be 0. IJ ¹ ( j)"0.5#<< # ) #<< # ) "sin h ", I J I I\H J J\H IJH where h "angle(<< , << ) and #<< # is the IJH I I\H J J\H I I\H length of vector << . ¹ ( j) is in e!ect the magnitude I I\H IJ of the cross-product of the two vectors. In Fig. 2(b), we align the vectors << and << and made the point J J\H I I\H < localized on the point <. This way, the descriptor J I ¹ ( j) is equal to the area of the triangle with the three IJ vertices <, < and (< ) . (Notice that the de"nition of I I\H J\H ¹ ( j ) is relatively a$ne-invariant.) But in practice, for I J example, for hand-drawn parallel curves or curves extracted by image processing routines, two curves are seldom exactly parallel. Fig. 3 gives examples for such inexact parallelism. We can determine the degree of local parallelism ¹ beI J tween the two curve segments from their corresponding Tree descriptor, ¹ree(<) and ¹ree(<), as follows: I J + ¹ " ¹ ( j) I J I J H\+ H$ + " 0.5#<< # ) #<< # ) "sin h ", I I\H J J\H IJH H\+ H$ where h "angle(<< , << ). IJH I I\H J J\H
1913
Similarly, we can normalize ¹ as follow, and render I J ¹K exactly a$ne invariant. I J ¹ I J ¹K "1! . I J Max ¹ I J I J
(2)
3.1.3. Azne-invariant local similarity matrix These two measures of local similarity, derived, respectively, from our Semi-local shape descriptor and Tree descriptor, can be combined and used to de"ne the azneinvariant local similarity measure as follows: E "(1!g)SK #g¹K , I J I J I J
(3)
where g is a regularization parameter used for controlling the contribution from the two terms. In our experiments, we use g"0.75. Notice the value of E is between 0 I J and 1. We can therefore compute the above similarity measures for every pair of feature points, respectively, on the two curves to form an azne-invariant local similarity matrix. Each element of such a matrix gives a measure of local similarity for a pair of points, respectively, on the two curves. We show in the following that using the azne-invariant local similarity matrix, parallelism detection between two curves can be achieved by detecting
Fig. 3. An example showing that in practice two perceptually parallel curves are not always exactly parallel, but their corresponding Tree descriptors are similar (see right-corner sub"gure, where one curve has been moved to match another curve).
1914
D. Shen et al. / Pattern Recognition 33 (2000) 1909}1918
a line with slope #1 within the matrix. This greatly simpli"es the parallelism detection process. Let us assume that one segment (Seg ) on the "rst curve r is parallel to another segment (Seg ) on the second curve r, and also the kth feature point on the segment Seg of the "rst curve r corresponds to the lth feature point on the segment Seg of the second curve r. Under this ideal condition, the a$ne-invariant local similarity measure should be E "1. This way, if we I J
visualize the local similarity matrix, E"[E ], as an I J image with intensities between 0 and 1 (level 1 means white), then, in the ideal case, we would observe in this image a continuous white line (i.e. line of 1s) passing through the position (k, l) with slope equal to#1 (see Fig. 4(c1) for example). In practice, since noise and deformation on the curves may occur, the a$ne-invariant local similarity measure, E , may not be exactly 1. To cater for noise, as well as I J
Fig. 4. Parallelism detection on two thick curves in (a1). Sub"gures (a1}a4) shows the a$ne-transformed curves and their parallelism detection results by our method, where thin lines link the corresponding points. Sub"gure (b1) represents the shape similarity measures based on Semi-local shape descriptor, while sub"gure (b2) represents the parallel measures based on the Tree-descriptor. The combined a$ne-invariant local similarity matrix is shown in (c1). The binarized a$ne invariant local similarity matrix is given in (c2). Sub"gure (c3) gives the result of line segmentation, while in (c4) the grey line is the "tting line for this segmented region.
D. Shen et al. / Pattern Recognition 33 (2000) 1909}1918
variability in the sample point selection and to make allowance for deformation, we instead look for high intensity lines with slope #1 in the image produced by the a$ne-invariant local similarity matrix (also see Fig. 4. (c1}c4) for example). The reason why we suggest combining the two similarity measures for parallelism detection can be explained by the example described below. For instance, if the shapes of two curves are identical then the similarity measure based on the semi-local shape descriptor alone is su$cient to detect such occurrence. However, in general, two curves having identical shape cannot be considered parallel, e.g. when one curve has undergone a rotational transform with respect to the other. So the second similarity measure based on Tree descriptor is needed in order to constrain that the two curves to have the similar shape as well as orientation for parallelism detection. 3.2. Parallelism detection based on the azne-invariant local similarity matrix An important conclusion of the above discussions is that lines of 1s with slope #1 in the azne-invariant local similarity matrix indicates the location and existence of parallel segments, respectively, on the two curves. It follows that parallelism detection can be formulated as a line detection problem within the similarity matrix, more importantly, the orientation of the line becomes a known a priori under this formulation. Many algorithms exist for line detection from image intensities. Here, we "rst binarize the a$ne-invariant local similarity matrix at a suitable threshold value. In our experiments, we set our threshold at S "0.55. UsG J ing this threshold, we can get a binarized local similarity matrix, where E "0 if E (0.55. (Fig. 4(c2) shows a I J I J binarized local similarity image from Fig. 4(c1).) Given the binarized local similarity matrix, it is easy to detect all line segments with slope #1 within it through suitable spatial "ltering such as morphological opening and closing with a line structuring element (Fig. 4(c3)). For every connected regions, a line "tting procedure is employed to obtain the "tted line. See Fig. 4(c4) for example. Interestingly, there may exist a set of lines within the matrix, which gives the set of potential parallel curve segments in the two curves. In fact, we can easily obtain the transformation relationship between every pair of segments, since we already know their correspondence [10,11]. Such transformation information can be used to determine whether one curve segment has been rotated through a large orientation with respect to another curve segment. We can "lter out the pair of curve segments which are separated by a signi"cant angle of rotation between them if necessary.
1915
Based on the above development, we can de"ne our parallelism detection algorithm as follows: (1) Binarize the a$ne-invariant local similarity matrix. (2) Detect line segments within it with slope #1 using line-"tting procedure. (3) Select "tted lines satisfying the constraint condition that the pairs of curve segments should be related with only a small rotation angle, and with at least a certain length. (4) Display the lines joining the corresponding points, respectively, on the two curves.
4. Experimental results In this section, we will provide a series of experimental results on parallelism detection. The proposed algorithm is tested by (a) curves obtained from natural or handdrawn images (Figs. 4}7), and (b) curves from handwritten digits (Fig. 8). Fig. 4 illustrates the process of parallelism detection. The two curves being studied are the two thick curves in (a1). Sub"gure (b1) represents the Semi-local similarity measures SK calculated by Semi-local shape descriptor, G J while sub"gure (b2) represents the Tree similarity measures ¹K by the Tree descriptor. Observing these I J
Fig. 5. Parallelism detection on two thick curves. Tests are performed under random a$ne-transformations.
For some applications, we may be unable to guarantee that the sampling orders, respectively, for the two studied curves are the same. In these cases, we have to operate the above algorithm on the same curves twice. During the xrst operation, the algorithm is performed on the two curves with the sampling orders provided originally. If there exist any pair of parallel segments on these two curves, they can be directly detected. During the second operation, we reverse one curve's sampling order and then perform the algorithm again.
1916
D. Shen et al. / Pattern Recognition 33 (2000) 1909}1918
two similarity images, it is easy to see that the Semi-local similarity measure SK and the Tree similarity measure G J ¹K in fact compare di!erent shape information localized I J on the two curves. The a$ne-invariant local similarity
matrix is shown in (c1). The binarized similarity matrix is given in (c2). The result of line detection is given in sub"gure (c3), while the "tted line for detected line segments is shown as a grey line in (c4). Sub"gures (a1}a4) shows the a$ne-transformed curves (thick curves) and the results of parallelism detection by our method, where thin lines link the corresponding points. Other two parallelism detection experiments are given in Figs. 5 and 6. There may exist more than one signi"cant pairs of parallel segments on two curves. Fig. 7 shows an example of the detection of two pairs of parallel segments. As an application, we also show experimentally that our algorithm can be applied to the skeletons of
Fig. 6. Another example on parallelism detection, where two thick curves are the matching curves. All the tests are performed under random a$ne-transformations on a pair of curves.
Fig. 7. Two pairs of segments on the two thick curves are, respectively, parallel. The thin lines are the resulted parallelism detected by our method.
Fig. 8. Parallelism detection experiments on handwriting digits under a$ne-transformations.
D. Shen et al. / Pattern Recognition 33 (2000) 1909}1918
handwriting digits, which may be useful for online or o!-line handwriting digits checking. Since the result of parallelism detection and the total matching error (a summation of the combined local similarity measures) on the detected pair of segments can be used for determining the similarity between the input handwriting digit and the models in the database. Fig. 8 shows some preliminary results on hand-written digits.
5. Conclusion In this paper, we present two new shape descriptors, Semi-local shape descriptor and Tree descriptor, which can be used for parallelism detection between two 3D planar curves. The Semi-local shape descriptor captures both local and semi-local features for every feature point on the curves, and the Tree descriptor serves to give measure on the degree of local parallelism for the segments on a curve with respect to another curve. Based on these two descriptors, we develop an a$ne-invariant local similarity matrix and simplify the problem of parallelism detection into a line detection problem within the a$ne-invariant local similarity matrix, computed for the two curves. Since these two shape descriptors provide su$cient local and semi-local shape information at every feature point along a curve, the process of detecting parallelism is thus robust against both noise and deformations. Moreover, the technique allows all signi"cant pairs of parallel segments within two curves to be detected. In our experiments, we applied the proposed technique to detecting randomly a$ne-transformed curves. Some of the testing curves are obtained from natural images, while others are manually drawn. We also applied the proposed technique to the skeletons of handwriting digits, which may be interesting if parallelism detection were used as an approach for recognizing online/o!-line handwriting digits.
1917
References [1] F. Ulupinar, R. Nevatia, Inferring shape from contour for curved surfaces, in: Proceedings of the International Conference on Pattern Recognition, 1990, pp. 147}154. [2] F. Ulupinar, R. Nevatia, Recovering shape from contour for constant cross section generalized cylinders, in: Proceedings of the IEEE Conference Computing Vision Pattern Recognition, Maui, HI, 1991, pp. 674}676. [3] T. Kanade, Recovery of the three-dimensional shape of an object from a single view, Arti"cial Intell. 17 (1981) 409}460. [4] M. Wertheimer, Principles of perceptual organisation (translated), in: D. Beardslee, M. Wertheimer (Eds.), Readings in Perception, Van Nostrand, Princeton, 1958. [5] P. Saintmarc, H. Rom, G. Medioni, B-spline contour representation and symmetry detection, IEEE Trans. Pattern Anal. Mach. Intell. 15 (11) (1993) 1191}1197. [6] A.D. Gross, T.E. Boult, Analyzing skewed symmetries, Int. J. Comput. Vision 13 (1994) 91}111. [7] A.M. Bruckstein, D. Shaked, Skew-symmetry detection via invariant signatures, Pattern Recognition 31 (2) (1998) 181}192. [8] J. Sato, R. Cipolla, A$ne integral invariants for extracting symmetry axes, Image Vision Comput. 15 (8) (1997) 627}635. [9] D. Shen, H.H.S. Ip, E.K. Teoh, Robust detection of skewed symmetries by combining local and semi-local a$ne invariants, IEEE Trans. PAMI, 1998, submitted for publication. [10] D. Shen, W.H. Wong, H.H.S. Ip, A$ne invariant image retrieval by correspondence matching of shapes. Image Vision Comput. 17 (7) (1999) 489}499. [11] H.H.S. Ip, D. Shen, An a$ne-invariant active contour model (AI-snake) for model-based segmentation, Image Vision Comput. 16 (2) (1998) 135}146. [12] H.H.S. Ip, W.H. Wong, Detecting perceptually parallel curves * criteria and force-driven optimization, Comput. Vision Image Understanding 68 (2) (1997) 190}208. [13] L. van Gool, T. Moons, D. Ungureanu, E. Pauwels, Symmetry from shape and shape from symmetry, Int. J. Robotics Res. 14 (5) (1995) 407}424. [14] W.H. Wong, D. Shen, H.H.S. Ip, A$ne invariant shape matching for content-based retrieval, ICARCV'96, Singapore, December 1996.
About the Author*DINGGANG SHEN received his B.S., M.S. and Ph.D. degrees in Electronics Engineering from Shanghai JiaoTong University (SJTU) in 1990, 1992 and 1995, respectively. He worked as Research Assistant in the Department of Computer Science at the Hong Kong University of Science & Technology from December 1994 to June 1995. From September 1995 to February 1996, he was a Lecturer of Communication Engineering at Shanghai JiaoTong University. He was a Research Fellow in the Department of Computer Science at City University of Hong Kong, from February 1996 to August 1997. From June 1997 to February 1999, he worked "rst as Post-Doctoral Fellow and then as Research Fellow in the School of Electrical and Electronic Engineering at Nanyang Technological University, Singapore. Since January 1999, he worked as Postdoctoral Research Fellow in School of Medicine at Johns Hopkins University, doing medical imaging. His research interests are in the areas of computer vision, pattern recognition, image processing, neural network and image indexing and retrieval.
About the Author*HORACE H.S. IP received his B.Sc. (First Class Honours) degree in Applied Physics and Ph.D. degree in Image Processing from University College London, United Kingdom, in 1980 and 1983, respectively. Presently, he is the Professor and Head of the Computer Science Department and also heads the Image Computing Research Group, at City University of Hong Kong. His research interests include image processing and analysis, pattern recognition, hypermedia computing systems and computer graphics. Prof. Ip is a member of the Editorial Boards for Pattern Recognition (Elsevier), the International Journal of Multimedia Tools and
1918
D. Shen et al. / Pattern Recognition 33 (2000) 1909}1918
Applications (Kluwer Academic), and the Chinese Journal of CAD and Computer Graphics (The Chinese Academy of Science) and a guest editor of the international journal of Real-Time Imaging (Academic Press). Prof. Ip serves on the IAPR Governing Board and co-chairs its Technical Committee on Multimedia Systems. He was the Chairman of the IEEE (HK) Computer chapter, and the Founding President of the Hong Kong Society for Multimedia and Image Computing. He has published over 80 papers in international journals and conference proceedings. About the Author*EAM KHWANG TEOH received his B.E. and M.E. degrees in Electrical Engineering from the University of Auckland, New Zealand in 1980 and 1982, respectively, and the Ph.D. degree in Electrical & Computer Engineering from the University of Newcastle, New South Wales in 1986. Presently, he is a Vice-Dean and an Associate Professor in the School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore. He is a co-inventor of an Australian Patent on An Adaptive Thickness Controller for a Rolling Mill. His current research interests are in the "eld of Computer Vision and Pattern Recognition, Intelligent Systems, Robotics and Industrial Automation. He has published more than 150 journal and conference papers in these areas.