- Email: [email protected]
COP: a new corner detector
Pattern Recognition Letters 23 (2002) 1349–1360 www.elsevier.com/locate/patrec
COP: a new corner detector Sun Cheol Bae a
a,*
, In So Kweon b, Choong Don Yoo
a
Department of Mechanical Engineering, Korea Advanced Institute of Science and Technology, 373-1, Kusong-dong, Yusung-ku, Taejon 305-701, South Korea b Department of Electrical Engineering, Korea Advanced Institute of Science and Technology, 373-1, Kusong-dong, Yusung-ku, Taejon 305-701, South Korea Received 29 March 2001; received in revised form 9 November 2001
Abstract Most conventional derivative-based corner detectors have shortcomings such as missing junctions, poor localization, sensitivity to noise and high computational cost. This paper presents a new, simple and effective low level processing method to detect corners. In this corner detection algorithm, two oriented cross operators called crosses as oriented pair (COP) are used, which provide useful information to extract low-level features due to its characteristics, preference for edge with different direction and simple direction determination. Fast, accurate and noise-robust corner detection is accomplished with the COP. Ó 2002 Elsevier Science B.V. All rights reserved. Keywords: COP; USAN; Corner dominant direction
1. Introduction Corners play an important role in computer vision for many applications such as object recognition, motion analysis, scene analysis, stereo matching, etc. Much of the work on two-dimensional features have concentrated on corners, that is, features formed at boundaries between two significantly dissimilar image brightness regions, where the boundary curvature is sufficiently high. However, many other types of localized structure
*
Corresponding author. Tel.: +82-428693081. E-mail addresses: [email protected] (S.C. Bae), [email protected] (I.S. Kweon), [email protected] (C.D. Yoo).
arise, for example, ‘‘T’’,‘‘X’’ and ‘‘Y’’ junctions, and these multi-region junctions should be also considered. Corners are intrinsically second order properties of a surface. Therefore, most of the earliest methods of corner detection used convolution masks to approximate the first and second order derivatives of the image brightness function, but derivatives are sensitive to noise. They tend to suffer from poor localization, missing junctions, sensitivity to noise and high computational cost. There has been an abundance of work on different approaches to the detection of these features in images. Recently, Smith and Brady (1997) proposed a new low-level processing, SUSAN. They used only masking operations to obtain lowlevel features. In this paper, we also use the same concept, univalue segment assimilating nucleus
0167-8655/02/$ - see front matter Ó 2002 Elsevier Science B.V. All rights reserved. PII: S 0 1 6 7 - 8 6 5 5 ( 0 2 ) 0 0 0 8 3 - 1
1350
S.C. Bae et al. / Pattern Recognition Letters 23 (2002) 1349–1360
(USAN) and propose a new corner detection algorithm using two oriented cross operators called crosses as oriented pair (COP) to detect such features more easily and effectively. This paper begins with COP principle, and continues with a proposed corner detection algorithm, including results of testing algorithm in contrast to other wellknown corner detectors. 2. Literature review Two approaches are most widely used for corner detection. The first approach is to initially segment the image into regions, represent the object boundary with a chain code, and search for points having maximum curvature or perform a polygonal approximation on the chains and then search for the line segment intersections. The major drawback of this approach is that the performance of corner detection depends on the preceding image segmentation result. In order to overcome this, the second approach, which works directly on grey-level image, has been employed recently. This second approach mainly uses gradients and surface curvatures and essentially defines the following cornerness measure: C¼
Ixx Iy2 þ Iyy Ix2 2Ixy Ix Iy : ðIx2 þ Iy2 Þ
ð1Þ
Points are then declared corners if the value of cornerness measure meets some threshold conditions. One of such corner detectors is that proposed by Dreschler and Nagel (1981), who defined corners to be points lying between extrema of Gaussian curvature. Kitchen and Rosenfeld (1982) used a local quadratic surface fit to find corners. These methods also suffer from poor localization, missing junctions and sensitivity to noise. Wang and Brady (1995) developed a curvature based method. They filtered the image by 3 3 Sobel operators to obtain the edge strength and direction, and derived the equation approximating the total curvature. However, this algorithm also suffers from sensitivity to noise, and setting two thresholds is very difficult. Moravec (1977) developed the idea of using ‘‘points of interest’’, which
are defined as occurring when there are large intensity variations in every direction. But the response is noisy and anisotropic. Beaudet (1978) used a rotationally invariant DET operator. This method found saddle points in the image brightness surface by calculating Gaussian curvature, that is, the product of the two principle curvatures. Harris and Stephens (1988) introduced a corner operator by modifying the Moravec’s interest operator, using only the first order derivatives to approximate the second derivatives. Their method produces consistent corner responses though they are not well localized. Noble (1988) explained how the Harris corner detector estimates image curvature and characterized two-dimensional surface features such as junctions by the differential geometry of a simple facet model. Deriche and Giraudon (1993) used a new scale space based approach that combines useful properties from Laplacian and Beaudet’s cornerness measure to obtain good localization and solved the problem of trihedral vertex model. Kweon and Kanade (1994) found corners in the level curve and derived the following equation: max
Itt
ðx;yÞ on c In
¼ max K;
ð2Þ
where t and n represent the tangential and perpendicular direction, respectively. This means that local extrema in the curvature on the contour occur at the points where the local extrema in Itt =In occur. Chen et al. (1995) used two scales wavelet transformation for corner detection. But test for real scene is not presented. Robbins and Owens (1997) used Gabor filters and analyzed energy in phase domain to detect corners. They have good localization accuracy but suffer from many false alarms. Smith and Brady (1997) proposed a new low-level processing, SUSAN, which utilize simple masking operations instead of gradient convolution. In corner detection, they used a circular window, composed of 37 pixels, to obtain USAN area whose centroid is then used to suppress false positives. Finally, non-maximum suppression finds corners. Although the corner localization and noise robustness are good or better than other algorithms, it takes much computational cost in
S.C. Bae et al. / Pattern Recognition Letters 23 (2002) 1349–1360
obtaining USAN area and finding corners due to the large window size. Trajkovic and Hedley (1998) proposed minimum intensity change (MIC) algorithm. They defined corner as a point where the change of image intensity is high in all directions and found them as local maxima of minimum change of intensity. Although this method is very simple and fast, it sometimes gives false responses on the straight lines and is highly sensitive to noise. Parida and Geiger (1998) performed the modeling of junctions and their detection. They modeled a junction as wedge shaped regions having constant intensities and constant angular directions of each radial boundary. To detect corners, they minimized an energy function defined as the sum of two measures of distances, fitted function from the data and the gradient using L2 norm. They have good stability and noise robustness, but time complexity is high. Mokhtarian and Suomela (1998) defined corners as points where image edges have their maxima of absolute curvature and used curvature scale space (CSS) technique to improve localization and increase robustness to noise. To implement, they initially used a Canny edge detector to obtain edge contour, linked broken edges, and then defined the evolved version of the curve parameterized by the function of the arc length parameter u and scale r. They computed the curvature j given by
1351
3. Corner detection 3.1. COP The COP corner detector is composed of two oriented crosses as shown in Fig. 1. COP is developed from the ideas that a rotationally symmetric operator is optimal and two masks are widely used in the feature detection like Sobel, Gaussian, steering filter and so on. The left one responds strongly to 45° edges, and the right one to the horizontal and vertical edges. Fig. 2 illustrates these characteristics of each of the COP with simple images, respectively. Each response shows the inverted USAN area, which is the number of pixels dissimilar to the mask’s nucleus, and each of the COP shows peak values near the corners. Fig. 3 shows two typical step edges and responses of the COP. Considering each response of the COP, R2 produces a larger value than R1. Along the vertical edge, R2 is always larger than R1 and these phenomena also take place when horizontal edge exists. By the proposed algorithm, we discard case 1-3. For the image with )45° edge,
Fig. 1. Two crosses of the COP.
jðu; rÞ ¼
Xu ðu; rÞYuu ðu; rÞ Xuu ðu; rÞYu ðu; rÞ 2
2 1:5
ðXu ðu; rÞ þ Yu ðu; rÞ Þ
ð3Þ
at the highest scale rhigh to obtain corner candidates and finally tracked the corners to be the lowest scale to improve localization. This algorithm performs well in detection and is robust to noise, but time complexity is high. Recently, Ando (2000) attempted to analyze one-dimensional and two-dimensional features such as edges, ridges and corners in image using gradient covariance matrix. They proposed two measures, functions of two eigenvalues of that matrix, to categorize image field into three regions, uni-directionally varying region (UNIVAR), omni-directionally varying region (OMNIVAR) and non-varying region. Corner localization and noise robustness are good, but the false alarm rate is high.
Fig. 2. Response of the COP to the simple images: (a) horizontal and vertical edges; (b) 45° edges.
1352
S.C. Bae et al. / Pattern Recognition Letters 23 (2002) 1349–1360
Fig. 3. Typical step edge and responses of the COP.
R1 is larger than R2 except the discarding case 2-3, and this also holds in +45° edge case. The real scene includes many noisy and blurred edges as shown in Fig. 4. Edges in this case are similar to line edges. Therefore, we can also obtain single response in the actual line edge case, which does not happen in many edge finding algorithms using derivative convolution masks. Fig. 5 shows more general edge whose orientation changes. Excepting two upper cases by the proposed algorithm, case 5-3 always occurs when local multiple orientations exist at the same time.
Fig. 5. Typical edge where orientation changes and responses of the COP.
3.2. Probabilistic dominant direction According to the COP response, we can predefine the raw directions, d i ði ¼ 1; 2; . . . ; 15Þ as shown in Fig. 6. Directions of d 3 , d 6 , d 7 , d 8 , d 9 and d 10 have four similar types, such that, d 3 , the
combination of d 1 and d 2 , can have four similar geometries. Raw direction initially obtained in 3 3 window is so locally noisy that the following steps are
Fig. 4. Typical blurred edge and responses of the COP.
S.C. Bae et al. / Pattern Recognition Letters 23 (2002) 1349–1360
1353
Fig. 6. Pre-defined 15 raw directions.
needed to provide smoothing effect in real scene process. Firstly, we define each probability, pi of probabilistic dominant directions, pd i from the raw direction distribution as: pi ¼ wik nk ;
ð4Þ
where wik is the weight factor and nk the frequency, that is, the number of occurrences of d k , in a 3 3 window. It leads to the following matrix form: P ¼ WN ;
ð5Þ
Fig. 7. Four probabilistic dominant directions.
where 1 p1 Bp C B 2C P ¼ B C; @ p4 A 0
0
p5 4 0 2 0
3.3. Corner dominant direction and response function
1
0 0
2 2 0
0 2
0 1 1
1
4 2 0 0 0 4
0 0 0 2
0 0 2 0 2 0
2 0 2 1
2 1 1 1 1 2
1C C C; 0A
0 0 0 0 1 n1 C B B C C B C N ¼B B C: C B @ A n15
4 2
2 0 2
0 1
1 1 0
2
B0 B W ¼B @0 0
ð6Þ
Six corner types are defined as shown in Fig. 8, where each corner type has multiple corner dominant directions, dom i and dom j ði 6¼ j; i; j ¼ 1; 2; 4; 5Þ. ‘‘T’’,‘‘Y’’ and ‘‘X’’ multi-region junctions can be considered because they have at least two dominant directions. Corners are detected using the proposed algorithm and overlaid on a given simple image.
All the values of the matrix W are scaled to consider the effects of every raw direction on four dominant directions and the value 4 means 100% effect of d k on each pi . Probabilistic dominant directions, pd i are then defined as pd i ¼ max fpi g: i¼1;2;4;5
ð7Þ
Four probabilistic dominant directions, pd i are shown in Fig. 7, and it is possible to consider eight directions used in common image processing.
Fig. 8. Six corner types.
1354
S.C. Bae et al. / Pattern Recognition Letters 23 (2002) 1349–1360
dom i ¼ maxfGk g; 1;2
1;2
k ¼ 1; 2; 4; 5:
ð10Þ
The corner response function is defined as CRF ¼ Gi Gj ði 6¼ jÞ:
ð11Þ
The non-maximum suppression is needed because more than one point will have high CRF around a corner, and only the largest one becomes the corner point. (b)
(a)
3.4. Corner test
Fig. 9. Eight sector regions in a circular window and Tij : (a) eight sector regions in a circular window; (b) Tij .
Since corner is a point where orientation changes abruptly, it needs at least two dominant orientations. From the distribution of pd i in a given circular window, which has eight sector regions, the digital diameter 7 and consists of 37pixels in Fig. 9, we thus define Tij and Gi as Z Z Tij ¼ pi dXji ði ¼ 1; 2; 4; 5; j ¼ 1; 2Þ; ð8Þ j
When a corner candidate is in a circular search window as shown in Fig. 10, we consider some of 16 neighboring pixels in the circular boundary to obtain the angle of each dominant direction. If conditions are satisfied, the angle of corner dominant directions, h1 and h2 are obtained and then the corner is tested by comparing the included angle, jh1 h2 j to the given threshold, thh , which is geometrically defined value of 22.5°. 3.5. Algorithm
Xi
where Tij represents the probability of pd i in a sector, Xji , its area size is about 6 pixels. The rate of change of Tij in i-direction, Gi is also given by Gi ¼ absðTi1 Ti2 Þ;
i ¼ 1; 2; 4; 5;
ð9Þ
from which the corner dominant direction is determined by
The proposed corner detection algorithm is mainly divided into the following steps: Step 1. Calculate inverted USAN area using COP, Step 2. Obtain probabilistic dominant directions, Step 3. Classify corner candidates, Step 4. Test corner, Step 5. Find local maximum in a search window.
Fig. 10. Corner test in a circular window.
S.C. Bae et al. / Pattern Recognition Letters 23 (2002) 1349–1360
1355
In Step 1, the inverted USAN area is calculated using COP, and each output of the COP has the value ranging between 0 and 4. In Step 2, four probabilistic dominant directions are obtained based on probabilities of 15 pre-defined raw directions from COP response. In Step 3, corner candidates are classified to have multiple corner dominant directions according to six corner types. In Step 4, a simple corner test is applied to the corner candidates to discard false alarms. Finally, corners are identified by taking local maxima in the search window.
4. Results The results of testing the proposed corner algorithm are demonstrated and discussed. Sequential outputs of the algorithm are given, and simple images as well as smoothing and noisy ones are used to test the performance. Results are also compared with the outputs of four other corner detectors such as Plessy, Kitchen and Rosenfeld, SUSAN and CSS corner detectors. Finally, one example is added to test stability of the proposed corner finder, for given time series images. Our algorithm produces satisfactory results for all cases. As stated earlier, our algorithm works in several steps. The output in each intermediate step for a well-known house image is shown in Fig. 11. The initial inverted USAN areas are shown in Fig. 11(b). Approximately 10% of all image pixels are reported here for further processing. Corner candidates with multiple corner dominant directions
Fig. 11. Step by step process in corner detection algorithm: (a) input image; (b) inverted USAN area; (c) corner candidates; (d) detected corner.
are shown in Fig. 11(c). All these candidates are classified into six corner structures and they numbered about 1.5% of pixels. Finally, corners are declared in Fig. 11(d) through the corner test and non-maximum suppression, and they numbered about 0.1% of pixels. In order to test the localization accuracy of detected corners, We vary the radius of Gaussian smoothing as shown in Fig. 12. Real corner coordinates are given as fx; yg ¼ fð23; 16Þ; ð23; 43Þ; ð46; 17Þ; ð46; 43Þg:
Fig. 12. Outputs of COP corner detector for a given image with a varying radius of Gaussian smoothing: (a) radius ¼ 0; (b) radius ¼ 1; (c) radius ¼ 2; (d) radius ¼ 3.
1356
S.C. Bae et al. / Pattern Recognition Letters 23 (2002) 1349–1360
(a)
(b)
(c)
(d)
(e)
(f)
Fig. 13. Outputs of COP corner detector for a given image with blurring and two kinds of varying noises: (a) input image; (b) blurring; (c) uniform ðr ¼ 5Þ; (d) uniform ðr ¼ 10Þ; (e) Gaussian ðr ¼ 5Þ; (f) Gaussian ðr ¼ 10Þ.
(a)
(b)
(c)
(d)
Fig. 14. Outputs of COP corner detector for given images: (a) synthetic image; (b) character image; (c) block image; (d) panel image.
S.C. Bae et al. / Pattern Recognition Letters 23 (2002) 1349–1360
1357
(a)
(b)
(c)
(d)
(e)
(f)
Fig. 15. Outputs of corner detectors for a given block image: (a) input image; (b) Plessy; (c) K&R; (d) SUSAN; (e) CSS; (f) COP.
As Gaussian smoothing increases with fixed brightness threshold, obtained corner coordinates varies: fxa ; ya g ¼ fð23; 16Þ; ð23; 44Þ; ð46; 17Þ; ð46; 43Þg; fxb ; yb g ¼ fð24; 18Þ; ð24; 42Þ; ð45; 18Þ; ð45; 42Þg and fxc ; yc g ¼ fð24; 17Þ; ð23; 42Þ; ð46; 18Þ; ð46; 42Þg: Localization accuracy is satisfactory while some deviations exist. However, any corners are not detected in Fig. 12(d) where the size of Gaussian smoothing window is 7 7. The reason is that the distribution of brightness is too smooth to obtain inverted USAN areas, which can be obtained through the brightness dissimilarity. Noise test is also carried out for a simple one with varying uniform noise and Gaussian noise added to the image as shown in Fig. 13. In general, more false
alarms appear with increasing noise but the proposed algorithm is insensitive to noise. Fig. 14 shows the results of proposed corner detector for a simple synthetic image used in calibration, character image captured in lab environment, block image and panel image used in sub-assembly process of shipbuilding. While the proposed corner detector gives good performance with respect to detection, some multiple corners are declared in rounded corner caused by too much blurring in Fig. 14(b). Two corners are not detected due to no significant change of brightness in Fig. 14(c) as illustrated. Four well-known corner detectors are added to compare their performances with COP in Figs. 15–17. Three common images are used such as real block image with much texture and noise, house image with lots of small details and texture in the brick wall and lab image with a wide range of brightness. The performance varies from image to image, and also depends on the several parameter
1358
S.C. Bae et al. / Pattern Recognition Letters 23 (2002) 1349–1360
(a)
(b)
(c)
(d)
(e)
(f)
Fig. 16. Outputs of corner detectors for a given house image: (a) input image; (b) Plessy; (c) K&R; (d) SUSAN; (e) CSS; (f) COP.
selections. Therefore, we tried to obtain the best possible results for each detector and each detector is tested by varying parameter values to obtain the best results. In Fig. 15, CSS produces the best result and the other three corner detectors does not perform well with respect to detection. Although some false alarms take place due to the texture and noise, COP works well with respect to detection. In Fig. 16, CSS, SUSAN and COP perform better than Plessy, which suffers from sensitivity to noise. Kitchen and Rosenfeld has lower detection rate. Finally, the results on the lab image in Fig. 17 show that CSS and COP have better performance than the other corner detectors, which have lower detection rate. The stability of the proposed algorithm with fixed intensity threshold is tested using the time series images in Fig. 18. An image frame is processed to find corners and then lists of corners are matched to each other
using cross correlation to obtain motion flow. The result shows that the black vehicle has little motion, the background moves to the right and the ambulance has motion converging to the left even though there exist some false motion flow vectors. The mark ‘‘*’’, ‘‘ ’’ and ‘‘’’ denote the matched corners in three consecutive frames, respectively. We only use the brightness at the position of corners as an attribute in rough matching. The value of stability measure, that is the ratio of the number of matched corners to the number of corners in the first frame, is about 66.3% for the first two frames and 55.8% for three consecutive frames. All corner algorithms have at least one threshold. Although it is necessary for feature check, algorithms should be stable with threshold variation. If the threshold value is lower than some optimal threshold, false positives show up. In the opposite case, false negatives show up. Perfor-
S.C. Bae et al. / Pattern Recognition Letters 23 (2002) 1349–1360
1359
(a)
(b)
(c)
(d)
(e)
(f)
Fig. 17. Outputs of corner detectors for a given lab image: (a) input image; (b) Plessy; (c) K&R; (d) SUSAN; (e) CSS; (f) COP.
Fig. 18. Extracted motion flow from sequential images.
mance of the COP corner detector is not affected much by threshold variation. Due to non-derivative masking operations and the probabilistic approach, COP worked well in noisy test images. Although some false alarms show up near blunt corners, COP has good performance in detection and localization points of view. In Table 1, the performance comparison of all corner detectors is summarized with the detection rate, the false alarm rate and the average execution time. The detection rate is defined as the ratio of true positive (TP count) to the sum of TP and false negative (FN count) and the false alarm rate is defined as the ratio of the sum of false positive (FP count) and FN to the sum of TP, FP and FN. Each algorithm is executed on a Pentium PC (700 MHz). The execution time varies from image to image and also depends on the several parameters. COP and Kitchen and Rosenfeld detectors are faster than other detectors. The overall result shows that COP
1360
S.C. Bae et al. / Pattern Recognition Letters 23 (2002) 1349–1360
Table 1 The performance comparison of the each algorithm Algorithm
Plessy
K&R
SUSAN
CSS
COP
Detection rate (%) False alarm rate (%) Execution time (ms)
66.9 31.6 300–400
74.3 29.4 150–200
81.6 28.4 300–350
84.6 16.7 350–450
85.2 25.2 200–300
and CSS have better performance than other corner detectors.
5. Conclusion In this paper, a new corner detection method using the COP is presented, which provides useful information to extract low-level features due to its characteristics, preference for different edges and simple orientation determination. In order to obtain corner, probabilistic dominant directions and several corner types having two dominant directions are defined, and a simple corner test is applied to reduce false alarms. COP can handle other types of two-dimensional features such as X, Y and T multiple junctions because they also have at least two dominant directions. Experimental results indicate the satisfactory performance of the proposed algorithm in detection and stability. Small window size also lowers computational cost and increases localization accuracy. Simple orientation selection and noise robustness make it possible to produce effective and good performance.
Acknowledgements This research is supported by BK21 and POSCO.
References Ando, S., 2000. Image field categorization and edge/corner detection from gradient covariance. PAMI 22 (2), 179–190. Beaudet, P., 1978. Rotational invariant image operators. ICPR, 579–583. Chen, C., Lee, J., Sun, Y., 1995. Wavelet transformation for graylevel corner detection. Pattern Recognition 28 (6), 853–861. Deriche, R., Giraudon, G., 1993. A computational approach for corners and vertex detection. IJCV 10 (2), 101–124. Dreschler, L., Nagel, H., 1981. Volumetric model and 3D trajectory of a moving car derived from monocular TVframe sequence of a street scene. CVGIP 20 (3), 199–228. Harris, C., Stephens, M., 1988. A combined corner and edge detector. In: 4th Alvey Vision Conf., pp. 147–151. Kitchen, L., Rosenfeld, A., 1982. Gray-level corner detection. Pattern Recognition Lett. 1, 95–102. Kweon, I., Kanade, T., 1994. Extracting topographic terrain features from elevation maps. CVGIP 59 (2), 171–182. Mokhtarian, F., Suomela, R., 1998. Robust image corner detection through curvature scale space. PAMI 20 (12), 1376–1381. Moravec, H., 1977. Towards automatic visual obstacle avoidance. Proc. IJCAI, 584. Noble, A., 1988. Finding corners. Image Vision Comput. 6 (2), 121–128. Parida, L., Geiger, D., 1998. Junction: detection classification and reconstruction. PAMI 20 (7), 687–698. Robbins, B., Owens, R., 1997. 2D feature detection via local energy. Image Vision Comput. 15, 353–368. Smith, S., Brady, M., 1997. A new approach to low level image processing. IJCV 23 (1), 45–78. Trajkovic, M., Hedley, M., 1998. Fast corner detection. Image Vision Comput. 16, 75–87. Wang, H., Brady, M., 1995. Real-time corner detection algorithm for motion estimation. Image Vision Comput. 13 (9), 695–703.
COP: a new corner detector Sun Cheol Bae a
a,*
, In So Kweon b, Choong Don Yoo
a
Department of Mechanical Engineering, Korea Advanced Institute of Science and Technology, 373-1, Kusong-dong, Yusung-ku, Taejon 305-701, South Korea b Department of Electrical Engineering, Korea Advanced Institute of Science and Technology, 373-1, Kusong-dong, Yusung-ku, Taejon 305-701, South Korea Received 29 March 2001; received in revised form 9 November 2001
Abstract Most conventional derivative-based corner detectors have shortcomings such as missing junctions, poor localization, sensitivity to noise and high computational cost. This paper presents a new, simple and effective low level processing method to detect corners. In this corner detection algorithm, two oriented cross operators called crosses as oriented pair (COP) are used, which provide useful information to extract low-level features due to its characteristics, preference for edge with different direction and simple direction determination. Fast, accurate and noise-robust corner detection is accomplished with the COP. Ó 2002 Elsevier Science B.V. All rights reserved. Keywords: COP; USAN; Corner dominant direction
1. Introduction Corners play an important role in computer vision for many applications such as object recognition, motion analysis, scene analysis, stereo matching, etc. Much of the work on two-dimensional features have concentrated on corners, that is, features formed at boundaries between two significantly dissimilar image brightness regions, where the boundary curvature is sufficiently high. However, many other types of localized structure
*
Corresponding author. Tel.: +82-428693081. E-mail addresses: [email protected] (S.C. Bae), [email protected] (I.S. Kweon), [email protected] (C.D. Yoo).
arise, for example, ‘‘T’’,‘‘X’’ and ‘‘Y’’ junctions, and these multi-region junctions should be also considered. Corners are intrinsically second order properties of a surface. Therefore, most of the earliest methods of corner detection used convolution masks to approximate the first and second order derivatives of the image brightness function, but derivatives are sensitive to noise. They tend to suffer from poor localization, missing junctions, sensitivity to noise and high computational cost. There has been an abundance of work on different approaches to the detection of these features in images. Recently, Smith and Brady (1997) proposed a new low-level processing, SUSAN. They used only masking operations to obtain lowlevel features. In this paper, we also use the same concept, univalue segment assimilating nucleus
0167-8655/02/$ - see front matter Ó 2002 Elsevier Science B.V. All rights reserved. PII: S 0 1 6 7 - 8 6 5 5 ( 0 2 ) 0 0 0 8 3 - 1
1350
S.C. Bae et al. / Pattern Recognition Letters 23 (2002) 1349–1360
(USAN) and propose a new corner detection algorithm using two oriented cross operators called crosses as oriented pair (COP) to detect such features more easily and effectively. This paper begins with COP principle, and continues with a proposed corner detection algorithm, including results of testing algorithm in contrast to other wellknown corner detectors. 2. Literature review Two approaches are most widely used for corner detection. The first approach is to initially segment the image into regions, represent the object boundary with a chain code, and search for points having maximum curvature or perform a polygonal approximation on the chains and then search for the line segment intersections. The major drawback of this approach is that the performance of corner detection depends on the preceding image segmentation result. In order to overcome this, the second approach, which works directly on grey-level image, has been employed recently. This second approach mainly uses gradients and surface curvatures and essentially defines the following cornerness measure: C¼
Ixx Iy2 þ Iyy Ix2 2Ixy Ix Iy : ðIx2 þ Iy2 Þ
ð1Þ
Points are then declared corners if the value of cornerness measure meets some threshold conditions. One of such corner detectors is that proposed by Dreschler and Nagel (1981), who defined corners to be points lying between extrema of Gaussian curvature. Kitchen and Rosenfeld (1982) used a local quadratic surface fit to find corners. These methods also suffer from poor localization, missing junctions and sensitivity to noise. Wang and Brady (1995) developed a curvature based method. They filtered the image by 3 3 Sobel operators to obtain the edge strength and direction, and derived the equation approximating the total curvature. However, this algorithm also suffers from sensitivity to noise, and setting two thresholds is very difficult. Moravec (1977) developed the idea of using ‘‘points of interest’’, which
are defined as occurring when there are large intensity variations in every direction. But the response is noisy and anisotropic. Beaudet (1978) used a rotationally invariant DET operator. This method found saddle points in the image brightness surface by calculating Gaussian curvature, that is, the product of the two principle curvatures. Harris and Stephens (1988) introduced a corner operator by modifying the Moravec’s interest operator, using only the first order derivatives to approximate the second derivatives. Their method produces consistent corner responses though they are not well localized. Noble (1988) explained how the Harris corner detector estimates image curvature and characterized two-dimensional surface features such as junctions by the differential geometry of a simple facet model. Deriche and Giraudon (1993) used a new scale space based approach that combines useful properties from Laplacian and Beaudet’s cornerness measure to obtain good localization and solved the problem of trihedral vertex model. Kweon and Kanade (1994) found corners in the level curve and derived the following equation: max
Itt
ðx;yÞ on c In
¼ max K;
ð2Þ
where t and n represent the tangential and perpendicular direction, respectively. This means that local extrema in the curvature on the contour occur at the points where the local extrema in Itt =In occur. Chen et al. (1995) used two scales wavelet transformation for corner detection. But test for real scene is not presented. Robbins and Owens (1997) used Gabor filters and analyzed energy in phase domain to detect corners. They have good localization accuracy but suffer from many false alarms. Smith and Brady (1997) proposed a new low-level processing, SUSAN, which utilize simple masking operations instead of gradient convolution. In corner detection, they used a circular window, composed of 37 pixels, to obtain USAN area whose centroid is then used to suppress false positives. Finally, non-maximum suppression finds corners. Although the corner localization and noise robustness are good or better than other algorithms, it takes much computational cost in
S.C. Bae et al. / Pattern Recognition Letters 23 (2002) 1349–1360
obtaining USAN area and finding corners due to the large window size. Trajkovic and Hedley (1998) proposed minimum intensity change (MIC) algorithm. They defined corner as a point where the change of image intensity is high in all directions and found them as local maxima of minimum change of intensity. Although this method is very simple and fast, it sometimes gives false responses on the straight lines and is highly sensitive to noise. Parida and Geiger (1998) performed the modeling of junctions and their detection. They modeled a junction as wedge shaped regions having constant intensities and constant angular directions of each radial boundary. To detect corners, they minimized an energy function defined as the sum of two measures of distances, fitted function from the data and the gradient using L2 norm. They have good stability and noise robustness, but time complexity is high. Mokhtarian and Suomela (1998) defined corners as points where image edges have their maxima of absolute curvature and used curvature scale space (CSS) technique to improve localization and increase robustness to noise. To implement, they initially used a Canny edge detector to obtain edge contour, linked broken edges, and then defined the evolved version of the curve parameterized by the function of the arc length parameter u and scale r. They computed the curvature j given by
1351
3. Corner detection 3.1. COP The COP corner detector is composed of two oriented crosses as shown in Fig. 1. COP is developed from the ideas that a rotationally symmetric operator is optimal and two masks are widely used in the feature detection like Sobel, Gaussian, steering filter and so on. The left one responds strongly to 45° edges, and the right one to the horizontal and vertical edges. Fig. 2 illustrates these characteristics of each of the COP with simple images, respectively. Each response shows the inverted USAN area, which is the number of pixels dissimilar to the mask’s nucleus, and each of the COP shows peak values near the corners. Fig. 3 shows two typical step edges and responses of the COP. Considering each response of the COP, R2 produces a larger value than R1. Along the vertical edge, R2 is always larger than R1 and these phenomena also take place when horizontal edge exists. By the proposed algorithm, we discard case 1-3. For the image with )45° edge,
Fig. 1. Two crosses of the COP.
jðu; rÞ ¼
Xu ðu; rÞYuu ðu; rÞ Xuu ðu; rÞYu ðu; rÞ 2
2 1:5
ðXu ðu; rÞ þ Yu ðu; rÞ Þ
ð3Þ
at the highest scale rhigh to obtain corner candidates and finally tracked the corners to be the lowest scale to improve localization. This algorithm performs well in detection and is robust to noise, but time complexity is high. Recently, Ando (2000) attempted to analyze one-dimensional and two-dimensional features such as edges, ridges and corners in image using gradient covariance matrix. They proposed two measures, functions of two eigenvalues of that matrix, to categorize image field into three regions, uni-directionally varying region (UNIVAR), omni-directionally varying region (OMNIVAR) and non-varying region. Corner localization and noise robustness are good, but the false alarm rate is high.
Fig. 2. Response of the COP to the simple images: (a) horizontal and vertical edges; (b) 45° edges.
1352
S.C. Bae et al. / Pattern Recognition Letters 23 (2002) 1349–1360
Fig. 3. Typical step edge and responses of the COP.
R1 is larger than R2 except the discarding case 2-3, and this also holds in +45° edge case. The real scene includes many noisy and blurred edges as shown in Fig. 4. Edges in this case are similar to line edges. Therefore, we can also obtain single response in the actual line edge case, which does not happen in many edge finding algorithms using derivative convolution masks. Fig. 5 shows more general edge whose orientation changes. Excepting two upper cases by the proposed algorithm, case 5-3 always occurs when local multiple orientations exist at the same time.
Fig. 5. Typical edge where orientation changes and responses of the COP.
3.2. Probabilistic dominant direction According to the COP response, we can predefine the raw directions, d i ði ¼ 1; 2; . . . ; 15Þ as shown in Fig. 6. Directions of d 3 , d 6 , d 7 , d 8 , d 9 and d 10 have four similar types, such that, d 3 , the
combination of d 1 and d 2 , can have four similar geometries. Raw direction initially obtained in 3 3 window is so locally noisy that the following steps are
Fig. 4. Typical blurred edge and responses of the COP.
S.C. Bae et al. / Pattern Recognition Letters 23 (2002) 1349–1360
1353
Fig. 6. Pre-defined 15 raw directions.
needed to provide smoothing effect in real scene process. Firstly, we define each probability, pi of probabilistic dominant directions, pd i from the raw direction distribution as: pi ¼ wik nk ;
ð4Þ
where wik is the weight factor and nk the frequency, that is, the number of occurrences of d k , in a 3 3 window. It leads to the following matrix form: P ¼ WN ;
ð5Þ
Fig. 7. Four probabilistic dominant directions.
where 1 p1 Bp C B 2C P ¼ B C; @ p4 A 0
0
p5 4 0 2 0
3.3. Corner dominant direction and response function
1
0 0
2 2 0
0 2
0 1 1
1
4 2 0 0 0 4
0 0 0 2
0 0 2 0 2 0
2 0 2 1
2 1 1 1 1 2
1C C C; 0A
0 0 0 0 1 n1 C B B C C B C N ¼B B C: C B @ A n15
4 2
2 0 2
0 1
1 1 0
2
B0 B W ¼B @0 0
ð6Þ
Six corner types are defined as shown in Fig. 8, where each corner type has multiple corner dominant directions, dom i and dom j ði 6¼ j; i; j ¼ 1; 2; 4; 5Þ. ‘‘T’’,‘‘Y’’ and ‘‘X’’ multi-region junctions can be considered because they have at least two dominant directions. Corners are detected using the proposed algorithm and overlaid on a given simple image.
All the values of the matrix W are scaled to consider the effects of every raw direction on four dominant directions and the value 4 means 100% effect of d k on each pi . Probabilistic dominant directions, pd i are then defined as pd i ¼ max fpi g: i¼1;2;4;5
ð7Þ
Four probabilistic dominant directions, pd i are shown in Fig. 7, and it is possible to consider eight directions used in common image processing.
Fig. 8. Six corner types.
1354
S.C. Bae et al. / Pattern Recognition Letters 23 (2002) 1349–1360
dom i ¼ maxfGk g; 1;2
1;2
k ¼ 1; 2; 4; 5:
ð10Þ
The corner response function is defined as CRF ¼ Gi Gj ði 6¼ jÞ:
ð11Þ
The non-maximum suppression is needed because more than one point will have high CRF around a corner, and only the largest one becomes the corner point. (b)
(a)
3.4. Corner test
Fig. 9. Eight sector regions in a circular window and Tij : (a) eight sector regions in a circular window; (b) Tij .
Since corner is a point where orientation changes abruptly, it needs at least two dominant orientations. From the distribution of pd i in a given circular window, which has eight sector regions, the digital diameter 7 and consists of 37pixels in Fig. 9, we thus define Tij and Gi as Z Z Tij ¼ pi dXji ði ¼ 1; 2; 4; 5; j ¼ 1; 2Þ; ð8Þ j
When a corner candidate is in a circular search window as shown in Fig. 10, we consider some of 16 neighboring pixels in the circular boundary to obtain the angle of each dominant direction. If conditions are satisfied, the angle of corner dominant directions, h1 and h2 are obtained and then the corner is tested by comparing the included angle, jh1 h2 j to the given threshold, thh , which is geometrically defined value of 22.5°. 3.5. Algorithm
Xi
where Tij represents the probability of pd i in a sector, Xji , its area size is about 6 pixels. The rate of change of Tij in i-direction, Gi is also given by Gi ¼ absðTi1 Ti2 Þ;
i ¼ 1; 2; 4; 5;
ð9Þ
from which the corner dominant direction is determined by
The proposed corner detection algorithm is mainly divided into the following steps: Step 1. Calculate inverted USAN area using COP, Step 2. Obtain probabilistic dominant directions, Step 3. Classify corner candidates, Step 4. Test corner, Step 5. Find local maximum in a search window.
Fig. 10. Corner test in a circular window.
S.C. Bae et al. / Pattern Recognition Letters 23 (2002) 1349–1360
1355
In Step 1, the inverted USAN area is calculated using COP, and each output of the COP has the value ranging between 0 and 4. In Step 2, four probabilistic dominant directions are obtained based on probabilities of 15 pre-defined raw directions from COP response. In Step 3, corner candidates are classified to have multiple corner dominant directions according to six corner types. In Step 4, a simple corner test is applied to the corner candidates to discard false alarms. Finally, corners are identified by taking local maxima in the search window.
4. Results The results of testing the proposed corner algorithm are demonstrated and discussed. Sequential outputs of the algorithm are given, and simple images as well as smoothing and noisy ones are used to test the performance. Results are also compared with the outputs of four other corner detectors such as Plessy, Kitchen and Rosenfeld, SUSAN and CSS corner detectors. Finally, one example is added to test stability of the proposed corner finder, for given time series images. Our algorithm produces satisfactory results for all cases. As stated earlier, our algorithm works in several steps. The output in each intermediate step for a well-known house image is shown in Fig. 11. The initial inverted USAN areas are shown in Fig. 11(b). Approximately 10% of all image pixels are reported here for further processing. Corner candidates with multiple corner dominant directions
Fig. 11. Step by step process in corner detection algorithm: (a) input image; (b) inverted USAN area; (c) corner candidates; (d) detected corner.
are shown in Fig. 11(c). All these candidates are classified into six corner structures and they numbered about 1.5% of pixels. Finally, corners are declared in Fig. 11(d) through the corner test and non-maximum suppression, and they numbered about 0.1% of pixels. In order to test the localization accuracy of detected corners, We vary the radius of Gaussian smoothing as shown in Fig. 12. Real corner coordinates are given as fx; yg ¼ fð23; 16Þ; ð23; 43Þ; ð46; 17Þ; ð46; 43Þg:
Fig. 12. Outputs of COP corner detector for a given image with a varying radius of Gaussian smoothing: (a) radius ¼ 0; (b) radius ¼ 1; (c) radius ¼ 2; (d) radius ¼ 3.
1356
S.C. Bae et al. / Pattern Recognition Letters 23 (2002) 1349–1360
(a)
(b)
(c)
(d)
(e)
(f)
Fig. 13. Outputs of COP corner detector for a given image with blurring and two kinds of varying noises: (a) input image; (b) blurring; (c) uniform ðr ¼ 5Þ; (d) uniform ðr ¼ 10Þ; (e) Gaussian ðr ¼ 5Þ; (f) Gaussian ðr ¼ 10Þ.
(a)
(b)
(c)
(d)
Fig. 14. Outputs of COP corner detector for given images: (a) synthetic image; (b) character image; (c) block image; (d) panel image.
S.C. Bae et al. / Pattern Recognition Letters 23 (2002) 1349–1360
1357
(a)
(b)
(c)
(d)
(e)
(f)
Fig. 15. Outputs of corner detectors for a given block image: (a) input image; (b) Plessy; (c) K&R; (d) SUSAN; (e) CSS; (f) COP.
As Gaussian smoothing increases with fixed brightness threshold, obtained corner coordinates varies: fxa ; ya g ¼ fð23; 16Þ; ð23; 44Þ; ð46; 17Þ; ð46; 43Þg; fxb ; yb g ¼ fð24; 18Þ; ð24; 42Þ; ð45; 18Þ; ð45; 42Þg and fxc ; yc g ¼ fð24; 17Þ; ð23; 42Þ; ð46; 18Þ; ð46; 42Þg: Localization accuracy is satisfactory while some deviations exist. However, any corners are not detected in Fig. 12(d) where the size of Gaussian smoothing window is 7 7. The reason is that the distribution of brightness is too smooth to obtain inverted USAN areas, which can be obtained through the brightness dissimilarity. Noise test is also carried out for a simple one with varying uniform noise and Gaussian noise added to the image as shown in Fig. 13. In general, more false
alarms appear with increasing noise but the proposed algorithm is insensitive to noise. Fig. 14 shows the results of proposed corner detector for a simple synthetic image used in calibration, character image captured in lab environment, block image and panel image used in sub-assembly process of shipbuilding. While the proposed corner detector gives good performance with respect to detection, some multiple corners are declared in rounded corner caused by too much blurring in Fig. 14(b). Two corners are not detected due to no significant change of brightness in Fig. 14(c) as illustrated. Four well-known corner detectors are added to compare their performances with COP in Figs. 15–17. Three common images are used such as real block image with much texture and noise, house image with lots of small details and texture in the brick wall and lab image with a wide range of brightness. The performance varies from image to image, and also depends on the several parameter
1358
S.C. Bae et al. / Pattern Recognition Letters 23 (2002) 1349–1360
(a)
(b)
(c)
(d)
(e)
(f)
Fig. 16. Outputs of corner detectors for a given house image: (a) input image; (b) Plessy; (c) K&R; (d) SUSAN; (e) CSS; (f) COP.
selections. Therefore, we tried to obtain the best possible results for each detector and each detector is tested by varying parameter values to obtain the best results. In Fig. 15, CSS produces the best result and the other three corner detectors does not perform well with respect to detection. Although some false alarms take place due to the texture and noise, COP works well with respect to detection. In Fig. 16, CSS, SUSAN and COP perform better than Plessy, which suffers from sensitivity to noise. Kitchen and Rosenfeld has lower detection rate. Finally, the results on the lab image in Fig. 17 show that CSS and COP have better performance than the other corner detectors, which have lower detection rate. The stability of the proposed algorithm with fixed intensity threshold is tested using the time series images in Fig. 18. An image frame is processed to find corners and then lists of corners are matched to each other
using cross correlation to obtain motion flow. The result shows that the black vehicle has little motion, the background moves to the right and the ambulance has motion converging to the left even though there exist some false motion flow vectors. The mark ‘‘*’’, ‘‘ ’’ and ‘‘’’ denote the matched corners in three consecutive frames, respectively. We only use the brightness at the position of corners as an attribute in rough matching. The value of stability measure, that is the ratio of the number of matched corners to the number of corners in the first frame, is about 66.3% for the first two frames and 55.8% for three consecutive frames. All corner algorithms have at least one threshold. Although it is necessary for feature check, algorithms should be stable with threshold variation. If the threshold value is lower than some optimal threshold, false positives show up. In the opposite case, false negatives show up. Perfor-
S.C. Bae et al. / Pattern Recognition Letters 23 (2002) 1349–1360
1359
(a)
(b)
(c)
(d)
(e)
(f)
Fig. 17. Outputs of corner detectors for a given lab image: (a) input image; (b) Plessy; (c) K&R; (d) SUSAN; (e) CSS; (f) COP.
Fig. 18. Extracted motion flow from sequential images.
mance of the COP corner detector is not affected much by threshold variation. Due to non-derivative masking operations and the probabilistic approach, COP worked well in noisy test images. Although some false alarms show up near blunt corners, COP has good performance in detection and localization points of view. In Table 1, the performance comparison of all corner detectors is summarized with the detection rate, the false alarm rate and the average execution time. The detection rate is defined as the ratio of true positive (TP count) to the sum of TP and false negative (FN count) and the false alarm rate is defined as the ratio of the sum of false positive (FP count) and FN to the sum of TP, FP and FN. Each algorithm is executed on a Pentium PC (700 MHz). The execution time varies from image to image and also depends on the several parameters. COP and Kitchen and Rosenfeld detectors are faster than other detectors. The overall result shows that COP
1360
S.C. Bae et al. / Pattern Recognition Letters 23 (2002) 1349–1360
Table 1 The performance comparison of the each algorithm Algorithm
Plessy
K&R
SUSAN
CSS
COP
Detection rate (%) False alarm rate (%) Execution time (ms)
66.9 31.6 300–400
74.3 29.4 150–200
81.6 28.4 300–350
84.6 16.7 350–450
85.2 25.2 200–300
and CSS have better performance than other corner detectors.
5. Conclusion In this paper, a new corner detection method using the COP is presented, which provides useful information to extract low-level features due to its characteristics, preference for different edges and simple orientation determination. In order to obtain corner, probabilistic dominant directions and several corner types having two dominant directions are defined, and a simple corner test is applied to reduce false alarms. COP can handle other types of two-dimensional features such as X, Y and T multiple junctions because they also have at least two dominant directions. Experimental results indicate the satisfactory performance of the proposed algorithm in detection and stability. Small window size also lowers computational cost and increases localization accuracy. Simple orientation selection and noise robustness make it possible to produce effective and good performance.
Acknowledgements This research is supported by BK21 and POSCO.
References Ando, S., 2000. Image field categorization and edge/corner detection from gradient covariance. PAMI 22 (2), 179–190. Beaudet, P., 1978. Rotational invariant image operators. ICPR, 579–583. Chen, C., Lee, J., Sun, Y., 1995. Wavelet transformation for graylevel corner detection. Pattern Recognition 28 (6), 853–861. Deriche, R., Giraudon, G., 1993. A computational approach for corners and vertex detection. IJCV 10 (2), 101–124. Dreschler, L., Nagel, H., 1981. Volumetric model and 3D trajectory of a moving car derived from monocular TVframe sequence of a street scene. CVGIP 20 (3), 199–228. Harris, C., Stephens, M., 1988. A combined corner and edge detector. In: 4th Alvey Vision Conf., pp. 147–151. Kitchen, L., Rosenfeld, A., 1982. Gray-level corner detection. Pattern Recognition Lett. 1, 95–102. Kweon, I., Kanade, T., 1994. Extracting topographic terrain features from elevation maps. CVGIP 59 (2), 171–182. Mokhtarian, F., Suomela, R., 1998. Robust image corner detection through curvature scale space. PAMI 20 (12), 1376–1381. Moravec, H., 1977. Towards automatic visual obstacle avoidance. Proc. IJCAI, 584. Noble, A., 1988. Finding corners. Image Vision Comput. 6 (2), 121–128. Parida, L., Geiger, D., 1998. Junction: detection classification and reconstruction. PAMI 20 (7), 687–698. Robbins, B., Owens, R., 1997. 2D feature detection via local energy. Image Vision Comput. 15, 353–368. Smith, S., Brady, M., 1997. A new approach to low level image processing. IJCV 23 (1), 45–78. Trajkovic, M., Hedley, M., 1998. Fast corner detection. Image Vision Comput. 16, 75–87. Wang, H., Brady, M., 1995. Real-time corner detection algorithm for motion estimation. Image Vision Comput. 13 (9), 695–703.