- Email: [email protected]
Multi scale Harris corner detector based on Differential Morphological Decomposition
Pattern Recognition Letters 32 (2011) 1714–1719
Contents lists available at SciVerse ScienceDirect
Pattern Recognition Letters journal homepage: www.elsevier.com/locate/patrec
Multi scale Harris corner detector based on Differential Morphological Decomposition L. Gueguen ⇑, M. Pesaresi Global Security and Crisis Management Unit, IPSC, Joint Research Center-European Commission, Via E. Fermi, 2749, I-21027 Ispra (Va), Italy
a r t i c l e
i n f o
Article history: Received 24 September 2010 Available online 6 August 2011 Communicated by S. Aksoy Keywords: Corner detector Mathematical morphology Opening by reconstruction Scale space
a b s t r a c t In this paper, a novel method for multi scale corner analysis and detection is presented. First, stateof-the-art Harris–Laplace corner detector is reminded, which benefits from linear scale-space analysis. Secondly, a non-linear scale-space transform, namely Differential Morphological Decomposition, is described. This multi-scale transform is used jointly with the Harris corner indicator to build a new multi scale corner detector. Both corner detectors are visually assessed on synthetic and satellite images, highlighting the advantages of such a method. 2011 Elsevier B.V. All rights reserved.
1. Introduction Extraction of salient points from images has been demonstrated to be an efficient paradigm for matching and recognition, as it provides robustness to some transforms. Various solutions for extracting salient points have been developed over the past few years, such that they satisfy invariance properties. One amongst them is the Harris detector (Harris and Stephens, 1988) which identifies corners of image and was designed to be invariant to rotations. Then, the Harris–Laplace detector was proposed in (Mikolajczyk and Schmid, 2004), such that it is invariant to scale and affine transforms. These methods are reviewed hereafter. When the scale of a salient image point is known, the Harris detector is an efficient detector. However, other strategies have to be considered, when the scale is unknown. This issue is solved in (Mikolajczyk and Schmid, 2004; Shi et al., 2009), by computing a corner indicator at consecutive scales, and then selecting the best scale locally. As output, positions and scales of detected corners are obtained, and can be used for matching or analyzing images. Such an approach uses a low pass filtering of the original signal in order to detect large scale corners, and this simplification process destroys edges delineating corners. It results in bad detection of corners in large scales. In this paper, we propose to use a non-linear scale-space transform, namely a Differential Morphological Decomposition (DMD), in synergy with the Harris indicator. This non-linear transform preserves edges, while simplifying the input monochromatic image. In
⇑ Corresponding author. E-mail address: [email protected] (L. Gueguen). 0167-8655/$ - see front matter 2011 Elsevier B.V. All rights reserved. doi:10.1016/j.patrec.2011.07.021
addition, a DMD analyses an image by scale, such that the choice of the Harris indicator integration scale becomes straightforward. Combining both the advantages yields a multi-scale corner detector which is localized in scale and space. The paper is organized as follows. Section 2 reviews the single scale Harris corner indicator, and its extended multi-scale version: the Harris–Laplace corner detector. In Section 3, a new multi-scale corner detector is presented. It includes the description of Differential Morphological Decompositions, and their use with the Harris corner indicator. Then, the Harris–Laplace and the proposed detectors are applied to synthetic and satellite images in Section 4. Finally, Section 5 concludes. 2. Harris corner detector In this section, we remind the Harris and the Harris–Laplace corner indicators and detectors. The presentation of such well known technique is intended to make an analogy with the proposed idea. 2.1. Single scale Harris corner indicator The Harris corner indicator (Harris and Stephens, 1988) is a well established technique making use of several linear filtering of an image. Given an image f, we denote by L(p, r) = g(p, r) w f(p), the low pass filtering by a Gaussian function g(p, r) of scale r, where p is the position in the image grid. Let Lx(p, rD), Ly(p, rD) be the respective horizontal and vertical image derivatives at position p, obtained with Gaussian filter of local scale rD (the differentiation scale). Then these intermediate results are composed to obtain
L. Gueguen, M. Pesaresi / Pattern Recognition Letters 32 (2011) 1714–1719
three images L2x ðp; rD Þ; L2y ðp; rD Þ; Lx Ly ðp; rD Þ, where the multiplicative operation is performed pixel wise. These images are then filtered by a Gaussian function g(p, rI) of scale rI (the integration scale):
"
lðp; rI ; rD Þ ¼ r2D gðrI ÞH
L2x ðp;
rD Þ Lx Ly ðp; rD Þ ; Lx Ly ðp; rD Þ L2y ðp; rD Þ
Hðp; rI ; rD Þ ¼ detðlðp; rI ; rD ÞÞ j tr2 ðlðp; rI ; rD ÞÞ;
ð1Þ
ð2Þ
where j is a trade-off scalar, experimentally lying in the interval [0.04, 0.15]. High value of the Harris measurement indicates the presence of a corner of scale rI at position p in the image f. As a remark, the choice of the integration scale rI is critical, as a small scale results in missing the corners of large objects, while choosing a large scale results in missing small object corners. 2.2. Harris–Laplace corner detector As mentioned previously, the integration scale used for performing corner detection is crucial. To circumvent this issue, a multi-scale Harris corner detection was proposed in (Mikolajczyk and Schmid, 2004), by benefiting of a linear scale space decomposition. In such a paradigm, the Harris corner indicator is applied at successive integration scales, before selecting locally the characteristic scales of image blobs. In (Lindeberg, 1998), the Laplacian-of-Gaussian (LoG) is used, as a scale space representation, for selecting the image blob characteristic scales, and the scale normalized Laplacian operator is defined by:
LoGðp; rN Þ ¼ r
DMD is a scale space decomposition, the selection of the integration scale of the Harris corner indicator is straightforward. 3.1. Differential Morphological Decomposition
#
where w is the convolution operator. Finally, the single scale Harris corner indicator H(p, rI, rD) is given by:
2 N ðLxx ðp;
1715
rN Þ þ Lyy ðp; rN ÞÞ;
ð3Þ
where Lxx, Lyy are the second image derivatives in the horizontal and vertical directions, respectively. The characteristic scale at some position p is defined by the scale rN which hits the maximum of jLoG(p, rN)j. In (Mikolajczyk and Schmid, 2004), the authors propose to combine the Harris corner indicator and the scale normalized Laplacian operator in order to detect the salient positions in the scale space grid. First, a set of increasing analysis scales is chosen: r1 < r2 < < rn, enabling them the computation of a multi scale n stack of Harris indicators pfHðp; ffiffi rk ; crk Þgk¼1 , where c is a scalar ly ing in the range 1=2; 1= ð2Þ . For each scale rk, the local maxima of the Harris indicator H(p, rk, crk) are computed. Secondly, these spatial locations are analyzed through the LoG operator, in order to discard the ones which are not salient in the scale dimension:
^; rÞg ¼ argmaxlocalp Hðp; r; crÞ; fðp
ð4Þ
r^ ¼ argmaxr jLoGðp^; crÞj:
ð5Þ
^ which do no hit a scale maximum are disThe spatial locations p carded. The result of the Harris–Laplace corner detector is the set ^; r ^ Þg. of retained scale space positions fðp
This section describes the Differential Morphological Decomposition (DMD) in the context of multiresolution pyramid transform. Then, a specific DMD is described, which is based on the opening or closing by reconstruction operators. Differential Morphological Decompositions (DMD) belong to the family of multiresolution pyramid transforms, which analyze images by reducing or coarsing information at each pyramid level. In such a scheme, the removed information, which is the difference between the original and the coarse images, is stored in a detail image, which compounds the differential decomposition. In the following, the complete lattice L of discrete images is considered and is defined over the grid Z2 . A pyramid scheme analyzes sequentially an image reducing the information at each step (Goutsias and Heijmans, 1998). Let U"j : Lj ! Ljþ1 be the analysis operator at pyramid level j, and let U#j : Ljþ1 ! Lj be the corresponding synthesis operator. The successive complete lattices Lj are defined as the images of L by the successive analysis operators U"j , and they follow the inclusion relation Lj Ljþ1 . As information is to be reduced at each level, a pyramid condition is introduced in (Goutsias and Heijmans, 1998) and it is defined by U"j U#j ðf Þ ¼ f , 8f 2 Ljþ1 . The approximation at level j is ^f ¼ U#j U"j ðf Þ where f 2 Lj , and the detail image at level j is obtained by subtraction: g ¼ f ^f . Therefore, a differential decomposition of an image f can be obtained by the following recursive scheme:
8 f ¼ f 2 L; > < 0 fjþ1 ¼ U"j ðfj Þ 2 Ljþ1 ; > : g j ¼ fj U#j ðfjþ1 Þ:
ð6Þ
The differential decomposition of f is graphically represented in Fig. 1 and it is the compound of the detail images plus the last analysis result: {g0, . . . , gj;fj+1}. If the pyramid condition is satisfied, the decomposition enables a perfect reconstruction of the original image, meaning that all the information is kept during the analysis. A couple of operators (U, W) constitutes an adjunction of L if 8f ; g 2 L; Uðf Þ 6 g () f 6 WðgÞ, and these operators verify the pyramid condition (Goutsias and Heijmans, 1998). For example, (U, Id) is an adjunction for any morphological filter U, which follows the property of idempotence, i.e. U2 = U, and increasingness, i.e. "f, g, f 6 g ) U(f) 6 U(g). Therefore, any morphological filter can be used to define a Differential Morphological Decomposition. To satisfy the property of information decreasingness, series of morphological openings 6 Uj+1 6 Uj 6 with increasing size Structuring Elements (SE) can be considered. This decomposition is commonly called granulometry (Maragos, 1989), and is a non linear scale space representation. However, the shape of the structuring element impacts mainly the shape of structures retained in the detail images.
3. Differential Morphological Decomposition based corner detection In this section, a non linear scale-space decomposition is described thanks to morphological filtering by reconstruction (Soille, 2003). Then, a new method for indicating corners is proposed, taking benefit of the Differential Morphological Decomposition by reconstruction (DMD). Unlike linear scale-space decomposition, DMD preserve edges in higher scales, enabling a more robust detection of corners with the Harris indicator. In addition, as
Fig. 1. The Differential Morphological Decomposition is depicted for two pyramid levels, resulting in the representation g0, g1; f2. Thanks to the pyramid condition, the original signal f can be perfectly reconstructed.
1716
L. Gueguen, M. Pesaresi / Pattern Recognition Letters 32 (2011) 1714–1719
In (Pesaresi and Benediktsson, 2001; Gueguen et al., 2010), the use of morphological filters by reconstruction with increasing radius was proposed to circumvent this effect. This type of filtering is part of the family of connected operators which preserve edges while decreasing image structure contrasts (Soille, 2003). Opening or closing by reconstruction are morphological connected filters, which remove or threshold the local maxima or minima which cannot contain the structuring element, while keeping the rest of the image. The opening by reconstruction is defined thanks to the reconstruction by dilation Vincent (1993), which is expressed between two images respecting the following inequality f 6 g:
dgC ðf Þ ¼ dC ðf Þ ^ g; Rdg
¼
ðdgC Þi ;
9i P 1;
ð7Þ g i g iþ1 dC ¼ dC ;
ð8Þ
where dC is the dilation by the C-connectivity SE (the 4- or 8-connectivity), and ^ is the pointwise minimum operator. Then, the opening by reconstruction with the SE B is defined by:
cB ðf Þ ¼ Rdf ðB ðf ÞÞ;
ð9Þ
where B is the erosion by the SE B. A dual definition holds for expressing the closing by reconstruction. Hereafter, series of disk-shape SE Bj Bj+1 with increasing radius < rj < rj+1 < are considered to define a DMD by reconstruction from the respective series of morphological openings by reconstruction 6 cBj 6 cBjþ1 6 . The detailed image gj contains the image bright structures which can contain the disk Bj1, but cannot contain the successive one Bj. Therefore, the r þr average radius sj ¼ j12 j of both SE Bj1 and Bj can be understood as the characteristic scale of the structures contained in gj. Because of edges preservation and image structure scale analysis, the DMD based on openings/closings by reconstruction is an appealing scale-space representation for detecting corners. 3.2. DMD based Harris corner detector In this section, we define a new corner detector using the DMD by reconstruction {g0, . . . , gj; fj+1} of an image f, associated to the increasing scales < sj < sj+1 < . Such corner detector, as the Harris–Laplace indicator, benefits from the scale-space representation for selecting the integration scale ri of the Harris indicator expressed by (2). At scale sj, only image structures of this scale are present in the detail image gj. Since the noise is spatially unstructured, it remains in the first decomposition level g0. The upper levels being almost free of noise and preserving edges, a low pass filtering before the estimation of derivatives is unnecessary. Thus, the Harris corner indicator can be applied directly to the detail image gj, "j > 0. Let Hj(p, rI, 0) be the Harris detector applied to the detail image gj, with r þr an integration scale rI. Given the characteristic scale sj ¼ j12 j of structures in the detail image gj, the choice of the Gaussian filter scale becomes straightforward, and its optimal value is given by rI = sj/2. A stack of corners indicators {Hj(p, sj/2, 0)}j>0 is obtained by applying the Harris indicator to each detail image, except g0 containing the noise part. The DMD by reconstruction based Harris indicator is localized in space and scale by construction. Therefore, the DMD based corner detector consists in identifying the positions of the scale-space which reach a local maximum of {Hj(p, sj/2, 0)}j>0:
^; ^sÞg ¼ argmaxlocalp;s Hi ðp; sj =2; 0Þ: fðp j
ð10Þ
Due to the image acquisition limitations, the same image structure appears sometimes in contiguous detail images. Therefore, allowing to take the maximum scales enables to eliminate duplicate corners in contiguous scales.
Fig. 2. Synthetic image representing a distorted chess board. The image highlight rectangles at various scales.
As a remark, the proposed DMD by reconstruction analyzes either bright or dark structures depending on the choice of the morphological filter series. If openings by reconstruction are selected, only bright structures are analyzed. On the contrary, if closings by reconstruction are chosen, only dark structures are kept. Depending on the application, one or the other option has to be considered. 4. Experiments and discussion In this section, we compare both corner indicators/detectors on synthetic and satellite images. Then, we discuss the advantages and drawbacks of the Harris-DMD corner detector. 4.1. Experiments on synthetic images We propose to assess the indicators on the synthetic image displayed in Fig. 2, which represents a distorted chessboard. This image is selected because it contains rectangles with various scales, which makes it a good example for comparing the Harris–Laplace and the Harris-DMD corner detectors. The scale of rectangles decreases while going from the left to the right of the image. First, the pyramid of Gaussians of the image is displayed in Fig. 3(a). To each level of the pyramid, the corresponding Harris indicator is displayed. One can observe that, as the analysis scale becomes bigger, the Harris indicator becomes spatially unlocalized. Secondly, the DMD by opening by reconstruction is displayed in Fig. 3(b). In this case only bright objects of some scale are kept in the corresponding detail image. The selected scales are comparable to the ones used in the pyramid of Gaussian, such that the Harris integration scales are the same in both cases. On the right of each detail image the corresponding Harris indicator is displayed. One can observe that this last corner indicator remains spatially localized while increasing the analysis scale, because the edges are not smooth. Unlike the Harris–Laplace method which indicates corners at many consecutive scales, the DMD by reconstruction based corner indicator is localized in scale by construction. In the proposed example, a corner highlighted in one scale is not present in the other scales. From these stacks of corner indicators, corner positions and scales are retrieved thanks to (5) and (10). The results in both cases are showed in Fig. 4, where disk sizes encode corner scales. While, both the methods provide similar results in terms of corner detection, we observe that the Harris–Laplace detector generates more false alarms than the Harris-DMD. Indeed, the Harris–Laplace detector fails in selecting the optimal scale. This drawback is highlighted in Fig. 5, which displays the scale space positions of corners with respect to the horizontal image axis and the scale dimension. In Fig. 5(a), there is not any correlation between the horizontal and
L. Gueguen, M. Pesaresi / Pattern Recognition Letters 32 (2011) 1714–1719
1717
Fig. 3. (a) The left column represents the pyramid of Gaussian decomposition for selected increasing scales. The right column represents the corresponding Harris indicators. (b) The left column represents the DMD decomposition by opening by reconstruction for selected increasing scales. The right column represents the corresponding Harris indicators.
Fig. 4. (a) The result of the Harris–Laplace detector. (b) The result of the Harris-DMD corner detector. The size of disks represents the scale of the detected corner.
Fig. 5. The horizontal and scale positions of corners are displayed for the Harris–Laplace and the Harris-DMD detectors, respectively
1718
L. Gueguen, M. Pesaresi / Pattern Recognition Letters 32 (2011) 1714–1719
rectangle scales, going from left to right. This last example highlights the localization property of the Harris-DMD corner detector in space and scale. 4.2. Experiments on satellite images
Fig. 6. A panchromatic WorldView2 image of size 512 1024 pixels. DigitalGlobe, Inc.
the scale locations, meaning that the scale detector fails in capturing the rectangles scale behavior. On the contrary, in Fig. 5(b), the Harris-DMD detector achieves to follow the decreasingness of
The ability to detect corners is crucial for the analysis of urban area in very high resolution satellite images (Martinez-Fonte et al., 2005). In this example, a panchromatic Worldview image of resolution 0.5 m is used and displayed in Fig. 6. The pyramid of Gaussians and the DMD by opening by reconstruction of the satellite image are displayed in Fig. 7(a) and (b), with the corresponding stacks of Harris corner indicator. In the Harris–Laplace case, the corner indicators spatially overlap, such that the lower scales are privileged. On the contrary, the DMD separates sharply the image structures by scale, resulting in a reduced overlap of the corner indicators. Such property enables to obtain a good discrimination of corner scales. In the Harris–Laplace method, the noise impact is reduced by smoothing the image before computing edges,
Fig. 7. (a) The left column represents the pyramid of Gaussian decomposition for selected scales: {3, 7, 11, 15, 21} pixels. The right column represents the corresponding Harris indicators. (b) The left column represents the DMD decomposition by opening by reconstruction for the radius {3, 7, 11, 15, 21} pixels. The top image is the first detail image g0 and is considered as noise. The right column represents the corresponding Harris indicator stack.
Fig. 8. The Harris–Laplace (a) and the Harris-DMD (b) corner detectors are applied to the satellite image. The disk sizes represent the scales at which corners are detected.
L. Gueguen, M. Pesaresi / Pattern Recognition Letters 32 (2011) 1714–1719
resulting in unlocalized corner indicators. On the contrary, the DMD by reconstruction keeps the noise component in the first detail image, such that it does not impact the successive corner indicators. Avoiding a smoothing of edges in the successive detail images, a better localization of corners is obtained. The results of both corner detectors are given in Fig. 8(a) and (b), respectively. In the case of the Harris–Laplace detector, a series of scales going from 3 pixels to 40 pixels by step of 2 has been considered. In the case of the Harris-DMD detector, the disk shape SE radius series goes from 3 until 40 pixels by step of 2. Comparing both the detection results, the methods provide similar detection rates, where the majority of corners are detected. The Harris-DMD detector seems to have a better discrimination of corner scales than the Harris–Laplace method. However, as the DMD by reconstruction keeps edges in large scales, some false alarms appear at this level, while being discarded by the Harris–Laplace detector. 5. Conclusion This paper dealt with multi scale corner detection. First, the Harris–Laplace corner indicator was reminded. This indicator is based on the synergy of the Harris corner detector and a linear scale-space decomposition: the Laplacian of Gaussian. Secondly, Differential Morphological Decompositions by reconstruction were presented as non-linear scale-space representation. Combining this decomposition with the Harris corner indicator yields a new method for multi scale corner detection. Finally, both the methods were assessed visually on synthetic and satellite images, showing comparable efficiencies. Nevertheless, the proposed method was
1719
shown to discriminate the corner scales better, by exploiting scale-space locality of DMD by reconstruction. In conclusion, the Harris-DMD corner detector is a promising alternative for characterizing the salient points in satellite images. References Goutsias, J., Heijmans, H., 1998. Multiresolution signal decomposition schemes. Part 1: Linear and morphological pyramids. Tech. rep., CWI (Centre for Mathematics and Computer Science) Amsterdam, The Netherlands. Gueguen, L., Soille, P., Pesaresi, M., 2010. Differential morphological decomposition segmentation: A multi-scale object based image description. In: 20th International Conference on Pattern Recognition (ICPR) 2010, pp. 938 – 941. Harris, C., Stephens, M., 1988. A combined corner and edge detector. In: Proc. 4th Alvey Vision Conf. pp. 147–151. Lindeberg, T., 1998. Feature detection with automatic scale selection. Int. J. Comput. Vision 30, 79–116. Maragos, P., 1989. Pattern spectrum and multiscale shape representation. IEEE Trans. Pattern Anal. Machine Intell. 11 (7), 701–716. Martinez-Fonte, L., Gautama, S., Philips, W., Goeman, W., 2005. Evaluating corner detectors for the extraction of man-made structures in urban areas. In: Proc. IEEE International Geoscience and Remote Sensing Symp. IGARSS ’05, vol. 1. Mikolajczyk, K., Schmid, C., 2004. Scale and affine invariant interest point detectors. Int. J. Comput. Vision 60, 63–86. doi:10.1023/B:VISI.0000027790.02288.f2 . Pesaresi, M., Benediktsson, J., 2001. A new approach for the morphological segmentation of high-resolution satellite imagery. IEEE Trans. Geosci. Remote Sensing 39 (2), 309–320. Shi, F., Huang, X., Duan, Y., 2009. Robust Harris–Laplace detector by scale multiplication. In: Advances in Visual Computing. Lecture Notes in Computer Science, Vol. 5875. Springer, Berlin/ Heidelberg, pp. 265–274 . Soille, P., 2003. Morphological image analysis. Springer-Verlag. Vincent, L., 1993. Morphological grayscale reconstruction in image analysis: Applications and efficient algorithms. IEEE Trans. Image Process. 2 (2), 176–201.
Contents lists available at SciVerse ScienceDirect
Pattern Recognition Letters journal homepage: www.elsevier.com/locate/patrec
Multi scale Harris corner detector based on Differential Morphological Decomposition L. Gueguen ⇑, M. Pesaresi Global Security and Crisis Management Unit, IPSC, Joint Research Center-European Commission, Via E. Fermi, 2749, I-21027 Ispra (Va), Italy
a r t i c l e
i n f o
Article history: Received 24 September 2010 Available online 6 August 2011 Communicated by S. Aksoy Keywords: Corner detector Mathematical morphology Opening by reconstruction Scale space
a b s t r a c t In this paper, a novel method for multi scale corner analysis and detection is presented. First, stateof-the-art Harris–Laplace corner detector is reminded, which benefits from linear scale-space analysis. Secondly, a non-linear scale-space transform, namely Differential Morphological Decomposition, is described. This multi-scale transform is used jointly with the Harris corner indicator to build a new multi scale corner detector. Both corner detectors are visually assessed on synthetic and satellite images, highlighting the advantages of such a method. 2011 Elsevier B.V. All rights reserved.
1. Introduction Extraction of salient points from images has been demonstrated to be an efficient paradigm for matching and recognition, as it provides robustness to some transforms. Various solutions for extracting salient points have been developed over the past few years, such that they satisfy invariance properties. One amongst them is the Harris detector (Harris and Stephens, 1988) which identifies corners of image and was designed to be invariant to rotations. Then, the Harris–Laplace detector was proposed in (Mikolajczyk and Schmid, 2004), such that it is invariant to scale and affine transforms. These methods are reviewed hereafter. When the scale of a salient image point is known, the Harris detector is an efficient detector. However, other strategies have to be considered, when the scale is unknown. This issue is solved in (Mikolajczyk and Schmid, 2004; Shi et al., 2009), by computing a corner indicator at consecutive scales, and then selecting the best scale locally. As output, positions and scales of detected corners are obtained, and can be used for matching or analyzing images. Such an approach uses a low pass filtering of the original signal in order to detect large scale corners, and this simplification process destroys edges delineating corners. It results in bad detection of corners in large scales. In this paper, we propose to use a non-linear scale-space transform, namely a Differential Morphological Decomposition (DMD), in synergy with the Harris indicator. This non-linear transform preserves edges, while simplifying the input monochromatic image. In
⇑ Corresponding author. E-mail address: [email protected] (L. Gueguen). 0167-8655/$ - see front matter 2011 Elsevier B.V. All rights reserved. doi:10.1016/j.patrec.2011.07.021
addition, a DMD analyses an image by scale, such that the choice of the Harris indicator integration scale becomes straightforward. Combining both the advantages yields a multi-scale corner detector which is localized in scale and space. The paper is organized as follows. Section 2 reviews the single scale Harris corner indicator, and its extended multi-scale version: the Harris–Laplace corner detector. In Section 3, a new multi-scale corner detector is presented. It includes the description of Differential Morphological Decompositions, and their use with the Harris corner indicator. Then, the Harris–Laplace and the proposed detectors are applied to synthetic and satellite images in Section 4. Finally, Section 5 concludes. 2. Harris corner detector In this section, we remind the Harris and the Harris–Laplace corner indicators and detectors. The presentation of such well known technique is intended to make an analogy with the proposed idea. 2.1. Single scale Harris corner indicator The Harris corner indicator (Harris and Stephens, 1988) is a well established technique making use of several linear filtering of an image. Given an image f, we denote by L(p, r) = g(p, r) w f(p), the low pass filtering by a Gaussian function g(p, r) of scale r, where p is the position in the image grid. Let Lx(p, rD), Ly(p, rD) be the respective horizontal and vertical image derivatives at position p, obtained with Gaussian filter of local scale rD (the differentiation scale). Then these intermediate results are composed to obtain
L. Gueguen, M. Pesaresi / Pattern Recognition Letters 32 (2011) 1714–1719
three images L2x ðp; rD Þ; L2y ðp; rD Þ; Lx Ly ðp; rD Þ, where the multiplicative operation is performed pixel wise. These images are then filtered by a Gaussian function g(p, rI) of scale rI (the integration scale):
"
lðp; rI ; rD Þ ¼ r2D gðrI ÞH
L2x ðp;
rD Þ Lx Ly ðp; rD Þ ; Lx Ly ðp; rD Þ L2y ðp; rD Þ
Hðp; rI ; rD Þ ¼ detðlðp; rI ; rD ÞÞ j tr2 ðlðp; rI ; rD ÞÞ;
ð1Þ
ð2Þ
where j is a trade-off scalar, experimentally lying in the interval [0.04, 0.15]. High value of the Harris measurement indicates the presence of a corner of scale rI at position p in the image f. As a remark, the choice of the integration scale rI is critical, as a small scale results in missing the corners of large objects, while choosing a large scale results in missing small object corners. 2.2. Harris–Laplace corner detector As mentioned previously, the integration scale used for performing corner detection is crucial. To circumvent this issue, a multi-scale Harris corner detection was proposed in (Mikolajczyk and Schmid, 2004), by benefiting of a linear scale space decomposition. In such a paradigm, the Harris corner indicator is applied at successive integration scales, before selecting locally the characteristic scales of image blobs. In (Lindeberg, 1998), the Laplacian-of-Gaussian (LoG) is used, as a scale space representation, for selecting the image blob characteristic scales, and the scale normalized Laplacian operator is defined by:
LoGðp; rN Þ ¼ r
DMD is a scale space decomposition, the selection of the integration scale of the Harris corner indicator is straightforward. 3.1. Differential Morphological Decomposition
#
where w is the convolution operator. Finally, the single scale Harris corner indicator H(p, rI, rD) is given by:
2 N ðLxx ðp;
1715
rN Þ þ Lyy ðp; rN ÞÞ;
ð3Þ
where Lxx, Lyy are the second image derivatives in the horizontal and vertical directions, respectively. The characteristic scale at some position p is defined by the scale rN which hits the maximum of jLoG(p, rN)j. In (Mikolajczyk and Schmid, 2004), the authors propose to combine the Harris corner indicator and the scale normalized Laplacian operator in order to detect the salient positions in the scale space grid. First, a set of increasing analysis scales is chosen: r1 < r2 < < rn, enabling them the computation of a multi scale n stack of Harris indicators pfHðp; ffiffi rk ; crk Þgk¼1 , where c is a scalar ly ing in the range 1=2; 1= ð2Þ . For each scale rk, the local maxima of the Harris indicator H(p, rk, crk) are computed. Secondly, these spatial locations are analyzed through the LoG operator, in order to discard the ones which are not salient in the scale dimension:
^; rÞg ¼ argmaxlocalp Hðp; r; crÞ; fðp
ð4Þ
r^ ¼ argmaxr jLoGðp^; crÞj:
ð5Þ
^ which do no hit a scale maximum are disThe spatial locations p carded. The result of the Harris–Laplace corner detector is the set ^; r ^ Þg. of retained scale space positions fðp
This section describes the Differential Morphological Decomposition (DMD) in the context of multiresolution pyramid transform. Then, a specific DMD is described, which is based on the opening or closing by reconstruction operators. Differential Morphological Decompositions (DMD) belong to the family of multiresolution pyramid transforms, which analyze images by reducing or coarsing information at each pyramid level. In such a scheme, the removed information, which is the difference between the original and the coarse images, is stored in a detail image, which compounds the differential decomposition. In the following, the complete lattice L of discrete images is considered and is defined over the grid Z2 . A pyramid scheme analyzes sequentially an image reducing the information at each step (Goutsias and Heijmans, 1998). Let U"j : Lj ! Ljþ1 be the analysis operator at pyramid level j, and let U#j : Ljþ1 ! Lj be the corresponding synthesis operator. The successive complete lattices Lj are defined as the images of L by the successive analysis operators U"j , and they follow the inclusion relation Lj Ljþ1 . As information is to be reduced at each level, a pyramid condition is introduced in (Goutsias and Heijmans, 1998) and it is defined by U"j U#j ðf Þ ¼ f , 8f 2 Ljþ1 . The approximation at level j is ^f ¼ U#j U"j ðf Þ where f 2 Lj , and the detail image at level j is obtained by subtraction: g ¼ f ^f . Therefore, a differential decomposition of an image f can be obtained by the following recursive scheme:
8 f ¼ f 2 L; > < 0 fjþ1 ¼ U"j ðfj Þ 2 Ljþ1 ; > : g j ¼ fj U#j ðfjþ1 Þ:
ð6Þ
The differential decomposition of f is graphically represented in Fig. 1 and it is the compound of the detail images plus the last analysis result: {g0, . . . , gj;fj+1}. If the pyramid condition is satisfied, the decomposition enables a perfect reconstruction of the original image, meaning that all the information is kept during the analysis. A couple of operators (U, W) constitutes an adjunction of L if 8f ; g 2 L; Uðf Þ 6 g () f 6 WðgÞ, and these operators verify the pyramid condition (Goutsias and Heijmans, 1998). For example, (U, Id) is an adjunction for any morphological filter U, which follows the property of idempotence, i.e. U2 = U, and increasingness, i.e. "f, g, f 6 g ) U(f) 6 U(g). Therefore, any morphological filter can be used to define a Differential Morphological Decomposition. To satisfy the property of information decreasingness, series of morphological openings 6 Uj+1 6 Uj 6 with increasing size Structuring Elements (SE) can be considered. This decomposition is commonly called granulometry (Maragos, 1989), and is a non linear scale space representation. However, the shape of the structuring element impacts mainly the shape of structures retained in the detail images.
3. Differential Morphological Decomposition based corner detection In this section, a non linear scale-space decomposition is described thanks to morphological filtering by reconstruction (Soille, 2003). Then, a new method for indicating corners is proposed, taking benefit of the Differential Morphological Decomposition by reconstruction (DMD). Unlike linear scale-space decomposition, DMD preserve edges in higher scales, enabling a more robust detection of corners with the Harris indicator. In addition, as
Fig. 1. The Differential Morphological Decomposition is depicted for two pyramid levels, resulting in the representation g0, g1; f2. Thanks to the pyramid condition, the original signal f can be perfectly reconstructed.
1716
L. Gueguen, M. Pesaresi / Pattern Recognition Letters 32 (2011) 1714–1719
In (Pesaresi and Benediktsson, 2001; Gueguen et al., 2010), the use of morphological filters by reconstruction with increasing radius was proposed to circumvent this effect. This type of filtering is part of the family of connected operators which preserve edges while decreasing image structure contrasts (Soille, 2003). Opening or closing by reconstruction are morphological connected filters, which remove or threshold the local maxima or minima which cannot contain the structuring element, while keeping the rest of the image. The opening by reconstruction is defined thanks to the reconstruction by dilation Vincent (1993), which is expressed between two images respecting the following inequality f 6 g:
dgC ðf Þ ¼ dC ðf Þ ^ g; Rdg
¼
ðdgC Þi ;
9i P 1;
ð7Þ g i g iþ1 dC ¼ dC ;
ð8Þ
where dC is the dilation by the C-connectivity SE (the 4- or 8-connectivity), and ^ is the pointwise minimum operator. Then, the opening by reconstruction with the SE B is defined by:
cB ðf Þ ¼ Rdf ðB ðf ÞÞ;
ð9Þ
where B is the erosion by the SE B. A dual definition holds for expressing the closing by reconstruction. Hereafter, series of disk-shape SE Bj Bj+1 with increasing radius < rj < rj+1 < are considered to define a DMD by reconstruction from the respective series of morphological openings by reconstruction 6 cBj 6 cBjþ1 6 . The detailed image gj contains the image bright structures which can contain the disk Bj1, but cannot contain the successive one Bj. Therefore, the r þr average radius sj ¼ j12 j of both SE Bj1 and Bj can be understood as the characteristic scale of the structures contained in gj. Because of edges preservation and image structure scale analysis, the DMD based on openings/closings by reconstruction is an appealing scale-space representation for detecting corners. 3.2. DMD based Harris corner detector In this section, we define a new corner detector using the DMD by reconstruction {g0, . . . , gj; fj+1} of an image f, associated to the increasing scales < sj < sj+1 < . Such corner detector, as the Harris–Laplace indicator, benefits from the scale-space representation for selecting the integration scale ri of the Harris indicator expressed by (2). At scale sj, only image structures of this scale are present in the detail image gj. Since the noise is spatially unstructured, it remains in the first decomposition level g0. The upper levels being almost free of noise and preserving edges, a low pass filtering before the estimation of derivatives is unnecessary. Thus, the Harris corner indicator can be applied directly to the detail image gj, "j > 0. Let Hj(p, rI, 0) be the Harris detector applied to the detail image gj, with r þr an integration scale rI. Given the characteristic scale sj ¼ j12 j of structures in the detail image gj, the choice of the Gaussian filter scale becomes straightforward, and its optimal value is given by rI = sj/2. A stack of corners indicators {Hj(p, sj/2, 0)}j>0 is obtained by applying the Harris indicator to each detail image, except g0 containing the noise part. The DMD by reconstruction based Harris indicator is localized in space and scale by construction. Therefore, the DMD based corner detector consists in identifying the positions of the scale-space which reach a local maximum of {Hj(p, sj/2, 0)}j>0:
^; ^sÞg ¼ argmaxlocalp;s Hi ðp; sj =2; 0Þ: fðp j
ð10Þ
Due to the image acquisition limitations, the same image structure appears sometimes in contiguous detail images. Therefore, allowing to take the maximum scales enables to eliminate duplicate corners in contiguous scales.
Fig. 2. Synthetic image representing a distorted chess board. The image highlight rectangles at various scales.
As a remark, the proposed DMD by reconstruction analyzes either bright or dark structures depending on the choice of the morphological filter series. If openings by reconstruction are selected, only bright structures are analyzed. On the contrary, if closings by reconstruction are chosen, only dark structures are kept. Depending on the application, one or the other option has to be considered. 4. Experiments and discussion In this section, we compare both corner indicators/detectors on synthetic and satellite images. Then, we discuss the advantages and drawbacks of the Harris-DMD corner detector. 4.1. Experiments on synthetic images We propose to assess the indicators on the synthetic image displayed in Fig. 2, which represents a distorted chessboard. This image is selected because it contains rectangles with various scales, which makes it a good example for comparing the Harris–Laplace and the Harris-DMD corner detectors. The scale of rectangles decreases while going from the left to the right of the image. First, the pyramid of Gaussians of the image is displayed in Fig. 3(a). To each level of the pyramid, the corresponding Harris indicator is displayed. One can observe that, as the analysis scale becomes bigger, the Harris indicator becomes spatially unlocalized. Secondly, the DMD by opening by reconstruction is displayed in Fig. 3(b). In this case only bright objects of some scale are kept in the corresponding detail image. The selected scales are comparable to the ones used in the pyramid of Gaussian, such that the Harris integration scales are the same in both cases. On the right of each detail image the corresponding Harris indicator is displayed. One can observe that this last corner indicator remains spatially localized while increasing the analysis scale, because the edges are not smooth. Unlike the Harris–Laplace method which indicates corners at many consecutive scales, the DMD by reconstruction based corner indicator is localized in scale by construction. In the proposed example, a corner highlighted in one scale is not present in the other scales. From these stacks of corner indicators, corner positions and scales are retrieved thanks to (5) and (10). The results in both cases are showed in Fig. 4, where disk sizes encode corner scales. While, both the methods provide similar results in terms of corner detection, we observe that the Harris–Laplace detector generates more false alarms than the Harris-DMD. Indeed, the Harris–Laplace detector fails in selecting the optimal scale. This drawback is highlighted in Fig. 5, which displays the scale space positions of corners with respect to the horizontal image axis and the scale dimension. In Fig. 5(a), there is not any correlation between the horizontal and
L. Gueguen, M. Pesaresi / Pattern Recognition Letters 32 (2011) 1714–1719
1717
Fig. 3. (a) The left column represents the pyramid of Gaussian decomposition for selected increasing scales. The right column represents the corresponding Harris indicators. (b) The left column represents the DMD decomposition by opening by reconstruction for selected increasing scales. The right column represents the corresponding Harris indicators.
Fig. 4. (a) The result of the Harris–Laplace detector. (b) The result of the Harris-DMD corner detector. The size of disks represents the scale of the detected corner.
Fig. 5. The horizontal and scale positions of corners are displayed for the Harris–Laplace and the Harris-DMD detectors, respectively
1718
L. Gueguen, M. Pesaresi / Pattern Recognition Letters 32 (2011) 1714–1719
rectangle scales, going from left to right. This last example highlights the localization property of the Harris-DMD corner detector in space and scale. 4.2. Experiments on satellite images
Fig. 6. A panchromatic WorldView2 image of size 512 1024 pixels. DigitalGlobe, Inc.
the scale locations, meaning that the scale detector fails in capturing the rectangles scale behavior. On the contrary, in Fig. 5(b), the Harris-DMD detector achieves to follow the decreasingness of
The ability to detect corners is crucial for the analysis of urban area in very high resolution satellite images (Martinez-Fonte et al., 2005). In this example, a panchromatic Worldview image of resolution 0.5 m is used and displayed in Fig. 6. The pyramid of Gaussians and the DMD by opening by reconstruction of the satellite image are displayed in Fig. 7(a) and (b), with the corresponding stacks of Harris corner indicator. In the Harris–Laplace case, the corner indicators spatially overlap, such that the lower scales are privileged. On the contrary, the DMD separates sharply the image structures by scale, resulting in a reduced overlap of the corner indicators. Such property enables to obtain a good discrimination of corner scales. In the Harris–Laplace method, the noise impact is reduced by smoothing the image before computing edges,
Fig. 7. (a) The left column represents the pyramid of Gaussian decomposition for selected scales: {3, 7, 11, 15, 21} pixels. The right column represents the corresponding Harris indicators. (b) The left column represents the DMD decomposition by opening by reconstruction for the radius {3, 7, 11, 15, 21} pixels. The top image is the first detail image g0 and is considered as noise. The right column represents the corresponding Harris indicator stack.
Fig. 8. The Harris–Laplace (a) and the Harris-DMD (b) corner detectors are applied to the satellite image. The disk sizes represent the scales at which corners are detected.
L. Gueguen, M. Pesaresi / Pattern Recognition Letters 32 (2011) 1714–1719
resulting in unlocalized corner indicators. On the contrary, the DMD by reconstruction keeps the noise component in the first detail image, such that it does not impact the successive corner indicators. Avoiding a smoothing of edges in the successive detail images, a better localization of corners is obtained. The results of both corner detectors are given in Fig. 8(a) and (b), respectively. In the case of the Harris–Laplace detector, a series of scales going from 3 pixels to 40 pixels by step of 2 has been considered. In the case of the Harris-DMD detector, the disk shape SE radius series goes from 3 until 40 pixels by step of 2. Comparing both the detection results, the methods provide similar detection rates, where the majority of corners are detected. The Harris-DMD detector seems to have a better discrimination of corner scales than the Harris–Laplace method. However, as the DMD by reconstruction keeps edges in large scales, some false alarms appear at this level, while being discarded by the Harris–Laplace detector. 5. Conclusion This paper dealt with multi scale corner detection. First, the Harris–Laplace corner indicator was reminded. This indicator is based on the synergy of the Harris corner detector and a linear scale-space decomposition: the Laplacian of Gaussian. Secondly, Differential Morphological Decompositions by reconstruction were presented as non-linear scale-space representation. Combining this decomposition with the Harris corner indicator yields a new method for multi scale corner detection. Finally, both the methods were assessed visually on synthetic and satellite images, showing comparable efficiencies. Nevertheless, the proposed method was
1719
shown to discriminate the corner scales better, by exploiting scale-space locality of DMD by reconstruction. In conclusion, the Harris-DMD corner detector is a promising alternative for characterizing the salient points in satellite images. References Goutsias, J., Heijmans, H., 1998. Multiresolution signal decomposition schemes. Part 1: Linear and morphological pyramids. Tech. rep., CWI (Centre for Mathematics and Computer Science) Amsterdam, The Netherlands. Gueguen, L., Soille, P., Pesaresi, M., 2010. Differential morphological decomposition segmentation: A multi-scale object based image description. In: 20th International Conference on Pattern Recognition (ICPR) 2010, pp. 938 – 941. Harris, C., Stephens, M., 1988. A combined corner and edge detector. In: Proc. 4th Alvey Vision Conf. pp. 147–151. Lindeberg, T., 1998. Feature detection with automatic scale selection. Int. J. Comput. Vision 30, 79–116. Maragos, P., 1989. Pattern spectrum and multiscale shape representation. IEEE Trans. Pattern Anal. Machine Intell. 11 (7), 701–716. Martinez-Fonte, L., Gautama, S., Philips, W., Goeman, W., 2005. Evaluating corner detectors for the extraction of man-made structures in urban areas. In: Proc. IEEE International Geoscience and Remote Sensing Symp. IGARSS ’05, vol. 1. Mikolajczyk, K., Schmid, C., 2004. Scale and affine invariant interest point detectors. Int. J. Comput. Vision 60, 63–86. doi:10.1023/B:VISI.0000027790.02288.f2