- Email: [email protected]
Using selective correlation coefficient for robust image registration
Pattern Recognition 36 (2003) 1165 – 1173
www.elsevier.com/locate/patcog
Using selective correlation coe cient for robust image registration Shun’ichi Kanekoa; ∗ , Yutaka Satohb , Satoru Igarashia a Department
of Control and Information Engineering, Graduate School of Engineering, Hokkaido University, N13-W8 Sapporo, Japan b Softopia Japan, 4-1-7, Kagano, Ogaki, Japan Received 23 February 2001; accepted 1 April 2002
Abstract A new method is proposed for robust image registration named Selective Correlation Coe)cient in order to search images under ill-conditioned illumination or partial occlusion. A correlation mask-image is generated for selecting pixels of an image before matching. The mask-image can be derived from a binary-coded increment sign-image de3ned from any object-image and the template. The mask-rate of occluded regions is theoretically expected to be 0.5, while unoccluded regions have much lower rate than 0.5. Robustness for ill-conditioned environment can be realized since inconsistent brightness of occluded regions can be omitted from the mask operation. Furthermore, the mask enhancement procedure is proposed to get more stable robustness. The e6ectiveness of masking increases by the procedure, resulting in the rate around 0.7 for masking of occluded regions. This paper includes a theoretical modeling and analysis of the proposed method and some experimental results with real images. ? 2002 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. Keywords: Normalized cross correlation; Selective correlation coe cient; Image registration; Template matching; Robust statistics
1. Introduction In pattern recognition, image registration techniques to check image similarity based on a rational quantitative measure and to locate a template at the place of its optimum value is one of the oldest and fundamental image processing [1]. These techniques construct an important foundation that should be a kernel for image understanding systems in a lot of 3elds such as visual inspection, robot vision, environment monitoring, and general-purpose image-based measurement systems. High-speed and inexpensive computing facility and image processing hardware, furthermore, can recently be of popular utilization, and rather expensive algorithms for image processing with numerous computation get to be feasible even in real-world applications. There have been several approaches for image registration, such as a feature-based approach, brightness-based or ∗
Corresponding author. Tel.=fax: +81-11-706-6436. E-mail address: [email protected] (S. Kaneko).
image-based approach, and an aggregated measure-based approach like color or gray level histograms. Many non-brightness feature-based approaches have been proposed to robust image matching. GHT [2] is based on detected local features utilized as voting electors that pose limited scores and vote them for models, while the indexing approaches [3] adopt combinations of geometric features, such as edge points or line segments as key for indexing objects in the dictionary and locally performs searching them to pick up candidates of the objects. But we cannot expect 3ne matching results without detecting effective features by steps of preprocessing which need some heuristic scheme for selection of threshold values and may have a kind of instability. Moment-based approaches [4] are e6ective for rotated object patterns, but they need much cost of computation and have less robustness for brightness change and occlusion. Color or histogram indexing methods [5] are another approach to rotation-invariant object search in real scene, but they are not robust for occlusion and brightness change.
0031-3203/03/$30.00 ? 2002 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. PII: S 0 0 3 1 - 3 2 0 3 ( 0 2 ) 0 0 0 8 1 - X
1166
S. Kaneko et al. / Pattern Recognition 36 (2003) 1165 – 1173
In this paper, we focus our attention to brightness-based methods for image registration because in order to aim high precision of location and rich robustness for various ill-conditions, the approach might be better than the other ones [6,7]. Normalized cross correlation, sum of square di6erence, statistical sign change, increment sign correlation, and rank statistics have been proposed as brightness-based template matching techniques so far. Normalized cross correlation (CC) [8,9] has been used very widely because it can cope with noisy images and uniform change in brightness. But, when the scene involves partial occlusion of objects or saturation (highlight), this sum-of-multiplication-based correlation technique sometimes fails to capture true positions, while it is good to measure some slight change in brightness. Sum of squared di6erence (SSD) or sum of absolute di6erence [9] evaluates a sum of squared or absolute di6erences in brightness de3ned between two distinct images. Though it is an e cient algorithm, it is very feeble for change in brightness caused by illumination Kuctuation, occlusion and saturation. Statistical sign change [10] is derived as the sum of the number of sign changes between every two adjacent pairs of pixels, one of which belongs to a template-image and the other of which comes from an object-image at the same position. The object-image of the maximum value is considered as the solution. It interestingly utilizes statistical characteristics of noise and is available for large noises rather uniformly distributed over images. However, neither local brightness change nor occlusion can be coped with by this measure. Deterministic images also force it to be improved by an ad hoc unreasonable algorithm. Increment sign correlation (ISC) [11,12] is developed as a robust method for image registration, which can cope with occlusion, shadow, saturation or highlight. This method is based on a similarity measure which is evaluated by aggregating increment sign codes at each pixel, which represent +=− signs between any two adjacent pixels. ISC is proved to be robust because it does not directly utilize brightness of pixels but their incremental tendency, and it is e cient even in comparison with CC. Due to the information reduction through neglecting brightness values, however, there have been some problems such that the size of a template cannot be reduced and it cannot discriminate slight changes in brightness of images. Robust similarity measure using rank statistics [13] is based on an alternative approach for brightness variation. This kind of similarity measure is based on the fact that brightness itself changes in the case of rather uniform variation in illumination, but their ranks or orders can remain unchanged. However, it is largely inKuenced by a not small change of local perturbation of ranks caused by occlusion, shading and saturation. In this paper, a novel method for robust image-registration or image matching based on a halftone image named selective correlation coe cient (SCC) is proposed in order to obtain robustness especially for occlusion and saturation, ill-conditioned illumination and partial occlusion. The pro-
posed method is designed as an improved CC with an enhancement by ISC which is so robust for the ill-conditions above mentioned. In this method, an increment sign (IS) image transformed from the original hale-tone image is used for masking irrelevant pixels to template pixels. Because of the statistical nature of IS codes, parameters are not necessary in order to realize the masking operation which can automatically select consistent pixels to the template ones and consequently decrease wrong inKuence to taking the CC between the template and an object-image, and then it becomes possible to 3nd defective regions in the object-image.
2. Selective correlation coecient 2.1. De7nition Simple subtraction and check of their absolute di6erence, for example, causes pseudoknowledge corresponding to brightness change in object-images due to illumination change which occurs often in real situations. In order to have robustness in image registration, we need to focus our attention to the nature of brightness change in the ill-condition. This is a motivation of this work. Fig. 1 shows the overview of the proposed scheme to calculate SCC. It generates a selecting mask for each pixel before calculating correlation. The mask can be derived from the binary-coded IS between an object-image and a template-image. Then correlation is calculated not only from the template and the object-image but also from the mask-image based on IS which can be assumed as pre-3ltering by consistency between matched pixels. Each pixel in the mask-image takes a value of ‘0’ or ‘1’ and consequently they can select pixels whose brightness should be adopted for similarity evaluation between two images. The normalized CC function or correlation coe cient (CC) is 3rst considered. A one-dimensional ordered list, into which a two-dimensional image is rearranged according to some order, is utilized in this paper for simplicity without any loss of generality. Images might be often decomposed into each row or column and then stacked in their natu−1 ral order. Let F = {fn }Ni=0 de3ne a template-image and −1 G = {gn }Ni=0 an object-image which is of the same size N as the template and may be extracted from the larger scene. CC is expressed by the formula N −1
N N n=0 (fn − f)(gn − g) rcc = ; N −1 N −1 N 2 N2 n=0 (fn − f) n=0 (gn − g)
(1)
where fN and gN mean the averages of brightness for F and G, respectively. SCC is de3ned as an extension of CC with a masking factor for every pixel pair by the equation
S. Kaneko et al. / Pattern Recognition 36 (2003) 1165 – 1173
Template
1167
Template
Masking
Object
*
*
Correlation
Mask(IS)
*
0 1 1 1 1 0 1 0
Scene
bn
1 0 0 1 0 1 1 1 Fig. 1. Overview of proposed method: SCC.
0
0
0
0
0
occluded pairs
below. N −1
N N n=0 cn (fn − f)(gn − g) : rscc = N −1 N −1 N 2 N2 n=0 cn (fn − f) n=0 cn (gn − g)
0
1
1
0 0 0 1 0 1 0 1 1 0 0 1 0 1 0 1
cn
0
0
1
1
1
1
1
1
matched pairs
Fig. 2. Codes bn and cn from images.
(2)
In this de3nition, brightness values fn and gn of the template and the object-image are multiplied by the coe cient cn , and then the sum of all the products is normalized by the standard deviations in each image, where the coe cients cn are also introduced. The mask coe cient or code cn reveals concordance of increment signs for check consistency between the template and the object-image in the same way as feature evaluation [14], and then is de3ned by 1 − |bn − bn | (n = 0 or even); cn = (3) (n = odd); cn−1 where the increment sign bn for F is de3ned similarly in Ref. [12] 1 (fn+1 ¿ fn ); bn = (4) 0 (otherwise) and bn is also de3ned for G as well. Increment signs bn and bn are binary digits which correspond to brightness change between adjacent pixels. The values ‘1’ and ‘0’, on the other hand, reveal non-negative and negative signs of brightness increment de3ned in succeeding neighborhood at each pixel. The de3nition of orientation in neighborhood pixels is not important for calculation of the codes. In this paper, we make one-dimensional brightness lists through horizontal decomposition of a two-dimensional image array and linking the rows into the list, and consequently horizontally adjacent pixels are compared. It is possible to adopt another way to tailor the list. The mask coe cient cn is constructed through the rule that it takes ‘1’ if bn and bn are concordant and ‘0’ otherwise. Consequently, the image M = {cn } can mask pixels of probable inconsistency out of the calculation of correlation following the mask operation. We call this mask
a ‘basic mask’ to distinguish the di6erence between this operation and the one which we will de3ne in conjunction with ‘enhancement operation’ described later. Fig. 2 shows schematic examples of the codes bn and cn . In this 3gure, we can 3nd typical examples of occlusion and brightness change due to shadow of the box in front of Winnie-the-pooh on the toy train. We assign the same value to two adjacent coe cients by de3nition, for example c0 = c1 or c14 = c15 , because for computation of an increment sign code we need two adjacent pixels and then in order to make cn independent, any two pixel pairs should not be overlapped. ISC has been proposed for robust image registration in the case of occlusion and=or change in illumination. The correlation can be modeled as a statistic that has a binomial or normal distribution under a reasonable assumption and approximation. From the statistical analysis and modeling, ISC has the following fundamental characteristics [11,12]: • The deviation is generally small. • The mean is the probability of sign reversal between bn and bn . • It has an expected value of mean 0.5 for uncorrelated images. • It has an expected value more than 0.5 for correlated images, which depends on the amount of noise included in the scene. These statistical characteristics of ISC guarantee the performance of the basic mask-image M which is expected to exclusively choose probably consistent pixels and otherwise remove inconsistent ones. The basic mask-rate of occluded regions is theoretically expected to be 0.5 because of the ISC characteristics for uncorrelated images. On the other hand, for unoccluded or similar regions, it can have the possible mask-rate below 0.5 in the case of correct pair of images.
1168
S. Kaneko et al. / Pattern Recognition 36 (2003) 1165 – 1173
0.223 (8998/40354) 0.504
( 6428 /12744 ( (a)
(b)
(c)
(d)
Fig. 3. An example of basic mask-image, (a) template (b) scene (c) mask-image: M , (d) mask-rate.
2.2. Characteristics of basic mask-image
2.4. Computational cost
Fig. 3 shows a real example of the basic mask-image. A template-image in Fig. 3(a) and the similar scene with occlusion in (b) are given, and they generate a basic mask-image in (c) where two distinct mask-rates can be observed and computed as shown in (d). The scene includes around 24% occluded area at the lower right corner. In the mask-image shown in Fig. 3(c), pixels in black show the value ‘0’, on the other hand pixels in white, the value ‘1’. In the similar region in the upper half and the lower left, the mask-rate is 0.223, while in the occluded region in the lower right the rate is 0.504 just as theoretically expected. We can recognize a probabilistic tendency to have much more masked pixels in occluded regions than in unoccluded ones.
The computational cost of SCC is examined in comparison with CC which has been very popular in many real applications in laboratory and industry. From Eqs. (1) and (2), SCC seems to need summations of three-term products while CC has the ones of two-terms. Because cn has ‘1’ or ‘0’ values in nature and the multiplication can be replaced by logical operation, the cost of multiplication is not essentially increased. SCC has a rather small number of products since it selects pixels from all the pixels. Then, the main issue of computational cost is the one for variance computation. In CC, the variance or the self-correlation of the template-image can be calculated only once in an oRine stage because it does not change throughout the calculation of a CC, but in SCC, three correlations should be calculated every time because they can be changed due to the coe cients cn . SCC needs an extra calculation for obtaining signs cn at every position. The cost, however, can be small and negligible because it depends on only comparing operations between two brightness values.
2.3. Stability of SCC In SCC, the amount of pixels which have contributed to the 3nal correlation value is lower than the one CC has. From the fundamental consideration in the previous section, we have to discuss about two kinds of e6ect of the mask operation: the e6ect in occluded regions and the one in other regions. First, in occluded regions, the mask-rate will be around 0.5, for example 0.504 in Fig. 3, and then a half of irrelevant pixels can be successfully excluded from correlation. This is a positive merit of SCC, however, the other half remains a wrong contribution to correlation, however, this might be the limitation of our SCC. Second, in normal regions, the mask-rate can be expected to be lower than the rate above mentioned, for example 0.223 in Fig. 3. This mask-rate varies mainly with two factors: magnitude of additive noise and contrast in the template [12]. In real situation, we should be certain of the reduction of pixels in normal regions. Reduction of pixels, for example small template-images, might generally harm statistical stability of the measure. In the case of SCC, however, we can assume for the pixels excluded by the mask operation with M to be inconsistent between the template and the object-image, therefore, the total performance should be evaluated through the real experiments.
3. Enhancement of mask-image 3.1. Basic procedure As also shown in Fig. 3, in real images, the mask-rates of unoccluded regions are generally more than 0 because of quantization error and=or additive noise introduced by an imaging process and illumination Kuctuation. On the other hand, in occluded regions, since the correlation is always low, inKuence of noise is small and the mask-rates are around 0.5 as expected. When noise is larger, however, the di6erence between them becomes smaller and then it should be improved through some enhancement procedure. The following procedure for mask enhancement is introduced in this paper. The procedure is fundamentally based on a majority rule as illustrated in Fig. 4. The 3gure shows the four-pixel majority rule in neighborhood. In this rule, there can be 3ve classes {C#k}k=0; 1; :::; 4 of pixel neighborhoods in M according to k pixels or codes having ‘0’. Pixels in the 3rst two classes in Fig. 4 are transformed to ‘1’ to
S. Kaneko et al. / Pattern Recognition 36 (2003) 1165 – 1173
C#0
0 0 1 1 0 1 1 0
C#1
0 1 0 0 1 1 0
0 1 0 0 1 1 0
0 0 1 1 0 1 0
0 0 1 1 0 1 0
0 0 0 1 1 0
0 1 0 0 1 0
0 1 0 0 1 0
0 1 0 0 1 0
0 1 0 0 1 0
0 0 1 1 0 0
C#3
0 0 0 0 1
0 1 0 0 0
C#4
0 0 0 0
C#2
0 1 0 0 0
0 0 0 1 0
1169
0.243 (9808/40354)
0 0 1 1 0 1 1 0
0.696
( 8876 /12744 ( Fig. 6. Enhanced mask-image.
0 0 0 0
Fig. 4. Mask enhancement scheme: four-pixel majority rule.
1
(a)
(b)
30%
60%
Fig. 7. Examples of occlusion, (a) template, (b) scene, 30%, 60%.
0.8
Pe
0.6
0.4
: pe2 : pe4 : pe6
0.2
0 0
0.2
0.4
0.6
0.8
1
P Fig. 5. Mask enhancement curves.
reduce masked pixels in unoccluded regions. Otherwise, all the pixels are transformed to the code ‘0’. The enhancement procedure consists of these two kinds of operations which realize reduction and accretion of the code ‘0’. Let p and pem (m = 2; 4; 6; : : :) be the basic mask-rate and the enhanced one based on the m-pixel majority rule, respectively. The rate pe2 , pe4 and pe6 , for example, can be obtained as follows: pe2 = 1 − (1 − p)2 ;
(5)
pe4 = 1 − (1 − p)4 − 4(1 − p)3 p;
(6)
pe6 = 1 − (1 − p)6 − 6(1 − p)5 p − 15(1 − p)4 p2 :
(7)
Fig. 5 shows the mathematical characteristics of the enhanced mask-rates. As shown in the 3gure, the four-pixel rule and the six-pixel rule have both the e6ects of reduction
and accretion, while the two-pixel rule always enhances any pixel with ‘0’. We adopt the four-pixel majority rule for mask enhancement for basic mask-images in this paper as considering tradeo6 of enhancement e6ect and spatial resolution. The larger the former is, in general, the smaller the latter becomes. Fig. 6 shows the enhanced version of the basic mask-image in Fig. 3. We notice that the mask-rate in the occluded region is enhanced from 0.504 to 0.696 as also shown the mask-image to the left, while the remaining region has very slight change in the rate from 0.223 to 0.243. Theoretical expectation in the e6ect of enhancement is 0.217 for 0.243 and 0.688 for 0.696 according to Eq. (7). These expectation can estimate the real values of the enhanced mask-rate. 3.2. Characteristics for occlusion In this section, we investigate the characteristics of SCC with regard to occlusion in template-images. As shown in Fig. 7, for the experiments, the template-image and the scene that were used in Fig. 3 were exchanged for generality. Occlusion was changed in degree from 0% to 100%. Fig. 8 shows the characteristics of SCC values versus occlusion. We use SCC+EM to call the selective correlation coe cient with enhancement. The values of SCC, CC and SCC+EM are reported for comparison. First, all of them have the same values of −0:3 at 100% occlusion, and so the original images can be considered as nearly uncorrelated. As occlusion increase, SCC+EM always holds the highest correlation and SCC also maintain high correlation compared with CC. The decreasing rates of both SCC and SCC+EM are lower than the one of CC, which reveal their robustness for occlusion.
1170
S. Kaneko et al. / Pattern Recognition 36 (2003) 1165 – 1173 1
0.8 Correlation
the generation of a basic mask-image consumes 15% of the total cost, the mask enhancement 5% and then the summation of products 80%. We call hereafter SCC+EM as SCC for simplicity.
: SCC+Enhanced mask : SCC+Mask : CC
0.6 0.4
4. Experiments
0.2
4.1. Occlusion
0
The e6ectiveness of SCC in the real scene containing occlusion and highlight is investigated in this section. Fig. 9 shows the result of the experiment with a simple template-image of a toy-bird. Occlusion increases from the left to the right scenes. SCC detected the right position while CC failed because of the occlusion from around 20% to 40%. Also shown in the right upper corners in each of the scenes in Fig. 9(b) – (d), SCC could make the enhanced mask-images that could almost correctly capture occluded regions. Table 2 shows the calculated values of SCC and CC, where we can 3nd that SCC could retain rather reasonable values while CC su6ered from occlusion in some cases, losing the correct position which reveals higher correlation values than the one at the correct position as shown in the table. The di6erences in correlation values range from 14% to 17% in CC. Fig. 10 indicates the pro3les of correlation values in the neighborhood of correct position in Fig. 9(b). In Fig. 10, the upper part shows the pro3le of SCC, in which the prominent highest peak could be recognized at the correct position, while in the lower of CC also seemed to have such kind of peak at the same part, however, it was not the maximum all over the scene. Since it has been known that ISC has similar characteristics of SCC [12], SCC is expected to have somewhat neutral nature between ISC and CC. Thus, we can utilize SCC in the case that a scene possibly has occlusion and simultaneously we want
-0.2 0
20
40 60 Occlusion Ratio (%)
80
100
Fig. 8. Correlation values versus occlusion.
Table 1 Comparing computational cost (s=pixel) SCC SCC+EM CC SSD
0.30 0.26 0.28 0.25
3.3. Comparing computational cost Table 1 shows computation time per pixel for SCC, SCC+EM, CC and SSD. They are almost of the same cost. In CC, we calculated the brightness variance of the template N 2 once in advance. SCC+EM excels SCC (fn − f) and CC in computation time, because masked pixels do not need to be multiplied. From this fact, we can expect the sum-of-product operation has the main e6ect for computational cost, and then, in the case of the images in Fig. 7,
(a)
(b)
(c)
(d)
(e)
(f )
(g)
Fig. 9. Registration under occlusion (640 × 320), (a) template image (120 × 92), (b) SCC (20% occlusion), (c) SCC (30%), (d) SCC (40%), (e) CC (20% occlusion), (f) CC (30%), (g) CC (40%).
S. Kaneko et al. / Pattern Recognition 36 (2003) 1165 – 1173 Table 2 SCC values at correct position. Values in brackets are calculated at matched positions
SCC CC
(b),(e)
(c),(f)
(d),(g)
0.82( – ) 0.69( – )
0.58( – ) 0.35(0.42)
0.58( – ) 0.36(0.42)
rSCC 0.6 0.4
134
0.2
129 0 124
412 417 119
422 427
rCC 0.6 0.4
134
0.2
129 0 124 Row
412 417 Column
119 422 427
Fig. 10. Pro3les of correlation values.
to test the brightness of objects which keeps even important information for image registration.
1171
4.2. Highlight Fig. 11 includes example images of glossy package including highlighted regions. The highlight or saturation due to intense illumination or its reKection can be often observed in real-world scenes, for example, many wrapped industrial products on production lines. Highlighted scenes can be dealt with by SCC as well as in the case of occlusion. Results of image registration by SCC and CC were shown by black frames in Fig. 11(a) and (b). In both the 3gures, SCC could report the correct objects while CC reported wrong portions of the similar distribution of brightness to the one of the template. ISC is known to have robustness as well as SCC has [12], but it sometimes fails to search the correct instances of a template in the case of poor texture like binary images as shown in Fig. 11. In such cases, SCC can be expected to have much more robustness even in comparison with ISC, because it is based on both robust coding of brightness increment and brightness. Fig. 11(c) and (d) show the enhanced mask-images corresponding to the scenes 1 and 2, respectively, where black pixels could capture highlighted regions. These examples reveal the robustness of SCC for highlighted object-images and for change in brightness occurred in object-images, which can be found through comparing the templates and matched object-images especially in Fig. 11(a) and (c). 4.3. On size of template-image The proposed similarity measure SCC is designed to improve CC by introducing a mask operation by IS which is a basis of the robustness for ill-conditions. In the ISC application, however, there have been the problem that because reduction of information amount through rejection of brightness it is somewhat hard to utilize template-images of small size. We should con3rm the characteristics of SCC for template size in comparison with the other criteria ISC and CC. Fig. 12 shows the original image and the degraded version which was derived from the original image through blur of 3 × 3 box 3ltering followed by additive noise of Gaussian
Fig. 11. Matching for glossy object, (a) scene 1, (b) scene 2,(c) enhanced mask 1, (d) enhanced mask 2.
1172
S. Kaneko et al. / Pattern Recognition 36 (2003) 1165 – 1173
Fig. 12. Data images for size problem, (a) original image, (b) degraded version.
extension or modi3cation of SCC is also interesting from the theoretical point of view and we will have to continue our consideration on this kind of modi3cation. In the 3eld of statistics, some robust estimators have been proposed [15]. SCC is related to M-estimators which utilize masking coe cients to stabilize the evaluation measure against outlying noise, and then the proposed method can provide a robust scheme for detecting essential similarity or consistency between pixels independent of their brightness. The robust estimators in statistics will give us some inspiration or motivation to design another robust measure for image registration. 6. Conclusions
1
Overlap rate
0.8 0.6 : ISC : CC : SCC
0.4 0.2 0 64 60
50 40 30 20 Template side length N
10
4
Fig. 13. Overlap ratio versus template size.
distribution of the mean 0 and the variance 64. The performance of registration is measured by the score of overlap ratio de3ned as a=A, where a and A are the overlapped area and the template size from 64 × 64 to 4 × 4, respectively. For each template size, 100 template-images were extracted randomly from the original image in Fig. 12(a) and all of them were searched over Fig. 12(b). Overlap ratios at the correct positions for each matching were reported and their means were depicted in Fig. 13. The curve of ISC retained its upper limit 1 and began to fall at around 48 and decreased monotonously, while the one of SCC kept the highest limit up to 28. In this experiment, since there was neither occlusion nor other irregularity, CC had the best performance and could keep the upper limit up to 16. Furthermore, the pro3les of SCC and CC are similar to each other, which indicate their similar characteristics for template size. According to this analysis, SCC can have somewhat intermediate performance for template size between ISC and CC. 5. Discussions The mask coe cients {cn } can be of real value between the interval [0; 1] in order to represent a level of consistency between corresponding pixels over the two images. This
An image registration method by the selective correlation coe cient named SCC, which has the nature of robustness for occlusion, highlight and ill-conditioned illuminations, has been proposed. Using basic mask-images based on increment signs, a brightness-based correlation is e6ectively improved, reducing inKuence of occlusion. Furthermore, a simple procedure for enhancing the basic mask-image has also been proposed. It enables us to enhance the di6erence between the mask-rates in occluded regions and the one in unoccluded regions, obtaining more stable performance of matching. It is able to estimate the degree of enhancement by use of a mathematical model. The computational cost of SCC is almost the same as the one of CC even if it includes a step of generating a basic mask-image ad enhancing. By the experiments with the real images, the excellent performance with robustness for occlusion and highlight was observed. References [1] A.K. Jain, R.P.W. Duin, J. Mao, Statistical pattern recognition: a review IEEE Trans. Pattern Anal. Machine Intell. 22 (1) (2000) 4–37. [2] D.H. Ballard, Generalizing the hough transform to detect arbitrary shapes, Pattern Recognition 13 (2) (1981) 111–122. [3] Y. Lamdan, H.J. Wolfson, Geometric hashing: a general and e cient model-based recognition scheme, Proc. ICCV (1988) 238–249. [4] S.X. Liao, M. Pawlak, On image analysis by moments, IEEE Trans. Pattern Anal. Machine Intell. 18 (3) (1996) 254–266. [5] M.J. Swain, D.H. Ballard, Color indexing, Internat. J. Comput. Vision 7 (1) (1991) 11–32. [6] J.C. Russ (Ed.), The Image Processing Handbook, CRC Press, Tokyo, 1994. [7] L.G. Brown, A survey of image registration techniques, ACM Computing Surveys 24 (4) (1992) 325–376. [8] J.K. Aggarwal, L.S. Davis, W.N. Martin, Correspondence processes in dynamic scene analysis, Proc. IEEE 69 (5) (1981) 562–571. [9] D.I. Barnea, H.F. Silverman, A class of algorithms for fast digital image registration, IEEE Trans. Comput. 21 (2) (1972) 179–186.
S. Kaneko et al. / Pattern Recognition 36 (2003) 1165 – 1173 [10] A. Venot, J.F. Lebruchec, J.C. Roucayrol, A new class of similarity measures for robust image registration, CVGIP 28 (1984) 176–184. [11] I. Murase, S. Kaneko, S. Igarashi, Robust matching by increment sign correlation, IEICE Trans. (D-II) 83-D-II (5) (2000) 1323–1331 (in Japanese). [12] S. Kaneko, I. Murase, S. Igarashi, Robust image registration by increment sign correlation, Pattern Recognition 35 (2002) 2223–2234.
1173
[13] D.N. Bhat, S.K. Nayar, Ordinal measures for image correspondence, IEEE Trans. Pattern Anal. Machine Intell. 20 (4) (1998) 415–423. [14] V. Lashkia, S. Kaneko, S. Aleshin, Textural region location in complex images using test feature classi3ers, Can. J. ECE 24 (2) (1999) 65–71. [15] P.J. Rousseeuw, A.M. Leroy, Robust Regression and Outlier Detection, Wiley, New York, 1987.
About the Author—SHUN’ICHI KANEKO received the B.S. degree in Precision Engineering and the M.S. degree in Information Engineering from Hokkaido University, Japan, in 1978 and 1980, respectively, and then the Ph.D. degree in Systems Engineering from the University of Tokyo, Japan, in 1990. He had been a research assistant of the Department of Computer Science since 1980 to 1991, an associate professor of the Department of Electronic Engineering since 1991 to 1995, and an associate professor of the Department of Bio-application and Systems Engineering since 1996 to 1996, in Tokyo University of Agriculture and Technology, Japan. He is an associate professor of the Department of Control and Information Engineering in Hokkaido University from 1996. He received the Best Paper Award in 1990, the Society Award in 1998, respectively, from Japan Society of Precision Engineering. His research interest includes machine vision, image sensing and understanding, robust image registration. He is a member of IEICE, JSPE, IEEJ, SICE and the IEEE computer society. About the Author—YUTAKA SATOH received the B.S. and M.S. degrees in Computer Science from Tokyo University of Agriculture and Technology, Japan, in 1996 and 1998, respectively, and then the Ph.D. degree in Systems Engineering from Hokkaido University, Japan, in 2001. He joined Softpia Japan as a researcher in the project of Human-Object Information Processing (HOIP) in 2001. He is a member of IEICE and JSPE. About the Author—SATORU IGARASHI received the B.S., M.S. and Ph.D. degrees in Precision Engineering from Hokkaido University, Japan, in 1963, 1965, and 1971, respectively. He is a professor of the Graduate School of Engineering in Hokkaido University. He received the Best Presentation Award from Japanese Society of Die and Mould Technology in 1995 and the Society Award from Japan Society of Precision Engineering in 1998. His research interest includes production system, mechanics, machine vision. He is a member of JSPE, SGE and the Society of Life Support Technology.
www.elsevier.com/locate/patcog
Using selective correlation coe cient for robust image registration Shun’ichi Kanekoa; ∗ , Yutaka Satohb , Satoru Igarashia a Department
of Control and Information Engineering, Graduate School of Engineering, Hokkaido University, N13-W8 Sapporo, Japan b Softopia Japan, 4-1-7, Kagano, Ogaki, Japan Received 23 February 2001; accepted 1 April 2002
Abstract A new method is proposed for robust image registration named Selective Correlation Coe)cient in order to search images under ill-conditioned illumination or partial occlusion. A correlation mask-image is generated for selecting pixels of an image before matching. The mask-image can be derived from a binary-coded increment sign-image de3ned from any object-image and the template. The mask-rate of occluded regions is theoretically expected to be 0.5, while unoccluded regions have much lower rate than 0.5. Robustness for ill-conditioned environment can be realized since inconsistent brightness of occluded regions can be omitted from the mask operation. Furthermore, the mask enhancement procedure is proposed to get more stable robustness. The e6ectiveness of masking increases by the procedure, resulting in the rate around 0.7 for masking of occluded regions. This paper includes a theoretical modeling and analysis of the proposed method and some experimental results with real images. ? 2002 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. Keywords: Normalized cross correlation; Selective correlation coe cient; Image registration; Template matching; Robust statistics
1. Introduction In pattern recognition, image registration techniques to check image similarity based on a rational quantitative measure and to locate a template at the place of its optimum value is one of the oldest and fundamental image processing [1]. These techniques construct an important foundation that should be a kernel for image understanding systems in a lot of 3elds such as visual inspection, robot vision, environment monitoring, and general-purpose image-based measurement systems. High-speed and inexpensive computing facility and image processing hardware, furthermore, can recently be of popular utilization, and rather expensive algorithms for image processing with numerous computation get to be feasible even in real-world applications. There have been several approaches for image registration, such as a feature-based approach, brightness-based or ∗
Corresponding author. Tel.=fax: +81-11-706-6436. E-mail address: [email protected] (S. Kaneko).
image-based approach, and an aggregated measure-based approach like color or gray level histograms. Many non-brightness feature-based approaches have been proposed to robust image matching. GHT [2] is based on detected local features utilized as voting electors that pose limited scores and vote them for models, while the indexing approaches [3] adopt combinations of geometric features, such as edge points or line segments as key for indexing objects in the dictionary and locally performs searching them to pick up candidates of the objects. But we cannot expect 3ne matching results without detecting effective features by steps of preprocessing which need some heuristic scheme for selection of threshold values and may have a kind of instability. Moment-based approaches [4] are e6ective for rotated object patterns, but they need much cost of computation and have less robustness for brightness change and occlusion. Color or histogram indexing methods [5] are another approach to rotation-invariant object search in real scene, but they are not robust for occlusion and brightness change.
0031-3203/03/$30.00 ? 2002 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. PII: S 0 0 3 1 - 3 2 0 3 ( 0 2 ) 0 0 0 8 1 - X
1166
S. Kaneko et al. / Pattern Recognition 36 (2003) 1165 – 1173
In this paper, we focus our attention to brightness-based methods for image registration because in order to aim high precision of location and rich robustness for various ill-conditions, the approach might be better than the other ones [6,7]. Normalized cross correlation, sum of square di6erence, statistical sign change, increment sign correlation, and rank statistics have been proposed as brightness-based template matching techniques so far. Normalized cross correlation (CC) [8,9] has been used very widely because it can cope with noisy images and uniform change in brightness. But, when the scene involves partial occlusion of objects or saturation (highlight), this sum-of-multiplication-based correlation technique sometimes fails to capture true positions, while it is good to measure some slight change in brightness. Sum of squared di6erence (SSD) or sum of absolute di6erence [9] evaluates a sum of squared or absolute di6erences in brightness de3ned between two distinct images. Though it is an e cient algorithm, it is very feeble for change in brightness caused by illumination Kuctuation, occlusion and saturation. Statistical sign change [10] is derived as the sum of the number of sign changes between every two adjacent pairs of pixels, one of which belongs to a template-image and the other of which comes from an object-image at the same position. The object-image of the maximum value is considered as the solution. It interestingly utilizes statistical characteristics of noise and is available for large noises rather uniformly distributed over images. However, neither local brightness change nor occlusion can be coped with by this measure. Deterministic images also force it to be improved by an ad hoc unreasonable algorithm. Increment sign correlation (ISC) [11,12] is developed as a robust method for image registration, which can cope with occlusion, shadow, saturation or highlight. This method is based on a similarity measure which is evaluated by aggregating increment sign codes at each pixel, which represent +=− signs between any two adjacent pixels. ISC is proved to be robust because it does not directly utilize brightness of pixels but their incremental tendency, and it is e cient even in comparison with CC. Due to the information reduction through neglecting brightness values, however, there have been some problems such that the size of a template cannot be reduced and it cannot discriminate slight changes in brightness of images. Robust similarity measure using rank statistics [13] is based on an alternative approach for brightness variation. This kind of similarity measure is based on the fact that brightness itself changes in the case of rather uniform variation in illumination, but their ranks or orders can remain unchanged. However, it is largely inKuenced by a not small change of local perturbation of ranks caused by occlusion, shading and saturation. In this paper, a novel method for robust image-registration or image matching based on a halftone image named selective correlation coe cient (SCC) is proposed in order to obtain robustness especially for occlusion and saturation, ill-conditioned illumination and partial occlusion. The pro-
posed method is designed as an improved CC with an enhancement by ISC which is so robust for the ill-conditions above mentioned. In this method, an increment sign (IS) image transformed from the original hale-tone image is used for masking irrelevant pixels to template pixels. Because of the statistical nature of IS codes, parameters are not necessary in order to realize the masking operation which can automatically select consistent pixels to the template ones and consequently decrease wrong inKuence to taking the CC between the template and an object-image, and then it becomes possible to 3nd defective regions in the object-image.
2. Selective correlation coecient 2.1. De7nition Simple subtraction and check of their absolute di6erence, for example, causes pseudoknowledge corresponding to brightness change in object-images due to illumination change which occurs often in real situations. In order to have robustness in image registration, we need to focus our attention to the nature of brightness change in the ill-condition. This is a motivation of this work. Fig. 1 shows the overview of the proposed scheme to calculate SCC. It generates a selecting mask for each pixel before calculating correlation. The mask can be derived from the binary-coded IS between an object-image and a template-image. Then correlation is calculated not only from the template and the object-image but also from the mask-image based on IS which can be assumed as pre-3ltering by consistency between matched pixels. Each pixel in the mask-image takes a value of ‘0’ or ‘1’ and consequently they can select pixels whose brightness should be adopted for similarity evaluation between two images. The normalized CC function or correlation coe cient (CC) is 3rst considered. A one-dimensional ordered list, into which a two-dimensional image is rearranged according to some order, is utilized in this paper for simplicity without any loss of generality. Images might be often decomposed into each row or column and then stacked in their natu−1 ral order. Let F = {fn }Ni=0 de3ne a template-image and −1 G = {gn }Ni=0 an object-image which is of the same size N as the template and may be extracted from the larger scene. CC is expressed by the formula N −1
N N n=0 (fn − f)(gn − g) rcc = ; N −1 N −1 N 2 N2 n=0 (fn − f) n=0 (gn − g)
(1)
where fN and gN mean the averages of brightness for F and G, respectively. SCC is de3ned as an extension of CC with a masking factor for every pixel pair by the equation
S. Kaneko et al. / Pattern Recognition 36 (2003) 1165 – 1173
Template
1167
Template
Masking
Object
*
*
Correlation
Mask(IS)
*
0 1 1 1 1 0 1 0
Scene
bn
1 0 0 1 0 1 1 1 Fig. 1. Overview of proposed method: SCC.
0
0
0
0
0
occluded pairs
below. N −1
N N n=0 cn (fn − f)(gn − g) : rscc = N −1 N −1 N 2 N2 n=0 cn (fn − f) n=0 cn (gn − g)
0
1
1
0 0 0 1 0 1 0 1 1 0 0 1 0 1 0 1
cn
0
0
1
1
1
1
1
1
matched pairs
Fig. 2. Codes bn and cn from images.
(2)
In this de3nition, brightness values fn and gn of the template and the object-image are multiplied by the coe cient cn , and then the sum of all the products is normalized by the standard deviations in each image, where the coe cients cn are also introduced. The mask coe cient or code cn reveals concordance of increment signs for check consistency between the template and the object-image in the same way as feature evaluation [14], and then is de3ned by 1 − |bn − bn | (n = 0 or even); cn = (3) (n = odd); cn−1 where the increment sign bn for F is de3ned similarly in Ref. [12] 1 (fn+1 ¿ fn ); bn = (4) 0 (otherwise) and bn is also de3ned for G as well. Increment signs bn and bn are binary digits which correspond to brightness change between adjacent pixels. The values ‘1’ and ‘0’, on the other hand, reveal non-negative and negative signs of brightness increment de3ned in succeeding neighborhood at each pixel. The de3nition of orientation in neighborhood pixels is not important for calculation of the codes. In this paper, we make one-dimensional brightness lists through horizontal decomposition of a two-dimensional image array and linking the rows into the list, and consequently horizontally adjacent pixels are compared. It is possible to adopt another way to tailor the list. The mask coe cient cn is constructed through the rule that it takes ‘1’ if bn and bn are concordant and ‘0’ otherwise. Consequently, the image M = {cn } can mask pixels of probable inconsistency out of the calculation of correlation following the mask operation. We call this mask
a ‘basic mask’ to distinguish the di6erence between this operation and the one which we will de3ne in conjunction with ‘enhancement operation’ described later. Fig. 2 shows schematic examples of the codes bn and cn . In this 3gure, we can 3nd typical examples of occlusion and brightness change due to shadow of the box in front of Winnie-the-pooh on the toy train. We assign the same value to two adjacent coe cients by de3nition, for example c0 = c1 or c14 = c15 , because for computation of an increment sign code we need two adjacent pixels and then in order to make cn independent, any two pixel pairs should not be overlapped. ISC has been proposed for robust image registration in the case of occlusion and=or change in illumination. The correlation can be modeled as a statistic that has a binomial or normal distribution under a reasonable assumption and approximation. From the statistical analysis and modeling, ISC has the following fundamental characteristics [11,12]: • The deviation is generally small. • The mean is the probability of sign reversal between bn and bn . • It has an expected value of mean 0.5 for uncorrelated images. • It has an expected value more than 0.5 for correlated images, which depends on the amount of noise included in the scene. These statistical characteristics of ISC guarantee the performance of the basic mask-image M which is expected to exclusively choose probably consistent pixels and otherwise remove inconsistent ones. The basic mask-rate of occluded regions is theoretically expected to be 0.5 because of the ISC characteristics for uncorrelated images. On the other hand, for unoccluded or similar regions, it can have the possible mask-rate below 0.5 in the case of correct pair of images.
1168
S. Kaneko et al. / Pattern Recognition 36 (2003) 1165 – 1173
0.223 (8998/40354) 0.504
( 6428 /12744 ( (a)
(b)
(c)
(d)
Fig. 3. An example of basic mask-image, (a) template (b) scene (c) mask-image: M , (d) mask-rate.
2.2. Characteristics of basic mask-image
2.4. Computational cost
Fig. 3 shows a real example of the basic mask-image. A template-image in Fig. 3(a) and the similar scene with occlusion in (b) are given, and they generate a basic mask-image in (c) where two distinct mask-rates can be observed and computed as shown in (d). The scene includes around 24% occluded area at the lower right corner. In the mask-image shown in Fig. 3(c), pixels in black show the value ‘0’, on the other hand pixels in white, the value ‘1’. In the similar region in the upper half and the lower left, the mask-rate is 0.223, while in the occluded region in the lower right the rate is 0.504 just as theoretically expected. We can recognize a probabilistic tendency to have much more masked pixels in occluded regions than in unoccluded ones.
The computational cost of SCC is examined in comparison with CC which has been very popular in many real applications in laboratory and industry. From Eqs. (1) and (2), SCC seems to need summations of three-term products while CC has the ones of two-terms. Because cn has ‘1’ or ‘0’ values in nature and the multiplication can be replaced by logical operation, the cost of multiplication is not essentially increased. SCC has a rather small number of products since it selects pixels from all the pixels. Then, the main issue of computational cost is the one for variance computation. In CC, the variance or the self-correlation of the template-image can be calculated only once in an oRine stage because it does not change throughout the calculation of a CC, but in SCC, three correlations should be calculated every time because they can be changed due to the coe cients cn . SCC needs an extra calculation for obtaining signs cn at every position. The cost, however, can be small and negligible because it depends on only comparing operations between two brightness values.
2.3. Stability of SCC In SCC, the amount of pixels which have contributed to the 3nal correlation value is lower than the one CC has. From the fundamental consideration in the previous section, we have to discuss about two kinds of e6ect of the mask operation: the e6ect in occluded regions and the one in other regions. First, in occluded regions, the mask-rate will be around 0.5, for example 0.504 in Fig. 3, and then a half of irrelevant pixels can be successfully excluded from correlation. This is a positive merit of SCC, however, the other half remains a wrong contribution to correlation, however, this might be the limitation of our SCC. Second, in normal regions, the mask-rate can be expected to be lower than the rate above mentioned, for example 0.223 in Fig. 3. This mask-rate varies mainly with two factors: magnitude of additive noise and contrast in the template [12]. In real situation, we should be certain of the reduction of pixels in normal regions. Reduction of pixels, for example small template-images, might generally harm statistical stability of the measure. In the case of SCC, however, we can assume for the pixels excluded by the mask operation with M to be inconsistent between the template and the object-image, therefore, the total performance should be evaluated through the real experiments.
3. Enhancement of mask-image 3.1. Basic procedure As also shown in Fig. 3, in real images, the mask-rates of unoccluded regions are generally more than 0 because of quantization error and=or additive noise introduced by an imaging process and illumination Kuctuation. On the other hand, in occluded regions, since the correlation is always low, inKuence of noise is small and the mask-rates are around 0.5 as expected. When noise is larger, however, the di6erence between them becomes smaller and then it should be improved through some enhancement procedure. The following procedure for mask enhancement is introduced in this paper. The procedure is fundamentally based on a majority rule as illustrated in Fig. 4. The 3gure shows the four-pixel majority rule in neighborhood. In this rule, there can be 3ve classes {C#k}k=0; 1; :::; 4 of pixel neighborhoods in M according to k pixels or codes having ‘0’. Pixels in the 3rst two classes in Fig. 4 are transformed to ‘1’ to
S. Kaneko et al. / Pattern Recognition 36 (2003) 1165 – 1173
C#0
0 0 1 1 0 1 1 0
C#1
0 1 0 0 1 1 0
0 1 0 0 1 1 0
0 0 1 1 0 1 0
0 0 1 1 0 1 0
0 0 0 1 1 0
0 1 0 0 1 0
0 1 0 0 1 0
0 1 0 0 1 0
0 1 0 0 1 0
0 0 1 1 0 0
C#3
0 0 0 0 1
0 1 0 0 0
C#4
0 0 0 0
C#2
0 1 0 0 0
0 0 0 1 0
1169
0.243 (9808/40354)
0 0 1 1 0 1 1 0
0.696
( 8876 /12744 ( Fig. 6. Enhanced mask-image.
0 0 0 0
Fig. 4. Mask enhancement scheme: four-pixel majority rule.
1
(a)
(b)
30%
60%
Fig. 7. Examples of occlusion, (a) template, (b) scene, 30%, 60%.
0.8
Pe
0.6
0.4
: pe2 : pe4 : pe6
0.2
0 0
0.2
0.4
0.6
0.8
1
P Fig. 5. Mask enhancement curves.
reduce masked pixels in unoccluded regions. Otherwise, all the pixels are transformed to the code ‘0’. The enhancement procedure consists of these two kinds of operations which realize reduction and accretion of the code ‘0’. Let p and pem (m = 2; 4; 6; : : :) be the basic mask-rate and the enhanced one based on the m-pixel majority rule, respectively. The rate pe2 , pe4 and pe6 , for example, can be obtained as follows: pe2 = 1 − (1 − p)2 ;
(5)
pe4 = 1 − (1 − p)4 − 4(1 − p)3 p;
(6)
pe6 = 1 − (1 − p)6 − 6(1 − p)5 p − 15(1 − p)4 p2 :
(7)
Fig. 5 shows the mathematical characteristics of the enhanced mask-rates. As shown in the 3gure, the four-pixel rule and the six-pixel rule have both the e6ects of reduction
and accretion, while the two-pixel rule always enhances any pixel with ‘0’. We adopt the four-pixel majority rule for mask enhancement for basic mask-images in this paper as considering tradeo6 of enhancement e6ect and spatial resolution. The larger the former is, in general, the smaller the latter becomes. Fig. 6 shows the enhanced version of the basic mask-image in Fig. 3. We notice that the mask-rate in the occluded region is enhanced from 0.504 to 0.696 as also shown the mask-image to the left, while the remaining region has very slight change in the rate from 0.223 to 0.243. Theoretical expectation in the e6ect of enhancement is 0.217 for 0.243 and 0.688 for 0.696 according to Eq. (7). These expectation can estimate the real values of the enhanced mask-rate. 3.2. Characteristics for occlusion In this section, we investigate the characteristics of SCC with regard to occlusion in template-images. As shown in Fig. 7, for the experiments, the template-image and the scene that were used in Fig. 3 were exchanged for generality. Occlusion was changed in degree from 0% to 100%. Fig. 8 shows the characteristics of SCC values versus occlusion. We use SCC+EM to call the selective correlation coe cient with enhancement. The values of SCC, CC and SCC+EM are reported for comparison. First, all of them have the same values of −0:3 at 100% occlusion, and so the original images can be considered as nearly uncorrelated. As occlusion increase, SCC+EM always holds the highest correlation and SCC also maintain high correlation compared with CC. The decreasing rates of both SCC and SCC+EM are lower than the one of CC, which reveal their robustness for occlusion.
1170
S. Kaneko et al. / Pattern Recognition 36 (2003) 1165 – 1173 1
0.8 Correlation
the generation of a basic mask-image consumes 15% of the total cost, the mask enhancement 5% and then the summation of products 80%. We call hereafter SCC+EM as SCC for simplicity.
: SCC+Enhanced mask : SCC+Mask : CC
0.6 0.4
4. Experiments
0.2
4.1. Occlusion
0
The e6ectiveness of SCC in the real scene containing occlusion and highlight is investigated in this section. Fig. 9 shows the result of the experiment with a simple template-image of a toy-bird. Occlusion increases from the left to the right scenes. SCC detected the right position while CC failed because of the occlusion from around 20% to 40%. Also shown in the right upper corners in each of the scenes in Fig. 9(b) – (d), SCC could make the enhanced mask-images that could almost correctly capture occluded regions. Table 2 shows the calculated values of SCC and CC, where we can 3nd that SCC could retain rather reasonable values while CC su6ered from occlusion in some cases, losing the correct position which reveals higher correlation values than the one at the correct position as shown in the table. The di6erences in correlation values range from 14% to 17% in CC. Fig. 10 indicates the pro3les of correlation values in the neighborhood of correct position in Fig. 9(b). In Fig. 10, the upper part shows the pro3le of SCC, in which the prominent highest peak could be recognized at the correct position, while in the lower of CC also seemed to have such kind of peak at the same part, however, it was not the maximum all over the scene. Since it has been known that ISC has similar characteristics of SCC [12], SCC is expected to have somewhat neutral nature between ISC and CC. Thus, we can utilize SCC in the case that a scene possibly has occlusion and simultaneously we want
-0.2 0
20
40 60 Occlusion Ratio (%)
80
100
Fig. 8. Correlation values versus occlusion.
Table 1 Comparing computational cost (s=pixel) SCC SCC+EM CC SSD
0.30 0.26 0.28 0.25
3.3. Comparing computational cost Table 1 shows computation time per pixel for SCC, SCC+EM, CC and SSD. They are almost of the same cost. In CC, we calculated the brightness variance of the template N 2 once in advance. SCC+EM excels SCC (fn − f) and CC in computation time, because masked pixels do not need to be multiplied. From this fact, we can expect the sum-of-product operation has the main e6ect for computational cost, and then, in the case of the images in Fig. 7,
(a)
(b)
(c)
(d)
(e)
(f )
(g)
Fig. 9. Registration under occlusion (640 × 320), (a) template image (120 × 92), (b) SCC (20% occlusion), (c) SCC (30%), (d) SCC (40%), (e) CC (20% occlusion), (f) CC (30%), (g) CC (40%).
S. Kaneko et al. / Pattern Recognition 36 (2003) 1165 – 1173 Table 2 SCC values at correct position. Values in brackets are calculated at matched positions
SCC CC
(b),(e)
(c),(f)
(d),(g)
0.82( – ) 0.69( – )
0.58( – ) 0.35(0.42)
0.58( – ) 0.36(0.42)
rSCC 0.6 0.4
134
0.2
129 0 124
412 417 119
422 427
rCC 0.6 0.4
134
0.2
129 0 124 Row
412 417 Column
119 422 427
Fig. 10. Pro3les of correlation values.
to test the brightness of objects which keeps even important information for image registration.
1171
4.2. Highlight Fig. 11 includes example images of glossy package including highlighted regions. The highlight or saturation due to intense illumination or its reKection can be often observed in real-world scenes, for example, many wrapped industrial products on production lines. Highlighted scenes can be dealt with by SCC as well as in the case of occlusion. Results of image registration by SCC and CC were shown by black frames in Fig. 11(a) and (b). In both the 3gures, SCC could report the correct objects while CC reported wrong portions of the similar distribution of brightness to the one of the template. ISC is known to have robustness as well as SCC has [12], but it sometimes fails to search the correct instances of a template in the case of poor texture like binary images as shown in Fig. 11. In such cases, SCC can be expected to have much more robustness even in comparison with ISC, because it is based on both robust coding of brightness increment and brightness. Fig. 11(c) and (d) show the enhanced mask-images corresponding to the scenes 1 and 2, respectively, where black pixels could capture highlighted regions. These examples reveal the robustness of SCC for highlighted object-images and for change in brightness occurred in object-images, which can be found through comparing the templates and matched object-images especially in Fig. 11(a) and (c). 4.3. On size of template-image The proposed similarity measure SCC is designed to improve CC by introducing a mask operation by IS which is a basis of the robustness for ill-conditions. In the ISC application, however, there have been the problem that because reduction of information amount through rejection of brightness it is somewhat hard to utilize template-images of small size. We should con3rm the characteristics of SCC for template size in comparison with the other criteria ISC and CC. Fig. 12 shows the original image and the degraded version which was derived from the original image through blur of 3 × 3 box 3ltering followed by additive noise of Gaussian
Fig. 11. Matching for glossy object, (a) scene 1, (b) scene 2,(c) enhanced mask 1, (d) enhanced mask 2.
1172
S. Kaneko et al. / Pattern Recognition 36 (2003) 1165 – 1173
Fig. 12. Data images for size problem, (a) original image, (b) degraded version.
extension or modi3cation of SCC is also interesting from the theoretical point of view and we will have to continue our consideration on this kind of modi3cation. In the 3eld of statistics, some robust estimators have been proposed [15]. SCC is related to M-estimators which utilize masking coe cients to stabilize the evaluation measure against outlying noise, and then the proposed method can provide a robust scheme for detecting essential similarity or consistency between pixels independent of their brightness. The robust estimators in statistics will give us some inspiration or motivation to design another robust measure for image registration. 6. Conclusions
1
Overlap rate
0.8 0.6 : ISC : CC : SCC
0.4 0.2 0 64 60
50 40 30 20 Template side length N
10
4
Fig. 13. Overlap ratio versus template size.
distribution of the mean 0 and the variance 64. The performance of registration is measured by the score of overlap ratio de3ned as a=A, where a and A are the overlapped area and the template size from 64 × 64 to 4 × 4, respectively. For each template size, 100 template-images were extracted randomly from the original image in Fig. 12(a) and all of them were searched over Fig. 12(b). Overlap ratios at the correct positions for each matching were reported and their means were depicted in Fig. 13. The curve of ISC retained its upper limit 1 and began to fall at around 48 and decreased monotonously, while the one of SCC kept the highest limit up to 28. In this experiment, since there was neither occlusion nor other irregularity, CC had the best performance and could keep the upper limit up to 16. Furthermore, the pro3les of SCC and CC are similar to each other, which indicate their similar characteristics for template size. According to this analysis, SCC can have somewhat intermediate performance for template size between ISC and CC. 5. Discussions The mask coe cients {cn } can be of real value between the interval [0; 1] in order to represent a level of consistency between corresponding pixels over the two images. This
An image registration method by the selective correlation coe cient named SCC, which has the nature of robustness for occlusion, highlight and ill-conditioned illuminations, has been proposed. Using basic mask-images based on increment signs, a brightness-based correlation is e6ectively improved, reducing inKuence of occlusion. Furthermore, a simple procedure for enhancing the basic mask-image has also been proposed. It enables us to enhance the di6erence between the mask-rates in occluded regions and the one in unoccluded regions, obtaining more stable performance of matching. It is able to estimate the degree of enhancement by use of a mathematical model. The computational cost of SCC is almost the same as the one of CC even if it includes a step of generating a basic mask-image ad enhancing. By the experiments with the real images, the excellent performance with robustness for occlusion and highlight was observed. References [1] A.K. Jain, R.P.W. Duin, J. Mao, Statistical pattern recognition: a review IEEE Trans. Pattern Anal. Machine Intell. 22 (1) (2000) 4–37. [2] D.H. Ballard, Generalizing the hough transform to detect arbitrary shapes, Pattern Recognition 13 (2) (1981) 111–122. [3] Y. Lamdan, H.J. Wolfson, Geometric hashing: a general and e cient model-based recognition scheme, Proc. ICCV (1988) 238–249. [4] S.X. Liao, M. Pawlak, On image analysis by moments, IEEE Trans. Pattern Anal. Machine Intell. 18 (3) (1996) 254–266. [5] M.J. Swain, D.H. Ballard, Color indexing, Internat. J. Comput. Vision 7 (1) (1991) 11–32. [6] J.C. Russ (Ed.), The Image Processing Handbook, CRC Press, Tokyo, 1994. [7] L.G. Brown, A survey of image registration techniques, ACM Computing Surveys 24 (4) (1992) 325–376. [8] J.K. Aggarwal, L.S. Davis, W.N. Martin, Correspondence processes in dynamic scene analysis, Proc. IEEE 69 (5) (1981) 562–571. [9] D.I. Barnea, H.F. Silverman, A class of algorithms for fast digital image registration, IEEE Trans. Comput. 21 (2) (1972) 179–186.
S. Kaneko et al. / Pattern Recognition 36 (2003) 1165 – 1173 [10] A. Venot, J.F. Lebruchec, J.C. Roucayrol, A new class of similarity measures for robust image registration, CVGIP 28 (1984) 176–184. [11] I. Murase, S. Kaneko, S. Igarashi, Robust matching by increment sign correlation, IEICE Trans. (D-II) 83-D-II (5) (2000) 1323–1331 (in Japanese). [12] S. Kaneko, I. Murase, S. Igarashi, Robust image registration by increment sign correlation, Pattern Recognition 35 (2002) 2223–2234.
1173
[13] D.N. Bhat, S.K. Nayar, Ordinal measures for image correspondence, IEEE Trans. Pattern Anal. Machine Intell. 20 (4) (1998) 415–423. [14] V. Lashkia, S. Kaneko, S. Aleshin, Textural region location in complex images using test feature classi3ers, Can. J. ECE 24 (2) (1999) 65–71. [15] P.J. Rousseeuw, A.M. Leroy, Robust Regression and Outlier Detection, Wiley, New York, 1987.
About the Author—SHUN’ICHI KANEKO received the B.S. degree in Precision Engineering and the M.S. degree in Information Engineering from Hokkaido University, Japan, in 1978 and 1980, respectively, and then the Ph.D. degree in Systems Engineering from the University of Tokyo, Japan, in 1990. He had been a research assistant of the Department of Computer Science since 1980 to 1991, an associate professor of the Department of Electronic Engineering since 1991 to 1995, and an associate professor of the Department of Bio-application and Systems Engineering since 1996 to 1996, in Tokyo University of Agriculture and Technology, Japan. He is an associate professor of the Department of Control and Information Engineering in Hokkaido University from 1996. He received the Best Paper Award in 1990, the Society Award in 1998, respectively, from Japan Society of Precision Engineering. His research interest includes machine vision, image sensing and understanding, robust image registration. He is a member of IEICE, JSPE, IEEJ, SICE and the IEEE computer society. About the Author—YUTAKA SATOH received the B.S. and M.S. degrees in Computer Science from Tokyo University of Agriculture and Technology, Japan, in 1996 and 1998, respectively, and then the Ph.D. degree in Systems Engineering from Hokkaido University, Japan, in 2001. He joined Softpia Japan as a researcher in the project of Human-Object Information Processing (HOIP) in 2001. He is a member of IEICE and JSPE. About the Author—SATORU IGARASHI received the B.S., M.S. and Ph.D. degrees in Precision Engineering from Hokkaido University, Japan, in 1963, 1965, and 1971, respectively. He is a professor of the Graduate School of Engineering in Hokkaido University. He received the Best Presentation Award from Japanese Society of Die and Mould Technology in 1995 and the Society Award from Japan Society of Precision Engineering in 1998. His research interest includes production system, mechanics, machine vision. He is a member of JSPE, SGE and the Society of Life Support Technology.