- Email: [email protected]
Rough surface classification using point statistics from photometric stereo
Pattern Recognition Letters 21 (2000) 593±604
www.elsevier.nl/locate/patrec
Rough surface classi®cation using point statistics from photometric stereo G. McGunnigle *, M.J. Chantler Department of Computing and Electrical Engineering, Heriot Watt University, Riccarton, Edinburgh EH14 4AS, UK Received 21 May 1999; received in revised form 16 February 2000
Abstract Rough surfaces can be classi®ed by the point statistics of their derivative ®elds, estimated using photometric stereo. Such a scheme is proposed and found to be more accurate and robust than image-intensity-based classi®cation. It is particularly eective when applied to directional surfaces, even under rotation. The scheme is therefore robust and economic ± suitable for many applications and worthy of further investigation. Ó 2000 Elsevier Science B.V. All rights reserved. Keywords: Surface classi®cation; Texture analysis; Rotation invariance; Photometric stereo
1. Introduction This correspondence is concerned with the segmentation of rough surfaces into regions that vary in either the degree, or type, of surface roughness. Our thesis is that it is preferable to classify on the basis of the surface derivatives estimates rather than image intensities. A technique that discriminates between regions on the basis of estimates of their roughness and directionality is proposed. It uses the point, or zero lag, statistics of the estimated surface partial derivatives for classi®cation. Using statistics extracted from the ®rstorder probability density function is by de®nition less powerful than using statistics derived from the second- or higher-order densities. However, it is
*
Corresponding author. Tel.: +44-131-449-5111; fax: +44131-451-3327. E-mail address: [email protected] (G. McGunnigle).
also computationally cheaper and resort to higherorder statistics should only be made after point statistics have proven to be inadequate. In inspection tasks, where imaging conditions are strictly controlled, we believe that there is a good case for a classi®er based on point statistics. Since the proposed scheme is pixel-wise in nature, we will compare it with a pixel-wise scheme based on image intensity. First, we use simulation to investigate the validity of our assertions. This technique allows a degree of controllability over the surfaces and imaging process that would be dicult to obtain with real data. The most serious and signi®cant assertion ± the rotation invariance of our scheme ± is veri®ed using real data. This correspondence reports promising results for the proposed technique. The scheme oers signi®cant improvements in the levels of accuracy and robustness for discrimination between isotropic surfaces. Moreover, greatest improvement is oered in the case of directional surfaces,
0167-8655/00/$ - see front matter Ó 2000 Elsevier Science B.V. All rights reserved. PII: S 0 1 6 7 - 8 6 5 5 ( 0 0 ) 0 0 0 2 4 - 6
594
G. McGunnigle, M.J. Chantler / Pattern Recognition Letters 21 (2000) 593±604
especially when these are subject to rotations that are a priori unknown to the system.
Ssay x
ksay ; x3
Smal x kmal
2. Theory We present a model for the statistics of an image formed by a rough surface. This model will be used to develop two classi®ers used in the experimental section. 2.1. Image formation Several surface topography models have been proposed that are de®ned in terms of the power spectrum. Three such models will be used to generate test surfaces in this paper: the two-dimensional forms of the Sayles (Sayles and Thomas, 1978) Mulvanney (Mulvanney et al., 1989) and Ripple (Linnett, 1991) models are shown in Eqs. (1)±(3), respectively. The Sayles and Mulvanney surfaces are isotropic, the Ripple texture is directional. Images of these surfaces rendered with a Lambertian model are shown in Fig. 1. The test surfaces are assumed to have random phase. The height distributions are therefore Gaussian as a consequence of the central limit theorem and their broadband spectra. The assumption that surface height distributions are Gaussian is common in the rough surface theoretical literature. Experimental results suggest that though not universal, the Gaussian assumption is valid for many natural surfaces (Ogilvy, 1991, p. 11):
1 x2 1 x2c
ÿ3=2 ;
2
where S x is the two-dimensional power spectrum, x the radial frequency, xc the cut-o frequency, ksay and kmal are constants; krip Srip u; v q3 ; 2 2 u ÿ xc v ÿ xd
3
where u and v are the Cartesian frequency coordinates, xc and xd the cut-o frequencies in the x and y directions, krip is a constant. Since we are interested in the images of surfaces, we are more concerned with facet slope distribution than with the height distribution. We use two slope parameters, prms and qrms , corresponding to the standard deviation (S.D.) of the surface partial derivatives, p and q, measured along the x and y axes, respectively. If we assume the surface is globally ¯at, then p and q will both have mean values of zero. Since dierentiation is a linear operation p and q follow a bivariate Gaussian distribution: 1 1 T ÿ1 m exp ÿ C m ; 4 S p; q 2 2 2p jCj where 2 C4
p2
pq
pq
q2
3 5;
m
Fig. 1. Rendered Sayles, Mulvanney and Ripple test surfaces.
p : q
5
G. McGunnigle, M.J. Chantler / Pattern Recognition Letters 21 (2000) 593±604
595
Fig. 2. Distribution of facet slope for Sayles, Mulvanney and Ripple test surfaces.
The slope distributions for the test surfaces are shown in Fig. 2. The isotropic surfaces have equal and uncorrelated slope parameters and a completely symmetrical distribution. For the purposes of surface description we shall assume that, in the case of directional surfaces, the surface directionality is aligned with one of the coordinate axes. In this case the slope parameters are unequal and the random variables uncorrelated. The slope distribution of the directional surface is asymmetrical. This paper is restricted to surfaces with a Lambertian re¯ectance function. The intensity i of a surface facet with partial derivatives p and q with re¯ectivity q, illuminated from a tilt angle of s and a slant angle of r is given by
The Lambertian re¯ectance function may be linearised over a limited region of the pq plane (Kube and Pentland, 1988). Most surfaces consist largely of facets with moderate slopes and can be accurately rendered using a linear approximation to a limited region of the re¯ectance map, centred on p 0 and q 0. In this case the re¯ectance map can be approximated using (9), assuming q is equal to unity. Furthermore, the assumptions of a Gaussian height distribution and a linear re¯ectance function predict the distribution of image intensity will be Gaussian:
i p; q; q; s; r
Now, assume the surface is rotated such that the axes of the distribution are aligned with the coordinate axes. In this case the bivariate slope distribution may be resolved into the product of two one-dimensional probability density functions, (10). Since intensity is the sum of linear functions of p and q, the intensity distribution can be obtained by convolving the distributions of those functions. Since both distributions are normal, the intensity distribution will be normal with variance equal to the sum of variances of the original functions. The intensity distribution will have a mean (11) and a S.D. as shown in (12): 1 p2 S p; q exp ÿ 0:5 2 2pprms qrms prms 2 q ; 10 exp ÿ 0:5 2 qrms
q
ÿp cos s sin r ÿ q sin s sin r cos r p : p2 q2 1
6
The re¯ectance function may be expressed using a re¯ectance map r p; q. This assumes ®xed illumination and viewing geometry and expresses observed intensity as a function of the surface partial derivatives, it is therefore de®ned on the same plane as the scatter plots shown in Fig. 2. By combining (4) and (6) we can predict the point statistics of the surface image, Eqs. (7) and (8): Z Z r p; qS p; q dp dq; 7 li r2i
Z Z
2
r p; q ÿ li S p; q dp dq:
8
r p; q ÿp cos s sin r ÿ q sin s sin r cos r: 9
596
G. McGunnigle, M.J. Chantler / Pattern Recognition Letters 21 (2000) 593±604
li cos r;
11
qÿ 2 cos2 s q2rms sin2 s : ri sin r prms
12
2.2. Surface classi®cation on image intensity Now consider two regions with identical surface re¯ectivities, q 1, illuminated from the same direction, s; r that dier in their topographical characteristics. The images of both surfaces will have a common mean (11) and will only dier in terms of their S.D. The image may be segmented by classifying each pixel on the basis of the estimated S.D. of the surrounding region. In practice, this can be estimated by squaring, low pass ®ltering and taking the square root of each pixel value, and is similar to post-®ltering used in many texture analysis schemes, e.g., Laws (1979). In many applications, the inspection system may be presented with a surface with an arbitrarily oriented directionality. If a surface is rotated, then its slope distribution will also rotate. In the case of directional surfaces, this will aect the value of the integral, (8). That is, for a directional surface, the variance of image intensity is dependent on the orientation of the surface. The quantitative eect can be clari®ed without loss of generality if, instead of rotating the slope distribution by an angle / we rotate the re¯ectance map by an angle ÿ/. For an isotropic surface, prms qrms , and the rotational terms in (12) sum to one, regardless of the surface orientation. For a directional surface, prms kqrms , and the sum of the rotational terms in (12) is still a function of surface orientation. That is, in the case of a directional surface expression (12) may be modi®ed to the form shown in (13). This eect of rotation of the Ripple test surface on the S.D. of image intensity is shown in Fig. 3. From this it can be seen that image variance is not a surface rotation invariant feature for directional surfaces: q ÿ ri qrms sin r k 2 cos2 s ÿ / sin2 s ÿ / : 13
Fig. 3. The variation of image S.D. with Ripple surface rotation.
2.3. Using photometric stereo for surface classi®cation If the topography of the surface can be estimated, it may be possible to improve on the performance of the intensity classi®er instead by classifying on the surface derivatives. Shape from shading is an under-constrained problem, most algorithms overcome this diculty by imposing a smoothness constraint on the recovered surface. This is clearly not appropriate in this application. Photometric stereo (Woodham, 1980) constrains the problem by using several images of the same scene under dierent illuminantion orientations. Originally developed for Lambertian surfaces the technique had been extended to surfaces with arbitrary re¯ectance functions (Rajaram et al., 1995) and more recently to surfaces where the re¯ectance function was unknown a priori (Kay and Caelli, 1995). When applied to Lambertian surfaces photometric stereo estimates three quantities: q the facet albedo and the partial derivatives p and q. In this paper, we will retain the Lambertian assumption and use the simple photometric algorithm described in (McGunnigle and Chantler, 1997). This algorithm could be replaced by a more complex scheme with less restrictive assumptions. However, any photometric technique will be restricted to surfaces which do not have signi®cant self or cast shadowing. Very rough surfaces will therefore have to be illuminated from high slant
G. McGunnigle, M.J. Chantler / Pattern Recognition Letters 21 (2000) 593±604
angles, with some loss in the ability of the algorithm to discriminate between facets of similar gradient. The ®rst aim of this paper is to ascertain whether the extra information obtained using photometric stereo improves the accuracy of surface classi®cation over that of the intensity classi®er. An improvement may take the form of either a reduction in the Bayes error or an improvement in the robustness of the classi®er to some variable. As with image intensity, if we use partial derivatives we will have to discriminate between distributions that share a common mean. A further complication is that we wish our classi®er to be rotation invariant. If the surface rotates, the slope distribution will also rotate. We adopt the following approach; at each pixel, a local estimate of the bivariate distribution is made. The eigenvalues of this distribution are then calculated and these form the basis for classi®cation. The approach is illustrated in Fig. 4. Although the distributions may rotate the eigenvalues will be constant. A classi®er based on this approach, which we shall call a k classi®er, should therefore be rotation invariant. The absolute values of the eigenvalues give a measure of the roughness of the surface. Their relative values give an indication of the degree of the surface directionality.
597
3. Synthetic experiments Two groups of experiments were carried out. In the ®rst group, the ability of the proposed classi®er to discriminate between surfaces with dierent degrees of roughness was compared with a classi®er based on a local estimate the S.D. of pixel intensity. The classi®ers' robustness to white noise was assessed. In the second group, the classi®ers' ability to discriminate on the basis of surface directionality was compared. The classi®ers' robustness to surface rotation was also assessed. 3.1. Surface roughness Two isotropic Sayles surfaces were generated, the ®rst was scaled to have an rms slope of 0.25, the rms slope of the other was varied in the course of the experiment. The visual eect to scaling the surface is shown in Fig. 5. The intensity and k classi®ers were applied to discriminate between the surfaces. The results shown in Fig. 6 show that both the intensity and k classi®ers are able to discriminate between surfaces that dier signi®cantly
Fig. 5. Gradated increase in roughness 0±0.5.
Fig. 4. Physical signi®cance of proposed features.
Fig. 6. The variation of classi®cation accuracy with the rms slope of the second test surface.
598
G. McGunnigle, M.J. Chantler / Pattern Recognition Letters 21 (2000) 593±604
in roughness. However, for dicult classi®cations, i.e., where the surfaces are very similar, the k classi®er does oer a signi®cant improvement in classi®cation accuracy. Since the features for the classi®ers are calculated in dierent ways, it is possible that the classi®ers will be aected by noise to dierent degrees. In this experiment the surfaces have rms slopes of 0.25 and 0.23, respectively, and the S/N ratio is varied. The results, Fig. 7, show that both classi®ers are able to function at low S/N levels and that the photometric classi®er is still superior. The ®nal experiment in this group is complementary to the ®rst; in this case roughness is varied indirectly by varying the spectrum of one of the surfaces. Where the break frequency is low the Mulvanney model approximates Sayle's model. Sayles noted that many surfaces have their longer wavelengths attenuated by surface processing (Sayles and Thomas, 1978). In the third experiment, varying degrees of processing were simulated by varying the break frequency of a Mulvanney surface. The visual eect of varying the cut-o frequency is shown in Fig. 8. The classi®ers were used to discriminate between two surfaces, the
®rst had a break frequency of 32 cycles and rms slope 0.2, the break frequency of the second surface was varied and the surface was rescaled so that the magnitudes of the high frequency components were equal to those in the ®rst surface. The results are shown in Fig. 9. Once again the classi®ers show a similar dependency on the variable, with the k classi®er giving improved classi®cation accuracy. 3.2. Surface directionality The directionality of a surface is an important cue to its identi®cation, however, in many applications the orientation of the surface is not under the control of the designer. In this section, two experiments are described, the ®rst compares the ability of the classi®ers to discriminate on directionality alone, the second assesses the robustness of the classi®er to surface rotation. The isotropic Sayles surface is a special case of the Ripple surface with a centre frequency of zero cycles per image (c/i). As the centre frequency increases, so does the directionality of the surface. In the next simulation we compare the ability of the classi®ers to discriminate on surface directionality. Two Ripple surfaces are used, the ®rst has a centre frequency of 0 c/i, the frequency of the second is increased from 0 to 32 c/i over the course of the experiment (see Figs. 10 and 11). However, throughout the experiment, the second surface is
Fig. 7. The robustness of the classi®ers to white noise.
Fig. 8. Gradated increase in cut-o frequency.
Fig. 9. The eect of varying cut-o frequency on classi®cation accuracy.
G. McGunnigle, M.J. Chantler / Pattern Recognition Letters 21 (2000) 593±604
599
scaled so that its rms slope measured along its principal directionality is equal to that of the ®rst surface. The experiment is performed twice: in the ®rst the grain of the material runs vertically ± perpendicular to the illuminant vector and in the second it runs horizontally ± collinear with the illuminant. The results presented in Fig. 12 show the k classi®er to have a performance that is equal to that of the intensity classi®er in the ®rst experiment. However, if the surface is rotated 90°, and the experiment repeated, the performance of the intensity classi®er is signi®cantly reduced. This degradation is due to the fact that the illuminant is collinear with the surface grain, and the directionality visible in the image is attenuated. This is
re¯ected in a reduction in the S.D. of image intensity. Since the k classi®er operates on an estimate of properties of the surface, it is unaected by the interaction of the illuminant and surface directionality, it is therefore capable of discriminating on surface directionality, regardless of its orientation. The previous experiment highlighted that a classi®er based on intensity is sensitive to the orientation of the surface; the k classi®er in contrast, appears to be much more robust to the surface orientation. To test whether the intensity classi®er is rotation dependent and whether the k is rotation invariant we measure the robustness of the intensity and k features to rotation of the Ripple surface, Fig. 13. The intensity feature varies signi®cantly, however, the k features are much more stable and therefore represent a more robust basis for a classi®er. In order to ascertain the eect of surface rotation on surface classi®cation we conduct our ®nal simulation. Both classi®ers are trained, at 0° of rotation, to discriminate between an isotropic surface with prms 0:2 and a directional surface with prms 0:25 and qrms 0:1. The directional surface was then rotated in 10° increments and the classi®ers applied, without retraining, to segment the image, the results are shown in Fig. 14. The results indicate that the intensity classi®er is highly sensitive to surface rotation whereas the k classi®er is rotation invariant.
Fig. 12. The variation of classi®cation accuracy with centre frequency of the second surface.
Fig. 13. The variation of feature means with rotation of the Ripple surface.
Fig. 10. Gradated increase in centre frequency, surface illuminated from s 0°.
Fig. 11. The same surface illuminated from s 90°.
600
G. McGunnigle, M.J. Chantler / Pattern Recognition Letters 21 (2000) 593±604
Fig. 14. The variation of k and intensity classi®cation accuracy with rotation.
4. Experiments on real data The most signi®cant result of the synthetic experiments was that the intensity classi®er is not
robust to the rotation of directional surfaces, whereas the k classi®er is. In this section, we describe a series of experiments carried out on two pairs of real, directional surfaces. The ®rst pair Striate and Slate are naturally occurring rock surfaces. The second pair Ground 1 and Ground 2 are manmade surfaces formed by grinding a ¯at plaster surface. By varying the length of grinding stroke it is possible to vary the directionality of the surface. The ®rst pair are shown under 0°, 45° and 90° of rotation in Figs. 15 and 16, the visual eect of the change in the relative orientation of surface directionality and illuminant is clear. In the case of Striate the grain of the surface is originally perpendicular to the illuminant, as the surface is rotated and the surface grain becomes collinear with the illuminant tilt direction, the directionality visible in the image is attenuated. In the case of Slate, the surface directionality is initially collinear with the illuminant vector and only becomes visible
Fig. 15. Striate surface at 0°, 45° and 90° of rotation.
Fig. 16. Slate surface at 0°, 45° and 90°.
G. McGunnigle, M.J. Chantler / Pattern Recognition Letters 21 (2000) 593±604
when the surface is rotated. This is re¯ected in the change of the S.D. of image intensity with surface rotation. The textures Ground 1 and Ground 2 also exhibit this phenomenon, Figs. 17 and 18. If we examine the distributions of slopes estimated from the Striate and Slate surfaces, Figs. 19 and 20, we can see that the distributions re¯ect the
601
directionality of the surfaces and their relative roughnesses. Rotation of the surface leads to a corresponding rotation of the slope distribution. If we plot the k features (Fig. 21), we can see that these are much more stable than the intensitybased features, Fig. 22. This is borne out by the classi®cation results, Fig. 23: the intensity classi®er
Fig. 17. Ground 1 surface 0°, 45° and 90°.
Fig. 18. Ground 2 surface 0°, 45° and 90°.
Fig. 19. Scatter plot of the estimated partial derivatives of the Striate surface at 0°, 45° and 90° of rotation.
602
G. McGunnigle, M.J. Chantler / Pattern Recognition Letters 21 (2000) 593±604
Fig. 20. Scatter plots of partial derivatives estimated from Slate surface at 0°, 45° and 90° of rotation.
Fig. 21. The variation with rotation of k feature mean for Striate and Slate surfaces.
Fig. 23. Comparison of classi®ers for rotation of Striate and Slate surfaces.
Fig. 22. The variation with rotation of intensity feature mean for Striate and Slate surfaces.
Fig. 24. The variation with rotation of intensity feature mean for Ground 1 and Ground 2 surfaces.
G. McGunnigle, M.J. Chantler / Pattern Recognition Letters 21 (2000) 593±604
Fig. 25. The variation with rotation of k feature mean for Ground 1 and Ground 2 surfaces.
603
conditions can be strictly controlled, the point statistics of either the image, or the surface, are often sucient for accurate surface segmentation. This paper has highlighted one serious problem with classi®ers that classify rough surfaces on the basis of their image: image-intensity-based features are not robust to the rotation of directional surfaces. In this correspondence, a simple scheme that extracts rotation invariant statistics from photometric estimates of the surface derivatives was proposed. This scheme was found to be rotation invariant and was also found to oer improved classi®cation for isotropic surfaces. Set against these advantages, the proposed scheme does require additional resources. The computational requirements of our simple photometric scheme and the principal components analysis are small relative to the other operations; the most signi®cant increase is due to the fact that it is necessary to low pass ®lter three ®elds instead of just one for the intensity classi®er. In any case, the computational requirements are still modest compared to those of a second-order classi®er. However, in practice the main overhead of this scheme is the requirement for photometric image sets. The authors believe this is the most signi®cant restriction to this approach.
References Fig. 26. Comparison of classi®ers for rotation of surfaces.
rapidly fails as the surfaces are rotated, however the k classi®er is able to maintain a low level of misclassi®cation for all angles of rotation. We repeated the previous experiments with the manmade surfaces and obtained similar results: Figs. 24±26.
5. Discussion This correspondence has described an approach to rough surface classi®cation. It is our argument that in an inspection task, i.e., where the imaging
Kay, G., Caelli, T., 1995. Estimating the parameters of an illumination model using photometric stereo. Graphical Models and Image Processing 57 (5), 365±388. Kube, P., Pentland, A.P., 1988. On the imaging of fractal surfaces. IEEE Trans. Pattern Analysis Machine Intelligence 10 (5), 704±707. Laws, K.I., 1979. Texture energy measures. In: Proceedings of the Workshop on Image Understanding, pp. 47±51. Linnett, L.M., 1991. Multi-texture image segementation. Ph.D. thesis, Department of Electrical and Electronic Engineering, Heriot Watt University. McGunnigle, G., Chantler, M.J., 1997. A Model-based technique for the classi®cation of textured surfaces with illuminant direction invariance. In: Clark, A.F. (Ed.), BMVC97, Proceedings of the Conference on British Machine Vision, Vol. 2, pp. 470±480. Mulvanney, D.J., Newland, D.E., Gill, K.F., 1989. A complete description of surface texture pro®les. Wear 132, 173±182. Ogilvy, J.A., 1991. Theory of Wave Scattering from Random Rough Surfaces. Adam Hilger, Bristol.
604
G. McGunnigle, M.J. Chantler / Pattern Recognition Letters 21 (2000) 593±604
Rajaram, K.V., Parthasarth, G., Faruqui, M.A., 1995. A neural network approach to photometric stereo inversion of real-world re¯ectance maps for extracting 3-D shapes of objects. IEEE Trans. Systems, Man Cybernet. 25 (9), 1289±1300.
Sayles, R.S., Thomas, T.R., 1978. Surface topography as a nonstationary random process. Nature 271 (2), 431±434. Woodham, R., 1980. Photometric method for determining surface orientation from multiple images. Opt. Eng. 19 (1), 139±144.
www.elsevier.nl/locate/patrec
Rough surface classi®cation using point statistics from photometric stereo G. McGunnigle *, M.J. Chantler Department of Computing and Electrical Engineering, Heriot Watt University, Riccarton, Edinburgh EH14 4AS, UK Received 21 May 1999; received in revised form 16 February 2000
Abstract Rough surfaces can be classi®ed by the point statistics of their derivative ®elds, estimated using photometric stereo. Such a scheme is proposed and found to be more accurate and robust than image-intensity-based classi®cation. It is particularly eective when applied to directional surfaces, even under rotation. The scheme is therefore robust and economic ± suitable for many applications and worthy of further investigation. Ó 2000 Elsevier Science B.V. All rights reserved. Keywords: Surface classi®cation; Texture analysis; Rotation invariance; Photometric stereo
1. Introduction This correspondence is concerned with the segmentation of rough surfaces into regions that vary in either the degree, or type, of surface roughness. Our thesis is that it is preferable to classify on the basis of the surface derivatives estimates rather than image intensities. A technique that discriminates between regions on the basis of estimates of their roughness and directionality is proposed. It uses the point, or zero lag, statistics of the estimated surface partial derivatives for classi®cation. Using statistics extracted from the ®rstorder probability density function is by de®nition less powerful than using statistics derived from the second- or higher-order densities. However, it is
*
Corresponding author. Tel.: +44-131-449-5111; fax: +44131-451-3327. E-mail address: [email protected] (G. McGunnigle).
also computationally cheaper and resort to higherorder statistics should only be made after point statistics have proven to be inadequate. In inspection tasks, where imaging conditions are strictly controlled, we believe that there is a good case for a classi®er based on point statistics. Since the proposed scheme is pixel-wise in nature, we will compare it with a pixel-wise scheme based on image intensity. First, we use simulation to investigate the validity of our assertions. This technique allows a degree of controllability over the surfaces and imaging process that would be dicult to obtain with real data. The most serious and signi®cant assertion ± the rotation invariance of our scheme ± is veri®ed using real data. This correspondence reports promising results for the proposed technique. The scheme oers signi®cant improvements in the levels of accuracy and robustness for discrimination between isotropic surfaces. Moreover, greatest improvement is oered in the case of directional surfaces,
0167-8655/00/$ - see front matter Ó 2000 Elsevier Science B.V. All rights reserved. PII: S 0 1 6 7 - 8 6 5 5 ( 0 0 ) 0 0 0 2 4 - 6
594
G. McGunnigle, M.J. Chantler / Pattern Recognition Letters 21 (2000) 593±604
especially when these are subject to rotations that are a priori unknown to the system.
Ssay x
ksay ; x3
Smal x kmal
2. Theory We present a model for the statistics of an image formed by a rough surface. This model will be used to develop two classi®ers used in the experimental section. 2.1. Image formation Several surface topography models have been proposed that are de®ned in terms of the power spectrum. Three such models will be used to generate test surfaces in this paper: the two-dimensional forms of the Sayles (Sayles and Thomas, 1978) Mulvanney (Mulvanney et al., 1989) and Ripple (Linnett, 1991) models are shown in Eqs. (1)±(3), respectively. The Sayles and Mulvanney surfaces are isotropic, the Ripple texture is directional. Images of these surfaces rendered with a Lambertian model are shown in Fig. 1. The test surfaces are assumed to have random phase. The height distributions are therefore Gaussian as a consequence of the central limit theorem and their broadband spectra. The assumption that surface height distributions are Gaussian is common in the rough surface theoretical literature. Experimental results suggest that though not universal, the Gaussian assumption is valid for many natural surfaces (Ogilvy, 1991, p. 11):
1 x2 1 x2c
ÿ3=2 ;
2
where S x is the two-dimensional power spectrum, x the radial frequency, xc the cut-o frequency, ksay and kmal are constants; krip Srip u; v q3 ; 2 2 u ÿ xc v ÿ xd
3
where u and v are the Cartesian frequency coordinates, xc and xd the cut-o frequencies in the x and y directions, krip is a constant. Since we are interested in the images of surfaces, we are more concerned with facet slope distribution than with the height distribution. We use two slope parameters, prms and qrms , corresponding to the standard deviation (S.D.) of the surface partial derivatives, p and q, measured along the x and y axes, respectively. If we assume the surface is globally ¯at, then p and q will both have mean values of zero. Since dierentiation is a linear operation p and q follow a bivariate Gaussian distribution: 1 1 T ÿ1 m exp ÿ C m ; 4 S p; q 2 2 2p jCj where 2 C4
p2
pq
pq
q2
3 5;
m
Fig. 1. Rendered Sayles, Mulvanney and Ripple test surfaces.
p : q
5
G. McGunnigle, M.J. Chantler / Pattern Recognition Letters 21 (2000) 593±604
595
Fig. 2. Distribution of facet slope for Sayles, Mulvanney and Ripple test surfaces.
The slope distributions for the test surfaces are shown in Fig. 2. The isotropic surfaces have equal and uncorrelated slope parameters and a completely symmetrical distribution. For the purposes of surface description we shall assume that, in the case of directional surfaces, the surface directionality is aligned with one of the coordinate axes. In this case the slope parameters are unequal and the random variables uncorrelated. The slope distribution of the directional surface is asymmetrical. This paper is restricted to surfaces with a Lambertian re¯ectance function. The intensity i of a surface facet with partial derivatives p and q with re¯ectivity q, illuminated from a tilt angle of s and a slant angle of r is given by
The Lambertian re¯ectance function may be linearised over a limited region of the pq plane (Kube and Pentland, 1988). Most surfaces consist largely of facets with moderate slopes and can be accurately rendered using a linear approximation to a limited region of the re¯ectance map, centred on p 0 and q 0. In this case the re¯ectance map can be approximated using (9), assuming q is equal to unity. Furthermore, the assumptions of a Gaussian height distribution and a linear re¯ectance function predict the distribution of image intensity will be Gaussian:
i p; q; q; s; r
Now, assume the surface is rotated such that the axes of the distribution are aligned with the coordinate axes. In this case the bivariate slope distribution may be resolved into the product of two one-dimensional probability density functions, (10). Since intensity is the sum of linear functions of p and q, the intensity distribution can be obtained by convolving the distributions of those functions. Since both distributions are normal, the intensity distribution will be normal with variance equal to the sum of variances of the original functions. The intensity distribution will have a mean (11) and a S.D. as shown in (12): 1 p2 S p; q exp ÿ 0:5 2 2pprms qrms prms 2 q ; 10 exp ÿ 0:5 2 qrms
q
ÿp cos s sin r ÿ q sin s sin r cos r p : p2 q2 1
6
The re¯ectance function may be expressed using a re¯ectance map r p; q. This assumes ®xed illumination and viewing geometry and expresses observed intensity as a function of the surface partial derivatives, it is therefore de®ned on the same plane as the scatter plots shown in Fig. 2. By combining (4) and (6) we can predict the point statistics of the surface image, Eqs. (7) and (8): Z Z r p; qS p; q dp dq; 7 li r2i
Z Z
2
r p; q ÿ li S p; q dp dq:
8
r p; q ÿp cos s sin r ÿ q sin s sin r cos r: 9
596
G. McGunnigle, M.J. Chantler / Pattern Recognition Letters 21 (2000) 593±604
li cos r;
11
qÿ 2 cos2 s q2rms sin2 s : ri sin r prms
12
2.2. Surface classi®cation on image intensity Now consider two regions with identical surface re¯ectivities, q 1, illuminated from the same direction, s; r that dier in their topographical characteristics. The images of both surfaces will have a common mean (11) and will only dier in terms of their S.D. The image may be segmented by classifying each pixel on the basis of the estimated S.D. of the surrounding region. In practice, this can be estimated by squaring, low pass ®ltering and taking the square root of each pixel value, and is similar to post-®ltering used in many texture analysis schemes, e.g., Laws (1979). In many applications, the inspection system may be presented with a surface with an arbitrarily oriented directionality. If a surface is rotated, then its slope distribution will also rotate. In the case of directional surfaces, this will aect the value of the integral, (8). That is, for a directional surface, the variance of image intensity is dependent on the orientation of the surface. The quantitative eect can be clari®ed without loss of generality if, instead of rotating the slope distribution by an angle / we rotate the re¯ectance map by an angle ÿ/. For an isotropic surface, prms qrms , and the rotational terms in (12) sum to one, regardless of the surface orientation. For a directional surface, prms kqrms , and the sum of the rotational terms in (12) is still a function of surface orientation. That is, in the case of a directional surface expression (12) may be modi®ed to the form shown in (13). This eect of rotation of the Ripple test surface on the S.D. of image intensity is shown in Fig. 3. From this it can be seen that image variance is not a surface rotation invariant feature for directional surfaces: q ÿ ri qrms sin r k 2 cos2 s ÿ / sin2 s ÿ / : 13
Fig. 3. The variation of image S.D. with Ripple surface rotation.
2.3. Using photometric stereo for surface classi®cation If the topography of the surface can be estimated, it may be possible to improve on the performance of the intensity classi®er instead by classifying on the surface derivatives. Shape from shading is an under-constrained problem, most algorithms overcome this diculty by imposing a smoothness constraint on the recovered surface. This is clearly not appropriate in this application. Photometric stereo (Woodham, 1980) constrains the problem by using several images of the same scene under dierent illuminantion orientations. Originally developed for Lambertian surfaces the technique had been extended to surfaces with arbitrary re¯ectance functions (Rajaram et al., 1995) and more recently to surfaces where the re¯ectance function was unknown a priori (Kay and Caelli, 1995). When applied to Lambertian surfaces photometric stereo estimates three quantities: q the facet albedo and the partial derivatives p and q. In this paper, we will retain the Lambertian assumption and use the simple photometric algorithm described in (McGunnigle and Chantler, 1997). This algorithm could be replaced by a more complex scheme with less restrictive assumptions. However, any photometric technique will be restricted to surfaces which do not have signi®cant self or cast shadowing. Very rough surfaces will therefore have to be illuminated from high slant
G. McGunnigle, M.J. Chantler / Pattern Recognition Letters 21 (2000) 593±604
angles, with some loss in the ability of the algorithm to discriminate between facets of similar gradient. The ®rst aim of this paper is to ascertain whether the extra information obtained using photometric stereo improves the accuracy of surface classi®cation over that of the intensity classi®er. An improvement may take the form of either a reduction in the Bayes error or an improvement in the robustness of the classi®er to some variable. As with image intensity, if we use partial derivatives we will have to discriminate between distributions that share a common mean. A further complication is that we wish our classi®er to be rotation invariant. If the surface rotates, the slope distribution will also rotate. We adopt the following approach; at each pixel, a local estimate of the bivariate distribution is made. The eigenvalues of this distribution are then calculated and these form the basis for classi®cation. The approach is illustrated in Fig. 4. Although the distributions may rotate the eigenvalues will be constant. A classi®er based on this approach, which we shall call a k classi®er, should therefore be rotation invariant. The absolute values of the eigenvalues give a measure of the roughness of the surface. Their relative values give an indication of the degree of the surface directionality.
597
3. Synthetic experiments Two groups of experiments were carried out. In the ®rst group, the ability of the proposed classi®er to discriminate between surfaces with dierent degrees of roughness was compared with a classi®er based on a local estimate the S.D. of pixel intensity. The classi®ers' robustness to white noise was assessed. In the second group, the classi®ers' ability to discriminate on the basis of surface directionality was compared. The classi®ers' robustness to surface rotation was also assessed. 3.1. Surface roughness Two isotropic Sayles surfaces were generated, the ®rst was scaled to have an rms slope of 0.25, the rms slope of the other was varied in the course of the experiment. The visual eect to scaling the surface is shown in Fig. 5. The intensity and k classi®ers were applied to discriminate between the surfaces. The results shown in Fig. 6 show that both the intensity and k classi®ers are able to discriminate between surfaces that dier signi®cantly
Fig. 5. Gradated increase in roughness 0±0.5.
Fig. 4. Physical signi®cance of proposed features.
Fig. 6. The variation of classi®cation accuracy with the rms slope of the second test surface.
598
G. McGunnigle, M.J. Chantler / Pattern Recognition Letters 21 (2000) 593±604
in roughness. However, for dicult classi®cations, i.e., where the surfaces are very similar, the k classi®er does oer a signi®cant improvement in classi®cation accuracy. Since the features for the classi®ers are calculated in dierent ways, it is possible that the classi®ers will be aected by noise to dierent degrees. In this experiment the surfaces have rms slopes of 0.25 and 0.23, respectively, and the S/N ratio is varied. The results, Fig. 7, show that both classi®ers are able to function at low S/N levels and that the photometric classi®er is still superior. The ®nal experiment in this group is complementary to the ®rst; in this case roughness is varied indirectly by varying the spectrum of one of the surfaces. Where the break frequency is low the Mulvanney model approximates Sayle's model. Sayles noted that many surfaces have their longer wavelengths attenuated by surface processing (Sayles and Thomas, 1978). In the third experiment, varying degrees of processing were simulated by varying the break frequency of a Mulvanney surface. The visual eect of varying the cut-o frequency is shown in Fig. 8. The classi®ers were used to discriminate between two surfaces, the
®rst had a break frequency of 32 cycles and rms slope 0.2, the break frequency of the second surface was varied and the surface was rescaled so that the magnitudes of the high frequency components were equal to those in the ®rst surface. The results are shown in Fig. 9. Once again the classi®ers show a similar dependency on the variable, with the k classi®er giving improved classi®cation accuracy. 3.2. Surface directionality The directionality of a surface is an important cue to its identi®cation, however, in many applications the orientation of the surface is not under the control of the designer. In this section, two experiments are described, the ®rst compares the ability of the classi®ers to discriminate on directionality alone, the second assesses the robustness of the classi®er to surface rotation. The isotropic Sayles surface is a special case of the Ripple surface with a centre frequency of zero cycles per image (c/i). As the centre frequency increases, so does the directionality of the surface. In the next simulation we compare the ability of the classi®ers to discriminate on surface directionality. Two Ripple surfaces are used, the ®rst has a centre frequency of 0 c/i, the frequency of the second is increased from 0 to 32 c/i over the course of the experiment (see Figs. 10 and 11). However, throughout the experiment, the second surface is
Fig. 7. The robustness of the classi®ers to white noise.
Fig. 8. Gradated increase in cut-o frequency.
Fig. 9. The eect of varying cut-o frequency on classi®cation accuracy.
G. McGunnigle, M.J. Chantler / Pattern Recognition Letters 21 (2000) 593±604
599
scaled so that its rms slope measured along its principal directionality is equal to that of the ®rst surface. The experiment is performed twice: in the ®rst the grain of the material runs vertically ± perpendicular to the illuminant vector and in the second it runs horizontally ± collinear with the illuminant. The results presented in Fig. 12 show the k classi®er to have a performance that is equal to that of the intensity classi®er in the ®rst experiment. However, if the surface is rotated 90°, and the experiment repeated, the performance of the intensity classi®er is signi®cantly reduced. This degradation is due to the fact that the illuminant is collinear with the surface grain, and the directionality visible in the image is attenuated. This is
re¯ected in a reduction in the S.D. of image intensity. Since the k classi®er operates on an estimate of properties of the surface, it is unaected by the interaction of the illuminant and surface directionality, it is therefore capable of discriminating on surface directionality, regardless of its orientation. The previous experiment highlighted that a classi®er based on intensity is sensitive to the orientation of the surface; the k classi®er in contrast, appears to be much more robust to the surface orientation. To test whether the intensity classi®er is rotation dependent and whether the k is rotation invariant we measure the robustness of the intensity and k features to rotation of the Ripple surface, Fig. 13. The intensity feature varies signi®cantly, however, the k features are much more stable and therefore represent a more robust basis for a classi®er. In order to ascertain the eect of surface rotation on surface classi®cation we conduct our ®nal simulation. Both classi®ers are trained, at 0° of rotation, to discriminate between an isotropic surface with prms 0:2 and a directional surface with prms 0:25 and qrms 0:1. The directional surface was then rotated in 10° increments and the classi®ers applied, without retraining, to segment the image, the results are shown in Fig. 14. The results indicate that the intensity classi®er is highly sensitive to surface rotation whereas the k classi®er is rotation invariant.
Fig. 12. The variation of classi®cation accuracy with centre frequency of the second surface.
Fig. 13. The variation of feature means with rotation of the Ripple surface.
Fig. 10. Gradated increase in centre frequency, surface illuminated from s 0°.
Fig. 11. The same surface illuminated from s 90°.
600
G. McGunnigle, M.J. Chantler / Pattern Recognition Letters 21 (2000) 593±604
Fig. 14. The variation of k and intensity classi®cation accuracy with rotation.
4. Experiments on real data The most signi®cant result of the synthetic experiments was that the intensity classi®er is not
robust to the rotation of directional surfaces, whereas the k classi®er is. In this section, we describe a series of experiments carried out on two pairs of real, directional surfaces. The ®rst pair Striate and Slate are naturally occurring rock surfaces. The second pair Ground 1 and Ground 2 are manmade surfaces formed by grinding a ¯at plaster surface. By varying the length of grinding stroke it is possible to vary the directionality of the surface. The ®rst pair are shown under 0°, 45° and 90° of rotation in Figs. 15 and 16, the visual eect of the change in the relative orientation of surface directionality and illuminant is clear. In the case of Striate the grain of the surface is originally perpendicular to the illuminant, as the surface is rotated and the surface grain becomes collinear with the illuminant tilt direction, the directionality visible in the image is attenuated. In the case of Slate, the surface directionality is initially collinear with the illuminant vector and only becomes visible
Fig. 15. Striate surface at 0°, 45° and 90° of rotation.
Fig. 16. Slate surface at 0°, 45° and 90°.
G. McGunnigle, M.J. Chantler / Pattern Recognition Letters 21 (2000) 593±604
when the surface is rotated. This is re¯ected in the change of the S.D. of image intensity with surface rotation. The textures Ground 1 and Ground 2 also exhibit this phenomenon, Figs. 17 and 18. If we examine the distributions of slopes estimated from the Striate and Slate surfaces, Figs. 19 and 20, we can see that the distributions re¯ect the
601
directionality of the surfaces and their relative roughnesses. Rotation of the surface leads to a corresponding rotation of the slope distribution. If we plot the k features (Fig. 21), we can see that these are much more stable than the intensitybased features, Fig. 22. This is borne out by the classi®cation results, Fig. 23: the intensity classi®er
Fig. 17. Ground 1 surface 0°, 45° and 90°.
Fig. 18. Ground 2 surface 0°, 45° and 90°.
Fig. 19. Scatter plot of the estimated partial derivatives of the Striate surface at 0°, 45° and 90° of rotation.
602
G. McGunnigle, M.J. Chantler / Pattern Recognition Letters 21 (2000) 593±604
Fig. 20. Scatter plots of partial derivatives estimated from Slate surface at 0°, 45° and 90° of rotation.
Fig. 21. The variation with rotation of k feature mean for Striate and Slate surfaces.
Fig. 23. Comparison of classi®ers for rotation of Striate and Slate surfaces.
Fig. 22. The variation with rotation of intensity feature mean for Striate and Slate surfaces.
Fig. 24. The variation with rotation of intensity feature mean for Ground 1 and Ground 2 surfaces.
G. McGunnigle, M.J. Chantler / Pattern Recognition Letters 21 (2000) 593±604
Fig. 25. The variation with rotation of k feature mean for Ground 1 and Ground 2 surfaces.
603
conditions can be strictly controlled, the point statistics of either the image, or the surface, are often sucient for accurate surface segmentation. This paper has highlighted one serious problem with classi®ers that classify rough surfaces on the basis of their image: image-intensity-based features are not robust to the rotation of directional surfaces. In this correspondence, a simple scheme that extracts rotation invariant statistics from photometric estimates of the surface derivatives was proposed. This scheme was found to be rotation invariant and was also found to oer improved classi®cation for isotropic surfaces. Set against these advantages, the proposed scheme does require additional resources. The computational requirements of our simple photometric scheme and the principal components analysis are small relative to the other operations; the most signi®cant increase is due to the fact that it is necessary to low pass ®lter three ®elds instead of just one for the intensity classi®er. In any case, the computational requirements are still modest compared to those of a second-order classi®er. However, in practice the main overhead of this scheme is the requirement for photometric image sets. The authors believe this is the most signi®cant restriction to this approach.
References Fig. 26. Comparison of classi®ers for rotation of surfaces.
rapidly fails as the surfaces are rotated, however the k classi®er is able to maintain a low level of misclassi®cation for all angles of rotation. We repeated the previous experiments with the manmade surfaces and obtained similar results: Figs. 24±26.
5. Discussion This correspondence has described an approach to rough surface classi®cation. It is our argument that in an inspection task, i.e., where the imaging
Kay, G., Caelli, T., 1995. Estimating the parameters of an illumination model using photometric stereo. Graphical Models and Image Processing 57 (5), 365±388. Kube, P., Pentland, A.P., 1988. On the imaging of fractal surfaces. IEEE Trans. Pattern Analysis Machine Intelligence 10 (5), 704±707. Laws, K.I., 1979. Texture energy measures. In: Proceedings of the Workshop on Image Understanding, pp. 47±51. Linnett, L.M., 1991. Multi-texture image segementation. Ph.D. thesis, Department of Electrical and Electronic Engineering, Heriot Watt University. McGunnigle, G., Chantler, M.J., 1997. A Model-based technique for the classi®cation of textured surfaces with illuminant direction invariance. In: Clark, A.F. (Ed.), BMVC97, Proceedings of the Conference on British Machine Vision, Vol. 2, pp. 470±480. Mulvanney, D.J., Newland, D.E., Gill, K.F., 1989. A complete description of surface texture pro®les. Wear 132, 173±182. Ogilvy, J.A., 1991. Theory of Wave Scattering from Random Rough Surfaces. Adam Hilger, Bristol.
604
G. McGunnigle, M.J. Chantler / Pattern Recognition Letters 21 (2000) 593±604
Rajaram, K.V., Parthasarth, G., Faruqui, M.A., 1995. A neural network approach to photometric stereo inversion of real-world re¯ectance maps for extracting 3-D shapes of objects. IEEE Trans. Systems, Man Cybernet. 25 (9), 1289±1300.
Sayles, R.S., Thomas, T.R., 1978. Surface topography as a nonstationary random process. Nature 271 (2), 431±434. Woodham, R., 1980. Photometric method for determining surface orientation from multiple images. Opt. Eng. 19 (1), 139±144.