- Email: [email protected]
Flexible colour point distribution models
WI n -IF
ELSEVIER
COMPUTING
Image and Vision Computing 14 (1996) 703-708
Short communication
Flexible colour point distribution models Christine
M. Onyango,
Silsoe Research Institute,
John A. Marchant
Wrest Park, Silsoe. Bedford MK45 4HS, UK
Received 17 July 1995; revised 3 November 1995
Abstract A technique is described which models the way the colour of a set of objects can change. Models are derived from a set of example images. They describe the average appearance of an object and the way in which that appearance can change. Two types of object have been modelled using this method, and the models have been fitted to objects not seen in the training data. Keywords:
Colour modelling; Flexible templates; Finite elements; Point distribution models
1. Introduction Segmenting images into regions of interest is generally more reliable if a model of the object is available. Biological objects exhibit a lot of variability which a model must accommodate. The surface characteristics of an object, like its colour and shading patterns, are as important as its shape because these are often the features that distinguish one object from another. Often it is difficult to obtain the edge information which describes the shape of the object, and the only distinguishing features are the shading and surface characteristics. Monochromatic images are adequate for a lot of tasks, but sometimes the colour of an object is required to provide accurate discrimination. For instance, the most important criterion when determining the ripeness of a tomato is whether it is red or green. Green potatoes, which contain a toxic chemical, can only be identified by their colour. The level of decay in a leaf in autumn can only be measured by its colour, since other factors like the moisture content cannot be readily assessed visually. In all these instances, a precise analytical description of how the colour and the shape of the objects change is difficult to derive. The most appropriate class of models for problems of this type are those which are trained by example. Cootes’ [l] Point Distribution Model (PDM), provides a compact way of encapsulating the variability of outline shape in natural objects using a set of training images. A companion technique by Marchant [2] has endowed the resulting model with a grey level 0262-8856/96/$15.00 0 1996 Elsevier Science B.V. All rights reserved SSDZ 0262-8856(95)01076-9
distribution. Previous work has extended the PDM to include variations in the grey level distribution of monochrome objects [3,4]. This paper describes the use of the method to model variations in colour.
2. Representation
of colour
Colour video cameras mimic the human eye by having three sensors which respond to different parts of the visual light spectrum, broadly red, green and blue. In this work we use a system that gives a different representation of colour. The HSI system divides colour into three components: hue, saturation and intensity. Hue is the attribute of colour perception denoted by blue, green, yellow, purple and so on. It corresponds broadly to the wavelength of the radiant energy. Saturation is the degree of purity of a colour. Maximum saturation implies the colour is pure while an unsaturated colour is grey or white. Intensity measures the brightness of a colour [5]. The following equations represent linear combinations of the parameters, and it is therefore simple to convert from one system to another. The RGB values are converted to the Y (intensity), I (inphase chrominance signal), Q (quadrature chrominance signal) using the following equations [6]:
= [I Y
0.299
I
0.596
Q
1
0.212
0.587 -0.275 -0.523
(1) 0.3111
LB1
704
CM.
H and S are further follows: H=arctan
Onyango, J.A. Marchantllmage
defined in cylindrical
co-ordinates
as
(2)
9 0
where arc tan resolves
angles in four quadrants.
S = arc tan
(3)
3. Point distribution models A detailed description of PDMs can be found in Cootes [l]: the basic principles are summarised here. PDMs are derived by placing points at corresponding positions on a set of shapes. The positions must be the same on each shape, so the points are normally placed on easily recognised parts of the image. The shapes are then aligned with respect to a reference, usually the first shape in the sequence, to remove differences in scale, orientation and translation. Cootes calculates the mean shape vector f of m example shape vectors Xi
and the covariance
matrix
about
the mean
I=1
The eigenvectors of the covariance matrix S, give a set of modes of variation in which the model can deform. In Cootes’ analysis each shape vector, xi, is a set of n points
To derive a model of the colour have replaced Eq. (6) with
Si1,Si2,...,Sin,gi1rgi2,...,gin
(6)
components
IT
I4 (1996)
703-708
model is used to search an image for examples of the shape of interest, a small number of modes of variation, which describe changes in all the parameters, will limit the extent to which any one parameter can be varied. The resulting fit will be a compromise between parameter values, and not necessarily the best fit in any one. Combined models of all the parameters have other drawbacks, because of the difference in scale between, for example, shape and intensity [4]. To avoid the problems associated with combined models, we take advantage of the fact that shape, and the three components of colour, are probably independent. Collecting enough data to obtain a model containing decoupled modes of variation is not always possible. A compromise solution involves manipulating the available data to obtain modes in parameters which we know to be independent. Instead of analysing the large covariance matrix derived from the vector containing values of x, y, h, s and g, the analysis can be done in four stages on the same set of data. The first stage is as described by Cootes [l] using the vector (6). In the second, third and fourth stages, x and y are replaced with values of hue, saturation and intensity, respectively. This results in four small covariance matrices which give four sets of eigenvalues and eigenvectors. The eigenvectors describe changes in shape, hue, saturation and intensity. A convenient method of implementing this in the software is to set Xi-X=0
s, = J&xi- .t)(Xi - i)T
Xi=(Xil,Xi2,...,Xin,yilrYi2r...,yi,)T
and Vision Computing
HSI, we
(7)
The covariance matrix now gives a set of eigenvectors which describe modes of variation in shape and colour. h, s and g are the hue, saturation and intensity of the pixel at a point (x, y). If the model is trained on a large data set, any genuine relationships between parameters will be reflected in the modes of variation. Equally, parameters which are decoupled in the training set will have separate modes in the model. The maximum number of possible modes NM,,, = m - 1, where m is the total number of shapes. Small data sets limit the number of possible modes of variation, and in the extreme case where m = 2, all the variation will be attributed to one mode. When the
for the three components the model. For example, only in x and y, we set hi-h^=si-.+gi-~=O
(8) which are to be excluded from to derive a model which varies
(9)
This gives a covariance matrix with columns and rows corresponding to h, s and g set to zero, and eigenvectors with components in x and y only. As the four components shape, hue, saturation and intensity do not interact, they can be fitted sequentially to objects in an image. Marchant’s [2] Finite Element method for rendering grey levels by interpolating between the grey level values at the control points is extended here to interpolate hue and saturation over the surface of the object. This paper describes the use of colour point distribution models to model simple shapes of a single colour, as well as more intricate shapes, leaves, which exhibit complex colour variations.
4. Training data The first set of training data consisted of rigid, plastic blocks. These were used as test objects because they had single, uniform colours and simple outline shapes.
C.M. Onyango, J.A. Marchantllmage
and Vision Computing I4 (1996)
.
. .
a
703-708
.
.
.
l
.
.
.
a
.
.
. .
. . . .
.
.
. . .
.
. .
b Fig. 2. (a) A typical leaf, (b) point positions in leaf model.
Fig. 1. (a) Simple coloured objects, (b) structure of block model.
Although each block had a single colour, variations in reflectance were caused by the illumination and the shapes of the object. The model contained 26 points and 4 elements. Fig. I(a) shows the set of blocks used in the experiment. Fig. l(b) illustrates the structure of the model, the position of the points and the elements. The second set of data consisted of leaves of Parthenocissus (Virginia creeper). The leaves were different shades of green, red, brown and yellow. The colours
varied both from one leaf to another and over individual leaves. Some of the changes were gradual with a slow transition from dark green to light green, for example. Others were sudden, for instance where the surface of the leaf was mottled. The leaves also produced specular reflections where the shiny surface of the leaf reflected all the incident light. Since the light source was white, these appeared as bright white patches on the leaf. The leaf veins were also markedly different in colour from the remainder of the leaf. Fig. 2(a) shows a typical leaf, and Fig. 2(b) illustrates the model, the position of the points and the elements. 5. Derivation of models
For each data set obtained by placing control points on the object, separate point distribution models for shape, hue, saturation and intensity were derived. Both data sets were fairly small, but large enough to do the analysis and illustrate the salient points.
706
CM.
Onyango, J.A. Marchantjlmage
and Vision Computing
Since the point distribution models do not accommodate changes in scale, position and orientation, these variables must be reintroduced into the model if it is to be fitted to an image. These variables only affect the shape of the object, so the shape model is a function of up to seven parameters, X,y, scale, angle and up to three modes of variation obtained from the point distribution model. Although the error in the colour component of the fit is theoretically a function of x,y, angle and scale, once the shape component of the model has been fitted, it is only necessary to alter the modes of variation in colour. The position, scale and angle can be fixed so the error in hue, saturation and intensity is only dependent on the modes of variation in colour.
The model is fitted to the image by optimising an objective function which measures the boundary correspondence, he (Eq. (10)) and the colour correspondence, An (Eq. (11))
(10) where he is the line integral of the second differential of intensity, G,,, normal to the boundary, around the boundary. (C, - Ci)dx dy
703-708
The objective function displays all the qualities discussed in previous work [7]. On the smooth cylindrical blocks the edge component has a narrow but well defined minimum. The model has to be fairly close to the true minimum for a minimisation technique to converge. On the leaves, the behaviour of the objective function away from the true boundary is unpredictable, as there is no characteristic shading pattern which relates to shape. Highlights and rapid colour changes create a lot of spurious edge data in regions that do not correspond to the true boundary. Since the body of the leaf contains a lot of spurious edge data, the fitting procedure is much improved by smoothing the image heavily. This averages out the shading in the body of the leaf without destroying the edge information at the boundary. The next stage in the process of model fitting assumes that the shape has been fitted accurately and no further attempt is made to move the model spatially. The error between the model and the image is integrated over the entire object using a technique called Gaussian quadrature [8]. Image data is sampled randomly at nine points in each element, and although none of the sample points is on the boundary of the object, it is important that the model is correctly aligned with the object and that it does not ,overlap the background. Any inaccuracies in the alignment will result in errors in the measure of fit. It is very important to model shape precisely. In these experiments it was found that, whilst the point distribution model gave a good global representation of shape and the ways in which shape could change, there was still a need for local refinement to the model, based on local image data. This is the subject of further investigations. The colour components of the model are fitted by fixing its shape, position, orientation and size. The objective function now measures the average error between one particular colour component of the model, hue, saturation or intensity, as described by Eq. (11). Since the shape is fixed, only the modes of variation, which describe
6. Model fitting
hn =
14 (1996)
(11)
where hn is the area integral of the error between the colour components, C,, in the model and the colour components, Ci, in the image over the area of the model. The object function used here is similar to that used by Marchant et al. [7]. It differs in that, instead of measuring the error in grey level only, it measures the error in hue, saturation or intensity, depending on what is being fitted at the time.
b
a Fig. 3. (a) Training
set for block model (blue block excluded),
(b) model fitted to the blue block.
CM.
Onyango, J.A. Marchantllmage
and Vision Computing 14 (1996)
107
703-708
model. The blue block was excluded from the training set and used to test the model. The aim of this test was to see how well the PDM modelled the colour of the object, so no attempt was made to fit the shape of the block. The model was placed manually in approximately the right position. The simplex optimisation method was used to minimise the objective function [ll]. x,y, scale and angle were restricted to prevent the simplex from minimising the error in the colour fit by changing these parameters. Fig. 3(b) depicts the model after it has been fitted to an object that was not in the training set. 7.2. Colourjtting
b Fig. 4. (a) Example
leaf, (b) model of leaf in Fig. 4(a).
allowable changes in colour, are used to minimise the objective function. The relationship between the objective function and the modes of variation in hue is extremely well behaved. It has a clearly defined global minimum, in most cases, and no significant local minima. The other colour components, saturation and intensity, exhibit similar characteristics. 7. Results 7.1. ColourJitting Four
to simple shapes
of the blocks in Fig. 3(a) were used to train the
to complex
shapes
A model consisting of five elements and 26 nodes was trained on five leaves which were various shades of green, red, orange and yellow. The shape component of the model was fitted by heavily smoothing the image to reduce the magnitude of the edges in the centre of the leaf, and then maximising the magnitude of the second differential of the grey level along the inward normal to the boundary. Initial values of orientation and position were obtained by manually selecting the apex and the stalk of the leaf. The orientation was set to the angle of the line joining these two points and the position to the midpoint of this line. Automatic initialisation of the model using corner detection on the boundary is currently under investigation. After fitting the boundary component, the parameters that controlled shape were fixed and the colour components of the model were then fitted by minimising the error in intensity, hue and saturation, respectively. Where the colour variation is more complex it is more important to fit the shape accurately. The model used here, which had been trained on five leaves, was not very successful in fitting the shape of the test leaves because the training set was not large enough, and all the modes of variation were not included in the model. A larger training set and the inclusion of more modes in the model would probably remedy this, but it is thought that in general the PDM might not be the best tool for modelling the finer variations between objects, and it might be better to use the PDM with some other more locally sensitive method. Fig. 4(b) shows the model after fitting to the leaf in Fig. 4(a). This result demonstrates the potential of the technique for modelling colour, as well as shape variations of an object. The model is quite coarse in that it only contains five elements, so the estimates of the colour are done on very sparse data. Nevertheless, the model makes a reasonable attempt at modelling the colours of the leaf. Further work is required to test the method on a wide range of objects. In tests the program took on average 16s on a Sun workstation to fit the model to an image of a leaf.
708
However, no attempt speed of operation.
C.M. Onyango. J.A. Marchantllmage and Vision Computing 14 (1996) 703-708
has been
made
to optimise
the
References Sl T.F. Cootes,
8. Conclusions 1. Colour Point Distribution models provide compact simple descriptions of complex shapes, shading patterns and colours. 2. This generic technique is potentially applicable to a wide range of objects. 3. Complex colour distributions have been modelled with this technique. 4. A strategy has been developed for fitting the model. By first fitting the shape and then intensity, hue and saturation, the strategy overcomes the difficulties of optimising a function of several variables. It also removes the requirement for having a scaling function on hue saturation and intensity. 5. Further work is being done to combine PDMs with other flexible models to refine local fit.
C.J. Taylor, D.H. Cooper and J. Graham, Training models of shape from sets of examples, Proc. 3rd British Machine Vision Conf., Leeds, UK, 22-24 September 1992, pp. 9918. I21 J.A. Marchant, Adding grey level information to point distribution models using finite elements, Proc. 4th British Machine Vision Conf., Guildford, UK, 21-23 September 1993, pp. 309-318. [31 T.F. Cootes and C.J. Taylor, Modelling object appearance using the grey-level surface, Proc. 5th British Machine Vision Conf., York, UK, 13-16 September 1994, pp. 479-488. Modelling grey-level surfaces [41 CM. Onyango and J.A. Marchant, using three dimensional point distribution models, Image and Vision Computing, 14( 10) (1996). 151G. Wyszecki and W.S. Stiles, Color Science: Concepts and Methods, Quantitative Data and Formulae, John Wiley and Sons, New York, 1982. McGraw Hill, 161K.B. Benson, Television Engineering Handbook, New York, 1986. 171J.A. Marchant and C.M. Onyango, Fitting grey level point distribution models to animals in scenes, Image and Vision Computing, 13(l) (1995) 3-12. PI O.C. Zienkiewicz, The Finite Element Method, 3rd edn., McGrawHill, Maidenhead, 1977. 191W.H. Press, B.P. Flannery, S.A. Teukolsky and W.T. Vetterling, Numerical Recipes in C, Cambridge University Press, Cambridge, 1988.
ELSEVIER
COMPUTING
Image and Vision Computing 14 (1996) 703-708
Short communication
Flexible colour point distribution models Christine
M. Onyango,
Silsoe Research Institute,
John A. Marchant
Wrest Park, Silsoe. Bedford MK45 4HS, UK
Received 17 July 1995; revised 3 November 1995
Abstract A technique is described which models the way the colour of a set of objects can change. Models are derived from a set of example images. They describe the average appearance of an object and the way in which that appearance can change. Two types of object have been modelled using this method, and the models have been fitted to objects not seen in the training data. Keywords:
Colour modelling; Flexible templates; Finite elements; Point distribution models
1. Introduction Segmenting images into regions of interest is generally more reliable if a model of the object is available. Biological objects exhibit a lot of variability which a model must accommodate. The surface characteristics of an object, like its colour and shading patterns, are as important as its shape because these are often the features that distinguish one object from another. Often it is difficult to obtain the edge information which describes the shape of the object, and the only distinguishing features are the shading and surface characteristics. Monochromatic images are adequate for a lot of tasks, but sometimes the colour of an object is required to provide accurate discrimination. For instance, the most important criterion when determining the ripeness of a tomato is whether it is red or green. Green potatoes, which contain a toxic chemical, can only be identified by their colour. The level of decay in a leaf in autumn can only be measured by its colour, since other factors like the moisture content cannot be readily assessed visually. In all these instances, a precise analytical description of how the colour and the shape of the objects change is difficult to derive. The most appropriate class of models for problems of this type are those which are trained by example. Cootes’ [l] Point Distribution Model (PDM), provides a compact way of encapsulating the variability of outline shape in natural objects using a set of training images. A companion technique by Marchant [2] has endowed the resulting model with a grey level 0262-8856/96/$15.00 0 1996 Elsevier Science B.V. All rights reserved SSDZ 0262-8856(95)01076-9
distribution. Previous work has extended the PDM to include variations in the grey level distribution of monochrome objects [3,4]. This paper describes the use of the method to model variations in colour.
2. Representation
of colour
Colour video cameras mimic the human eye by having three sensors which respond to different parts of the visual light spectrum, broadly red, green and blue. In this work we use a system that gives a different representation of colour. The HSI system divides colour into three components: hue, saturation and intensity. Hue is the attribute of colour perception denoted by blue, green, yellow, purple and so on. It corresponds broadly to the wavelength of the radiant energy. Saturation is the degree of purity of a colour. Maximum saturation implies the colour is pure while an unsaturated colour is grey or white. Intensity measures the brightness of a colour [5]. The following equations represent linear combinations of the parameters, and it is therefore simple to convert from one system to another. The RGB values are converted to the Y (intensity), I (inphase chrominance signal), Q (quadrature chrominance signal) using the following equations [6]:
= [I Y
0.299
I
0.596
Q
1
0.212
0.587 -0.275 -0.523
(1) 0.3111
LB1
704
CM.
H and S are further follows: H=arctan
Onyango, J.A. Marchantllmage
defined in cylindrical
co-ordinates
as
(2)
9 0
where arc tan resolves
angles in four quadrants.
S = arc tan
(3)
3. Point distribution models A detailed description of PDMs can be found in Cootes [l]: the basic principles are summarised here. PDMs are derived by placing points at corresponding positions on a set of shapes. The positions must be the same on each shape, so the points are normally placed on easily recognised parts of the image. The shapes are then aligned with respect to a reference, usually the first shape in the sequence, to remove differences in scale, orientation and translation. Cootes calculates the mean shape vector f of m example shape vectors Xi
and the covariance
matrix
about
the mean
I=1
The eigenvectors of the covariance matrix S, give a set of modes of variation in which the model can deform. In Cootes’ analysis each shape vector, xi, is a set of n points
To derive a model of the colour have replaced Eq. (6) with
Si1,Si2,...,Sin,gi1rgi2,...,gin
(6)
components
IT
I4 (1996)
703-708
model is used to search an image for examples of the shape of interest, a small number of modes of variation, which describe changes in all the parameters, will limit the extent to which any one parameter can be varied. The resulting fit will be a compromise between parameter values, and not necessarily the best fit in any one. Combined models of all the parameters have other drawbacks, because of the difference in scale between, for example, shape and intensity [4]. To avoid the problems associated with combined models, we take advantage of the fact that shape, and the three components of colour, are probably independent. Collecting enough data to obtain a model containing decoupled modes of variation is not always possible. A compromise solution involves manipulating the available data to obtain modes in parameters which we know to be independent. Instead of analysing the large covariance matrix derived from the vector containing values of x, y, h, s and g, the analysis can be done in four stages on the same set of data. The first stage is as described by Cootes [l] using the vector (6). In the second, third and fourth stages, x and y are replaced with values of hue, saturation and intensity, respectively. This results in four small covariance matrices which give four sets of eigenvalues and eigenvectors. The eigenvectors describe changes in shape, hue, saturation and intensity. A convenient method of implementing this in the software is to set Xi-X=0
s, = J&xi- .t)(Xi - i)T
Xi=(Xil,Xi2,...,Xin,yilrYi2r...,yi,)T
and Vision Computing
HSI, we
(7)
The covariance matrix now gives a set of eigenvectors which describe modes of variation in shape and colour. h, s and g are the hue, saturation and intensity of the pixel at a point (x, y). If the model is trained on a large data set, any genuine relationships between parameters will be reflected in the modes of variation. Equally, parameters which are decoupled in the training set will have separate modes in the model. The maximum number of possible modes NM,,, = m - 1, where m is the total number of shapes. Small data sets limit the number of possible modes of variation, and in the extreme case where m = 2, all the variation will be attributed to one mode. When the
for the three components the model. For example, only in x and y, we set hi-h^=si-.+gi-~=O
(8) which are to be excluded from to derive a model which varies
(9)
This gives a covariance matrix with columns and rows corresponding to h, s and g set to zero, and eigenvectors with components in x and y only. As the four components shape, hue, saturation and intensity do not interact, they can be fitted sequentially to objects in an image. Marchant’s [2] Finite Element method for rendering grey levels by interpolating between the grey level values at the control points is extended here to interpolate hue and saturation over the surface of the object. This paper describes the use of colour point distribution models to model simple shapes of a single colour, as well as more intricate shapes, leaves, which exhibit complex colour variations.
4. Training data The first set of training data consisted of rigid, plastic blocks. These were used as test objects because they had single, uniform colours and simple outline shapes.
C.M. Onyango, J.A. Marchantllmage
and Vision Computing I4 (1996)
.
. .
a
703-708
.
.
.
l
.
.
.
a
.
.
. .
. . . .
.
.
. . .
.
. .
b Fig. 2. (a) A typical leaf, (b) point positions in leaf model.
Fig. 1. (a) Simple coloured objects, (b) structure of block model.
Although each block had a single colour, variations in reflectance were caused by the illumination and the shapes of the object. The model contained 26 points and 4 elements. Fig. I(a) shows the set of blocks used in the experiment. Fig. l(b) illustrates the structure of the model, the position of the points and the elements. The second set of data consisted of leaves of Parthenocissus (Virginia creeper). The leaves were different shades of green, red, brown and yellow. The colours
varied both from one leaf to another and over individual leaves. Some of the changes were gradual with a slow transition from dark green to light green, for example. Others were sudden, for instance where the surface of the leaf was mottled. The leaves also produced specular reflections where the shiny surface of the leaf reflected all the incident light. Since the light source was white, these appeared as bright white patches on the leaf. The leaf veins were also markedly different in colour from the remainder of the leaf. Fig. 2(a) shows a typical leaf, and Fig. 2(b) illustrates the model, the position of the points and the elements. 5. Derivation of models
For each data set obtained by placing control points on the object, separate point distribution models for shape, hue, saturation and intensity were derived. Both data sets were fairly small, but large enough to do the analysis and illustrate the salient points.
706
CM.
Onyango, J.A. Marchantjlmage
and Vision Computing
Since the point distribution models do not accommodate changes in scale, position and orientation, these variables must be reintroduced into the model if it is to be fitted to an image. These variables only affect the shape of the object, so the shape model is a function of up to seven parameters, X,y, scale, angle and up to three modes of variation obtained from the point distribution model. Although the error in the colour component of the fit is theoretically a function of x,y, angle and scale, once the shape component of the model has been fitted, it is only necessary to alter the modes of variation in colour. The position, scale and angle can be fixed so the error in hue, saturation and intensity is only dependent on the modes of variation in colour.
The model is fitted to the image by optimising an objective function which measures the boundary correspondence, he (Eq. (10)) and the colour correspondence, An (Eq. (11))
(10) where he is the line integral of the second differential of intensity, G,,, normal to the boundary, around the boundary. (C, - Ci)dx dy
703-708
The objective function displays all the qualities discussed in previous work [7]. On the smooth cylindrical blocks the edge component has a narrow but well defined minimum. The model has to be fairly close to the true minimum for a minimisation technique to converge. On the leaves, the behaviour of the objective function away from the true boundary is unpredictable, as there is no characteristic shading pattern which relates to shape. Highlights and rapid colour changes create a lot of spurious edge data in regions that do not correspond to the true boundary. Since the body of the leaf contains a lot of spurious edge data, the fitting procedure is much improved by smoothing the image heavily. This averages out the shading in the body of the leaf without destroying the edge information at the boundary. The next stage in the process of model fitting assumes that the shape has been fitted accurately and no further attempt is made to move the model spatially. The error between the model and the image is integrated over the entire object using a technique called Gaussian quadrature [8]. Image data is sampled randomly at nine points in each element, and although none of the sample points is on the boundary of the object, it is important that the model is correctly aligned with the object and that it does not ,overlap the background. Any inaccuracies in the alignment will result in errors in the measure of fit. It is very important to model shape precisely. In these experiments it was found that, whilst the point distribution model gave a good global representation of shape and the ways in which shape could change, there was still a need for local refinement to the model, based on local image data. This is the subject of further investigations. The colour components of the model are fitted by fixing its shape, position, orientation and size. The objective function now measures the average error between one particular colour component of the model, hue, saturation or intensity, as described by Eq. (11). Since the shape is fixed, only the modes of variation, which describe
6. Model fitting
hn =
14 (1996)
(11)
where hn is the area integral of the error between the colour components, C,, in the model and the colour components, Ci, in the image over the area of the model. The object function used here is similar to that used by Marchant et al. [7]. It differs in that, instead of measuring the error in grey level only, it measures the error in hue, saturation or intensity, depending on what is being fitted at the time.
b
a Fig. 3. (a) Training
set for block model (blue block excluded),
(b) model fitted to the blue block.
CM.
Onyango, J.A. Marchantllmage
and Vision Computing 14 (1996)
107
703-708
model. The blue block was excluded from the training set and used to test the model. The aim of this test was to see how well the PDM modelled the colour of the object, so no attempt was made to fit the shape of the block. The model was placed manually in approximately the right position. The simplex optimisation method was used to minimise the objective function [ll]. x,y, scale and angle were restricted to prevent the simplex from minimising the error in the colour fit by changing these parameters. Fig. 3(b) depicts the model after it has been fitted to an object that was not in the training set. 7.2. Colourjtting
b Fig. 4. (a) Example
leaf, (b) model of leaf in Fig. 4(a).
allowable changes in colour, are used to minimise the objective function. The relationship between the objective function and the modes of variation in hue is extremely well behaved. It has a clearly defined global minimum, in most cases, and no significant local minima. The other colour components, saturation and intensity, exhibit similar characteristics. 7. Results 7.1. ColourJitting Four
to simple shapes
of the blocks in Fig. 3(a) were used to train the
to complex
shapes
A model consisting of five elements and 26 nodes was trained on five leaves which were various shades of green, red, orange and yellow. The shape component of the model was fitted by heavily smoothing the image to reduce the magnitude of the edges in the centre of the leaf, and then maximising the magnitude of the second differential of the grey level along the inward normal to the boundary. Initial values of orientation and position were obtained by manually selecting the apex and the stalk of the leaf. The orientation was set to the angle of the line joining these two points and the position to the midpoint of this line. Automatic initialisation of the model using corner detection on the boundary is currently under investigation. After fitting the boundary component, the parameters that controlled shape were fixed and the colour components of the model were then fitted by minimising the error in intensity, hue and saturation, respectively. Where the colour variation is more complex it is more important to fit the shape accurately. The model used here, which had been trained on five leaves, was not very successful in fitting the shape of the test leaves because the training set was not large enough, and all the modes of variation were not included in the model. A larger training set and the inclusion of more modes in the model would probably remedy this, but it is thought that in general the PDM might not be the best tool for modelling the finer variations between objects, and it might be better to use the PDM with some other more locally sensitive method. Fig. 4(b) shows the model after fitting to the leaf in Fig. 4(a). This result demonstrates the potential of the technique for modelling colour, as well as shape variations of an object. The model is quite coarse in that it only contains five elements, so the estimates of the colour are done on very sparse data. Nevertheless, the model makes a reasonable attempt at modelling the colours of the leaf. Further work is required to test the method on a wide range of objects. In tests the program took on average 16s on a Sun workstation to fit the model to an image of a leaf.
708
However, no attempt speed of operation.
C.M. Onyango. J.A. Marchantllmage and Vision Computing 14 (1996) 703-708
has been
made
to optimise
the
References Sl T.F. Cootes,
8. Conclusions 1. Colour Point Distribution models provide compact simple descriptions of complex shapes, shading patterns and colours. 2. This generic technique is potentially applicable to a wide range of objects. 3. Complex colour distributions have been modelled with this technique. 4. A strategy has been developed for fitting the model. By first fitting the shape and then intensity, hue and saturation, the strategy overcomes the difficulties of optimising a function of several variables. It also removes the requirement for having a scaling function on hue saturation and intensity. 5. Further work is being done to combine PDMs with other flexible models to refine local fit.
C.J. Taylor, D.H. Cooper and J. Graham, Training models of shape from sets of examples, Proc. 3rd British Machine Vision Conf., Leeds, UK, 22-24 September 1992, pp. 9918. I21 J.A. Marchant, Adding grey level information to point distribution models using finite elements, Proc. 4th British Machine Vision Conf., Guildford, UK, 21-23 September 1993, pp. 309-318. [31 T.F. Cootes and C.J. Taylor, Modelling object appearance using the grey-level surface, Proc. 5th British Machine Vision Conf., York, UK, 13-16 September 1994, pp. 479-488. Modelling grey-level surfaces [41 CM. Onyango and J.A. Marchant, using three dimensional point distribution models, Image and Vision Computing, 14( 10) (1996). 151G. Wyszecki and W.S. Stiles, Color Science: Concepts and Methods, Quantitative Data and Formulae, John Wiley and Sons, New York, 1982. McGraw Hill, 161K.B. Benson, Television Engineering Handbook, New York, 1986. 171J.A. Marchant and C.M. Onyango, Fitting grey level point distribution models to animals in scenes, Image and Vision Computing, 13(l) (1995) 3-12. PI O.C. Zienkiewicz, The Finite Element Method, 3rd edn., McGrawHill, Maidenhead, 1977. 191W.H. Press, B.P. Flannery, S.A. Teukolsky and W.T. Vetterling, Numerical Recipes in C, Cambridge University Press, Cambridge, 1988.