- Email: [email protected]
A morphological assessment system for ‘show quality’ bovine livestock based on image analysis
Computers and Electronics in Agriculture 78 (2011) 80–87
Contents lists available at ScienceDirect
Computers and Electronics in Agriculture journal homepage: www.elsevier.com/locate/compag
A morphological assessment system for ‘show quality’ bovine livestock based on image analysis Horacio M. González-Velasco a,⇑, Carlos J. García-Orellana b, Miguel Macías-Macías c, Ramón Gallardo-Caballero a, Antonio García-Manso a a b c
CAPI Research Group, Polytechnic School, University of Extremadura, Av. Universidad, s/n, 10003 Cáceres, Spain CAPI Research Group, Faculty of Sciences, University of Extremadura, Av. Elvas, s/n, 06006 Badajoz, Spain CAPI Research Group, Mérida University Center, University of Extremadura, Av. Santa Teresa de Jornet, 38, 06800 Mérida, Spain
a r t i c l e
i n f o
Article history: Received 10 June 2010 Received in revised form 24 May 2011 Accepted 12 June 2011
Keywords: Morphological assessment Bovine livestock Genetic algorithms Artificial neural networks Image analysis
a b s t r a c t Morphological assessment is one important parameter considered in conservation and improvement programs for bovine livestock. This assessment process consists of scoring an animal based on its morphology and is normally carried out by highly qualified staff. These animals are all of agreed ‘show quality’ and hence they are morphologically very similar. This paper presents a system designed to provide an assessment based on a lateral image of the cow. The system consists of two main parts: a feature extraction stage, to reduce the information on the cow in the image to a set of parameters, and a neural network stage to provide a score based on that set of parameters. For the image analysis section, a model of the animal is constructed by means of the point distribution model (PDM) technique. Later, that model is used in the searching process within each image, which is implemented using genetic algorithms (GAs). As a result of this stage, the vector of weights that describe the deviation of the given shape from the mean is obtained. This vector is used in the second stage, where a multilayer perceptron is trained to provide the desired assessment, using the scores given by experts for selected cows. The system has been tested with 138 images corresponding to 44 individuals of a special rustic breed, with very promising results, given that the information contained in only one view of the cow can not be considered complete. Ó 2011 Elsevier B.V. All rights reserved.
1. Introduction For the control and conservation of purity in certain breeds of bovine livestock, European legislation imposes the creation of herd-books, or records of the information stored about existing animals of that concrete breed. A morphological assessment is one of the parameters that are stored in those herd-books for each pure-bred animal and is very important in many tasks that are performed with them. In general, the morphological assessment assigns to an animal a score on the basis of its external morphology (hence, it is also known as a conformational evaluation or an external evaluation) and the similarity of its characteristics to the standard of the race. This score is usually normalized to between 0 and 100 points and is evaluated by highly qualified and experienced staff (known as assessors or, more generally, experts).
⇑ Corresponding author. Tel.: +34 927 257 245; fax: +34 927 257 203. E-mail addresses: [email protected] (H.M. González-Velasco), carlos@capi. unex.es (C.J. García-Orellana), [email protected] (M. Macías-Macías), ramon@ capi.unex.es (R. Gallardo-Caballero), [email protected] (A. García-Manso). 0168-1699/$ - see front matter Ó 2011 Elsevier B.V. All rights reserved. doi:10.1016/j.compag.2011.06.003
This personnel must behave in a ‘‘neutral way’’, and they must be trained in a centralized manner to maintain the criteria. There are several methods to carry out the morphological assessment, which usually considers measurable traits and traits that are directly evaluated by visual inspection. However, with rustic meat-producing breeds, normally bred in extensive farms, the morphological assessment is only based on visual inspection. For the most important breeds in our area (such as the Blanca Cacereña considered in the experiments), this method consists of scoring ten traits, each including several aspects of the standard of the race (see Table 1). Those traits are assessed by a continuous value between 1 and 10 points, and the global score is a weighted sum of the partial scores. The weights are determined by the characteristics of the breed and the objectives to be reached with its control and selection. With this method, the high subjectivity involved in morphological assessment is disturbing. This, together with the need of uniformity in the criteria, leads us to consider the utility of a semiautomatic system to help in morphological assessment, as previously discussed in Goyache et al. (2001) and Alonso et al. (2007).
H.M. González-Velasco et al. / Computers and Electronics in Agriculture 78 (2011) 80–87 Table 1 Traits to be individually assessed for the breed considered. The weights for the final sum are included. Trait to be assessed
Weight (males)
Weight (fem.)
General aspect and figure Development of the body Head Neck, chest, withers and back Thorax and belly Back and loins Hindquarters and tail Thighs and buttocks Extremities Genital organs Shape and quality of the udder
1.6 1.2 0.5 0.5 1.0 1.5 1.2 1.3 0.7 0.5 –
1.6 1.2 0.5 0.4 1.0 1.3 1.3 1.3 1.0 – 0.5
Goyache et al. (2001) proposed the use of Artificial Intelligence to develop a morphological assessment system, with the aim of reproducing, in a systematic way, the behavior of the experts. The assessment for each of the traits is based in a set of zoometric measurements obtained from three digital images of the animal in different positions. Using several machine learning techniques, the system is trained with the individual assessments provided by experts to give a value for each of the traits. Although the results presented are quite good, the main drawback of the method is the need to locate 63 key points in the images to obtain the zoometric measures for each of the animals to be assessed. On the other hand, Alonso et al. (2007) described an assessment method for young bulls of a specific breed that can be carried out using just 3 measures plus the appreciation of the curvature of the round profile in the lateral view. As in the previous paper, the assessment method was developed using Artificial Intelligence tools, with the aim of evaluating the economic value of the carcass at a constant age. These two works, as well as the information received from the experts consulted, suggest that the assessment could be done quite accurately using a fundamental analysis of the profiles of the animal in three pictures corresponding to three positions of the cow (frontal, lateral and rear). In this paper, we present a first approach to an automatic morphological assessment system based on images of only one of the views: the lateral position. At the beginning of our project we decided to work with important breeds for our region, especially the breed ‘‘Blanca Cacereña’’ cited above. Besides, after discussing with the experts, it was decided to deal only with individuals of agreed ‘show quality’ (animals whose assessment is higher than 65 points, according to the regulation for this breed), and to develop a system that is able to distinguish the best individuals from those not so good. This capability is very important in contests, where all the participants are very good and the subjectivity of the assessors gives rise to uncountable problems. However, this problem must be considered as a very challenging one, due to the similarity of all the animals to be assessed. Moreover, considering only ‘show quality’ individuals made easier the process of capturing the images, because this breed is in danger of extinction, and thus most individuals available to be photographed are reared and controlled in farms depending from the Regional Government, and almost all of then are considered as ‘show quality’ animals. The proposed method for the morphological assessment system has a classical two-stage structure. First, a feature extraction is performed on the image, with the aim of representing the profile of the cow with a reduced set of parameters. Later, using these parameters, a supervised neural network system is trained with samples of assessments made by the experts. This way, their knowledge about assessment can be extracted, and their assessments can be approximately reproduced.
81
In the first stage, the main tasks are the parameterization of the cow’s boundary and its location in the image. Our proposal is to use point distribution models (Cootes et al., 1995) to parameterize the contours and genetic algorithms (GAs) (Goldberg, 1989; Haupt and Haupt, 2004) to search within the image. Several methods have been proposed in the past for similar tasks, including that described in our previous paper (González et al., 2003), in which PDMs were used to parameterize the contours of cows and several objective functions for the genetic algorithms search were studied, always using a conventional edge detection method as a potential image to drive the search. Other methods proposed in the past are mainly related to medical imaging. Hill et al. (1994) used genetic algorithms in combination with active shape models (Cootes et al., 1995) as a method for segmenting medical images, and they successfully applied that method to echocardiograms, MR images and radiographs. Mishra et al. (2003) proposed a system for detecting the boundary of the left ventricle in a sequence of cardiac images, where the contours are detected by optimizing an active contour model with genetic algorithms. Li and Chutatape (2003) present a system for automatically detecting the boundary of the optic disk in color fundus images, in which principal component analysis is used to initialize a modified active shape model. Another interesting method for boundary extraction using genetic algorithms is described in Arámbula Cosío (2008). In that paper, the automatic location of the boundary of the prostate in transurethral ultrasound images is determined. First, an active shape model is initialized based on pixel classification to estimate the prostate region, and next, an automatic adjustment is carried out using a multi-population genetic algorithm. More recently, Vera et al. (2010) proposed a method for myocardial border detection in ventriculograms. In that work, support vector machines were used to detect several anatomical landmarks, which form the active contour that is later evolved using genetic algorithms. In our proposal, point distribution models are applied to describe the contour of the object to be segmented, as in Arámbula Cosío (2008). These models are later used to search within the image using genetic algorithms, with a methodology adapted to deal with our outdoor color images. What we propose is a pre-processing of the image, consisting of an edge detection and selection procedure like that described in González et al. (2004) so that the new image, together with a simple objective function, lets us carry out an efficient search using genetic algorithms. As a result, the contours extracted are represented by a reduced set of parameters: the vector of weights that describe the deviation of the considered shape from the mean in the training set. In the second stage, we propose to use the contours extracted for a selected set of cows, and the assessments made by experts on those cows, to train a supervised neural network (a multilayer perceptron, Bishop and Hinton, 1995). Once the neural network is trained, the complete system can provide an assessment for each image presented at its input. In Section 2, the technique used to search the contours within the images is described in detail (e.g., shape modeling and objective function). Next, Section 3 provides details about the neural network used and the training method. The results obtained by applying the whole system to a set of images can be found in Section 4, and finally, our conclusions and possible improvements of our work are presented in Section 5.
2. First stage: image analysis The contour extraction problem can be converted into an optimization problem by considering two steps: the parameterization of the shape we want to search and the definition of a cost function
82
H.M. González-Velasco et al. / Computers and Electronics in Agriculture 78 (2011) 80–87
that quantitatively determines whether or not a contour is adjusted to the object. The general outline of the system used is shown in Fig. 1. As illustrated, for the contour modeling and parameterization, the PDM technique (Cootes et al., 1995) has been applied, which lets us restrict all the parameters and precisely define the search space. On the other hand, the objective function proposed is designed to place the contour over those areas of the image where edges corresponding to the object are detected. 2.1. Shape modeling and search space As shown in Fig. 1, the first step consists of appropriately representing the desired shape we want to search along with its possible variations. In some previous works (Hill et al., 1994; González et al., 2003) a general method known as PDM has been used, which uses deformable models representing the contours by means of ordered groups of points (Fig. 2) located at specific positions on the object, which are constructed statistically based on a set of examples. As a result of the process, the average shape of our object is obtained, and each contour can be represented mathematically by a vector ~ x such as
~ b x ¼ x~m þ P ~
ð1Þ
where x~m is the average shape, P is the matrix of eigenvectors of the covariance matrix and ~ b is a vector containing the weights for each eigenvector, which properly defines the contour in our description. Fortunately, by considering only a few eigenvectors corresponding to the largest eigenvalues of the covariance matrix, we can describe almost all the variations taking place in the training set. This representation (model) of the contour, however, is defined in the normalized space. To project instances of this model to the space of our image, a transformation that preserves the shape is required (translation ~ t ¼ ðtx ; ty Þ, rotation h and scaling s). Thus, every permitted contour in the image can be represented by a reduced set fs; h;~ t; ~ bg of parameters.
For the search space to be precisely defined, the limits for these parameters are required. As stated in Cootes et al. (1995), to maintain a shape similar to those in the training set, the parameters bi pffiffiffiffi pffiffiffiffi must be restricted to values ki 6 bi 6 ki , where ki are the eigenvalues of the covariance matrix. The transformation parameters can be limited, considering that we approximately know the position of the object within the image: the animal is usually in a horizontal position (angle restriction), not arbitrarily small (restriction in scale) and more or less centered (restriction in ~ t). In our specific case (lateral position), the cow contour is described using a set of 126 points (Fig. 2), 35 of which are significant. The model was constructed using a training set of 45 photographs, which were previously selected from our database of 138 images, in an attempt to cover the variations in position as much as possible. Once the calculations were made, no more than 10 eigenvalues were needed to represent virtually every possible variation (more than 90%). As can be seen in Fig. 2b, only the limbs in the foreground have been considered in the model, and the others have been eliminated. This is due to the fact that, in most cases, the limbs in the background are partially or totally hidden by the limbs in the foreground (see Fig. 2a), and it is thus very difficult to construct a contour model to take into account all the possible relative positions. Moreover, the experts consulted agreed that the profile with only the limbs in the foreground contains enough information to carry out the assessment. 2.2. Objective function and potential image extraction Once the shape is parameterized, the main step of this method is the definition of the objective function. In several consulted works (e.g., Hill and Taylor, 1992; Hill et al., 1994), this function is constructed in such a way that it reaches a minimum when strong edges of similar magnitude are located near the points of a certain instance of the model in the image. We use a similar approach, generating an intermediate grayscale image that acts
Fig. 1. General outline of the contour location system using genetic algorithms.
Fig. 2. (a) The images used. (b) The model of the cow contour in the lateral position.
83
H.M. González-Velasco et al. / Computers and Electronics in Agriculture 78 (2011) 80–87
as a potential image (inverted, so we have to maximize the ‘‘energy’’). This potential image could be obtained using a conventional edge detector. However, to avoid the presence of many local maxima in the objective function, it is important that the edge maps do not contain many more edges than those corresponding to the searched object. This is not always possible because of the presence of background objects. For this reason, the method devised by González et al. (2004), for edge selection is applied. In this method, a multilayer perceptron neural network is trained using a set of images for which the edges of the searched object are known. The selection system acts over a previously obtained maximum edge map, resulting in only those edges that are more likely to correspond to the object we are looking for, according to the learned criterion. With this potential image, we propose the following objective function to be maximized by the GA:
fL ð~ XÞ ¼
n2 X j¼0
!1 Kj
n2 X j¼0
0 @
X
obtained with the alternative configuration and the following parameters: t ¼ 10 (we have 14 parameters to represent a contour in the image), P ¼ 2000 individuals, P m ¼ 0:4, P c ¼ 0:45 and S ¼ 60%. It is remarkable that the best results are obtained with few modes of variation (with ten modes of variation, around 92% of the variations observed in the training set are explained), although up to 20 modes were considered. The system described above has the drawback of not assuring a certain success rate in the search. A similar problem is described in Yuen et al. (2001), where a method is proposed to guarantee a given probability of success by a repeated execution of the GA (repeated GA). In that paper, a method is given to calculate the number N of executions needed to reach a given probability of success PRGA . In our case, we used this method to calculate the minimum number of executions needed to obtain PRGA ¼ 0:95, concluding that N must be at least 4 executions.
1 Dg ðX; YÞA
ð2Þ
ðX;YÞ2r~j
where ~ X ¼ ðX 0 ; Y 0 ; X 1 ; Y 1 ; . . . ; X n1 ; Y n1 Þ is the set of points (coordinates of the image, integer values) that form one instance of our model, which can be easily obtained from the parameters using Eq. (1) and a transformation; ~ r j is the set of points (pixels of the image) that form the straight line joining ðX j ; Y j Þ and ðX jþ1 ; Y jþ1 Þ; K j is the number of points contained in such a set; and Dg ðX; YÞ is a Gaussian function of the distance from the point ðX; YÞ to the nearest edge in the potential image. This function is defined to have a maximum value Imax for distance 0 and must be decreasing, so that a value less than or equal to Imax =100 is obtained for a given reach distance DA , in our case 20 pixels. To avoid calculating distances within the objective function (which is executed several times), a gray-level distance map is generated, over which the instances of our contour are projected.
2.4. Fixing points to drive the search: method Using the repeated GA, the results are not as good as expected, so we decided to drive the search in an interactive manner by providing one or two fixed points. In the method described above, we codify the parameters fs; h;~ t; ~ bg in the chromosome, and in the process of decoding, we calculate the projection of the contour into the image using those parameters. Let us denote by ~ x ¼ ðx0 ; y0 ; x1 ; y1 ; . . . ; xn1 ; yn1 ÞT a vector of points of the contour in the normalized space, and ~ X ¼ ðX 0 ; Y 0 ; X 1 ; Y 1 ; . . . ; X n1 ; Y n1 ÞT a vector of points of the contour projected into the image. The vector ~ x can be calculated using Eq. (1) with the parameters ~ b. In order to obtain the vector ~ X, all the points must be transformed into the image space:
Xj Yj
Standard configuration: binary coding (with 16 bits), roulette selection method (using raw fitness), two-point crossover and mutation operators, and steady-state replacement method. Alternative configuration: real coding, binary tournament selection method (using linear normalization to calculate fitness), two-point crossover operator, mutation operators, and steadystate replacement method. To determine the most suitable scheme for our problem, along with the best values for the parameters involved (modes of variation t, population P, probabilities of crossover and mutation P c and P m , and replacement rate S), a secondary GA was applied whose chromosomes included those parameters and whose objective function was related to the efficiency shown by the coded configuration over a concrete set of images, for which the correct position of the contour was known. As a result, the best performance was
¼ ¼
tx tx ty
þ
ty
2.3. Genetic algorithm configuration After defining the search space and the objective function, wee must define two more aspects in order to have an operating scheme for GA. On the one hand, the general outline of the GA must be decided (e.g., coding, replacement and selection methods). On the other hand, values for the parameters that determine the algorithm (e.g., population size and mutation rate) must be given. As there is no definitive conclusion on the optimal GA scheme to be used in this kind of problem, two configurations were considered, based on the ideas in Davis (1991). The first is similar to the basic GA in Goldberg (1989), and we called it the standard configuration, while the other uses real coding and is called the alternative. They have the following characteristics:
ax ay
xj
!
yj
s sin h
þ
s cos h s sin h s cos h ! xj ay yj ax
ð3Þ
To include information on the fixed points, our proposal is not to codify some of the parameters into the chromosome but to calculate them using the equations described above. Therefore, we reduce the dimensionality of the search space, making it easier for the algorithm to find a good solution. 2.4.1. Fixing one or two points First, consider that we know the position of the point p, ðX p ; Y p Þ. Then we can leave the parameters ðt x ; ty Þ without codifying, and calculate them using Eqs. (1) and (3). With Eq. (1), we calculate the vector ~ x, and applying Eq. (3), we obtain
(
X p ¼ tx þ ax xp ay yp Y p ¼ ty þ ay xp þ ax yp
( and then
t x ¼ X p ax xp þ ay yp t y ¼ Y p ay xp ax yp
ð4Þ
Suppose now that we know the positions of the points p and q, ðX p ; Y p Þ and ðX q ; Y q Þ. In this case, we can leave two more parameters without codifying: s and h, apart from ðt x ; t y Þ. With Eq. (1), we again calculate the vector ~ x, and applying Eq. (3), we obtain
8 Xp > > > < Yp > > Xq > : Yq
¼ tx þ ax xp ay yp ¼ ty þ ay xp þ ax yp ¼ tx þ ax xq ay yq
ð5Þ
¼ ty þ ay xq þ ax yq
This is a system of linear equations in the variables tx , ty , ax and ay , and it is straightforward to solve.
84
H.M. González-Velasco et al. / Computers and Electronics in Agriculture 78 (2011) 80–87
2.4.2. Fixing more than two points We did not consider more than two fixed points for of two reasons. First, if we fix the positions of more than two points, we have to leave some of the shape parameters (components of the vector ~ b) without coding, thus restricting the allowable shapes that can satisfy the fixed points. Second, the results with one or two fixed points were good enough, and to consider more than two points would be mathematically complicated, thus slowing down the process of decoding the chromosome.
new configuration as optimal if the MSE over the validation set is lower than the previous MSE.
2.4.3. Interactive scheme We propose to use an approach similar to that of van Ginneken et al. (2003). First, we start running the GA search without fixed points. Then the user decides if the result is acceptable or not. If not, he also has to decide based on the result whether to fix two, one or zero points, fix the points and run the algorithm again. This process can be repeated until an acceptable fitting is reached.
4.1. First stage
3. Second stage: neural network Once the contour of the animal in the image has been located and parameterized, a system is needed to provide an assessment as a function of those parameters. Although our contours are characterized by 14 parameters, we discarded four of them (s, h, tx and ty ) because they provide information about the position of the animal within the image but not about the morphology of the animal. Hence, we use the vector ~ b as input for a multilayer perceptron (Bishop and Hinton, 1995) with one hidden layer, ten input neurons at the input and one output neuron, which gives us the value for the assessment. To train the network, we use a database of images of cows whose morphological assessments by experts are known. After splitting this set of data into two subsets (training and validation), the training is carried out using the back-propagation algorithm with momentum and halts when the mean squared error (MSE) obtained over the validation set reaches a minimum. Later, we repeat this process, increasing the network size and we choose the
4. Results and discussion We conducted experiments to test both stages of our system. On the one hand, we wanted to assess the performance of the image analysis stage. On the other hand, we aimed to test the ability of the system to reproduce the assessments of the experts.
We tested our method with 93 images (640 480 pixels size) from our database, which were not used in the construction of the shape model (see Figs. 3–5 for examples). Our goal was to test the performance of the system both without and with fixed points and to compare the results. To evaluate the resulting contour quantitatively, the mean distance between its points and their proper position has been proposed as a figure of merit. This implies that you have to know the correct positions of all the points in all the images of the database, which is a huge amount of work to calculate. For this reason, the distance was estimated with only seven very significant points of the contour. The points used were f20; 25; 40; 50; 78; 99; 114g, as shown in Fig. 2. In some experiments carried out with the images used to build the model (for which all the points are precisely located by experts), the mean distance calculated with all the points and calculated with the seven points differed only by 1 pixel. Therefore, those seven points were precisely located for all the images in the database and could be used to calculate the figure of merit for all the adjusted contours. For example, the mean distance to the contour is shown in the images of Figs. 3–5. The mean distance from each point of the resulting contour to the nearest point of the labeled one (in the normal direction) could also be considered as a figure of merit, which has been used in other similar applications and could yield better results. However, considering that our model is not a closed contour but includes five
Fig. 3. Two examples of contours with a mean distance close to 10 pixels.
H.M. González-Velasco et al. / Computers and Electronics in Agriculture 78 (2011) 80–87
85
Fig. 4. Some examples of the contour extraction stage, along with the final mean distance (in pixels) to the correct position, considering the seven points described in the text.
open sections, many different contours could produce very good values for the figure of merit but have the extreme points of each section at a considerable distance of their correct position. For this reason, that figure of merit was not considered. In addition, to define if a contour extraction is a ‘‘success’’ or not, we applied a hard limit of ten pixels to the mean distance, which we considered enough to guarantee a very good fitting. In Fig. 3, two images are presented with the extracted contours superimposed (notice that, for each example in Figs. 3–5, a monochrome image has been included alongside the color image that was used for the tests, in order to improve the visibility of the annotated contour). For these examples, the mean distances for the contours are very close to ten pixels. In general, the fitting is very good, and there is only a slight deviation in some places. First, we applied the contour extraction system without fixing points. In Table 2, we present the success rate and the mean of the figure of merit calculated over the 93 images (‘‘mean distance’’). Fig. 4a and b shows two examples corresponding to two cases of failure. The rate of success is not very high (61.3%), and this rate of failures cannot be considered acceptable. Analyzing the resulting contours, we found many cases of errors similar to that in Fig. 4b, where the contour is very well adjusted except for the limbs. The problem resides in the confusion between the limbs in the foreground and the limbs in the background, when both can be seen in the image due to the position of the animal.
To overcome this difficulty, in Section 2.4, we proposed an interactive scheme based on fixing the positions of one or two points in their correct locations. In the experiments, we made a systematic test of all possible combinations of one and two fixed points, considering the seven significant points referred above. It is important to note that those points were selected because they give an idea of the location of significant parts of the animal (for example, the head, with points 20 and 25, or the limbs, with points 78 and 114). In Table 2, results are presented for one and two fixed points, with the best and worst cases marked with an asterisk. Examining the best cases, we can see that fixing points is clearly helpful because the results fixing the point 78 and the points 78 and 114 are significantly better than those obtained without fixed points. However, the selection of the points to fix appears to be very important because the performance in the worst cases is very similar to that of the original (without fixed points). Moreover, we also observed that the best of the cases is obtained by fixing the points corresponding to the limbs (78 and 114), indicating that the problem of the automatic approach with the limbs can be solved almost completely with this technique. In Figs. 4c and 5d–f, we present some examples of results obtained by fixing points 78 and 114. Cases c, d and f correspond to successful searches, and case e is one of the four images that failed. In this last case, the problem is that the position of the cow is quite ‘‘atypical’’, particularly the rear limb. Thus, it is impossible for our
86
H.M. González-Velasco et al. / Computers and Electronics in Agriculture 78 (2011) 80–87
Fig. 5. More examples of the contour extraction stage, along with the final mean distance to the correct position.
model, with an ‘‘acceptable’’ shape (i.e., bi parameters within their limits, indicated in Section 2.1), to properly locate both the limbs (forced) and also the head. It seems that this cow should have been considered to build the PDM. Finally, case f is also remarkable, as the fitting is very good even though the frontal limb is partially occluded. 4.2. Second stage: assessment In this stage, the aim was to test the ability and performance of our system in reproducing the morphological assessment carried out by experts, both globally and for each of the traits. It is important to note that all the 44 individuals we have worked with have a good assessment given by the experts, i.e., they are considered at least acceptable individuals of the breed (which means, for this breed, that they have an assessment higher than 65 points). For this reason, our system cannot be used to distinguish between acceptable and non-acceptable individuals, but can be useful separating very good individuals from those which are no so good, for instance in contests. With this main goal, the method described in Section 3 was used to train 11 neural networks that yield as their outputs the global assessment (one network) or the assessments of each of the aspects (10 networks). As stated in Section 3, the assessment given by an expert for the cows was used in the training process. As
the scale is different for the global assessment (0–100) and for the trait assessment (0–10), the data were normalized to the interval ½0; 1 in advance. For the tests, as we do not have a large amount of data, a leaveone-out strategy was followed. For each of the 134 images (for which we had the contour extracted), the remaining 133 were randomly divided into two subsets: training and validation. Using these subsets, networks were trained with between 5 and 40 neurons in the hidden layer, and the one offering the best results was selected. Finally, the contour from the original image was introduced to the network to be assessed, and the result was compared with the assessment provided by the expert to generate the error. As mentioned above, this process was carried out both with the global assessment and with the individual assessments for each of the traits. Table 3 presents the mean and the standard deviation of the relative value of the error (the absolute error divided by the maximum value of the scale, expressed as a percentage), both for the global assessment and for each of the individual traits. Fig. 6 also shows a box plot of the errors obtained in each of the experiments to give an idea of their distribution. We obtained a mean error of 5.297% (5.297 points) in the global score with a relatively high sigma, indicating that there is high variability in the errors of the assessments. This can also be observed in Fig. 6, where we see that the boxes and the whiskers are quite large. In our opinion, there are two main causes for these errors.
H.M. González-Velasco et al. / Computers and Electronics in Agriculture 78 (2011) 80–87 Table 2 The number of successes and the mean distance obtained by the method applied to our database of 93 images (the best and worst cases are marked with an asterisk). Fixed points
Successes
Mean
Fixed points
Successes
Mean
None 20 25 40⁄ 50 78⁄ 99 114 20, 25 20, 40 20, 50 20, 78 20, 99 20, 114 25, 40
57 56 61 52 58 80 60 71 60 63 57 84 61 76 55
10.03 9.84 9.05 10.49 9.54 6.52 9.69 8.14 9.72 8.95 9.16 5.78 9.09 7.22 9.42
25, 25, 25, 25, 40, 40, 40, 40, 50, 50, 50, 78, 78, 99,
60 86 65 77 57 83 53 73 86 54 71 83 89 72
8.82 5.46 8.46 6.92 10.11 6.13 10.02 7.44 5.74 10.33 7.45 6.17 5.42 7.60
(61.3%) (60.2%) (65.6%) (55.9%) (62.4%) (86.0%) (64.5%) (76.3%) (64.5%) (67.7%) (61.3%) (90.3%) (65.6%) (81.7%) (59.1%)
50 78 99 114 50 78 99 114 78 99⁄ 114 99 114⁄ 114
(64.5%) (92.5%) (69.9%) (82.8%) (61.3%) (89.2%) (57.0%) (78.5%) (92.5%) (58.1%) (76.3%) (89.2%) (95.7%) (77.4%)
Table 3 Mean and standard deviation of the relative value of the error, for the global assessment and for the individual traits. Trait
Description
Mean rel. error
r
1 2 3 4 5 6 7 8 9 10
Global assessment General aspect and figure Development of the body Head Neck, chest, withers and back Thorax and belly Back and loins Hindquarters and tail Thighs and buttocks Extremities Shape and quality of the udder
5.297 7.290 6.196 7.119 8.468 8.073 6.953 8.106 7.790 6.640 10.943
4.145 4.402 5.087 5.680 5.094 5.067 5.058 6.116 6.621 6.917 7.654
87
given in steps of 1 point (405 out of 440 partial assessments, 10 for each cow). If we consider the ‘‘resolution’’ as 1 point, then the corresponding ‘‘uncertainty’’ (or ‘‘error’’) in the assessments of the traits would be 0.5 points, and if we propagate that uncertainty to the global assessment (recall that it is calculated as a weighted sum), each global assessment can be considered to have an uncertainty of ±5 points. In that sense, our results can be considered as quite good and promising because with only partial input information, we obtain a low error in many cases, and the mean over all the cases is very close to 5 points. 5. Conclusions and future research In this paper, we have presented a system designed to provide a morphological assessment of a cow based on a lateral image of the animal. The system consists of two parts: one dedicated to the analysis of the image and the extraction of the profile of the animal, and the other designed to provide an assessment with the parameterized contour. The first part performs very well with little human intervention (fixing points), reaching a 95% of success rate. The second stage, based on a neural network, also gave very promising results, taking into account the fact that not all the information required for the morphological assessment can be observed in one view of the animal. To improve the method, two lines of research are being considered at this moment. First, we are working to incorporate the contours of the rear and frontal views of the cows as inputs to the neural network, complementing the information contained in the lateral view. Additionally, we intend to eliminate from the profiles all the parts that are affected by a change in the pose of the animal (for instance, the limbs in the lateral position). References
Fig. 6. A box plot of the errors obtained for each trait and for the global assessment.
First, the limbs were included in the contours, but the position of the limbs can change significantly from one image to another of the same (or a different) animal, causing a great difference between the shape parameters of the contours. Second, although many of the traits to be evaluated can be observed very well in the image of the lateral position, there are others that cannot be observed so well and even some that cannot be observed at all. The clearest example is trait 10 (shape and quality of the udder), which has the highest error and the largest boxes and whiskers because the udder is not visible in the contour that we have defined. In any event, it is important to note that in the data from the experts, the assessment for the individual traits is almost always
Alonso, J., Bahamonde, A., Villa, A., Rodríguez, A., 2007. Morphological assessment of beef cattle according to carcass value. Livestock Science 107, 265–273. Arámbula Cosío, F., 2008. Automatic initialization of an active shape model of the prostate. Medical Image Analysis 12, 469–483. Bishop, C.M., Hinton, G., 1995. Neural Networks for Pattern Recognition. Clarendon Press. Cootes, T.F., Taylor, C.J., Cooper, D.H., Graham, J., 1995. Active shape models – their training and application. Computer Vision and Image Understanding 61, 38–59. Davis, L., 1991. Handbook of Genetic Algorithms. Van Nostrand Reinhold. Goldberg, D.E., 1989. Genetic Algorithms in Search, Optimization and Machine Learning. Addison-Wesley. González, H.M., García, C.J., López, F.J., Macías, M., Gallardo, R., 2003. Segmentation of bovine livestock images using GA and ASM in a two-step approach. International Journal of Pattern Recognition and Artificial Intelligence 17, 601–616. González, H.M., García, C.J., Macías, M., López, F.J., Acevedo, M.I., 2004. Neuralnetworks-based edges selector for boundary extraction problems. Image and Vision Computing 22, 1129–1135. Goyache, F., Del Coz, J.J., Quevedo, J.R., et al., 2001. Using artificial intelligence to design and implement a morphological assessment system in beef cattle. Animal Science 73, 49–60. Haupt, R.L., Haupt, S.E., 2004. Practical Genetic Algorithms 2e. John Wiley. Hill, A., Cootes, T.F., Taylor, C.J., Lindley, K., 1994. Medical image interpretation: a generic approach using deformable templates. Journal of Medical Informatics 19, 47–59. Hill, A., Taylor, C.J., 1992. Model-based image interpretation using genetic algorithms. Image and Vision Computing 10, 295–300. Li, H., Chutatape, O., 2003. Boundary detection of optic disk by a modified ASM method. Pattern Recognition 36, 2093–2104. Mishra, A., Dutta, P.K., Ghosh, M.K., 2003. A ga based approach for boundary detection of left ventricle with echocardiographic image sequences. Image and Vision Computing 21, 967–976. van Ginneken, B., de Bruijne, M., Loog, M., Viergever, M., 2003. Interactive shape models. In: Sonka, M., Michael Fitzpatrick, J. (Eds.), Medical Imaging 2003: Image Processing. SPIE – The International Society for Optical Engineering. Proceedings of SPIE, vol. 5032, pp. 1206–1216. Vera, M., Bravo, A., Medina, R., 2010. Myocardial border detection from ventriculograms using support vector machines and real-coded genetic algorithms. Computers in Biology and Medicine 40, 446–455. Yuen, S.Y., Fong, C.K., Lam, H.S., 2001. Guaranteeing the probability of success using repeated runs of genetic algorithm. Image and Vision Computing 19, 551–560.
Contents lists available at ScienceDirect
Computers and Electronics in Agriculture journal homepage: www.elsevier.com/locate/compag
A morphological assessment system for ‘show quality’ bovine livestock based on image analysis Horacio M. González-Velasco a,⇑, Carlos J. García-Orellana b, Miguel Macías-Macías c, Ramón Gallardo-Caballero a, Antonio García-Manso a a b c
CAPI Research Group, Polytechnic School, University of Extremadura, Av. Universidad, s/n, 10003 Cáceres, Spain CAPI Research Group, Faculty of Sciences, University of Extremadura, Av. Elvas, s/n, 06006 Badajoz, Spain CAPI Research Group, Mérida University Center, University of Extremadura, Av. Santa Teresa de Jornet, 38, 06800 Mérida, Spain
a r t i c l e
i n f o
Article history: Received 10 June 2010 Received in revised form 24 May 2011 Accepted 12 June 2011
Keywords: Morphological assessment Bovine livestock Genetic algorithms Artificial neural networks Image analysis
a b s t r a c t Morphological assessment is one important parameter considered in conservation and improvement programs for bovine livestock. This assessment process consists of scoring an animal based on its morphology and is normally carried out by highly qualified staff. These animals are all of agreed ‘show quality’ and hence they are morphologically very similar. This paper presents a system designed to provide an assessment based on a lateral image of the cow. The system consists of two main parts: a feature extraction stage, to reduce the information on the cow in the image to a set of parameters, and a neural network stage to provide a score based on that set of parameters. For the image analysis section, a model of the animal is constructed by means of the point distribution model (PDM) technique. Later, that model is used in the searching process within each image, which is implemented using genetic algorithms (GAs). As a result of this stage, the vector of weights that describe the deviation of the given shape from the mean is obtained. This vector is used in the second stage, where a multilayer perceptron is trained to provide the desired assessment, using the scores given by experts for selected cows. The system has been tested with 138 images corresponding to 44 individuals of a special rustic breed, with very promising results, given that the information contained in only one view of the cow can not be considered complete. Ó 2011 Elsevier B.V. All rights reserved.
1. Introduction For the control and conservation of purity in certain breeds of bovine livestock, European legislation imposes the creation of herd-books, or records of the information stored about existing animals of that concrete breed. A morphological assessment is one of the parameters that are stored in those herd-books for each pure-bred animal and is very important in many tasks that are performed with them. In general, the morphological assessment assigns to an animal a score on the basis of its external morphology (hence, it is also known as a conformational evaluation or an external evaluation) and the similarity of its characteristics to the standard of the race. This score is usually normalized to between 0 and 100 points and is evaluated by highly qualified and experienced staff (known as assessors or, more generally, experts).
⇑ Corresponding author. Tel.: +34 927 257 245; fax: +34 927 257 203. E-mail addresses: [email protected] (H.M. González-Velasco), carlos@capi. unex.es (C.J. García-Orellana), [email protected] (M. Macías-Macías), ramon@ capi.unex.es (R. Gallardo-Caballero), [email protected] (A. García-Manso). 0168-1699/$ - see front matter Ó 2011 Elsevier B.V. All rights reserved. doi:10.1016/j.compag.2011.06.003
This personnel must behave in a ‘‘neutral way’’, and they must be trained in a centralized manner to maintain the criteria. There are several methods to carry out the morphological assessment, which usually considers measurable traits and traits that are directly evaluated by visual inspection. However, with rustic meat-producing breeds, normally bred in extensive farms, the morphological assessment is only based on visual inspection. For the most important breeds in our area (such as the Blanca Cacereña considered in the experiments), this method consists of scoring ten traits, each including several aspects of the standard of the race (see Table 1). Those traits are assessed by a continuous value between 1 and 10 points, and the global score is a weighted sum of the partial scores. The weights are determined by the characteristics of the breed and the objectives to be reached with its control and selection. With this method, the high subjectivity involved in morphological assessment is disturbing. This, together with the need of uniformity in the criteria, leads us to consider the utility of a semiautomatic system to help in morphological assessment, as previously discussed in Goyache et al. (2001) and Alonso et al. (2007).
H.M. González-Velasco et al. / Computers and Electronics in Agriculture 78 (2011) 80–87 Table 1 Traits to be individually assessed for the breed considered. The weights for the final sum are included. Trait to be assessed
Weight (males)
Weight (fem.)
General aspect and figure Development of the body Head Neck, chest, withers and back Thorax and belly Back and loins Hindquarters and tail Thighs and buttocks Extremities Genital organs Shape and quality of the udder
1.6 1.2 0.5 0.5 1.0 1.5 1.2 1.3 0.7 0.5 –
1.6 1.2 0.5 0.4 1.0 1.3 1.3 1.3 1.0 – 0.5
Goyache et al. (2001) proposed the use of Artificial Intelligence to develop a morphological assessment system, with the aim of reproducing, in a systematic way, the behavior of the experts. The assessment for each of the traits is based in a set of zoometric measurements obtained from three digital images of the animal in different positions. Using several machine learning techniques, the system is trained with the individual assessments provided by experts to give a value for each of the traits. Although the results presented are quite good, the main drawback of the method is the need to locate 63 key points in the images to obtain the zoometric measures for each of the animals to be assessed. On the other hand, Alonso et al. (2007) described an assessment method for young bulls of a specific breed that can be carried out using just 3 measures plus the appreciation of the curvature of the round profile in the lateral view. As in the previous paper, the assessment method was developed using Artificial Intelligence tools, with the aim of evaluating the economic value of the carcass at a constant age. These two works, as well as the information received from the experts consulted, suggest that the assessment could be done quite accurately using a fundamental analysis of the profiles of the animal in three pictures corresponding to three positions of the cow (frontal, lateral and rear). In this paper, we present a first approach to an automatic morphological assessment system based on images of only one of the views: the lateral position. At the beginning of our project we decided to work with important breeds for our region, especially the breed ‘‘Blanca Cacereña’’ cited above. Besides, after discussing with the experts, it was decided to deal only with individuals of agreed ‘show quality’ (animals whose assessment is higher than 65 points, according to the regulation for this breed), and to develop a system that is able to distinguish the best individuals from those not so good. This capability is very important in contests, where all the participants are very good and the subjectivity of the assessors gives rise to uncountable problems. However, this problem must be considered as a very challenging one, due to the similarity of all the animals to be assessed. Moreover, considering only ‘show quality’ individuals made easier the process of capturing the images, because this breed is in danger of extinction, and thus most individuals available to be photographed are reared and controlled in farms depending from the Regional Government, and almost all of then are considered as ‘show quality’ animals. The proposed method for the morphological assessment system has a classical two-stage structure. First, a feature extraction is performed on the image, with the aim of representing the profile of the cow with a reduced set of parameters. Later, using these parameters, a supervised neural network system is trained with samples of assessments made by the experts. This way, their knowledge about assessment can be extracted, and their assessments can be approximately reproduced.
81
In the first stage, the main tasks are the parameterization of the cow’s boundary and its location in the image. Our proposal is to use point distribution models (Cootes et al., 1995) to parameterize the contours and genetic algorithms (GAs) (Goldberg, 1989; Haupt and Haupt, 2004) to search within the image. Several methods have been proposed in the past for similar tasks, including that described in our previous paper (González et al., 2003), in which PDMs were used to parameterize the contours of cows and several objective functions for the genetic algorithms search were studied, always using a conventional edge detection method as a potential image to drive the search. Other methods proposed in the past are mainly related to medical imaging. Hill et al. (1994) used genetic algorithms in combination with active shape models (Cootes et al., 1995) as a method for segmenting medical images, and they successfully applied that method to echocardiograms, MR images and radiographs. Mishra et al. (2003) proposed a system for detecting the boundary of the left ventricle in a sequence of cardiac images, where the contours are detected by optimizing an active contour model with genetic algorithms. Li and Chutatape (2003) present a system for automatically detecting the boundary of the optic disk in color fundus images, in which principal component analysis is used to initialize a modified active shape model. Another interesting method for boundary extraction using genetic algorithms is described in Arámbula Cosío (2008). In that paper, the automatic location of the boundary of the prostate in transurethral ultrasound images is determined. First, an active shape model is initialized based on pixel classification to estimate the prostate region, and next, an automatic adjustment is carried out using a multi-population genetic algorithm. More recently, Vera et al. (2010) proposed a method for myocardial border detection in ventriculograms. In that work, support vector machines were used to detect several anatomical landmarks, which form the active contour that is later evolved using genetic algorithms. In our proposal, point distribution models are applied to describe the contour of the object to be segmented, as in Arámbula Cosío (2008). These models are later used to search within the image using genetic algorithms, with a methodology adapted to deal with our outdoor color images. What we propose is a pre-processing of the image, consisting of an edge detection and selection procedure like that described in González et al. (2004) so that the new image, together with a simple objective function, lets us carry out an efficient search using genetic algorithms. As a result, the contours extracted are represented by a reduced set of parameters: the vector of weights that describe the deviation of the considered shape from the mean in the training set. In the second stage, we propose to use the contours extracted for a selected set of cows, and the assessments made by experts on those cows, to train a supervised neural network (a multilayer perceptron, Bishop and Hinton, 1995). Once the neural network is trained, the complete system can provide an assessment for each image presented at its input. In Section 2, the technique used to search the contours within the images is described in detail (e.g., shape modeling and objective function). Next, Section 3 provides details about the neural network used and the training method. The results obtained by applying the whole system to a set of images can be found in Section 4, and finally, our conclusions and possible improvements of our work are presented in Section 5.
2. First stage: image analysis The contour extraction problem can be converted into an optimization problem by considering two steps: the parameterization of the shape we want to search and the definition of a cost function
82
H.M. González-Velasco et al. / Computers and Electronics in Agriculture 78 (2011) 80–87
that quantitatively determines whether or not a contour is adjusted to the object. The general outline of the system used is shown in Fig. 1. As illustrated, for the contour modeling and parameterization, the PDM technique (Cootes et al., 1995) has been applied, which lets us restrict all the parameters and precisely define the search space. On the other hand, the objective function proposed is designed to place the contour over those areas of the image where edges corresponding to the object are detected. 2.1. Shape modeling and search space As shown in Fig. 1, the first step consists of appropriately representing the desired shape we want to search along with its possible variations. In some previous works (Hill et al., 1994; González et al., 2003) a general method known as PDM has been used, which uses deformable models representing the contours by means of ordered groups of points (Fig. 2) located at specific positions on the object, which are constructed statistically based on a set of examples. As a result of the process, the average shape of our object is obtained, and each contour can be represented mathematically by a vector ~ x such as
~ b x ¼ x~m þ P ~
ð1Þ
where x~m is the average shape, P is the matrix of eigenvectors of the covariance matrix and ~ b is a vector containing the weights for each eigenvector, which properly defines the contour in our description. Fortunately, by considering only a few eigenvectors corresponding to the largest eigenvalues of the covariance matrix, we can describe almost all the variations taking place in the training set. This representation (model) of the contour, however, is defined in the normalized space. To project instances of this model to the space of our image, a transformation that preserves the shape is required (translation ~ t ¼ ðtx ; ty Þ, rotation h and scaling s). Thus, every permitted contour in the image can be represented by a reduced set fs; h;~ t; ~ bg of parameters.
For the search space to be precisely defined, the limits for these parameters are required. As stated in Cootes et al. (1995), to maintain a shape similar to those in the training set, the parameters bi pffiffiffiffi pffiffiffiffi must be restricted to values ki 6 bi 6 ki , where ki are the eigenvalues of the covariance matrix. The transformation parameters can be limited, considering that we approximately know the position of the object within the image: the animal is usually in a horizontal position (angle restriction), not arbitrarily small (restriction in scale) and more or less centered (restriction in ~ t). In our specific case (lateral position), the cow contour is described using a set of 126 points (Fig. 2), 35 of which are significant. The model was constructed using a training set of 45 photographs, which were previously selected from our database of 138 images, in an attempt to cover the variations in position as much as possible. Once the calculations were made, no more than 10 eigenvalues were needed to represent virtually every possible variation (more than 90%). As can be seen in Fig. 2b, only the limbs in the foreground have been considered in the model, and the others have been eliminated. This is due to the fact that, in most cases, the limbs in the background are partially or totally hidden by the limbs in the foreground (see Fig. 2a), and it is thus very difficult to construct a contour model to take into account all the possible relative positions. Moreover, the experts consulted agreed that the profile with only the limbs in the foreground contains enough information to carry out the assessment. 2.2. Objective function and potential image extraction Once the shape is parameterized, the main step of this method is the definition of the objective function. In several consulted works (e.g., Hill and Taylor, 1992; Hill et al., 1994), this function is constructed in such a way that it reaches a minimum when strong edges of similar magnitude are located near the points of a certain instance of the model in the image. We use a similar approach, generating an intermediate grayscale image that acts
Fig. 1. General outline of the contour location system using genetic algorithms.
Fig. 2. (a) The images used. (b) The model of the cow contour in the lateral position.
83
H.M. González-Velasco et al. / Computers and Electronics in Agriculture 78 (2011) 80–87
as a potential image (inverted, so we have to maximize the ‘‘energy’’). This potential image could be obtained using a conventional edge detector. However, to avoid the presence of many local maxima in the objective function, it is important that the edge maps do not contain many more edges than those corresponding to the searched object. This is not always possible because of the presence of background objects. For this reason, the method devised by González et al. (2004), for edge selection is applied. In this method, a multilayer perceptron neural network is trained using a set of images for which the edges of the searched object are known. The selection system acts over a previously obtained maximum edge map, resulting in only those edges that are more likely to correspond to the object we are looking for, according to the learned criterion. With this potential image, we propose the following objective function to be maximized by the GA:
fL ð~ XÞ ¼
n2 X j¼0
!1 Kj
n2 X j¼0
0 @
X
obtained with the alternative configuration and the following parameters: t ¼ 10 (we have 14 parameters to represent a contour in the image), P ¼ 2000 individuals, P m ¼ 0:4, P c ¼ 0:45 and S ¼ 60%. It is remarkable that the best results are obtained with few modes of variation (with ten modes of variation, around 92% of the variations observed in the training set are explained), although up to 20 modes were considered. The system described above has the drawback of not assuring a certain success rate in the search. A similar problem is described in Yuen et al. (2001), where a method is proposed to guarantee a given probability of success by a repeated execution of the GA (repeated GA). In that paper, a method is given to calculate the number N of executions needed to reach a given probability of success PRGA . In our case, we used this method to calculate the minimum number of executions needed to obtain PRGA ¼ 0:95, concluding that N must be at least 4 executions.
1 Dg ðX; YÞA
ð2Þ
ðX;YÞ2r~j
where ~ X ¼ ðX 0 ; Y 0 ; X 1 ; Y 1 ; . . . ; X n1 ; Y n1 Þ is the set of points (coordinates of the image, integer values) that form one instance of our model, which can be easily obtained from the parameters using Eq. (1) and a transformation; ~ r j is the set of points (pixels of the image) that form the straight line joining ðX j ; Y j Þ and ðX jþ1 ; Y jþ1 Þ; K j is the number of points contained in such a set; and Dg ðX; YÞ is a Gaussian function of the distance from the point ðX; YÞ to the nearest edge in the potential image. This function is defined to have a maximum value Imax for distance 0 and must be decreasing, so that a value less than or equal to Imax =100 is obtained for a given reach distance DA , in our case 20 pixels. To avoid calculating distances within the objective function (which is executed several times), a gray-level distance map is generated, over which the instances of our contour are projected.
2.4. Fixing points to drive the search: method Using the repeated GA, the results are not as good as expected, so we decided to drive the search in an interactive manner by providing one or two fixed points. In the method described above, we codify the parameters fs; h;~ t; ~ bg in the chromosome, and in the process of decoding, we calculate the projection of the contour into the image using those parameters. Let us denote by ~ x ¼ ðx0 ; y0 ; x1 ; y1 ; . . . ; xn1 ; yn1 ÞT a vector of points of the contour in the normalized space, and ~ X ¼ ðX 0 ; Y 0 ; X 1 ; Y 1 ; . . . ; X n1 ; Y n1 ÞT a vector of points of the contour projected into the image. The vector ~ x can be calculated using Eq. (1) with the parameters ~ b. In order to obtain the vector ~ X, all the points must be transformed into the image space:
Xj Yj
Standard configuration: binary coding (with 16 bits), roulette selection method (using raw fitness), two-point crossover and mutation operators, and steady-state replacement method. Alternative configuration: real coding, binary tournament selection method (using linear normalization to calculate fitness), two-point crossover operator, mutation operators, and steadystate replacement method. To determine the most suitable scheme for our problem, along with the best values for the parameters involved (modes of variation t, population P, probabilities of crossover and mutation P c and P m , and replacement rate S), a secondary GA was applied whose chromosomes included those parameters and whose objective function was related to the efficiency shown by the coded configuration over a concrete set of images, for which the correct position of the contour was known. As a result, the best performance was
¼ ¼
tx tx ty
þ
ty
2.3. Genetic algorithm configuration After defining the search space and the objective function, wee must define two more aspects in order to have an operating scheme for GA. On the one hand, the general outline of the GA must be decided (e.g., coding, replacement and selection methods). On the other hand, values for the parameters that determine the algorithm (e.g., population size and mutation rate) must be given. As there is no definitive conclusion on the optimal GA scheme to be used in this kind of problem, two configurations were considered, based on the ideas in Davis (1991). The first is similar to the basic GA in Goldberg (1989), and we called it the standard configuration, while the other uses real coding and is called the alternative. They have the following characteristics:
ax ay
xj
!
yj
s sin h
þ
s cos h s sin h s cos h ! xj ay yj ax
ð3Þ
To include information on the fixed points, our proposal is not to codify some of the parameters into the chromosome but to calculate them using the equations described above. Therefore, we reduce the dimensionality of the search space, making it easier for the algorithm to find a good solution. 2.4.1. Fixing one or two points First, consider that we know the position of the point p, ðX p ; Y p Þ. Then we can leave the parameters ðt x ; ty Þ without codifying, and calculate them using Eqs. (1) and (3). With Eq. (1), we calculate the vector ~ x, and applying Eq. (3), we obtain
(
X p ¼ tx þ ax xp ay yp Y p ¼ ty þ ay xp þ ax yp
( and then
t x ¼ X p ax xp þ ay yp t y ¼ Y p ay xp ax yp
ð4Þ
Suppose now that we know the positions of the points p and q, ðX p ; Y p Þ and ðX q ; Y q Þ. In this case, we can leave two more parameters without codifying: s and h, apart from ðt x ; t y Þ. With Eq. (1), we again calculate the vector ~ x, and applying Eq. (3), we obtain
8 Xp > > > < Yp > > Xq > : Yq
¼ tx þ ax xp ay yp ¼ ty þ ay xp þ ax yp ¼ tx þ ax xq ay yq
ð5Þ
¼ ty þ ay xq þ ax yq
This is a system of linear equations in the variables tx , ty , ax and ay , and it is straightforward to solve.
84
H.M. González-Velasco et al. / Computers and Electronics in Agriculture 78 (2011) 80–87
2.4.2. Fixing more than two points We did not consider more than two fixed points for of two reasons. First, if we fix the positions of more than two points, we have to leave some of the shape parameters (components of the vector ~ b) without coding, thus restricting the allowable shapes that can satisfy the fixed points. Second, the results with one or two fixed points were good enough, and to consider more than two points would be mathematically complicated, thus slowing down the process of decoding the chromosome.
new configuration as optimal if the MSE over the validation set is lower than the previous MSE.
2.4.3. Interactive scheme We propose to use an approach similar to that of van Ginneken et al. (2003). First, we start running the GA search without fixed points. Then the user decides if the result is acceptable or not. If not, he also has to decide based on the result whether to fix two, one or zero points, fix the points and run the algorithm again. This process can be repeated until an acceptable fitting is reached.
4.1. First stage
3. Second stage: neural network Once the contour of the animal in the image has been located and parameterized, a system is needed to provide an assessment as a function of those parameters. Although our contours are characterized by 14 parameters, we discarded four of them (s, h, tx and ty ) because they provide information about the position of the animal within the image but not about the morphology of the animal. Hence, we use the vector ~ b as input for a multilayer perceptron (Bishop and Hinton, 1995) with one hidden layer, ten input neurons at the input and one output neuron, which gives us the value for the assessment. To train the network, we use a database of images of cows whose morphological assessments by experts are known. After splitting this set of data into two subsets (training and validation), the training is carried out using the back-propagation algorithm with momentum and halts when the mean squared error (MSE) obtained over the validation set reaches a minimum. Later, we repeat this process, increasing the network size and we choose the
4. Results and discussion We conducted experiments to test both stages of our system. On the one hand, we wanted to assess the performance of the image analysis stage. On the other hand, we aimed to test the ability of the system to reproduce the assessments of the experts.
We tested our method with 93 images (640 480 pixels size) from our database, which were not used in the construction of the shape model (see Figs. 3–5 for examples). Our goal was to test the performance of the system both without and with fixed points and to compare the results. To evaluate the resulting contour quantitatively, the mean distance between its points and their proper position has been proposed as a figure of merit. This implies that you have to know the correct positions of all the points in all the images of the database, which is a huge amount of work to calculate. For this reason, the distance was estimated with only seven very significant points of the contour. The points used were f20; 25; 40; 50; 78; 99; 114g, as shown in Fig. 2. In some experiments carried out with the images used to build the model (for which all the points are precisely located by experts), the mean distance calculated with all the points and calculated with the seven points differed only by 1 pixel. Therefore, those seven points were precisely located for all the images in the database and could be used to calculate the figure of merit for all the adjusted contours. For example, the mean distance to the contour is shown in the images of Figs. 3–5. The mean distance from each point of the resulting contour to the nearest point of the labeled one (in the normal direction) could also be considered as a figure of merit, which has been used in other similar applications and could yield better results. However, considering that our model is not a closed contour but includes five
Fig. 3. Two examples of contours with a mean distance close to 10 pixels.
H.M. González-Velasco et al. / Computers and Electronics in Agriculture 78 (2011) 80–87
85
Fig. 4. Some examples of the contour extraction stage, along with the final mean distance (in pixels) to the correct position, considering the seven points described in the text.
open sections, many different contours could produce very good values for the figure of merit but have the extreme points of each section at a considerable distance of their correct position. For this reason, that figure of merit was not considered. In addition, to define if a contour extraction is a ‘‘success’’ or not, we applied a hard limit of ten pixels to the mean distance, which we considered enough to guarantee a very good fitting. In Fig. 3, two images are presented with the extracted contours superimposed (notice that, for each example in Figs. 3–5, a monochrome image has been included alongside the color image that was used for the tests, in order to improve the visibility of the annotated contour). For these examples, the mean distances for the contours are very close to ten pixels. In general, the fitting is very good, and there is only a slight deviation in some places. First, we applied the contour extraction system without fixing points. In Table 2, we present the success rate and the mean of the figure of merit calculated over the 93 images (‘‘mean distance’’). Fig. 4a and b shows two examples corresponding to two cases of failure. The rate of success is not very high (61.3%), and this rate of failures cannot be considered acceptable. Analyzing the resulting contours, we found many cases of errors similar to that in Fig. 4b, where the contour is very well adjusted except for the limbs. The problem resides in the confusion between the limbs in the foreground and the limbs in the background, when both can be seen in the image due to the position of the animal.
To overcome this difficulty, in Section 2.4, we proposed an interactive scheme based on fixing the positions of one or two points in their correct locations. In the experiments, we made a systematic test of all possible combinations of one and two fixed points, considering the seven significant points referred above. It is important to note that those points were selected because they give an idea of the location of significant parts of the animal (for example, the head, with points 20 and 25, or the limbs, with points 78 and 114). In Table 2, results are presented for one and two fixed points, with the best and worst cases marked with an asterisk. Examining the best cases, we can see that fixing points is clearly helpful because the results fixing the point 78 and the points 78 and 114 are significantly better than those obtained without fixed points. However, the selection of the points to fix appears to be very important because the performance in the worst cases is very similar to that of the original (without fixed points). Moreover, we also observed that the best of the cases is obtained by fixing the points corresponding to the limbs (78 and 114), indicating that the problem of the automatic approach with the limbs can be solved almost completely with this technique. In Figs. 4c and 5d–f, we present some examples of results obtained by fixing points 78 and 114. Cases c, d and f correspond to successful searches, and case e is one of the four images that failed. In this last case, the problem is that the position of the cow is quite ‘‘atypical’’, particularly the rear limb. Thus, it is impossible for our
86
H.M. González-Velasco et al. / Computers and Electronics in Agriculture 78 (2011) 80–87
Fig. 5. More examples of the contour extraction stage, along with the final mean distance to the correct position.
model, with an ‘‘acceptable’’ shape (i.e., bi parameters within their limits, indicated in Section 2.1), to properly locate both the limbs (forced) and also the head. It seems that this cow should have been considered to build the PDM. Finally, case f is also remarkable, as the fitting is very good even though the frontal limb is partially occluded. 4.2. Second stage: assessment In this stage, the aim was to test the ability and performance of our system in reproducing the morphological assessment carried out by experts, both globally and for each of the traits. It is important to note that all the 44 individuals we have worked with have a good assessment given by the experts, i.e., they are considered at least acceptable individuals of the breed (which means, for this breed, that they have an assessment higher than 65 points). For this reason, our system cannot be used to distinguish between acceptable and non-acceptable individuals, but can be useful separating very good individuals from those which are no so good, for instance in contests. With this main goal, the method described in Section 3 was used to train 11 neural networks that yield as their outputs the global assessment (one network) or the assessments of each of the aspects (10 networks). As stated in Section 3, the assessment given by an expert for the cows was used in the training process. As
the scale is different for the global assessment (0–100) and for the trait assessment (0–10), the data were normalized to the interval ½0; 1 in advance. For the tests, as we do not have a large amount of data, a leaveone-out strategy was followed. For each of the 134 images (for which we had the contour extracted), the remaining 133 were randomly divided into two subsets: training and validation. Using these subsets, networks were trained with between 5 and 40 neurons in the hidden layer, and the one offering the best results was selected. Finally, the contour from the original image was introduced to the network to be assessed, and the result was compared with the assessment provided by the expert to generate the error. As mentioned above, this process was carried out both with the global assessment and with the individual assessments for each of the traits. Table 3 presents the mean and the standard deviation of the relative value of the error (the absolute error divided by the maximum value of the scale, expressed as a percentage), both for the global assessment and for each of the individual traits. Fig. 6 also shows a box plot of the errors obtained in each of the experiments to give an idea of their distribution. We obtained a mean error of 5.297% (5.297 points) in the global score with a relatively high sigma, indicating that there is high variability in the errors of the assessments. This can also be observed in Fig. 6, where we see that the boxes and the whiskers are quite large. In our opinion, there are two main causes for these errors.
H.M. González-Velasco et al. / Computers and Electronics in Agriculture 78 (2011) 80–87 Table 2 The number of successes and the mean distance obtained by the method applied to our database of 93 images (the best and worst cases are marked with an asterisk). Fixed points
Successes
Mean
Fixed points
Successes
Mean
None 20 25 40⁄ 50 78⁄ 99 114 20, 25 20, 40 20, 50 20, 78 20, 99 20, 114 25, 40
57 56 61 52 58 80 60 71 60 63 57 84 61 76 55
10.03 9.84 9.05 10.49 9.54 6.52 9.69 8.14 9.72 8.95 9.16 5.78 9.09 7.22 9.42
25, 25, 25, 25, 40, 40, 40, 40, 50, 50, 50, 78, 78, 99,
60 86 65 77 57 83 53 73 86 54 71 83 89 72
8.82 5.46 8.46 6.92 10.11 6.13 10.02 7.44 5.74 10.33 7.45 6.17 5.42 7.60
(61.3%) (60.2%) (65.6%) (55.9%) (62.4%) (86.0%) (64.5%) (76.3%) (64.5%) (67.7%) (61.3%) (90.3%) (65.6%) (81.7%) (59.1%)
50 78 99 114 50 78 99 114 78 99⁄ 114 99 114⁄ 114
(64.5%) (92.5%) (69.9%) (82.8%) (61.3%) (89.2%) (57.0%) (78.5%) (92.5%) (58.1%) (76.3%) (89.2%) (95.7%) (77.4%)
Table 3 Mean and standard deviation of the relative value of the error, for the global assessment and for the individual traits. Trait
Description
Mean rel. error
r
1 2 3 4 5 6 7 8 9 10
Global assessment General aspect and figure Development of the body Head Neck, chest, withers and back Thorax and belly Back and loins Hindquarters and tail Thighs and buttocks Extremities Shape and quality of the udder
5.297 7.290 6.196 7.119 8.468 8.073 6.953 8.106 7.790 6.640 10.943
4.145 4.402 5.087 5.680 5.094 5.067 5.058 6.116 6.621 6.917 7.654
87
given in steps of 1 point (405 out of 440 partial assessments, 10 for each cow). If we consider the ‘‘resolution’’ as 1 point, then the corresponding ‘‘uncertainty’’ (or ‘‘error’’) in the assessments of the traits would be 0.5 points, and if we propagate that uncertainty to the global assessment (recall that it is calculated as a weighted sum), each global assessment can be considered to have an uncertainty of ±5 points. In that sense, our results can be considered as quite good and promising because with only partial input information, we obtain a low error in many cases, and the mean over all the cases is very close to 5 points. 5. Conclusions and future research In this paper, we have presented a system designed to provide a morphological assessment of a cow based on a lateral image of the animal. The system consists of two parts: one dedicated to the analysis of the image and the extraction of the profile of the animal, and the other designed to provide an assessment with the parameterized contour. The first part performs very well with little human intervention (fixing points), reaching a 95% of success rate. The second stage, based on a neural network, also gave very promising results, taking into account the fact that not all the information required for the morphological assessment can be observed in one view of the animal. To improve the method, two lines of research are being considered at this moment. First, we are working to incorporate the contours of the rear and frontal views of the cows as inputs to the neural network, complementing the information contained in the lateral view. Additionally, we intend to eliminate from the profiles all the parts that are affected by a change in the pose of the animal (for instance, the limbs in the lateral position). References
Fig. 6. A box plot of the errors obtained for each trait and for the global assessment.
First, the limbs were included in the contours, but the position of the limbs can change significantly from one image to another of the same (or a different) animal, causing a great difference between the shape parameters of the contours. Second, although many of the traits to be evaluated can be observed very well in the image of the lateral position, there are others that cannot be observed so well and even some that cannot be observed at all. The clearest example is trait 10 (shape and quality of the udder), which has the highest error and the largest boxes and whiskers because the udder is not visible in the contour that we have defined. In any event, it is important to note that in the data from the experts, the assessment for the individual traits is almost always
Alonso, J., Bahamonde, A., Villa, A., Rodríguez, A., 2007. Morphological assessment of beef cattle according to carcass value. Livestock Science 107, 265–273. Arámbula Cosío, F., 2008. Automatic initialization of an active shape model of the prostate. Medical Image Analysis 12, 469–483. Bishop, C.M., Hinton, G., 1995. Neural Networks for Pattern Recognition. Clarendon Press. Cootes, T.F., Taylor, C.J., Cooper, D.H., Graham, J., 1995. Active shape models – their training and application. Computer Vision and Image Understanding 61, 38–59. Davis, L., 1991. Handbook of Genetic Algorithms. Van Nostrand Reinhold. Goldberg, D.E., 1989. Genetic Algorithms in Search, Optimization and Machine Learning. Addison-Wesley. González, H.M., García, C.J., López, F.J., Macías, M., Gallardo, R., 2003. Segmentation of bovine livestock images using GA and ASM in a two-step approach. International Journal of Pattern Recognition and Artificial Intelligence 17, 601–616. González, H.M., García, C.J., Macías, M., López, F.J., Acevedo, M.I., 2004. Neuralnetworks-based edges selector for boundary extraction problems. Image and Vision Computing 22, 1129–1135. Goyache, F., Del Coz, J.J., Quevedo, J.R., et al., 2001. Using artificial intelligence to design and implement a morphological assessment system in beef cattle. Animal Science 73, 49–60. Haupt, R.L., Haupt, S.E., 2004. Practical Genetic Algorithms 2e. John Wiley. Hill, A., Cootes, T.F., Taylor, C.J., Lindley, K., 1994. Medical image interpretation: a generic approach using deformable templates. Journal of Medical Informatics 19, 47–59. Hill, A., Taylor, C.J., 1992. Model-based image interpretation using genetic algorithms. Image and Vision Computing 10, 295–300. Li, H., Chutatape, O., 2003. Boundary detection of optic disk by a modified ASM method. Pattern Recognition 36, 2093–2104. Mishra, A., Dutta, P.K., Ghosh, M.K., 2003. A ga based approach for boundary detection of left ventricle with echocardiographic image sequences. Image and Vision Computing 21, 967–976. van Ginneken, B., de Bruijne, M., Loog, M., Viergever, M., 2003. Interactive shape models. In: Sonka, M., Michael Fitzpatrick, J. (Eds.), Medical Imaging 2003: Image Processing. SPIE – The International Society for Optical Engineering. Proceedings of SPIE, vol. 5032, pp. 1206–1216. Vera, M., Bravo, A., Medina, R., 2010. Myocardial border detection from ventriculograms using support vector machines and real-coded genetic algorithms. Computers in Biology and Medicine 40, 446–455. Yuen, S.Y., Fong, C.K., Lam, H.S., 2001. Guaranteeing the probability of success using repeated runs of genetic algorithm. Image and Vision Computing 19, 551–560.