- Email: [email protected]
Sensor Fusion for Quality Grading of Melons
Copyright © IFAC Control Applications in Post-Harvest and Processing Technology, Ostend, Belgium, 1995
SENSOR FUSION FOR QUALITY GRADING OF MELONS
V. Steinmetz *. G. Rabatel *. M. Crochon * T. Talou **. B. Bourrounet **
*CEMAGREF - Agricultural Equipment and Food Process Engineering Division BP 5095 Montpellier 34033 Cedex 1- France Tel:(33) 67 046300 Fax:(33) 67635 7 95 Email: vincent.steinmetz@cemagreffr ** Ecole Nationale Superieure de Chimie. Laboratoire de Chimie Agro-Industrielle. 118. route de Narbonne. 31077 Toulouse. France. Tel:(33) 61 17 57 24. Fax:(33) 61 17 5 7 30
Abstract: Various destructive and non-destructive sensors are applied to melon in order to assess its quality components. Destructive measurements (finnness and sugar content) made on the fruits are used as references, and are compared to the non-destructive sensors measurements (vision system, electronic sniffer). Classes of melons are made based on an expert, but also on unsupervised classification (Kohonen network). It is shown that the classification provided by the expert is not completely related to the destructive measurements, and that a satisfying artificial classifier can be built based on the electronic sniffer. Keywords: Sensor fusion, quality control, melons
was higher than all over destructive and nondestructive applied techniques. External melon finnness was also studied (Aubert et al., 1977).
1. INTRODUCTION Melon is a highly perishable fruit, and about 2 millions of tons were harvested in Europe in 1992, mainly in Spain, Italy and France. Melons are mainly field produced and grading melons is a labour intensive task. Melons are mostly graded based on their weight or size with mechanical systems, and based on colour. The latter characteristic is so far perfonned by human graders in charge of removing the non-mature or advanced-mature fruits based upon subjective visual detennination (colour or changes in the physical exterior aspect such as the appearance of a slip near the stem).
Sensor fusion is analogous to the ongoing cognitive process used by humans to integrate data continually from their senses to make inferences about the external world. Though few sensor fusion models for classification are available in the literature, some research has already been conducted on architecture selection for sensor fusion (Steinmetz et al., 1994), and on the application of sensor fusion to tomato classification (Edan et al., 1994). Promising results in this area leads our study of sensor fusion techniques applied to melon quality assessment.
Research for non-destructive maturity evaluation of melons has already been conducted on different aspects of melon maturity. A sensor system, based on detection of concentrations of headspace gases, has been developed and tested on several cultivars of melons (Benady et al., 1992). The ability of this sensor for discriminating between ripeness stages
The goals of this study were to combine the sensors in order to match the classes made by the expert, to compare the perfonnance of the classifiers using different sensors, and to compare different fusion techniques.
201
2.2 Firmness
2. MATERIAL
Firmness was measured with a Magness-Taylor apparatus. Melons were cut in two halves and firmness was measured on the same half on two opposite points (Figure 2). The average of these measurements was used as an indicator of the fruit firmness . Since the same distance from the skin was always used, it may induce a bias in the measurement. Indeed smaller melon in size will appear softer than large melon, because the flesh IS softer close to the heart of the melon .
Five different sensors were used : a CCD camera, a near-infra-red sensor (Bellon, 1994), electronic sniffer (all non-destructive), and a refractometer and a manual penetrometer (both destructive).
2. J Vision system Inspection chamber: The inspection chamber includes consideration of melon position and size, lighting design and image resolution. Melon colour and size are one of the major elements in designing the inspection chamber. Background was selected to be black because it provided the best contrast. To provide uniform and diffuse lighting above the melon, an illumination chamber providing day light with ten 58W fluorescent lamps was used. These lamps were placed circularly around the inspection chamber. Views of the inspection chamber are shown in Figure I. The chamber is a half-sphere with a 2 meters radius. Using a 15 mm focal length, the vertical and horizontal resolutions were respectively 0.91 mm/pixel and 0.65 mm/pixel. The aspect ratio was 1.4: I. Distance of the camera to the melon was set to 0.6 m.
me asurement locations
1.5 cm Top view
Side view
Figure 2. Firmness measurement on melons .
2.3 Sugar content Sugar content was measured with a refractometer providing Brix degrees . Measurements were made with the same method than firmness, i.e. two measurements on each opposite side of a half-cut melon . Data were average and used as an indicator of the fruit sugar content.
PC
lighting
~op
2.4 Aromas
melon view for the camer.)
Aromas were measured with an electronic sniffer. The sensors used are of M.O.S. (Taguchi, 1971) type (Meta-Oxide Sensors) and were composed of sinfered tin dioxide deposit on an alumina ceramic tube with an heater coil inside. In presence of volatile compounds, the electrical resistance of the sensor was modified due to volatile adsorption on its surface. The adsorption kinetics is several seconds, is reversible at ambient temperature and proportional to volatiles concentration in the atmosphere. The sensor is coupled with a data acquisition hardware and the analog output signal is digitised and stored in the computer. The measurement system consists of a glass flask (0 .6 I capacity) in which are placed five commercial MOS . This flask is linked to a sample air input port and an air output port, a control unit which provides both constant tension of 5 V to the sensor circuit voltage and variable tension, (4 to 6 V) to the heater circuit, and a personal computer allowing continuous monitoring and measurement of all electrical signals via five AID converters.
Figure 1. Inspection chamber (not to scale).
Image acquisition: The image processing hardware includes a colour CCD camera, a microcomputer and acquisition board necessary to acquire and perform basic image processing operations. The camera was centred at the middle of the background and balanced by adjusting the colour gains. The aperture of the lens was set to 16 to allow the segmentation of the melon from the background and colour segmentation. Video signals from the camera were digitized with a colour frame grabber (Model Bytech) mounted in a personal microcomputer (PC 386). The camera was calibrated before each experiment with a colour reference frame. Average R, G and B values were computed for this reference frame in order to check the camera calibration.
202
3.2 Sensor fusion
3. METHODS
3. 1 Image analysis
Multisensor fusion is an evolving technology concerned with the problem of how to combine data from multiple sensors in order to make inferences about a physical event, activity, or situation (Hall, (992). The ultimate goal of multisensor fusion is to find a combined declaration that is more specific and more accurate than any of the declarations based on each individual sensor. The approach relies on the hypothesis that fusion of signals from several sensors (similar or disparate sensors) will give better results than the signal from one single sensor (Holmbom ,
Two views were acquired for each melon: the first view is a top view as presented in Figure I. A 90 degrees rotation was applied to the melon in order to acquire the side view. Each melon image was 256 x 256 pixels and covered 23.2 cm x 16.0 cm area. Each pixel was assigned three different values: R, G and B on an 8-bit scale. Background was almost completely black.
(989).
Figure 3 shows a description of the different steps performed during image analysis. Global segmentation involves segmentation of the whole melon from the background. Local segmentation involves segmentation of the greenest part of the melon. The main extracted characteristics were the diameters, the histograms of the top view for each colour component, and the average blue component of the top view.
The fusion process was applied to the features extracted from each individual sensor. Discriminant analysis and neural network approaches are proposed for testing the combination of the different sensors for class prediction. These classifiers were applied for separating the melons into the four categories commonly used by the expert ('green' (G), 'light green' (G I), 'light yellow' (Y I) and 'yellow' (Y» . Different classifiers were built by associating different kind of sensors, and performance of the different classifiers were compared. Since aromas sensor was applied only on a part of the samples, principal component analysis is also used for including the aromas sensor within the sensor fusion process description, because this method requires less numerous samples.
Feature extraction includes size and colour considerations. Three parameters were extracted from the image, i.e. two diameters from the top view and one diameter from the side view. The variables derived for colour are based on the red, green and blue histograms . Since the global colour of the melon is used by the human grader, the histogram was computed based on the top view of the melon . However, the melons present some greenest part which are placed symmetrically around it. These parts are schematically represented by the black lines on the melon of Figure I. These greenest parts may present some noise for the histogram colour, that is why they were eliminated before the computation of the histogram . Furthermore, the area around the stem of the melon is also important for the maturity evaluation, especially for mature melons. Skin of mature melons often splits apart around the stem, allowing the flesh to be seen. Therefore, the colour histogram around the stem was computed within a window which size was a function of the size of the melon.
Features extracted from the histogram are calculated based on the pixels values which lead to the best discrimination among the classes (Ros et aI., (990). This provided five different colour features extracted from the histograms.
3.3 Artificial classifiers Features used by the artificial classifiers are the features extracted from the different sensors. Tested classifiers include discriminant analysis (Duda, (973) followed by a nearest cluster center classifier, and a neural network. Some research has been conducted on the application of neural networks technology for inspection of fruits, vegetables and horticultural products. The properties of reflectance spectra for tomatoes have been studied (Thai et al., (992), and recent research on flower inspection includes work on grading cyclamens using an artificial expert (Brons et al., (991) and roses inspection (Steinmetz et al., (993).
r--
1', --_.0....-_-. ,
,, ,, ,
~
,,
,
_ _ _ _ _J
SI'cvlcw
'------'I
'----- ----------'
Grouping of the samples into classes was made based on Kohonen's algorithm (Kohonen, (990). This algorithm (KA) generates a topologically ordered mapping based on a competitive scheme.
Top Ylcw
Figure 3. Image analysis for the two views of the melon . 203
Topological ordering means that neurons that are close to each other represent similar input patterns. The results of this algorithm is a four 'natural' grouping of the samples.
1~r------------------------------:
COO'elation coetlicient 0.98 1400
n=174 A\/erage error. 25 9
1200
4. EXPERIMENTAL PROCEDURES One variety (Galoubet) representing a total of 176 samples were collected in early June 1994 at a grower's place located in the Camargue area (France). The samples were classified by the same expert in the 4 different classes previously described.
4OO2~----~4----~6~----~8----~10~--~12
The samples were collected at two different harvesting dates. Each sample was processed successively through the individual sensors for nondestructive measurements in the following order: weight, image acquisition, electronic sniffer (43 samples only), near-infra-red measurements. Destructive measurements, i.e. finnness and sugar content, were then processed. Melons were stored in a plastic bag during a 12 hours period before being sensed by the electronic sniffer. Atmosphere was sampled from the bag with a 50 cm) syringe, and then introduced into the gas flask .
x 10'
Figure 4. Observed weight (grams) versus the product of the three diameters .
o . o45r----------------~-~-~--__,
004
..:,\
;
"
,~
0.035 003
0025 002
0015 001
5. RESULTS AND DISCUSSION
"-
-\
-;'~',
I
\
/
JI
loght Yellow
Light Green
".- , . \.. /I
/, Yellow
'.
', '
\) ~
0005
5. 1 Weight prediction based on image analysis
~L--~20--4~0-~60~~8~0~~10=0~~~--~1~20~-714' 0 Pixel values
The purpose was to predict the weight based on image processing, i.e. the three diameters of the melon . Results are presented in Figure 4. This model was obtained with a linear regression between the weight and the product of the three diameters of the melon measured by image analysis. This linear regression relies on a constant density for the melons, which is obviously not correct. Indeed, melon density ranges from 0.90 to more than I, depending on the variety. The fairly good results presented here can be explained by the fact that all the melons used during the experiment belong to the same variety, and all of them come from the same growing place and the same field . The results would certainly not be as good if many different varieties of melons had been used. For example, the variety 'Canari' presents a more ellipsoid shape, and a different calibration would be necessary for this kind of variety.
Figure 5. Histogram of blue components for different samples of the four classes.
5.3 Sugar content and near-infra-red sensor No significant relationship was found between the sugar content measured by the refractometer and the spectra measured by the near-infra-red sensor. The main reason is due to the fact that the measurements was made on the skin of the melons, since sugar content was measured in the flesh . The near-infra-red sensor should then be equipped with a special device able to go through the skin until reaching the flesh . An other approach could be checking relationship between the sugar content of the skin and the sugar content of the flesh .
5.2 Colour histograms
5.4 Sugar content and firmness
Typical colour histograms from the top view of the melons are presented in Figure 5. Figure 5 is an illustration of the discrimination between the different classes around the pixel value 80.
Relationship between grader classification and fruit maturity was studied by comparing means and variances for each day and each class (Tables I & 2).
204
Table I Fjrmness (kes/O.5cnh for the classes made by the expert (mean - stand. d(.vjatjon)
Day I Day 2
Y 1.68 (0.37) 1.86 (0.51 )
YI 1.69 (0.48) 1.77 (0.43)
GI 1.76 (0.27) 1.85 (0.35)
Table 3, Su::ar content (de::ree Brjx) for the classes provjded by Kohonen ' s al::orjthm (mean - stand. devjation)
G 2.50 ( 1.23) 3.57 (1.39)
Class number Sample number Sugar content Firmness
In Table I, for both days, mean of class G is different (at a 5% significant level) from the means of the other classes.
2
3
4
89
9
27
51
12.78 (0.79) 1.69 (0.35)
12.40 (0.34) 2.69 (0.32)
10.73 (0.71) 1.47 (0.27)
10.23 ( 1.25) 2.86 ( 1.24)
Table 2 Su::ar content (deeree Brjx) for the classes made by the expert (mean - stand devjatjon) 5.5 Principal component analysis Day I Day 2
Y 12.23 ( 1.79) 12.3 ( 1.25)
YI 12.3 ( 1.05) 10.76 (0.98)
GI 12.29 (0.80) 12.05 ( 1.07)
G 10.84 ( 1.15) 10.18 ( 1.84)
Principal component analysis (PCA) was performed with 43 samples, i.e. the 43 samples that were measured with the electronic sniffer. Measurements used for the PCA were weight, size, sugar content, firmness, aromas and colour features extracted from the image analysis process. Figure 6 represents the correlation circle.
In Table 2, for day I, mean of class G is different (at a 5% significant level) from the means of the other classes. For day 2, means of the class G and mean of class Y I are not different, and means of the class G I and mean of class Y are not different.
OS •. .: .......... , .............. , ..... ......... , ......... :" .. .... ColD'
~
'.iAl.
Only the 'Green' class is highly related to the destructive measurements. These results tends to prove that the expert classification, mainly based on colour, is not really related to melon maturity, except for the 'Green' class. It has to be noticed that the mean values of the second day are not consistent with the mean values of the first day. This can be explained by the dates of sample collection: the first day was a Monday, that is to say, the day where the grader had the largest choice among melons for building the classes, because melons are not harvested during the week-end. On the other side, the second collecting day was a Wednesday where a smaller set of melons was available for building the classes. In other words, it is clear that the expert did not build the classes for the first day in the same way than he did it during the second day. These results are very similar to the ones exposed in (Guedalia, 1994), where it is shown the difficulty to correctly classify the components that form the complete manual classification.
,
t*.
.Arm~c..
o.... ...........!.............
:
:
.
+. . .... . .+.. . :
: SUla,
:~;;,~.
:
",
-{)6 .. :,: ......... ; ........... .. -j-............. j......... ;,,·· /
.. ;'
.~ .
·1 -1
-{) s
0
0.5
Figure 6. Correlation circle based on principal component analysis. Based on the PCA, the two axis represents 72 % of variance. The first axis (48 % of variance) represents the opposition between firmness and sugar content. The second axis (23% of variance) is more related to the weight. PCA was also processed with the 176 samples without including aromas measurements. Results were found to be sim ilar to the previous PCA with aromas (52% of variance on the first axis, and 24% on the second axis).
Based on these results, the Kohonen's algorithm was applied in order to build quality classes based on the destructive measurements. Results are presented in Table 3. Four classes are provided by the algorithm. Class I represents sweet and soft melons, while class 4 represents non-sweet and firm melons. Classes 3 and 4 represent intennediary quality classes (sweet and finn for class 2, and non-sweet and soft for class 3).
5.6 Neural network classifier Results obtained with a neural network classifier are presented in Table 4. These results were obtained with the following features as inputs for the network: vision system, the sugar content and firmness feature . The average error rate is 35%, and the error rate 205
within one class is 12%. Additional networks were built by combining some features as inputs of the network. Table 5 presents the results by combining the different sensors. Colour is one of the most important feature for the classes built by the experts. Including sugar content or firmness do not show any important improvement in the classification process. This is a confirmation of what has been shown in paragraph 5.4, where destructive measurements were shown to be poorly related to the classification of the expert.
from the other. However, the two intermediary classes ('light yellow' and 'light green') are poorly separated. This is not surprising because these two classes are rather similar one to the other from the colour point of view, and from the maturity point of view as well. Results from the cluster-minimum distance classifier applied after the discriminant analysis are presented in Table 7. The average error rate is 45% and the error rate within one class is 10%. The average error rate is larger compared to the results from the neural network. It has to be noticed in Table 7 that the 'Green' class was rather well separated from the other classes. Different combinations of sensors were also tested with the discriminant analysis. They led to similar conclusions than the ones obtained from the neural network classifier.
Table 4 Classification of the neural network based on vision sensor. refractometer and penetrometer
Classes made by the expert
Y YI GI G
Classes made by the network Y YI GI G 2 33 8 2 33 12 I1 27 4 12 3 6 23
Table 5. Error classification of the neural network dependin~ on the sensors Sensor combination Colour Color and refractometer Color and penetrometer Color, refractometer and penetrometer Refractometer and penetrometer
·2
Error classification rate 37% 38% 38% 35%
~·~5--.~lO--~ . 5--~--~~lO~~'~5~~
Figure 7. First two axis from the discriminant analysis Table 7 Classification of the discriminant analysis
62%
A backpropagation neural network was also trained based on the classes provided by Kohonen's algorithm . The five aromas components were used as inputs. Table 6 shows that the classification is greatly improved, the error rate being about 8%.
Classes made by the expert
Y YI GI G
Classes made based on discrim inant analysis Y YI GI 4 31 6 11 19 7 7 10 15 0 12
G 3 \1
11 31
Table 6 Neural netword classification for aromas Classification made by the neural network I 2 3 KA 3 0 0 24 3 Classes 2 0 3 0 0 6 4 0 0 0
6. CONCLUSIONS 4 0
Sensor fusion was studied in order to improve the classification performance of the sensors with respect to the expert classification. Features extracted from each individual sensors were used and two different classifiers were compared: a neural network classifier, and a minimal distance classifier to cluster center associated with a discriminant analysis. These classifiers provided the classification of melons among the four classes 'green', 'light green' and 'light yellow' and 'yellow'.
0 6
5. 7 Discriminant analysis
Results from the discriminant analysis applied on the features from the vision sensor, refractometer and manual penetrometer are presented in Figure 7, where the two first axis are sketched. It is visible that the two extreme classes, i.e. the greenest class and the more yel10w class are fairly wel1 separated one
Weight of melons could be easily computed based on vision system with an average error of 25 grams. It was shown that the vision sensor is the most related to expert classification, though this classification is 206
not completely related to melon maturity, except for the 'green' class. Classes made by t~e expert were shown to be highly subjective, based on colour histogram analysis and on maturity parameters as well. Aromas was shown with the PCA to be related to sugar content. Near-infra-red sensor was not measuring sugar content of the flesh . Neural networks classifier was shown to have a higher performance than discriminant analysis classifier, with 35% versus 45% classification error rate, respectively. However, this error rate was still high.
Edan, Y., H. Pasternack, D. Guedalia, N. Ozer. 1. Shmulevitch. D. Rachnmani. E. Fallik and S. Grinberg (1994). Multi-sensor quality classification of tomatoes . ASAE International Summer Meeting, Kansas City. Missouri . ASAE
paper 94-6032 . Guedalia, D., and Y. Edan (1994). A dynamic artificial neural network for coding and classification of multi-sensor quality information. ASAE International Summer Meeting. Kansas City, Missouri . ASAE paper 94-3053 . Hall, L. D. (1992). Mathematical techniques in mu/tisensor datafllsion. Artech House. Boston Holmbom. P.• O. Pedersen, B. Sandell and A. Lauber (1989) . Fusing sensor systems: promises and problems. Sensor review. 9(3): 143-152. Kohonen, T. (1990) The self-organizing map .
By removing the classes provided by the expert, and by using an unsupervised classification, it was shown that the classification could be improved by using aromas only. However, it must not be forgotten that only 43 samples were used by the electronic sniffer. and all samples belonged to the same variety. This is why further research should be done in this area, with a larger number of sample, including different varieties of melons.
Proceedings of the IEEE. (78)9 :1464-1480 . Ros, F.• S. Guillaume, G. Rabatel, F. Sevila, and D. Bertrand. (1994). The characterisation of granular product populations as a pattern recognItIOn problem : application to meal products classification. Accepted to Journal of Chemometrics. Steinmetz, V. and MJ . Delwiche. (1993). Neural network analysis of rose straighmess.
7. ACKNOWLEDGEMENTS We express our appreciation to the CEHM growers in Marsillargues, Gard, France, and particularly Mr. Yard, for providing the experimental samples.
Proceedings of the Artificial Intelligence for Food and Agriculture Conference, Nimes. France. Steinmetz, V., M. Crochon, V. Bellon-Maurel, J. L. GarciaFernandez, P. Barreiro Elorza and L. Verstreken (1994). Sensors for fruit firmness assessment: comparison and fusion . Submitted to Journal of Agricultural Engineering Research . Steinmetz, V.. M. Crochon , T. Talou and B. Bourrounet (1995). Sensor fusion for fruit quality assessment: application . to melon . Proceedings of the International Conference ASAE 'Harvest and post-harvest technologies for fresh fruits and vegetables'. Mexico. Taguchi. N. (1971). Method for making a gas sensing element. US Patent 3.625,756. Thai, N.C .• R.L. Shewfelt. and J.G . Latimer (1992) . Neural network analysis of tomato reflectance spectra. ASA E Paper No. 92-7057 .
REFERENCES Aubert, S. and R. Dumas de Vaulx . (1977). Undestructive firmness test for the maturity control of melons. Ann. Technol. Agric. 27(3):243-254. Bellon, V. (1994). Infrared and near-infrared technology for the food industry and agricultural uses. On line applications. Food Control 5:(1) Benady, M., J.E. Simon, DJ. Charles and G.E. Miles. (1992). Determining melon ripeness by analyzing heads pace gas emission. ASAE International Summer Meeting. Charlotte, North Carolina. ASAE paper 92-6055. Brons, A., G. Rabatel, F. Sevila and C. Touzet (1991). Neural network techniques to simulate human judgement on pot plant quality: an artificial expert for beauty evaluation . ASAE paper No. 91-9893 . Chen, H., J. De Baerdemaek.e r and V. Bellon. (1993). Finite element study of the melon for nondestructive sensing of melon quality. ASAE International Winter Meeting. Chicago. Illinois.
ASAE paper 93-6599 . Crochon. M. Eating quality of melons. (1986). EEC Conference. Aix-en-Provence, France. Duda, R. and P. Hart (1973). Pattern Classification and Scene Analysis. John Wiley & Sons. 207
SENSOR FUSION FOR QUALITY GRADING OF MELONS
V. Steinmetz *. G. Rabatel *. M. Crochon * T. Talou **. B. Bourrounet **
*CEMAGREF - Agricultural Equipment and Food Process Engineering Division BP 5095 Montpellier 34033 Cedex 1- France Tel:(33) 67 046300 Fax:(33) 67635 7 95 Email: vincent.steinmetz@cemagreffr ** Ecole Nationale Superieure de Chimie. Laboratoire de Chimie Agro-Industrielle. 118. route de Narbonne. 31077 Toulouse. France. Tel:(33) 61 17 57 24. Fax:(33) 61 17 5 7 30
Abstract: Various destructive and non-destructive sensors are applied to melon in order to assess its quality components. Destructive measurements (finnness and sugar content) made on the fruits are used as references, and are compared to the non-destructive sensors measurements (vision system, electronic sniffer). Classes of melons are made based on an expert, but also on unsupervised classification (Kohonen network). It is shown that the classification provided by the expert is not completely related to the destructive measurements, and that a satisfying artificial classifier can be built based on the electronic sniffer. Keywords: Sensor fusion, quality control, melons
was higher than all over destructive and nondestructive applied techniques. External melon finnness was also studied (Aubert et al., 1977).
1. INTRODUCTION Melon is a highly perishable fruit, and about 2 millions of tons were harvested in Europe in 1992, mainly in Spain, Italy and France. Melons are mainly field produced and grading melons is a labour intensive task. Melons are mostly graded based on their weight or size with mechanical systems, and based on colour. The latter characteristic is so far perfonned by human graders in charge of removing the non-mature or advanced-mature fruits based upon subjective visual detennination (colour or changes in the physical exterior aspect such as the appearance of a slip near the stem).
Sensor fusion is analogous to the ongoing cognitive process used by humans to integrate data continually from their senses to make inferences about the external world. Though few sensor fusion models for classification are available in the literature, some research has already been conducted on architecture selection for sensor fusion (Steinmetz et al., 1994), and on the application of sensor fusion to tomato classification (Edan et al., 1994). Promising results in this area leads our study of sensor fusion techniques applied to melon quality assessment.
Research for non-destructive maturity evaluation of melons has already been conducted on different aspects of melon maturity. A sensor system, based on detection of concentrations of headspace gases, has been developed and tested on several cultivars of melons (Benady et al., 1992). The ability of this sensor for discriminating between ripeness stages
The goals of this study were to combine the sensors in order to match the classes made by the expert, to compare the perfonnance of the classifiers using different sensors, and to compare different fusion techniques.
201
2.2 Firmness
2. MATERIAL
Firmness was measured with a Magness-Taylor apparatus. Melons were cut in two halves and firmness was measured on the same half on two opposite points (Figure 2). The average of these measurements was used as an indicator of the fruit firmness . Since the same distance from the skin was always used, it may induce a bias in the measurement. Indeed smaller melon in size will appear softer than large melon, because the flesh IS softer close to the heart of the melon .
Five different sensors were used : a CCD camera, a near-infra-red sensor (Bellon, 1994), electronic sniffer (all non-destructive), and a refractometer and a manual penetrometer (both destructive).
2. J Vision system Inspection chamber: The inspection chamber includes consideration of melon position and size, lighting design and image resolution. Melon colour and size are one of the major elements in designing the inspection chamber. Background was selected to be black because it provided the best contrast. To provide uniform and diffuse lighting above the melon, an illumination chamber providing day light with ten 58W fluorescent lamps was used. These lamps were placed circularly around the inspection chamber. Views of the inspection chamber are shown in Figure I. The chamber is a half-sphere with a 2 meters radius. Using a 15 mm focal length, the vertical and horizontal resolutions were respectively 0.91 mm/pixel and 0.65 mm/pixel. The aspect ratio was 1.4: I. Distance of the camera to the melon was set to 0.6 m.
me asurement locations
1.5 cm Top view
Side view
Figure 2. Firmness measurement on melons .
2.3 Sugar content Sugar content was measured with a refractometer providing Brix degrees . Measurements were made with the same method than firmness, i.e. two measurements on each opposite side of a half-cut melon . Data were average and used as an indicator of the fruit sugar content.
PC
lighting
~op
2.4 Aromas
melon view for the camer.)
Aromas were measured with an electronic sniffer. The sensors used are of M.O.S. (Taguchi, 1971) type (Meta-Oxide Sensors) and were composed of sinfered tin dioxide deposit on an alumina ceramic tube with an heater coil inside. In presence of volatile compounds, the electrical resistance of the sensor was modified due to volatile adsorption on its surface. The adsorption kinetics is several seconds, is reversible at ambient temperature and proportional to volatiles concentration in the atmosphere. The sensor is coupled with a data acquisition hardware and the analog output signal is digitised and stored in the computer. The measurement system consists of a glass flask (0 .6 I capacity) in which are placed five commercial MOS . This flask is linked to a sample air input port and an air output port, a control unit which provides both constant tension of 5 V to the sensor circuit voltage and variable tension, (4 to 6 V) to the heater circuit, and a personal computer allowing continuous monitoring and measurement of all electrical signals via five AID converters.
Figure 1. Inspection chamber (not to scale).
Image acquisition: The image processing hardware includes a colour CCD camera, a microcomputer and acquisition board necessary to acquire and perform basic image processing operations. The camera was centred at the middle of the background and balanced by adjusting the colour gains. The aperture of the lens was set to 16 to allow the segmentation of the melon from the background and colour segmentation. Video signals from the camera were digitized with a colour frame grabber (Model Bytech) mounted in a personal microcomputer (PC 386). The camera was calibrated before each experiment with a colour reference frame. Average R, G and B values were computed for this reference frame in order to check the camera calibration.
202
3.2 Sensor fusion
3. METHODS
3. 1 Image analysis
Multisensor fusion is an evolving technology concerned with the problem of how to combine data from multiple sensors in order to make inferences about a physical event, activity, or situation (Hall, (992). The ultimate goal of multisensor fusion is to find a combined declaration that is more specific and more accurate than any of the declarations based on each individual sensor. The approach relies on the hypothesis that fusion of signals from several sensors (similar or disparate sensors) will give better results than the signal from one single sensor (Holmbom ,
Two views were acquired for each melon: the first view is a top view as presented in Figure I. A 90 degrees rotation was applied to the melon in order to acquire the side view. Each melon image was 256 x 256 pixels and covered 23.2 cm x 16.0 cm area. Each pixel was assigned three different values: R, G and B on an 8-bit scale. Background was almost completely black.
(989).
Figure 3 shows a description of the different steps performed during image analysis. Global segmentation involves segmentation of the whole melon from the background. Local segmentation involves segmentation of the greenest part of the melon. The main extracted characteristics were the diameters, the histograms of the top view for each colour component, and the average blue component of the top view.
The fusion process was applied to the features extracted from each individual sensor. Discriminant analysis and neural network approaches are proposed for testing the combination of the different sensors for class prediction. These classifiers were applied for separating the melons into the four categories commonly used by the expert ('green' (G), 'light green' (G I), 'light yellow' (Y I) and 'yellow' (Y» . Different classifiers were built by associating different kind of sensors, and performance of the different classifiers were compared. Since aromas sensor was applied only on a part of the samples, principal component analysis is also used for including the aromas sensor within the sensor fusion process description, because this method requires less numerous samples.
Feature extraction includes size and colour considerations. Three parameters were extracted from the image, i.e. two diameters from the top view and one diameter from the side view. The variables derived for colour are based on the red, green and blue histograms . Since the global colour of the melon is used by the human grader, the histogram was computed based on the top view of the melon . However, the melons present some greenest part which are placed symmetrically around it. These parts are schematically represented by the black lines on the melon of Figure I. These greenest parts may present some noise for the histogram colour, that is why they were eliminated before the computation of the histogram . Furthermore, the area around the stem of the melon is also important for the maturity evaluation, especially for mature melons. Skin of mature melons often splits apart around the stem, allowing the flesh to be seen. Therefore, the colour histogram around the stem was computed within a window which size was a function of the size of the melon.
Features extracted from the histogram are calculated based on the pixels values which lead to the best discrimination among the classes (Ros et aI., (990). This provided five different colour features extracted from the histograms.
3.3 Artificial classifiers Features used by the artificial classifiers are the features extracted from the different sensors. Tested classifiers include discriminant analysis (Duda, (973) followed by a nearest cluster center classifier, and a neural network. Some research has been conducted on the application of neural networks technology for inspection of fruits, vegetables and horticultural products. The properties of reflectance spectra for tomatoes have been studied (Thai et al., (992), and recent research on flower inspection includes work on grading cyclamens using an artificial expert (Brons et al., (991) and roses inspection (Steinmetz et al., (993).
r--
1', --_.0....-_-. ,
,, ,, ,
~
,,
,
_ _ _ _ _J
SI'cvlcw
'------'I
'----- ----------'
Grouping of the samples into classes was made based on Kohonen's algorithm (Kohonen, (990). This algorithm (KA) generates a topologically ordered mapping based on a competitive scheme.
Top Ylcw
Figure 3. Image analysis for the two views of the melon . 203
Topological ordering means that neurons that are close to each other represent similar input patterns. The results of this algorithm is a four 'natural' grouping of the samples.
1~r------------------------------:
COO'elation coetlicient 0.98 1400
n=174 A\/erage error. 25 9
1200
4. EXPERIMENTAL PROCEDURES One variety (Galoubet) representing a total of 176 samples were collected in early June 1994 at a grower's place located in the Camargue area (France). The samples were classified by the same expert in the 4 different classes previously described.
4OO2~----~4----~6~----~8----~10~--~12
The samples were collected at two different harvesting dates. Each sample was processed successively through the individual sensors for nondestructive measurements in the following order: weight, image acquisition, electronic sniffer (43 samples only), near-infra-red measurements. Destructive measurements, i.e. finnness and sugar content, were then processed. Melons were stored in a plastic bag during a 12 hours period before being sensed by the electronic sniffer. Atmosphere was sampled from the bag with a 50 cm) syringe, and then introduced into the gas flask .
x 10'
Figure 4. Observed weight (grams) versus the product of the three diameters .
o . o45r----------------~-~-~--__,
004
..:,\
;
"
,~
0.035 003
0025 002
0015 001
5. RESULTS AND DISCUSSION
"-
-\
-;'~',
I
\
/
JI
loght Yellow
Light Green
".- , . \.. /I
/, Yellow
'.
', '
\) ~
0005
5. 1 Weight prediction based on image analysis
~L--~20--4~0-~60~~8~0~~10=0~~~--~1~20~-714' 0 Pixel values
The purpose was to predict the weight based on image processing, i.e. the three diameters of the melon . Results are presented in Figure 4. This model was obtained with a linear regression between the weight and the product of the three diameters of the melon measured by image analysis. This linear regression relies on a constant density for the melons, which is obviously not correct. Indeed, melon density ranges from 0.90 to more than I, depending on the variety. The fairly good results presented here can be explained by the fact that all the melons used during the experiment belong to the same variety, and all of them come from the same growing place and the same field . The results would certainly not be as good if many different varieties of melons had been used. For example, the variety 'Canari' presents a more ellipsoid shape, and a different calibration would be necessary for this kind of variety.
Figure 5. Histogram of blue components for different samples of the four classes.
5.3 Sugar content and near-infra-red sensor No significant relationship was found between the sugar content measured by the refractometer and the spectra measured by the near-infra-red sensor. The main reason is due to the fact that the measurements was made on the skin of the melons, since sugar content was measured in the flesh . The near-infra-red sensor should then be equipped with a special device able to go through the skin until reaching the flesh . An other approach could be checking relationship between the sugar content of the skin and the sugar content of the flesh .
5.2 Colour histograms
5.4 Sugar content and firmness
Typical colour histograms from the top view of the melons are presented in Figure 5. Figure 5 is an illustration of the discrimination between the different classes around the pixel value 80.
Relationship between grader classification and fruit maturity was studied by comparing means and variances for each day and each class (Tables I & 2).
204
Table I Fjrmness (kes/O.5cnh for the classes made by the expert (mean - stand. d(.vjatjon)
Day I Day 2
Y 1.68 (0.37) 1.86 (0.51 )
YI 1.69 (0.48) 1.77 (0.43)
GI 1.76 (0.27) 1.85 (0.35)
Table 3, Su::ar content (de::ree Brjx) for the classes provjded by Kohonen ' s al::orjthm (mean - stand. devjation)
G 2.50 ( 1.23) 3.57 (1.39)
Class number Sample number Sugar content Firmness
In Table I, for both days, mean of class G is different (at a 5% significant level) from the means of the other classes.
2
3
4
89
9
27
51
12.78 (0.79) 1.69 (0.35)
12.40 (0.34) 2.69 (0.32)
10.73 (0.71) 1.47 (0.27)
10.23 ( 1.25) 2.86 ( 1.24)
Table 2 Su::ar content (deeree Brjx) for the classes made by the expert (mean - stand devjatjon) 5.5 Principal component analysis Day I Day 2
Y 12.23 ( 1.79) 12.3 ( 1.25)
YI 12.3 ( 1.05) 10.76 (0.98)
GI 12.29 (0.80) 12.05 ( 1.07)
G 10.84 ( 1.15) 10.18 ( 1.84)
Principal component analysis (PCA) was performed with 43 samples, i.e. the 43 samples that were measured with the electronic sniffer. Measurements used for the PCA were weight, size, sugar content, firmness, aromas and colour features extracted from the image analysis process. Figure 6 represents the correlation circle.
In Table 2, for day I, mean of class G is different (at a 5% significant level) from the means of the other classes. For day 2, means of the class G and mean of class Y I are not different, and means of the class G I and mean of class Y are not different.
OS •. .: .......... , .............. , ..... ......... , ......... :" .. .... ColD'
~
'.iAl.
Only the 'Green' class is highly related to the destructive measurements. These results tends to prove that the expert classification, mainly based on colour, is not really related to melon maturity, except for the 'Green' class. It has to be noticed that the mean values of the second day are not consistent with the mean values of the first day. This can be explained by the dates of sample collection: the first day was a Monday, that is to say, the day where the grader had the largest choice among melons for building the classes, because melons are not harvested during the week-end. On the other side, the second collecting day was a Wednesday where a smaller set of melons was available for building the classes. In other words, it is clear that the expert did not build the classes for the first day in the same way than he did it during the second day. These results are very similar to the ones exposed in (Guedalia, 1994), where it is shown the difficulty to correctly classify the components that form the complete manual classification.
,
t*.
.Arm~c..
o.... ...........!.............
:
:
.
+. . .... . .+.. . :
: SUla,
:~;;,~.
:
",
-{)6 .. :,: ......... ; ........... .. -j-............. j......... ;,,·· /
.. ;'
.~ .
·1 -1
-{) s
0
0.5
Figure 6. Correlation circle based on principal component analysis. Based on the PCA, the two axis represents 72 % of variance. The first axis (48 % of variance) represents the opposition between firmness and sugar content. The second axis (23% of variance) is more related to the weight. PCA was also processed with the 176 samples without including aromas measurements. Results were found to be sim ilar to the previous PCA with aromas (52% of variance on the first axis, and 24% on the second axis).
Based on these results, the Kohonen's algorithm was applied in order to build quality classes based on the destructive measurements. Results are presented in Table 3. Four classes are provided by the algorithm. Class I represents sweet and soft melons, while class 4 represents non-sweet and firm melons. Classes 3 and 4 represent intennediary quality classes (sweet and finn for class 2, and non-sweet and soft for class 3).
5.6 Neural network classifier Results obtained with a neural network classifier are presented in Table 4. These results were obtained with the following features as inputs for the network: vision system, the sugar content and firmness feature . The average error rate is 35%, and the error rate 205
within one class is 12%. Additional networks were built by combining some features as inputs of the network. Table 5 presents the results by combining the different sensors. Colour is one of the most important feature for the classes built by the experts. Including sugar content or firmness do not show any important improvement in the classification process. This is a confirmation of what has been shown in paragraph 5.4, where destructive measurements were shown to be poorly related to the classification of the expert.
from the other. However, the two intermediary classes ('light yellow' and 'light green') are poorly separated. This is not surprising because these two classes are rather similar one to the other from the colour point of view, and from the maturity point of view as well. Results from the cluster-minimum distance classifier applied after the discriminant analysis are presented in Table 7. The average error rate is 45% and the error rate within one class is 10%. The average error rate is larger compared to the results from the neural network. It has to be noticed in Table 7 that the 'Green' class was rather well separated from the other classes. Different combinations of sensors were also tested with the discriminant analysis. They led to similar conclusions than the ones obtained from the neural network classifier.
Table 4 Classification of the neural network based on vision sensor. refractometer and penetrometer
Classes made by the expert
Y YI GI G
Classes made by the network Y YI GI G 2 33 8 2 33 12 I1 27 4 12 3 6 23
Table 5. Error classification of the neural network dependin~ on the sensors Sensor combination Colour Color and refractometer Color and penetrometer Color, refractometer and penetrometer Refractometer and penetrometer
·2
Error classification rate 37% 38% 38% 35%
~·~5--.~lO--~ . 5--~--~~lO~~'~5~~
Figure 7. First two axis from the discriminant analysis Table 7 Classification of the discriminant analysis
62%
A backpropagation neural network was also trained based on the classes provided by Kohonen's algorithm . The five aromas components were used as inputs. Table 6 shows that the classification is greatly improved, the error rate being about 8%.
Classes made by the expert
Y YI GI G
Classes made based on discrim inant analysis Y YI GI 4 31 6 11 19 7 7 10 15 0 12
G 3 \1
11 31
Table 6 Neural netword classification for aromas Classification made by the neural network I 2 3 KA 3 0 0 24 3 Classes 2 0 3 0 0 6 4 0 0 0
6. CONCLUSIONS 4 0
Sensor fusion was studied in order to improve the classification performance of the sensors with respect to the expert classification. Features extracted from each individual sensors were used and two different classifiers were compared: a neural network classifier, and a minimal distance classifier to cluster center associated with a discriminant analysis. These classifiers provided the classification of melons among the four classes 'green', 'light green' and 'light yellow' and 'yellow'.
0 6
5. 7 Discriminant analysis
Results from the discriminant analysis applied on the features from the vision sensor, refractometer and manual penetrometer are presented in Figure 7, where the two first axis are sketched. It is visible that the two extreme classes, i.e. the greenest class and the more yel10w class are fairly wel1 separated one
Weight of melons could be easily computed based on vision system with an average error of 25 grams. It was shown that the vision sensor is the most related to expert classification, though this classification is 206
not completely related to melon maturity, except for the 'green' class. Classes made by t~e expert were shown to be highly subjective, based on colour histogram analysis and on maturity parameters as well. Aromas was shown with the PCA to be related to sugar content. Near-infra-red sensor was not measuring sugar content of the flesh . Neural networks classifier was shown to have a higher performance than discriminant analysis classifier, with 35% versus 45% classification error rate, respectively. However, this error rate was still high.
Edan, Y., H. Pasternack, D. Guedalia, N. Ozer. 1. Shmulevitch. D. Rachnmani. E. Fallik and S. Grinberg (1994). Multi-sensor quality classification of tomatoes . ASAE International Summer Meeting, Kansas City. Missouri . ASAE
paper 94-6032 . Guedalia, D., and Y. Edan (1994). A dynamic artificial neural network for coding and classification of multi-sensor quality information. ASAE International Summer Meeting. Kansas City, Missouri . ASAE paper 94-3053 . Hall, L. D. (1992). Mathematical techniques in mu/tisensor datafllsion. Artech House. Boston Holmbom. P.• O. Pedersen, B. Sandell and A. Lauber (1989) . Fusing sensor systems: promises and problems. Sensor review. 9(3): 143-152. Kohonen, T. (1990) The self-organizing map .
By removing the classes provided by the expert, and by using an unsupervised classification, it was shown that the classification could be improved by using aromas only. However, it must not be forgotten that only 43 samples were used by the electronic sniffer. and all samples belonged to the same variety. This is why further research should be done in this area, with a larger number of sample, including different varieties of melons.
Proceedings of the IEEE. (78)9 :1464-1480 . Ros, F.• S. Guillaume, G. Rabatel, F. Sevila, and D. Bertrand. (1994). The characterisation of granular product populations as a pattern recognItIOn problem : application to meal products classification. Accepted to Journal of Chemometrics. Steinmetz, V. and MJ . Delwiche. (1993). Neural network analysis of rose straighmess.
7. ACKNOWLEDGEMENTS We express our appreciation to the CEHM growers in Marsillargues, Gard, France, and particularly Mr. Yard, for providing the experimental samples.
Proceedings of the Artificial Intelligence for Food and Agriculture Conference, Nimes. France. Steinmetz, V., M. Crochon, V. Bellon-Maurel, J. L. GarciaFernandez, P. Barreiro Elorza and L. Verstreken (1994). Sensors for fruit firmness assessment: comparison and fusion . Submitted to Journal of Agricultural Engineering Research . Steinmetz, V.. M. Crochon , T. Talou and B. Bourrounet (1995). Sensor fusion for fruit quality assessment: application . to melon . Proceedings of the International Conference ASAE 'Harvest and post-harvest technologies for fresh fruits and vegetables'. Mexico. Taguchi. N. (1971). Method for making a gas sensing element. US Patent 3.625,756. Thai, N.C .• R.L. Shewfelt. and J.G . Latimer (1992) . Neural network analysis of tomato reflectance spectra. ASA E Paper No. 92-7057 .
REFERENCES Aubert, S. and R. Dumas de Vaulx . (1977). Undestructive firmness test for the maturity control of melons. Ann. Technol. Agric. 27(3):243-254. Bellon, V. (1994). Infrared and near-infrared technology for the food industry and agricultural uses. On line applications. Food Control 5:(1) Benady, M., J.E. Simon, DJ. Charles and G.E. Miles. (1992). Determining melon ripeness by analyzing heads pace gas emission. ASAE International Summer Meeting. Charlotte, North Carolina. ASAE paper 92-6055. Brons, A., G. Rabatel, F. Sevila and C. Touzet (1991). Neural network techniques to simulate human judgement on pot plant quality: an artificial expert for beauty evaluation . ASAE paper No. 91-9893 . Chen, H., J. De Baerdemaek.e r and V. Bellon. (1993). Finite element study of the melon for nondestructive sensing of melon quality. ASAE International Winter Meeting. Chicago. Illinois.
ASAE paper 93-6599 . Crochon. M. Eating quality of melons. (1986). EEC Conference. Aix-en-Provence, France. Duda, R. and P. Hart (1973). Pattern Classification and Scene Analysis. John Wiley & Sons. 207