- Email: [email protected]
Data analysis for a hybrid sensor array
Sensors and Actuators B 106 (2005) 136–143
Data analysis for a hybrid sensor array M. Pardo a,∗ , L.G. Kwong b , G. Sberveglieri a , K. Brubaker c , J.F. Schneider c , W.R. Penrose d , J.R. Stetter d b
a INFM and University of Brescia, Brescia, Italy Xavier University, Cagayan de Oro City, Philippines c Argonne National Laboratory, Argonne, IL, USA d Illinois Institute of Technology, Chicago, IL, USA
Available online 2 July 2004
Abstract We present the results obtained in measuring diverse food products with the Moses II electronic nose (EN) equipped with three different classes of chemical sensors; namely, seven quartz micro-balances (QMB), eight semiconductor sensors (S) and four electrochemical cells (EC). Data are analyzed both with traditional PCA plots, pointing out the limits encountered by this technique and via exhaustive sensor selection. The principal sensor selection results are that: (a) the ranking of the sensor type with regard to discrimination is QCM > EC > S; (b) selected hybrid sensors have much better performances than selected sensors from any single sensor class (test set error lowered by circa 35%); (c) sensors selected from the hybrid array also have better performances than the complete set of hybrid sensors (test set error lowered by circa 25%); and in particular (d) a subsets of as few as two sensors (one QCM, one EC cell) give results similar or better to all 19 sensors. © 2004 Elsevier B.V. All rights reserved. Keywords: Data analysis; Hybrid sensor array; Electronic nose; Feature selection
1. Introduction The US Department of Agriculture (USDA) is responsible for the interdiction of agricultural products, especially fruits and meats, from abroad. They do this in order to prevent the importation of exotic pests and diseased or defective food products. At present, inspectors select passenger bags and shipping containers at random, and physically inspect them. Trained dogs are sometimes used, but little in the way of instrumentation has been exploited. We have begun preliminary work towards developing a hybrid sensor array (electronic nose, EN) device specifically to detect meat and fruit in baggage under a USDA contract. The rational design of a sensor array-based instrument for detecting contraband agricultural products in airline freight involves (a) identification of specific vapours typically found in the headspace above various samples of meat and fruit, and (b) identification of a minimum set of sensors that will allow efficient detection. Hybrid sensor systems have been often advocated (and rarely reported to be used) to improve EN’s selectivity, see, ∗ Corresponding author. Fax: +39 046 13 288969. E-mail address: [email protected] (M. Pardo).
0925-4005/$ – see front matter © 2004 Elsevier B.V. All rights reserved. doi:10.1016/j.snb.2004.05.045
e.g. [1–3]. The claimed advantages of such systems is the low(er) correlation between the responses of the different sensors types, which in turn is assured by the different construction techniques. Data analysis for the EN is often limited to the drawing of PCA plots. This is the case also for the proof of performance enhancement with an hybrid sensor array given by Ulmer et al. [1,2]. PCA score plots obtained with single sensor types and with the hybrid array are visually compared. This is fine for easy problems/ problems with few data, in which the advantage given by the hybrid array is macroscopic, which is not quite always the case. It may well be that a hybrid array is beneficial even if this is not apparent from PCA plots. What matters is that the actual quantitative indicator of performance, which is the test set error for some classifier, is lowered. Moreover, the higher the dimensionality of the measurement space, the less representative the PCA plot is, since it squashes all data in a 2D plane, irrespective of the original dimensionality. Holmberg et al. [3] go a step forward and determine a subset of the hybrid array with better performance than the whole hybrid array (actually, they do not show explicitly that the performance of the hybrid array is better than that of the single components). The sensor selection is accom-
M. Pardo et al. / Sensors and Actuators B 106 (2005) 136–143
plished through the use of cluster analysis (CA) and sensor correlation, though details are not given. This procedure still has a high degree of arbitrariness. For CA, a criterion has to be put forward to fix the number of clusters (branches) at which the hierarchical tree is cut and to select the sensor representative of any given sensor cluster. The sensors correlation matrix, on the other hand, examines only the pair wise correlation of variables. In other words, CA and the sensors correlation matrix can represent an aid for sensor selection but do not assure any optimality. The topic of feature selection has been devoted some attention in the sensor field mainly outside the analysis of hybrid arrays, see, e.g. [4–6]. Selecting good sensor subsets can help both in the understanding of the sensors themselves and in enhancing the classification performance through a more stable data representation. In statistical pattern recognition, the phenomenon of the curse of dimensionality has been often observed, where the sparseness of the data in high dimensional spaces causes a bad classification performance. For the case of hybrid arrays, it is possible that all the sensors taken together do not lower (or do not significantly lower) the test set error with respect to a single class of sensors, while a properly chosen subset does. In the present contribution, we first describe data qualitatively with PCA plots and then determine the best sensor subsets of dimensionality up to four through an exhaustive sensor subset selection.
137
Table 1 Samples for the four analyzed datasets #Set
Sample
Abbreviation
1
Blank Smoked ham Smoked salmon Fresh pork Fresh salmon Imitation crab meat
B SH SS P FS ICM
2
Blank Corned beef Vienna Corned Chicago Pork Roast beef
B CBV CBC PR RB
3
Cooked ham Hot head cheese Sliced bologna Cotto salami Blank
CH HHC SB CS B
4
Blank Plastic Paper Cotto salami wrapped in paper + plastic Cotto salami wrapped in plastic Cotto salami (unwrapped)
B Plc Ppr CS–Plc/Ppr CS–Plc CS
Abbreviations are used in the PCA plots legends.
food item (Salami) under conditions of attempted concealment. 2.2. PCA and sensor selection
2. Experimental and methods 2.1. The E-nose measurements Samples of cured meat products were purchased. All were frozen before use. Samples were prepared by weighing 1.00 ± 0.05 g samples of the meat into 20 ml headspace vials. These were placed in groups of 44 into the headspace sampler of a Moses II Electronic Nose (Lennartz Electronic SA, Tubingen, Germany). All samples were prepared in quadruplicate and only a single headspace measurement taken from each. We determined the optimal experimental conditions with regard to sampling temperature, HR control and sensor recovery time. Altogether, we measured 30 food items over a 45-day-period. The measurements were subdivided in 20 sets, each corresponding to a particular discrimination problem, e.g. fresh meat against smoked meat. Each sub-problem comprised 4–6 different sample types. The Moses II was equipped with three different classes of chemical sensors, namely seven quartz micro-balances (QMB), eight semiconductor sensors (S) and four electrochemical cells (EC). We analyze here the four sets, as described in Table 1. The first three sets have been measured the same day, while the fourth set has been measured after 40 days. The fourth set is representative of a real USDA problem since we measure a
PCA analysis is a useful tool for showing selected data slices, where the selection is decided on the basis of a priori knowledge of the data structure or from hindsight gained from data through successive application of PCA itself. Data can be sliced, e.g. by plotting the measurements on only a subset of samples, or by disregarding some sensors or features. PCA analysis, or some other form of multivariate data projection and plotting, is useful for gaining confidence in the data and as an a posteriori double check for results obtained with other non-visual, automated techniques (e.g. to look a the best sensor subsets derived via sensor selection). Feature selection needs two ingredients: a criterion for judging a feature subset (classical ones are the ratio between between-class class distances divided by within-class scatter or the test set error for a given classifier) and a search strategy through which successive subsets are examined (for an overview, see, e.g. [7,8]). In the present contribution, we performed Sensor Selection on dataset #4 of Table 1 since this is the dataset most representative of the real USDA problem. Sensors were selected via an exhaustive search of all 2,3,4 sensor subsets out of 19 sensors using the cross-validated (four-folds) test set error of a kNN (k = 3) classifier as performance measure. This means that, e.g. for the analysis of the four sensor subsets, 19 pick 4 = 3876 sensor sets have been ranked. Evidently, it would not have been possible to look at every sensor combination with PCA.
138
M. Pardo et al. / Sensors and Actuators B 106 (2005) 136–143
We divided the data in four-folds because there were only four data (replicates) for every class. In the fold used in turn for testing, by design, we put one point out of every class so that the test set is representative of all data. We considered kNN (with k fixed to k = 3 a priori, we tried also some other values of k and did not get different results) because it is both fast and flexible. A linear discriminant would have been too rigid, while the computational overhead for, say, multilayer perceptrons (ANN) did not allow their use.
3. Results We first draw PCA score plots for successive data slices, showing how far data can be interpreted by PCA. We then present the sensor selection results. 3.1. PCA analysis In the first PCA plot, Fig. 1, we show all the data (of the four analyzed sets) using all sensors. Data are mixed, yet some gross features can be recognized (numbers refer to the plot): fresh salmon (FS) is spread (1); other food types, like imitation of crab meat (ICM), are well clustered (2); cotto salami (CS) samples measured at an interval of 40 days produce different responses (3); blanks (B) and plastic (Plc) cluster together and blanks give constant responses even after 40 days (4).
In Fig. 2, we eliminated dataset #4. Reducing the number of classes, more information becomes readable: data of fresh foods are spread (1); treated (smoked, cooked) product cluster together (2); dataset #3 forms its own cluster (but note that blanks still are in the same cluster as the blanks of the other datasets). For getting an impression of the influence of the various sensor types on the discrimination, we plotted the PCA plots for all sensors and for each sensor type separately for dataset #1 (Figs. 3–6). Various deductions can be made by comparing the plots: (1) sensors inside the QCM and semiconductor class are correlated, since the first principal component for each sensor class contains more than 90% of the total variance (Figs. 3 and 5); (2) the PCA plot obtained with all sensors is similar to that obtained with the semiconductor sensors only; (3) QCM are a good choice if the task is discriminating blanks from food; (4) EC cells show a curious trend in the data, where smoked and fresh foods depart from the cluster of blanks in two orthogonal directions; (5) the spread of smoked products relative to the spread of fresh products is significantly smaller for EC cells than for QCM and semiconductor sensors. 3.2. Sensor selection The output of the feature selection algorithm is a ranking of every subset for every searched feature set cardinality. In Fig. 7, the ranking index (the cross validated classification ratio) for the top scoring subsets of cardinality 2,3,4 is shown. The performance of the best subset increases with
Fig. 1. PCA plot of all the samples described in Table 1. Numbers and arrows are explained in the text.
M. Pardo et al. / Sensors and Actuators B 106 (2005) 136–143
Fig. 2. PCA plot for the first three datasets described in Table 1. Numbers and arrows are explained in the text.
Fig. 3. PCA plot of dataset #1 described in Table 1.
139
140
M. Pardo et al. / Sensors and Actuators B 106 (2005) 136–143
Fig. 4. PCA plot of dataset #1 described in Table 1. Only EC cells were considered for drawing this plot.
Fig. 5. PCA plot of dataset #1 described in Table 1. Only QCM were considered for drawing this plot.
M. Pardo et al. / Sensors and Actuators B 106 (2005) 136–143
141
Fig. 6. PCA plot of dataset #1 described in Table 1. Only semiconductor sensors were considered for drawing this plot.
the cardinality of the subset. This holds true for the analyzed cardinalities. A scan of all cardinalities would be needed to asses the complete curve of best performance vs. subset cardinality; yet after a certain cardinality, the best perfor-
mance is bound to drop. The line in Fig. 7, in fact, shows the performance using the complete set of 19 sensors. We see that it is worse than the performance for the top scoring two-sensor-subsets. This is an instance of the curse of
Fig. 7. Cross validated test set performance (3-NN classifier) for the top scoring 2,3,4-sensor-subsets. The continuous line gives the performance obtained with all 19 sensors.
142
M. Pardo et al. / Sensors and Actuators B 106 (2005) 136–143
dimensionality and points out the need for feature selection, especially to take advantage of the increased number of sensors in the hybrid array. Increasing the subset cardinality also increases the number of sets with high performance, e.g. the there are only 4 three-sensor-subsets with a higher performance than that of the best two-sensor-subset while more than 50 four-sensor-subsets share this property. This clearly depends on the combinatorial grow of the number of subsets with the cardinality of the subsets (up to a cardinality of one half of the total number of sensors). Interestingly enough, the best two-sensor-subset is formed by sensors of two different classes, one QCM and one EC cell. Moreover, the best three-sensor-subsets adds one semiconductor sensor to the previous two. Some further analysis is required to compare the results obtained with the hybrid array to the best ones obtainable with any single class. In Fig. 8, we report the best performances for 2,3,4-sensor sets chosen from (a) all the hybrid array and (b) each single sensor class. We also show the performance obtained with all the sensors from the hybrid array and from each single sensor class, i.e. the results obtainable without feature selection (straight lines). The figure supports the following conclusions: 1. The ranking of single sensor types is QCM > EC > S, both by comparing the performance of the best selected sensors and the performance of all sensors (straight lines).
2. For any given sensor class, feature selection improves the performance by three to five percentage points (the classification error being around 20%) using as few as two or three sensors. 3. The performance obtained with the complete hybrid array (no feature selection) is better than the performance of the best selected sensor subsets for any single sensor type. The performance of all hybrid sensors is ∼83%, while the best subset of QCM sensors scores ∼80%. This means that even without performing feature selection on the hybrid array there is an advantage in going hybrid. 4. Still, if the best sensor subsets are selected from the hybrid array, the performance increases up to ∼87%. This corresponds to a decrease of ∼7% out of an error of ∼20%, i.e. to a 35% reduction of the classification error. It is to be noted that in another feature selection study [9], we found that point 4 above held, while this was not the case for point 3. In that study, different types of features were extracted from the response curve of each sensor (all of them semiconductor sensors) and a similar analysis to that performed in this paper was completed. While the best selected feature sets coming out of all features gave better performance that the best selected feature sets derived from each single feature class (like in point 4 above), the performance of kNN over all features (i.e. no selection) was worse than the best selected feature sets derived from some single feature class (unlike in point 3 above). This would hint to
Fig. 8. Cross validated test set performance (3-NN classifier) for the best 2,3,4-sensor subsets for four sensor classes: (1) QCM; (2) semiconductor; (3) EC cells and (4) QCM + S + EC. The continuous line gives the performance for all the sensors in these classes, i.e. seven QCM, eight S, four EC cells, the array of 19 sensors.
M. Pardo et al. / Sensors and Actuators B 106 (2005) 136–143
143
Fig. 9. PCA plot obtained from all sensors (left) and PCA plot obtained with just the best two-sensor-subset (one QCM and one EC) given by feature selection (right). Apart form a rotation around the first principal component, the plots are similar (though the classification performance is better for the two-sensor-subset).
the fact that in general feature selection is necessary when the number of features increases. In Fig. 9, we compare the PCA plot obtained from all sensors (left) with the PCA plot obtained with just the best two-sensor-subset (one QCM and one EC) given by feature selection (right). Apart form a rotation around the first principal component, the plots are quite similar (though the classification performance is better for the two-sensorsubset).
4. Conclusions We have performed preliminary measurements aimed at evaluating the possible use of an hybrid EN for detecting contraband food. The analysis of the data has been divided in two parts: PCA permits to get visual confidence, then feature selection gives quantitative classification figures for sensor subsets. In particular, we compared the best results obtained with any single sensor type with the best results obtained with the hybrid array. The outlined methodology allows to state that the fusion of different sensor types substantially boosts EN performances. The advantage given by the hybrid array is fully exploited by performing sensor selection. Just two sensors (one QCM and one EC cell) give better performance than all 19 sensors. Still, it has to be noted that sensor selection results have been obtained from only one dataset with few
points per class. Further analysis will allow more general conclusions.
References [1] H. Ulmer, J. Mitrovics, G. Noetzel, U. Weimar, W. Göpel, Odours and flavours identified with hybrid modular sensor systems, Sens. Actuators B 43 (1997) 24–33. [2] H. Ulmer, J. Mitrovics, U. Weimar, W. Göpel, Sensor arrays with only one or several transducer principles? The advantage of hybrid modular systems, Sens. Actuators B 65 (2000) 79–81. [3] M. Holmberg, F. Winquist, I. Lundström, J.W. Gardner, E.L. Hines, Identification of paper quality using a hybrid electronic nose, Sens. Actuators B 27 (1–3) (1995) 246–249. [4] A.N. Chaudry, T.M. Hawkins, P.J. Travers, A method for selecting an optimum sensor array, Sens. Actuators B 69 (3) (2000) 236–242. [5] P. Boilot, E.L. Hines, M.A. Gongora, R.S. Folland, Electronic noses inter-comparison, data fusion and sensor selection in discrimination of standard fruit solutions, Sens. Actuators B 88 (1) (2003) 80–88. [6] A. Pardo, S. Marco, A. Ortega, A. Perera, T. Sundic, J. Samitier, in: Proceedings of the Conference of International Symposium on Olfaction and Electronic Noses ISOEN, Institute of Physics Publishing, London, 2000, pp. 83–88. [7] R.O. Duda, P.E. Hart, D.G. Stork, Pattern Classification, second ed., Wiley, 2001. [8] A. Webb, Statistical Pattern Recognition, Arnold, 1999. [9] M. Della Torre, I. Riccò, F. Nardini, A. Bresciani, M. Falasconi, M. Pardo, G. Sberveglieri, Evaluating features for coffee seasoning estimation with the novel EOS835 electronic nose, in: Proceedings of the 10th International Symposium on Olfaction and Electronic Nose, Riga, 2003.
Data analysis for a hybrid sensor array M. Pardo a,∗ , L.G. Kwong b , G. Sberveglieri a , K. Brubaker c , J.F. Schneider c , W.R. Penrose d , J.R. Stetter d b
a INFM and University of Brescia, Brescia, Italy Xavier University, Cagayan de Oro City, Philippines c Argonne National Laboratory, Argonne, IL, USA d Illinois Institute of Technology, Chicago, IL, USA
Available online 2 July 2004
Abstract We present the results obtained in measuring diverse food products with the Moses II electronic nose (EN) equipped with three different classes of chemical sensors; namely, seven quartz micro-balances (QMB), eight semiconductor sensors (S) and four electrochemical cells (EC). Data are analyzed both with traditional PCA plots, pointing out the limits encountered by this technique and via exhaustive sensor selection. The principal sensor selection results are that: (a) the ranking of the sensor type with regard to discrimination is QCM > EC > S; (b) selected hybrid sensors have much better performances than selected sensors from any single sensor class (test set error lowered by circa 35%); (c) sensors selected from the hybrid array also have better performances than the complete set of hybrid sensors (test set error lowered by circa 25%); and in particular (d) a subsets of as few as two sensors (one QCM, one EC cell) give results similar or better to all 19 sensors. © 2004 Elsevier B.V. All rights reserved. Keywords: Data analysis; Hybrid sensor array; Electronic nose; Feature selection
1. Introduction The US Department of Agriculture (USDA) is responsible for the interdiction of agricultural products, especially fruits and meats, from abroad. They do this in order to prevent the importation of exotic pests and diseased or defective food products. At present, inspectors select passenger bags and shipping containers at random, and physically inspect them. Trained dogs are sometimes used, but little in the way of instrumentation has been exploited. We have begun preliminary work towards developing a hybrid sensor array (electronic nose, EN) device specifically to detect meat and fruit in baggage under a USDA contract. The rational design of a sensor array-based instrument for detecting contraband agricultural products in airline freight involves (a) identification of specific vapours typically found in the headspace above various samples of meat and fruit, and (b) identification of a minimum set of sensors that will allow efficient detection. Hybrid sensor systems have been often advocated (and rarely reported to be used) to improve EN’s selectivity, see, ∗ Corresponding author. Fax: +39 046 13 288969. E-mail address: [email protected] (M. Pardo).
0925-4005/$ – see front matter © 2004 Elsevier B.V. All rights reserved. doi:10.1016/j.snb.2004.05.045
e.g. [1–3]. The claimed advantages of such systems is the low(er) correlation between the responses of the different sensors types, which in turn is assured by the different construction techniques. Data analysis for the EN is often limited to the drawing of PCA plots. This is the case also for the proof of performance enhancement with an hybrid sensor array given by Ulmer et al. [1,2]. PCA score plots obtained with single sensor types and with the hybrid array are visually compared. This is fine for easy problems/ problems with few data, in which the advantage given by the hybrid array is macroscopic, which is not quite always the case. It may well be that a hybrid array is beneficial even if this is not apparent from PCA plots. What matters is that the actual quantitative indicator of performance, which is the test set error for some classifier, is lowered. Moreover, the higher the dimensionality of the measurement space, the less representative the PCA plot is, since it squashes all data in a 2D plane, irrespective of the original dimensionality. Holmberg et al. [3] go a step forward and determine a subset of the hybrid array with better performance than the whole hybrid array (actually, they do not show explicitly that the performance of the hybrid array is better than that of the single components). The sensor selection is accom-
M. Pardo et al. / Sensors and Actuators B 106 (2005) 136–143
plished through the use of cluster analysis (CA) and sensor correlation, though details are not given. This procedure still has a high degree of arbitrariness. For CA, a criterion has to be put forward to fix the number of clusters (branches) at which the hierarchical tree is cut and to select the sensor representative of any given sensor cluster. The sensors correlation matrix, on the other hand, examines only the pair wise correlation of variables. In other words, CA and the sensors correlation matrix can represent an aid for sensor selection but do not assure any optimality. The topic of feature selection has been devoted some attention in the sensor field mainly outside the analysis of hybrid arrays, see, e.g. [4–6]. Selecting good sensor subsets can help both in the understanding of the sensors themselves and in enhancing the classification performance through a more stable data representation. In statistical pattern recognition, the phenomenon of the curse of dimensionality has been often observed, where the sparseness of the data in high dimensional spaces causes a bad classification performance. For the case of hybrid arrays, it is possible that all the sensors taken together do not lower (or do not significantly lower) the test set error with respect to a single class of sensors, while a properly chosen subset does. In the present contribution, we first describe data qualitatively with PCA plots and then determine the best sensor subsets of dimensionality up to four through an exhaustive sensor subset selection.
137
Table 1 Samples for the four analyzed datasets #Set
Sample
Abbreviation
1
Blank Smoked ham Smoked salmon Fresh pork Fresh salmon Imitation crab meat
B SH SS P FS ICM
2
Blank Corned beef Vienna Corned Chicago Pork Roast beef
B CBV CBC PR RB
3
Cooked ham Hot head cheese Sliced bologna Cotto salami Blank
CH HHC SB CS B
4
Blank Plastic Paper Cotto salami wrapped in paper + plastic Cotto salami wrapped in plastic Cotto salami (unwrapped)
B Plc Ppr CS–Plc/Ppr CS–Plc CS
Abbreviations are used in the PCA plots legends.
food item (Salami) under conditions of attempted concealment. 2.2. PCA and sensor selection
2. Experimental and methods 2.1. The E-nose measurements Samples of cured meat products were purchased. All were frozen before use. Samples were prepared by weighing 1.00 ± 0.05 g samples of the meat into 20 ml headspace vials. These were placed in groups of 44 into the headspace sampler of a Moses II Electronic Nose (Lennartz Electronic SA, Tubingen, Germany). All samples were prepared in quadruplicate and only a single headspace measurement taken from each. We determined the optimal experimental conditions with regard to sampling temperature, HR control and sensor recovery time. Altogether, we measured 30 food items over a 45-day-period. The measurements were subdivided in 20 sets, each corresponding to a particular discrimination problem, e.g. fresh meat against smoked meat. Each sub-problem comprised 4–6 different sample types. The Moses II was equipped with three different classes of chemical sensors, namely seven quartz micro-balances (QMB), eight semiconductor sensors (S) and four electrochemical cells (EC). We analyze here the four sets, as described in Table 1. The first three sets have been measured the same day, while the fourth set has been measured after 40 days. The fourth set is representative of a real USDA problem since we measure a
PCA analysis is a useful tool for showing selected data slices, where the selection is decided on the basis of a priori knowledge of the data structure or from hindsight gained from data through successive application of PCA itself. Data can be sliced, e.g. by plotting the measurements on only a subset of samples, or by disregarding some sensors or features. PCA analysis, or some other form of multivariate data projection and plotting, is useful for gaining confidence in the data and as an a posteriori double check for results obtained with other non-visual, automated techniques (e.g. to look a the best sensor subsets derived via sensor selection). Feature selection needs two ingredients: a criterion for judging a feature subset (classical ones are the ratio between between-class class distances divided by within-class scatter or the test set error for a given classifier) and a search strategy through which successive subsets are examined (for an overview, see, e.g. [7,8]). In the present contribution, we performed Sensor Selection on dataset #4 of Table 1 since this is the dataset most representative of the real USDA problem. Sensors were selected via an exhaustive search of all 2,3,4 sensor subsets out of 19 sensors using the cross-validated (four-folds) test set error of a kNN (k = 3) classifier as performance measure. This means that, e.g. for the analysis of the four sensor subsets, 19 pick 4 = 3876 sensor sets have been ranked. Evidently, it would not have been possible to look at every sensor combination with PCA.
138
M. Pardo et al. / Sensors and Actuators B 106 (2005) 136–143
We divided the data in four-folds because there were only four data (replicates) for every class. In the fold used in turn for testing, by design, we put one point out of every class so that the test set is representative of all data. We considered kNN (with k fixed to k = 3 a priori, we tried also some other values of k and did not get different results) because it is both fast and flexible. A linear discriminant would have been too rigid, while the computational overhead for, say, multilayer perceptrons (ANN) did not allow their use.
3. Results We first draw PCA score plots for successive data slices, showing how far data can be interpreted by PCA. We then present the sensor selection results. 3.1. PCA analysis In the first PCA plot, Fig. 1, we show all the data (of the four analyzed sets) using all sensors. Data are mixed, yet some gross features can be recognized (numbers refer to the plot): fresh salmon (FS) is spread (1); other food types, like imitation of crab meat (ICM), are well clustered (2); cotto salami (CS) samples measured at an interval of 40 days produce different responses (3); blanks (B) and plastic (Plc) cluster together and blanks give constant responses even after 40 days (4).
In Fig. 2, we eliminated dataset #4. Reducing the number of classes, more information becomes readable: data of fresh foods are spread (1); treated (smoked, cooked) product cluster together (2); dataset #3 forms its own cluster (but note that blanks still are in the same cluster as the blanks of the other datasets). For getting an impression of the influence of the various sensor types on the discrimination, we plotted the PCA plots for all sensors and for each sensor type separately for dataset #1 (Figs. 3–6). Various deductions can be made by comparing the plots: (1) sensors inside the QCM and semiconductor class are correlated, since the first principal component for each sensor class contains more than 90% of the total variance (Figs. 3 and 5); (2) the PCA plot obtained with all sensors is similar to that obtained with the semiconductor sensors only; (3) QCM are a good choice if the task is discriminating blanks from food; (4) EC cells show a curious trend in the data, where smoked and fresh foods depart from the cluster of blanks in two orthogonal directions; (5) the spread of smoked products relative to the spread of fresh products is significantly smaller for EC cells than for QCM and semiconductor sensors. 3.2. Sensor selection The output of the feature selection algorithm is a ranking of every subset for every searched feature set cardinality. In Fig. 7, the ranking index (the cross validated classification ratio) for the top scoring subsets of cardinality 2,3,4 is shown. The performance of the best subset increases with
Fig. 1. PCA plot of all the samples described in Table 1. Numbers and arrows are explained in the text.
M. Pardo et al. / Sensors and Actuators B 106 (2005) 136–143
Fig. 2. PCA plot for the first three datasets described in Table 1. Numbers and arrows are explained in the text.
Fig. 3. PCA plot of dataset #1 described in Table 1.
139
140
M. Pardo et al. / Sensors and Actuators B 106 (2005) 136–143
Fig. 4. PCA plot of dataset #1 described in Table 1. Only EC cells were considered for drawing this plot.
Fig. 5. PCA plot of dataset #1 described in Table 1. Only QCM were considered for drawing this plot.
M. Pardo et al. / Sensors and Actuators B 106 (2005) 136–143
141
Fig. 6. PCA plot of dataset #1 described in Table 1. Only semiconductor sensors were considered for drawing this plot.
the cardinality of the subset. This holds true for the analyzed cardinalities. A scan of all cardinalities would be needed to asses the complete curve of best performance vs. subset cardinality; yet after a certain cardinality, the best perfor-
mance is bound to drop. The line in Fig. 7, in fact, shows the performance using the complete set of 19 sensors. We see that it is worse than the performance for the top scoring two-sensor-subsets. This is an instance of the curse of
Fig. 7. Cross validated test set performance (3-NN classifier) for the top scoring 2,3,4-sensor-subsets. The continuous line gives the performance obtained with all 19 sensors.
142
M. Pardo et al. / Sensors and Actuators B 106 (2005) 136–143
dimensionality and points out the need for feature selection, especially to take advantage of the increased number of sensors in the hybrid array. Increasing the subset cardinality also increases the number of sets with high performance, e.g. the there are only 4 three-sensor-subsets with a higher performance than that of the best two-sensor-subset while more than 50 four-sensor-subsets share this property. This clearly depends on the combinatorial grow of the number of subsets with the cardinality of the subsets (up to a cardinality of one half of the total number of sensors). Interestingly enough, the best two-sensor-subset is formed by sensors of two different classes, one QCM and one EC cell. Moreover, the best three-sensor-subsets adds one semiconductor sensor to the previous two. Some further analysis is required to compare the results obtained with the hybrid array to the best ones obtainable with any single class. In Fig. 8, we report the best performances for 2,3,4-sensor sets chosen from (a) all the hybrid array and (b) each single sensor class. We also show the performance obtained with all the sensors from the hybrid array and from each single sensor class, i.e. the results obtainable without feature selection (straight lines). The figure supports the following conclusions: 1. The ranking of single sensor types is QCM > EC > S, both by comparing the performance of the best selected sensors and the performance of all sensors (straight lines).
2. For any given sensor class, feature selection improves the performance by three to five percentage points (the classification error being around 20%) using as few as two or three sensors. 3. The performance obtained with the complete hybrid array (no feature selection) is better than the performance of the best selected sensor subsets for any single sensor type. The performance of all hybrid sensors is ∼83%, while the best subset of QCM sensors scores ∼80%. This means that even without performing feature selection on the hybrid array there is an advantage in going hybrid. 4. Still, if the best sensor subsets are selected from the hybrid array, the performance increases up to ∼87%. This corresponds to a decrease of ∼7% out of an error of ∼20%, i.e. to a 35% reduction of the classification error. It is to be noted that in another feature selection study [9], we found that point 4 above held, while this was not the case for point 3. In that study, different types of features were extracted from the response curve of each sensor (all of them semiconductor sensors) and a similar analysis to that performed in this paper was completed. While the best selected feature sets coming out of all features gave better performance that the best selected feature sets derived from each single feature class (like in point 4 above), the performance of kNN over all features (i.e. no selection) was worse than the best selected feature sets derived from some single feature class (unlike in point 3 above). This would hint to
Fig. 8. Cross validated test set performance (3-NN classifier) for the best 2,3,4-sensor subsets for four sensor classes: (1) QCM; (2) semiconductor; (3) EC cells and (4) QCM + S + EC. The continuous line gives the performance for all the sensors in these classes, i.e. seven QCM, eight S, four EC cells, the array of 19 sensors.
M. Pardo et al. / Sensors and Actuators B 106 (2005) 136–143
143
Fig. 9. PCA plot obtained from all sensors (left) and PCA plot obtained with just the best two-sensor-subset (one QCM and one EC) given by feature selection (right). Apart form a rotation around the first principal component, the plots are similar (though the classification performance is better for the two-sensor-subset).
the fact that in general feature selection is necessary when the number of features increases. In Fig. 9, we compare the PCA plot obtained from all sensors (left) with the PCA plot obtained with just the best two-sensor-subset (one QCM and one EC) given by feature selection (right). Apart form a rotation around the first principal component, the plots are quite similar (though the classification performance is better for the two-sensorsubset).
4. Conclusions We have performed preliminary measurements aimed at evaluating the possible use of an hybrid EN for detecting contraband food. The analysis of the data has been divided in two parts: PCA permits to get visual confidence, then feature selection gives quantitative classification figures for sensor subsets. In particular, we compared the best results obtained with any single sensor type with the best results obtained with the hybrid array. The outlined methodology allows to state that the fusion of different sensor types substantially boosts EN performances. The advantage given by the hybrid array is fully exploited by performing sensor selection. Just two sensors (one QCM and one EC cell) give better performance than all 19 sensors. Still, it has to be noted that sensor selection results have been obtained from only one dataset with few
points per class. Further analysis will allow more general conclusions.
References [1] H. Ulmer, J. Mitrovics, G. Noetzel, U. Weimar, W. Göpel, Odours and flavours identified with hybrid modular sensor systems, Sens. Actuators B 43 (1997) 24–33. [2] H. Ulmer, J. Mitrovics, U. Weimar, W. Göpel, Sensor arrays with only one or several transducer principles? The advantage of hybrid modular systems, Sens. Actuators B 65 (2000) 79–81. [3] M. Holmberg, F. Winquist, I. Lundström, J.W. Gardner, E.L. Hines, Identification of paper quality using a hybrid electronic nose, Sens. Actuators B 27 (1–3) (1995) 246–249. [4] A.N. Chaudry, T.M. Hawkins, P.J. Travers, A method for selecting an optimum sensor array, Sens. Actuators B 69 (3) (2000) 236–242. [5] P. Boilot, E.L. Hines, M.A. Gongora, R.S. Folland, Electronic noses inter-comparison, data fusion and sensor selection in discrimination of standard fruit solutions, Sens. Actuators B 88 (1) (2003) 80–88. [6] A. Pardo, S. Marco, A. Ortega, A. Perera, T. Sundic, J. Samitier, in: Proceedings of the Conference of International Symposium on Olfaction and Electronic Noses ISOEN, Institute of Physics Publishing, London, 2000, pp. 83–88. [7] R.O. Duda, P.E. Hart, D.G. Stork, Pattern Classification, second ed., Wiley, 2001. [8] A. Webb, Statistical Pattern Recognition, Arnold, 1999. [9] M. Della Torre, I. Riccò, F. Nardini, A. Bresciani, M. Falasconi, M. Pardo, G. Sberveglieri, Evaluating features for coffee seasoning estimation with the novel EOS835 electronic nose, in: Proceedings of the 10th International Symposium on Olfaction and Electronic Nose, Riga, 2003.