- Email: [email protected]
Application of neural-network systems to the dynamic response of polymer-based sensor arrays
B ELSEVIER
Sensors and Actuators B 26-27 (1995) 232-236
CHEMICAL
Application of neural-network systems to the dynamic response of polymer-based sensor arrays Markus Schweizer-Berberich b Josef G6ppert a, Andreas Hierlemann b Jan Mitrovics Udo Weimar b, Wolfgang Rosenstiel a, Wolfgang G6pel b
b,
=Wilhelm-Schickard-Institute of Computer Science, University of Tiibingen, Auf der Morgenstelle 8, D-72076 Tiibingen, Germany b Institute of Physical and Theoretical Chemistry and Center of Interface Analysis and Sensors, University of Tiibingen, Auf der Morgenstelle 8, D-72076 Tiibingen, Germany
Abstract
The conventional calibration method for sensor arrays uses steady-state signals that depend on the gas concentration. This method can be time consuming if many concentrations and compositions of a multicomponent mixture are required. Good experimental design may reduce the necessary effort so that the number of calibration experiments is minimized. Dynamic measurements may significantly reduce the time of each calibration experiment. In the present approach a random walk through the domain of the gas concentrations is chosen with each step of the walk adjusted for a short time only. The sensor array consists of six polymer (polysiloxanes with functional groups)-coated bulk acoustic wave (BAW) devices. The concentration domain is defined by a binary mixture of n-octane and toluene (150 to 800 ppm). Neural networks evaluate both qualitative and quantitative information frorfi the sensor response. In particular, the extensions of feed-forward nets towards recurrent or time-delay structures can be used to solve problems related to dynamic evaluations (e.g., no steady-state signal, parameter drift). These network architectures with different numbers of hidden neurons are applied to evaluate the data from the BAW device array. The networks are trained with back-propagation-like training algorithms and are validated with arbitrary gas mixtures. Keywords: Gas sensors; Calibration of array sensors; Multicomponent analysis; Artificial neural networks
I. Introduction Chemical or biochemical sensors can be useful in different applications, such as environmental monitoring, process analysis and quality control. Only for some special applications it is possible to use a single sensor set-up, because the cross-sensitivities of the sensor to other p a r a m e t e r s are negligible. In most other cases it is necessary to use sensor arrays for calculating the p a r a m e t e r s from sensor signals that are not linearly dependent. For such an approach classical chemometric algorithms are applied, such as principal components regression (PCR), partial least squares (PLS) or artificial neural networks (ANNs). For these algorithms steadystate signals are usually used as input values. As reported elsewhere, polymer-coated quartz microbalance sensors show a very good long-term stability and reproducibility for a period longer than half a year [1] if the differences (Af=fo-fgas) of steady-state signals are used. Several 0925-4005/95/$09.50 © 1995 Elsevier Science S.A. All rights reserved SSD1 0925-4005(94)01592-6
problems can occur in an evaluation of steady-state signals. The time taken to reach 90% of the signal (/9o) depends on kinetic parameters (e.g., diffusion rate, reaction rate, volume of the chamber, flow rate). Analyte-handling procedures are required to produce the reference value for the zero gas concentration. Additional valves, filters or reference gas are disadvantages for a portable system. Therefore we tried a dynamic calibration procedure. Any application of a sensor system in the field of environmental monitoring is characterized by a fluctuating concentration. A dynamic calibration procedure should simulate this behaviour in a m o r e realistic way than steady-state signals. Dynamic sensor response may have a further advantage because it includes information about the sensor kinetics that is different for each gas-polymer interaction. It can be evaluated from the transient of the sensor response to a change in the partial pressure [2,31.
M. Schweizer-Berberich et aL / Sensors and Actuators B 26-27 (1995) 232-236
2. Neural net model
The development of a neural net for an evaluation task consists of two major parts: first, the definition of the network structure and the network size (number of hidden layers and neurons, interconnection structure) and secondly, the training of the net (finding the optimal interconnection weights between neurons). The network structure is fixed in advance, according to known properties of the evaluation problem. Generally, the sensor signals include different properties of the system: (1) the sensitivity of the sensors to the different gases; (2) the dynamic properties of the system; (3) disturbing effects, like sensor drift or measuring noise. The sensitivity of the sensors may be evaluated by a simple feed-forward neural net of either linear neurons or non-linear hidden neurons for the estimation of the non-linear part of the model [4]. The evaluation of the dynamic properties requires dynamic behaviour of the neural net. Drift effects may also be considered as a long-term dynamic process, and are also taken into account by recurrent neurons. Noise may be centred in the set of training data. We used structured neural nets, consisting of two layers of neurons (Fig. 1). Input units store the values of the input vector. A layer of hidden neurons with a sigmoid transfer function performs mainly the nonlinear and the recurrent part of the data processing. The output layer consists of linear neurons, which are connected with the hidden neurons and by short-cut connections directly with the input units. It is able to combine linear data processing of the linear output neurons with the non-linear functions of the hidden neurons. Dynamic properties are introduced in the neural nets by recurrent connections, which create internal recurrent variables, or by a suitable representation of the sensor history in the input vector. This is performed by combining previous output values of the sensors, like a floating temporal window, in the (a)
Octane
Q
input vector. Both methods are used and compared in this paper. The training phase is performed by a back-propagation derivative called 'Quickpropagation' [5], a model that converges for the given data much faster than a standard back-propagation technique. The interconnection weights are randomly initialized in the range [ - 1... + 1]. During the training, the training vectors are presented in temporal order (this is needed for recurrent nets) and weights are updated after each presentation of all training vectors. Other neural-net architectures like self-organizing maps [6], using dynamic functions to evaluate dynamic data [7], represent other promising approaches.
3. Experimental
An array of piezoelectric quartz crystals was used to detect a binary mixture of volatile organic compounds: n-octane with toluene. This mixture serves as a model for other more interesting applications, such as the detection of benzene among a high level of hydrocarbons as is found at gasoline stations. 10 MHz bulk acoustic wave (BAW) quartz crystals were used as transducers and were coated with sidechain-modified polysiloxanes: polydimethylsiloxane (PDMS), poly(cyanopropyl)methylsiloxane (PCPMS), polyphenylmethylsiloxane (PPMS), poly(isopropylcarboxylic-acid)methylsiloxane (PiPCMS), poly(aminopropyl)methylsiloxane (PAPMS, 10% amino groups), poly [2- carboxy (b-valin- t-butylamide) propyl] methylsiloxane (Chirasil-Val, 10% valin groups). Solutions of these polymers in dichloromethane or propan-2-ol were sprayed onto the crystals. The frequency decrease for determining the layer thickness was monitored on-line. For more details see Ref. [1]. The frequency was measured with a frequency counter (HP 5334 B) connected to an 286-AT computer via an IEEE bus. Each channel was scanned with a multiplexer controlled by a PCL 726 interface card (Labtech, Wilmington, USA). Each experiment was stored with an offset defined by the first data point of this run.
(b)
Toluene
~
Octane Toluene
outputlayer hiddenlayer inputunits
recurrentvariables
sensorsignals
233
sensor number
Fig. 1. (a) R e c u r r e n t a n d (b) t i m e - d e l a y n e u r a l n e t w o r k s as u s e d for the e v a l u a t i o n .
~|~ 12 nm=6 -4
sensor 1 (9
sensorm(t)
M. Schweizer-Berbetich et al. / Sensors and Actuators B 26-27 (1995) 232-236
234
The array sensors were mounted inside a 200 ml brass chamber. The brass block and the gas were thermostatted at 303+0.1 K as well as the analyte gas before it entered the chamber. The concentrations of both gases were adjusted between 150 and 800 ppm by mass-flow controllers. Each mixture was fixed for 2 min. Such an experiment was called a random walk (RW). The frequency output was recorded every minute with 0.1 Hz resolution. Four random walks were used to train and test different neural networks.
in predicting the second random walk data, the performance on the three other walks is poor. The output value, especially on the first and the third random walks, is globally too small or too large. The error is quite small after recalibration at the beginning of each random walk and becomes larger towards the end. This type of error is probably caused by parametric drift of the sensors. Obviously the drift of the sensors did not behave similarly in the different data sets. In order to explore this behaviour, other ways of choosing training and validation vectors will now be compared.
4. Results and discussion
4.2. Reduction of drift
The time constants of the sensors are of the order of some seconds ( < 5 s). The PiPCMS sensor shows slower response. The time to reach 90% of the end concentration in the chamber is about 6 rain. Therefore the dominant time scale results from changing the gases or gas concentrations in the chamber. To test the potential of the following neural network, the root mean square (RMS) errors, calculated in ppm, were used:
E (%,ea.--C~¢~,)2
Eg., =
(1)
V ~ ' vector
E.,, = ~/~EgZ,~
(2)
V gas
4.1. First training A simple linear neural net was trained with the second random walk. The other three walks were used for the validation of the net. The output of this net is visualized in Fig. 2. Although the net is fairly good
800
By using the same data set for training and for validation, it will first be examined if neural networks are able to reduce the drift if represented in the training data. For this purpose, the linear neural net without hidden layers was trained with the measured data of all four data sets. Validation was performed using the same data. So the task of the neural net is to predict the trained data as well as possible. A closer look at the output shows a good approximation for all measurements (Fig. 3). A comparison of Figs. 3 and 2 reveals that the drift can be learned if presented in the training vectors. The influence of the network size and the network architecture on the error (RMS) is shown in Table 1. As the neural net used is comparatively small, the net cannot learn the training data perfectly. This is, however, desired in order to achieve better generalization properties. Increasing the complexity of the network by introducing recurrent or hidden neurons leads to a better prediction of the data. It can be noticed that the time-delay neural network seems to be more suitable for the given problem. We can conclude that if the same data set is used for training and for testing, the
Octane
800
predicted - -
E600
Octane
true
predicted
---
._=
E e~
600
true
~3 400
400
¢-
0 0
0 200
'H
RW !' ~' ',jl<
'~
RW2
RW3
);.(
c ~
U
><
200
0 o
RW4>
"< .
.
.
.
.
.
~,v~. ~.;.,,L~ . . . . . . . . i
0
i
2000
4000
time
<
RW3
lo
t~
0o~
03
i
~0
6000
in m i n
Fig. 2. E s t i m a t i o n of the c o n c e n t r a t i o n of n - o c t a n e with a l i n e a r n e u r a l net. T h e s e c o n d r a n d o m w a l k ( R W 2 ) w a s used for the training. G o o d a g r e e m e n t was f o u n d in the t r a i n i n g set but drift in the o t h e r sets.
0
2000
4000
6000
time in min Fig. 3. C a l c u l a t i o n of the n - o c t a n e c o n c e n t r a t i o n with a single l a y e r e d n e u r a l net w i t h o u t h i d d e n and r e c u r r e n t n e u r o n s . All r a n d o m w a l k s w e r e u s e d e i t h e r for the t r a i n i n g or the v a l i d a t i o n .
M. Schweizer-Berberich et aL / Sensors and Actuators B 26-27 (1995) 232-236 Table 1 Prediction properties (expressed as RMS errors) with changing size of the networks. Identical data sets were used for training and validation Cornpound
Linear
En-octa.~ 25.2 Eto]. . . . 25.8 E., 36.0
Recurrent
Time delay
2units
5 units
0hidden
2 hidden
5 hidden
25.4 24.4 35.2
24.0 23.0 33.2
21.4 24.7 32.7
21.9 23.5 32.1
22.0 22.8 31.7
alternating blocks of 50 min for training and validation set
E 32 Ct.
r/?
validation set ~
n-
28
(5
5
~
1'0
® 7 lime
1'5
20
25
number of hidden neurons Fig. 4. Prediction error (RMS) of the data with a time-delay net. All data points were divided into blocks of 50 min. The blocks were used alternately for training and validation. The influence of the hidden neurons on the error is pointed out.
drift can be suppressed and the error decreases with a more sophisticated network.
4.3. Prediction of new data Now the data set will be divided into two parts; one will be used for training and the second for the validation of the obtained model. This was done taking alternately blocks of 50 min for training and the following 50 min for validation. Training and validation data are distinct, but cover the same period of time. This procedure can be compared to a method where a reference concentration is used to recalibrate the system. Fig. 4 confirms for the time-delay net that with the given way of choosing the training data, a more complex neuron net model performs better on the training and on the test data sets. Surprisingly the error on the training set and on the validation set are in a similar range even though the validation set is smaller. This may be a result of the choice of training and validation sets, but obviously the error on the training set decreases more strongly than on the validation set by increasing the number of hidden neurons. An examination of the dynamic properties shows that the neural net output is able to follow the fast changes of the gas concentration. So, neural networks are suitable for the evaluation of dynamically acquired data and can reduce the effects of parameter drifts.
235
4.4. Training of temporal decorrelated data In fact, for future use of these sensors it is not satisfactory to have training and test data of the same time period. If training and validation data are temporally decorrelated, it is possible to improve the prediction by training with data from a longer period. It is expected that different types of drift are represented in a larger set of training data, which leads to a better generalization. The influence of using a larger set (RW2 + RW3 instead of RW2) for the training is shown in Table 2. The error is reduced for all data. It is not surprising that the error of the third random walk is drastically reduced because it is used for training. Comparison of different training cycles shows that the convergence is more stable. The second idea to prevent overlearning (overfitting) is to observe the training by means of an independent test set and stop the training at a convenient time. In Fig. 5 the evolution of the RMS error during a typical training cycle is shown for both the training set and a test set. Three phases are found: (1) A reduction of error in the training set as well as in the test set. The system is approximating characteristics which are valid for the training set as well as for the test set. (2) Further training leads to a reduction of the error in the training set but also to an increase of the error Table 2 Influence of the size of training data on the RMS error Validation sets
Training sets
ERw, ERw2 ERw3 Eaw4 E.,
RW 2
RW 2 and RW 3
82.5 56.5 85.1 64.8 75.2
62.9 45.6 27.2 57.6 48.8
time delay net 300
E o_ .~ 200
/
fast decrease
validation set: 1. random walk
IF-"
ffl
i
very slow decrease
,,(" 84 ppm after
100
70 training cycles I
;!'
~
training set: 2. random walk
2do
,;0
6;0
8do
,obo
training cycles Fig. 5. Overlearning of neural nets can be prevented by the minimum of a test set.
236
M. Schweizer-Berberich
et al. / Sensors and Actuators B 26-27 (1995) 232-236
Table 3 Prediction properties (expressed as RMS errors) by using a stop criterion on the validation set. Influence of the topology and the number of recurrent and hidden neurons Cornpound
Linear
En-octa,~ 32.7 EtoJ. . . . 36.3 E,, 48.8
Recurrent
Time delay
2units
5units
0hidden
2hidden
5hidden
34.8 39.2 52.5
34.3 36.8 50.5
37.3 38.1 53.4
36.6 38.4 53.4
36.4 37.5 52.5
in the test set. The training is now starting to approximate specific properties of the training set. (3) Stabilization of the training near the minimum. Only a small reduction of the training error and small changes on the error of the test set. In order to have the best generalization properties, it is recommended to stop the training at the end of phase one. For all further neural nets this training principle is applied. The stop criterion is determined by means of the first random walk. For a larger training set phase one (error on the training and the test data decreases) is lengthened. The longer training leads to a better overall performance. The hypothesis that a larger training set enables more information about the nature of the parameter drift to be gathered is confirmed. Neural nets trained in such a way are also fitter for the evaluation of data acquired before or after the training period. Table 3 shows the influence of the number of recurrent and hidden neurons on the RMS error. Contrary to the training with all random walks, these units do not seem to enable a better performance. The short training time is responsible for this due to the need for stopping the training at a very early stage. The time of about 100 training epochs is not sufficient to adapt these supplementary neurons in a satisfactory way.
polymer sensors. Then the sensor output first contains the fast dynamic properties of the sensors and the measuring cell, which varies on a scale of minutes, and secondly a slow parameter drift, which varies on a scale of hours or days. Both of these effects can be described if the training and the test data are chosen from a similar time period. Increasing the number of hidden and recurrent neurons decreases the evaluation error. 15 hidden neurons seem to be a good choice for the given evaluation problem. Evaluations become more difficult if the training data and the validation data are taken from different time periods. Probably parameter drift in this case reaches values that have not been presented in the training data. Increasing the size of training data is one way to reduce this effect. The second method consists of the observation of training by means of a second independent data set and stopping the training in an optimum configuration. In this case the increase of the size of the neural network does not lead to better performance. Further research will focus on parameter drifts. A prediction of this drift by means of neural networks will lead to a further increase of the output accuracy in dynamically calibrated sensor arrays.
References [1]
[2]
[3]
[4]
5. Conclusions
Different network topologies have been tested with regard to their possibility for evaluating dynamic sensor data. The dynamic response of the sensors includes information about the specific gas-polymer interaction. In large chambers the dynamic calibration method can shorten the calibration time, especially for sensors with more complicated concentration dependence than the
[5l
[6]
[7]
A. Hierlemann, U. Weimar, G. Kraus, M. Schweizer-Berberich and W. GOpel, Polymer based sensor arrays and multicomponent analysis for the detection of hazardous organic vapours in the environment, Sensors and Actuators B, 26-27 (1995) 126-134. S. Vaihinger, W. G6pel and J.R. Stetter. Detection of halogenated and other hydrocarbons in air: response functions of catalyst/electrochemical sensor systems, Sensors and Actuators B, 4 (1991) 337-343. J. Samitier, L.M. Lopez-Villegas, S. Marco, L. Camara, A. Pardo and O. Ruiz, A new method to analyse signal transients in chemical sensors, Sensors and Actuators B, 18-19 (1994) 308-312. J. G6ppert and W. Rosenstiel, Use of neural networks in online analysis, Fresenius' Z. Anal. Chem., 349 (1994) 367-371. S.E. Fahlmann. An empirical study of learning speed in backpropagation networks, Tech. Report CMU-SC-88-162, Carnegie-Mellon University, Pittsburgh, PA, USA, 1988. J. G6ppert and W. Rosenstiel, Self-organizing maps vs. backpropagation: an experimental study, Proc. Workshop on Design Methodologies for Microelectrons and Signal Processing, Silesian Technical University, Gliwice, Poland, 1993, pp. 153-162. J. G6ppert and W. Rosenstiel, Dynamic extensions of selforganizing maps, Proc. 1CANN "94, Sorrento, Italy, Springer, London, 1994, pp. 330-333.
Sensors and Actuators B 26-27 (1995) 232-236
CHEMICAL
Application of neural-network systems to the dynamic response of polymer-based sensor arrays Markus Schweizer-Berberich b Josef G6ppert a, Andreas Hierlemann b Jan Mitrovics Udo Weimar b, Wolfgang Rosenstiel a, Wolfgang G6pel b
b,
=Wilhelm-Schickard-Institute of Computer Science, University of Tiibingen, Auf der Morgenstelle 8, D-72076 Tiibingen, Germany b Institute of Physical and Theoretical Chemistry and Center of Interface Analysis and Sensors, University of Tiibingen, Auf der Morgenstelle 8, D-72076 Tiibingen, Germany
Abstract
The conventional calibration method for sensor arrays uses steady-state signals that depend on the gas concentration. This method can be time consuming if many concentrations and compositions of a multicomponent mixture are required. Good experimental design may reduce the necessary effort so that the number of calibration experiments is minimized. Dynamic measurements may significantly reduce the time of each calibration experiment. In the present approach a random walk through the domain of the gas concentrations is chosen with each step of the walk adjusted for a short time only. The sensor array consists of six polymer (polysiloxanes with functional groups)-coated bulk acoustic wave (BAW) devices. The concentration domain is defined by a binary mixture of n-octane and toluene (150 to 800 ppm). Neural networks evaluate both qualitative and quantitative information frorfi the sensor response. In particular, the extensions of feed-forward nets towards recurrent or time-delay structures can be used to solve problems related to dynamic evaluations (e.g., no steady-state signal, parameter drift). These network architectures with different numbers of hidden neurons are applied to evaluate the data from the BAW device array. The networks are trained with back-propagation-like training algorithms and are validated with arbitrary gas mixtures. Keywords: Gas sensors; Calibration of array sensors; Multicomponent analysis; Artificial neural networks
I. Introduction Chemical or biochemical sensors can be useful in different applications, such as environmental monitoring, process analysis and quality control. Only for some special applications it is possible to use a single sensor set-up, because the cross-sensitivities of the sensor to other p a r a m e t e r s are negligible. In most other cases it is necessary to use sensor arrays for calculating the p a r a m e t e r s from sensor signals that are not linearly dependent. For such an approach classical chemometric algorithms are applied, such as principal components regression (PCR), partial least squares (PLS) or artificial neural networks (ANNs). For these algorithms steadystate signals are usually used as input values. As reported elsewhere, polymer-coated quartz microbalance sensors show a very good long-term stability and reproducibility for a period longer than half a year [1] if the differences (Af=fo-fgas) of steady-state signals are used. Several 0925-4005/95/$09.50 © 1995 Elsevier Science S.A. All rights reserved SSD1 0925-4005(94)01592-6
problems can occur in an evaluation of steady-state signals. The time taken to reach 90% of the signal (/9o) depends on kinetic parameters (e.g., diffusion rate, reaction rate, volume of the chamber, flow rate). Analyte-handling procedures are required to produce the reference value for the zero gas concentration. Additional valves, filters or reference gas are disadvantages for a portable system. Therefore we tried a dynamic calibration procedure. Any application of a sensor system in the field of environmental monitoring is characterized by a fluctuating concentration. A dynamic calibration procedure should simulate this behaviour in a m o r e realistic way than steady-state signals. Dynamic sensor response may have a further advantage because it includes information about the sensor kinetics that is different for each gas-polymer interaction. It can be evaluated from the transient of the sensor response to a change in the partial pressure [2,31.
M. Schweizer-Berberich et aL / Sensors and Actuators B 26-27 (1995) 232-236
2. Neural net model
The development of a neural net for an evaluation task consists of two major parts: first, the definition of the network structure and the network size (number of hidden layers and neurons, interconnection structure) and secondly, the training of the net (finding the optimal interconnection weights between neurons). The network structure is fixed in advance, according to known properties of the evaluation problem. Generally, the sensor signals include different properties of the system: (1) the sensitivity of the sensors to the different gases; (2) the dynamic properties of the system; (3) disturbing effects, like sensor drift or measuring noise. The sensitivity of the sensors may be evaluated by a simple feed-forward neural net of either linear neurons or non-linear hidden neurons for the estimation of the non-linear part of the model [4]. The evaluation of the dynamic properties requires dynamic behaviour of the neural net. Drift effects may also be considered as a long-term dynamic process, and are also taken into account by recurrent neurons. Noise may be centred in the set of training data. We used structured neural nets, consisting of two layers of neurons (Fig. 1). Input units store the values of the input vector. A layer of hidden neurons with a sigmoid transfer function performs mainly the nonlinear and the recurrent part of the data processing. The output layer consists of linear neurons, which are connected with the hidden neurons and by short-cut connections directly with the input units. It is able to combine linear data processing of the linear output neurons with the non-linear functions of the hidden neurons. Dynamic properties are introduced in the neural nets by recurrent connections, which create internal recurrent variables, or by a suitable representation of the sensor history in the input vector. This is performed by combining previous output values of the sensors, like a floating temporal window, in the (a)
Octane
Q
input vector. Both methods are used and compared in this paper. The training phase is performed by a back-propagation derivative called 'Quickpropagation' [5], a model that converges for the given data much faster than a standard back-propagation technique. The interconnection weights are randomly initialized in the range [ - 1... + 1]. During the training, the training vectors are presented in temporal order (this is needed for recurrent nets) and weights are updated after each presentation of all training vectors. Other neural-net architectures like self-organizing maps [6], using dynamic functions to evaluate dynamic data [7], represent other promising approaches.
3. Experimental
An array of piezoelectric quartz crystals was used to detect a binary mixture of volatile organic compounds: n-octane with toluene. This mixture serves as a model for other more interesting applications, such as the detection of benzene among a high level of hydrocarbons as is found at gasoline stations. 10 MHz bulk acoustic wave (BAW) quartz crystals were used as transducers and were coated with sidechain-modified polysiloxanes: polydimethylsiloxane (PDMS), poly(cyanopropyl)methylsiloxane (PCPMS), polyphenylmethylsiloxane (PPMS), poly(isopropylcarboxylic-acid)methylsiloxane (PiPCMS), poly(aminopropyl)methylsiloxane (PAPMS, 10% amino groups), poly [2- carboxy (b-valin- t-butylamide) propyl] methylsiloxane (Chirasil-Val, 10% valin groups). Solutions of these polymers in dichloromethane or propan-2-ol were sprayed onto the crystals. The frequency decrease for determining the layer thickness was monitored on-line. For more details see Ref. [1]. The frequency was measured with a frequency counter (HP 5334 B) connected to an 286-AT computer via an IEEE bus. Each channel was scanned with a multiplexer controlled by a PCL 726 interface card (Labtech, Wilmington, USA). Each experiment was stored with an offset defined by the first data point of this run.
(b)
Toluene
~
Octane Toluene
outputlayer hiddenlayer inputunits
recurrentvariables
sensorsignals
233
sensor number
Fig. 1. (a) R e c u r r e n t a n d (b) t i m e - d e l a y n e u r a l n e t w o r k s as u s e d for the e v a l u a t i o n .
~|~ 12 nm=6 -4
sensor 1 (9
sensorm(t)
M. Schweizer-Berbetich et al. / Sensors and Actuators B 26-27 (1995) 232-236
234
The array sensors were mounted inside a 200 ml brass chamber. The brass block and the gas were thermostatted at 303+0.1 K as well as the analyte gas before it entered the chamber. The concentrations of both gases were adjusted between 150 and 800 ppm by mass-flow controllers. Each mixture was fixed for 2 min. Such an experiment was called a random walk (RW). The frequency output was recorded every minute with 0.1 Hz resolution. Four random walks were used to train and test different neural networks.
in predicting the second random walk data, the performance on the three other walks is poor. The output value, especially on the first and the third random walks, is globally too small or too large. The error is quite small after recalibration at the beginning of each random walk and becomes larger towards the end. This type of error is probably caused by parametric drift of the sensors. Obviously the drift of the sensors did not behave similarly in the different data sets. In order to explore this behaviour, other ways of choosing training and validation vectors will now be compared.
4. Results and discussion
4.2. Reduction of drift
The time constants of the sensors are of the order of some seconds ( < 5 s). The PiPCMS sensor shows slower response. The time to reach 90% of the end concentration in the chamber is about 6 rain. Therefore the dominant time scale results from changing the gases or gas concentrations in the chamber. To test the potential of the following neural network, the root mean square (RMS) errors, calculated in ppm, were used:
E (%,ea.--C~¢~,)2
Eg., =
(1)
V ~ ' vector
E.,, = ~/~EgZ,~
(2)
V gas
4.1. First training A simple linear neural net was trained with the second random walk. The other three walks were used for the validation of the net. The output of this net is visualized in Fig. 2. Although the net is fairly good
800
By using the same data set for training and for validation, it will first be examined if neural networks are able to reduce the drift if represented in the training data. For this purpose, the linear neural net without hidden layers was trained with the measured data of all four data sets. Validation was performed using the same data. So the task of the neural net is to predict the trained data as well as possible. A closer look at the output shows a good approximation for all measurements (Fig. 3). A comparison of Figs. 3 and 2 reveals that the drift can be learned if presented in the training vectors. The influence of the network size and the network architecture on the error (RMS) is shown in Table 1. As the neural net used is comparatively small, the net cannot learn the training data perfectly. This is, however, desired in order to achieve better generalization properties. Increasing the complexity of the network by introducing recurrent or hidden neurons leads to a better prediction of the data. It can be noticed that the time-delay neural network seems to be more suitable for the given problem. We can conclude that if the same data set is used for training and for testing, the
Octane
800
predicted - -
E600
Octane
true
predicted
---
._=
E e~
600
true
~3 400
400
¢-
0 0
0 200
'H
RW !' ~' ',jl<
'~
RW2
RW3
);.(
c ~
U
><
200
0 o
RW4>
"< .
.
.
.
.
.
~,v~. ~.;.,,L~ . . . . . . . . i
0
i
2000
4000
time
<
RW3
lo
t~
0o~
03
i
~0
6000
in m i n
Fig. 2. E s t i m a t i o n of the c o n c e n t r a t i o n of n - o c t a n e with a l i n e a r n e u r a l net. T h e s e c o n d r a n d o m w a l k ( R W 2 ) w a s used for the training. G o o d a g r e e m e n t was f o u n d in the t r a i n i n g set but drift in the o t h e r sets.
0
2000
4000
6000
time in min Fig. 3. C a l c u l a t i o n of the n - o c t a n e c o n c e n t r a t i o n with a single l a y e r e d n e u r a l net w i t h o u t h i d d e n and r e c u r r e n t n e u r o n s . All r a n d o m w a l k s w e r e u s e d e i t h e r for the t r a i n i n g or the v a l i d a t i o n .
M. Schweizer-Berberich et aL / Sensors and Actuators B 26-27 (1995) 232-236 Table 1 Prediction properties (expressed as RMS errors) with changing size of the networks. Identical data sets were used for training and validation Cornpound
Linear
En-octa.~ 25.2 Eto]. . . . 25.8 E., 36.0
Recurrent
Time delay
2units
5 units
0hidden
2 hidden
5 hidden
25.4 24.4 35.2
24.0 23.0 33.2
21.4 24.7 32.7
21.9 23.5 32.1
22.0 22.8 31.7
alternating blocks of 50 min for training and validation set
E 32 Ct.
r/?
validation set ~
n-
28
(5
5
~
1'0
® 7 lime
1'5
20
25
number of hidden neurons Fig. 4. Prediction error (RMS) of the data with a time-delay net. All data points were divided into blocks of 50 min. The blocks were used alternately for training and validation. The influence of the hidden neurons on the error is pointed out.
drift can be suppressed and the error decreases with a more sophisticated network.
4.3. Prediction of new data Now the data set will be divided into two parts; one will be used for training and the second for the validation of the obtained model. This was done taking alternately blocks of 50 min for training and the following 50 min for validation. Training and validation data are distinct, but cover the same period of time. This procedure can be compared to a method where a reference concentration is used to recalibrate the system. Fig. 4 confirms for the time-delay net that with the given way of choosing the training data, a more complex neuron net model performs better on the training and on the test data sets. Surprisingly the error on the training set and on the validation set are in a similar range even though the validation set is smaller. This may be a result of the choice of training and validation sets, but obviously the error on the training set decreases more strongly than on the validation set by increasing the number of hidden neurons. An examination of the dynamic properties shows that the neural net output is able to follow the fast changes of the gas concentration. So, neural networks are suitable for the evaluation of dynamically acquired data and can reduce the effects of parameter drifts.
235
4.4. Training of temporal decorrelated data In fact, for future use of these sensors it is not satisfactory to have training and test data of the same time period. If training and validation data are temporally decorrelated, it is possible to improve the prediction by training with data from a longer period. It is expected that different types of drift are represented in a larger set of training data, which leads to a better generalization. The influence of using a larger set (RW2 + RW3 instead of RW2) for the training is shown in Table 2. The error is reduced for all data. It is not surprising that the error of the third random walk is drastically reduced because it is used for training. Comparison of different training cycles shows that the convergence is more stable. The second idea to prevent overlearning (overfitting) is to observe the training by means of an independent test set and stop the training at a convenient time. In Fig. 5 the evolution of the RMS error during a typical training cycle is shown for both the training set and a test set. Three phases are found: (1) A reduction of error in the training set as well as in the test set. The system is approximating characteristics which are valid for the training set as well as for the test set. (2) Further training leads to a reduction of the error in the training set but also to an increase of the error Table 2 Influence of the size of training data on the RMS error Validation sets
Training sets
ERw, ERw2 ERw3 Eaw4 E.,
RW 2
RW 2 and RW 3
82.5 56.5 85.1 64.8 75.2
62.9 45.6 27.2 57.6 48.8
time delay net 300
E o_ .~ 200
/
fast decrease
validation set: 1. random walk
IF-"
ffl
i
very slow decrease
,,(" 84 ppm after
100
70 training cycles I
;!'
~
training set: 2. random walk
2do
,;0
6;0
8do
,obo
training cycles Fig. 5. Overlearning of neural nets can be prevented by the minimum of a test set.
236
M. Schweizer-Berberich
et al. / Sensors and Actuators B 26-27 (1995) 232-236
Table 3 Prediction properties (expressed as RMS errors) by using a stop criterion on the validation set. Influence of the topology and the number of recurrent and hidden neurons Cornpound
Linear
En-octa,~ 32.7 EtoJ. . . . 36.3 E,, 48.8
Recurrent
Time delay
2units
5units
0hidden
2hidden
5hidden
34.8 39.2 52.5
34.3 36.8 50.5
37.3 38.1 53.4
36.6 38.4 53.4
36.4 37.5 52.5
in the test set. The training is now starting to approximate specific properties of the training set. (3) Stabilization of the training near the minimum. Only a small reduction of the training error and small changes on the error of the test set. In order to have the best generalization properties, it is recommended to stop the training at the end of phase one. For all further neural nets this training principle is applied. The stop criterion is determined by means of the first random walk. For a larger training set phase one (error on the training and the test data decreases) is lengthened. The longer training leads to a better overall performance. The hypothesis that a larger training set enables more information about the nature of the parameter drift to be gathered is confirmed. Neural nets trained in such a way are also fitter for the evaluation of data acquired before or after the training period. Table 3 shows the influence of the number of recurrent and hidden neurons on the RMS error. Contrary to the training with all random walks, these units do not seem to enable a better performance. The short training time is responsible for this due to the need for stopping the training at a very early stage. The time of about 100 training epochs is not sufficient to adapt these supplementary neurons in a satisfactory way.
polymer sensors. Then the sensor output first contains the fast dynamic properties of the sensors and the measuring cell, which varies on a scale of minutes, and secondly a slow parameter drift, which varies on a scale of hours or days. Both of these effects can be described if the training and the test data are chosen from a similar time period. Increasing the number of hidden and recurrent neurons decreases the evaluation error. 15 hidden neurons seem to be a good choice for the given evaluation problem. Evaluations become more difficult if the training data and the validation data are taken from different time periods. Probably parameter drift in this case reaches values that have not been presented in the training data. Increasing the size of training data is one way to reduce this effect. The second method consists of the observation of training by means of a second independent data set and stopping the training in an optimum configuration. In this case the increase of the size of the neural network does not lead to better performance. Further research will focus on parameter drifts. A prediction of this drift by means of neural networks will lead to a further increase of the output accuracy in dynamically calibrated sensor arrays.
References [1]
[2]
[3]
[4]
5. Conclusions
Different network topologies have been tested with regard to their possibility for evaluating dynamic sensor data. The dynamic response of the sensors includes information about the specific gas-polymer interaction. In large chambers the dynamic calibration method can shorten the calibration time, especially for sensors with more complicated concentration dependence than the
[5l
[6]
[7]
A. Hierlemann, U. Weimar, G. Kraus, M. Schweizer-Berberich and W. GOpel, Polymer based sensor arrays and multicomponent analysis for the detection of hazardous organic vapours in the environment, Sensors and Actuators B, 26-27 (1995) 126-134. S. Vaihinger, W. G6pel and J.R. Stetter. Detection of halogenated and other hydrocarbons in air: response functions of catalyst/electrochemical sensor systems, Sensors and Actuators B, 4 (1991) 337-343. J. Samitier, L.M. Lopez-Villegas, S. Marco, L. Camara, A. Pardo and O. Ruiz, A new method to analyse signal transients in chemical sensors, Sensors and Actuators B, 18-19 (1994) 308-312. J. G6ppert and W. Rosenstiel, Use of neural networks in online analysis, Fresenius' Z. Anal. Chem., 349 (1994) 367-371. S.E. Fahlmann. An empirical study of learning speed in backpropagation networks, Tech. Report CMU-SC-88-162, Carnegie-Mellon University, Pittsburgh, PA, USA, 1988. J. G6ppert and W. Rosenstiel, Self-organizing maps vs. backpropagation: an experimental study, Proc. Workshop on Design Methodologies for Microelectrons and Signal Processing, Silesian Technical University, Gliwice, Poland, 1993, pp. 153-162. J. G6ppert and W. Rosenstiel, Dynamic extensions of selforganizing maps, Proc. 1CANN "94, Sorrento, Italy, Springer, London, 1994, pp. 330-333.