- Email: [email protected]
Two-level independent component regression model for multivariate spectroscopic calibration
Chemometrics and Intelligent Laboratory Systems 155 (2016) 160–169
Contents lists available at ScienceDirect
Chemometrics and Intelligent Laboratory Systems journal homepage: www.elsevier.com/locate/chemolab
Two-level independent component regression model for multivariate spectroscopic calibration Junhua Zheng, Zhihuan Song ⁎ State Key Laboratory of Industrial Control Technology, Institute of Industrial Process Control, Department of Control Science and Engineering, Zhejiang University, Hangzhou, 310027, China
a r t i c l e
i n f o
Article history: Received 21 August 2015 Received in revised form 2 April 2016 Accepted 5 April 2016 Available online 12 April 2016 Keywords: Multivariate calibration Independent component regression Two-level modeling Ensemble learning Bayesian inference
a b s t r a c t In this paper, a two-level independent component regression (ICR) model is developed for multivariate spectroscopic calibration. Compared to the traditionally used principal component regression and partial least squared regression model, the ICR model is more efficient to extract high order statistical information from the spectra data. To improve the calibration performance, an ensemble form of the ICR model is proposed. In the first level of the method, various subspaces are constructed based on the independent component decomposition of the original data space. Meanwhile, by defining a related index, the most important variables in each subspace are selected for ICR modeling, which form the second level of the proposed method. A Bayesian inference strategy is further developed for probabilistic combination of calibration results obtained from different subspaces. For performance evaluation, two case studies are carried out on a benchmark spectra dataset. © 2016 Elsevier B.V. All rights reserved.
1. Introduction In the past several decades, spectroscopic calibration modeling has become an effective tool for fast and non-invasive analysis in chemistry/biochemistry related areas, such as petrochemical and food industries and pharmaceutical and biological sectors [1–5]. In order to guarantee a high calibration performance for the spectroscopic device, various chemometrics methods have been incorporated. Commonly used ones include principal component regression (PCR), partial least squares (PLS), artificial neural networks (ANN), support vector regression (SVR), Gaussian process regression (GPR), etc. [6–20]. Among all those developed chemometrics modeling methods, the linear calibration model PCR and PLS may be two of the most widely used and accepted ones, which are efficient to provide fast and linear relationship analyses between the spectra and properties of the products. Though successful studies have demonstrated the efficiency of PCR and PLS based calibration methods, as Gustafsson pointed out, neither of these two methods can generally recover a true underlying linear latent model from the data [21]. In addition, PCR/PLS can only extract the first and second order statistics from the data, which means higher order statistical information has been ignored. For non-Gaussian data, high order statistics are necessary for information extraction and interpretation. As an emergent data analysis technique in recent years, independent component analysis (ICA) aims to decompose the original signals into different directions, which are independent to each other [22]. The extracted component by the ICA model is assumed to be ⁎ Corresponding author. E-mail addresses: [email protected] (J. Zheng), [email protected] (Z. Song).
http://dx.doi.org/10.1016/j.chemolab.2016.04.002 0169-7439/© 2016 Elsevier B.V. All rights reserved.
mutually independent instead of merely uncorrelated. Through probability interpretation, independence is a much stronger condition than uncorrelatedness, which can make use of higher order statistical information. Compared to PCR/PLS, it has been demonstrated that the ICA regression method (ICR) can recover the true underlying sources much better, depending on which an improved statistical interpretation of the data can be obtained. Recent works on ICA or ICA regression (ICR) have been done for blind source signal separation, image processing, process monitoring, spectra data analysis and quality prediction [23–33]. However, most ICR model based spectroscopic calibration works have been carried out on constructing a single model, no matter how complicated the spectra data performed. This may cause unsatisfactory performance, especially when the number of training data samples is relatively small, compared to the number of spectra data variables. Inspired by the idea of ensemble learning from the area of machine learning, the calibration performance could probably be improved through constructing multiple regression models for the same purpose. Typical ensemble learning strategies include bagging, random subspace, random forest, etc. [34–40]. In this paper, the random subspace method is employed and combined with ICR for model calibration purpose. The main idea of the random subspace method is to build various individual models based on different variable subsets which are randomly selected from the original variables space. Then, the final calibration result is obtained by combining the results of different individual models. However, a critical shortcoming of this method is that the diversity among different individual models cannot be well guaranteed through a random selection manner, which is a quite important issue in the ensemble learning method.
J. Zheng, Z. Song / Chemometrics and Intelligent Laboratory Systems 155 (2016) 160–169
To improve the calibration performance of the random subspace ICR model, a two-level ICR model is developed in this paper. In the first level of the model, an ICA model is constructed on the original spectra data. Based on this model, various modeling directions can be determined, which are independent with each other. The diversity property of the random subspace method can be greatly improved if we build individual models along those independent directions. For construction of those subspaces, a related index is defined for variable selection in each subspace, which is based on the absolute values of the separating matrix in the ICA model. In the second level, an ICR model can be developed for each subspace. For online calibration purpose, an important issue is how to combine the results from different individual models. While one can resort to a simple average combination strategy, a more effective Bayesian based probabilistic combination strategy is proposed for results combination in this paper. The rest of this paper is organized as follows. In Section 2, the basic ICR model is briefly introduced. Detailed demonstration of the twolevel ICR model is provided in Section 3, followed by illustrations of two benchmark spectra data examples in the Section 4. Finally, conclusions are made. 2. Independent component regression (ICR) Based on the ICA modeling method, the ICR model can be built between the extracted independent components and quality variables. In the ICA algorithm, it is assumed that the measured process variables x ∈ Rm× 1 can be expressed as linear combinations of r(≤m) unknown independent components s ∈ Rr × 1, the relationship between them is given by [22] x ¼ As þ e
ð1Þ
where A ∈ Rm × r is the mixing matrix, e ∈ Rm × 1 is the residual vector. The basic problem of ICA is to estimate the original component s and the mixing matrix A from x. Therefore, the objective of ICA is to calculate a separating matrix W so that the components of the reconstructed data matrix ŝ become as independent of each other as possible, given as ŝ ¼ Wx:
ð2Þ
After the independent components have been estimated from the process data, the linear regression can be carried out between two datasets: the independent component dataset Ŝ ¼ ½ŝ1 ; ŝ2 ; ⋯; ŝn T ∈ Rnr and the quality variable dataset Y = [y1, y2, ⋯ , yn]T ∈ Rn × p. Therefore, the linear regression matrix can be calculated as T −1 T Q ¼ Ŝ Ŝ Ŝ Y:
ð3Þ
If we denote the dataset of process variables as X = [x1, x2, ⋯ , xn]T ∈ Rn × m, and combine the two steps of ICR modeling procedures, the ICR regression matrix can be determined as RICR ¼ Q T W:
non-Gaussianity measurement, etc. Based on the independent behavior of the extracted components, a subspace can be defined through each independent component direction, which are orthogonal to each other. Therefore, to build the ICR model in each subspace, the importance of each variable in different subspaces should be measured, depending on which the most informational ones should be retained in their corresponding subspace. To this end, an independent component related index (RI) is defined as follows RIði; jÞ ¼
w ij jwi1 j þ ⋯ þ wij þ ⋯ þ jwim j
ð6Þ
where i = 1 , 2 , ⋯ , k , j = 1 , 2 , ⋯ , m, wij is the j-th element of the i-th independent component direction in the separating matrix W. Therefore, the larger the value of the j-th element, the more significant contribution it has provided through the i-th independent component direction. Based on this defined index, the importance values of different process variables through each independent component direction can be measured and ranked from the most important one to the least important one. An appropriate number of variables can be selected to form each subspace, depending on the selection scheme. The ICR model-based subspaces can be represented as follows 8 X ¼ XðS1 Þ→subspace #1 > > < 1 X2 ¼ XðS2 Þ→subspace #2 X→ ⋮ > > : Xk ¼ XðSk Þ→subspace #k
ð7Þ
where Si , i = 1 , 2 , ⋯ , k is the column vector, which related to each subspace along the corresponding IC direction. A diagram of the proposed subspace modeling approach is given in Fig. 1. 3.1. ICR modeling in each subspace Suppose the whole variable set has been divided into k subspaces, and mb variables have been selected in each subspace, where b = 1 , 2 , ⋯ , k, the corresponding dataset for each subspace can be represented as {Xb}b = 1 , 2 , ⋯ , k. Denote the quality variable dataset as Y ∈ Rn×p, the subspace ICR model can be constructed as follows Xb ¼ Ab Sb þ Eb
ð8Þ
−1 STb Yb : Q b ¼ STb Sb
ð9Þ
Therefore, the regression form of the subspace ICR model can be developed as Yb ¼ Q Tb Ŝb ¼ Q Tb Wb Xb ¼ RICR;b Xb :
ð10Þ
3.2. Online calibration based on two-level ICR model ð4Þ
3. Two-level ICR for multivariate calibration Denote the whole variable dataset as X ∈ Rn× m, where m is the number of process variables, and n is the sample number for each variable. An initial ICA decomposition can be carried out on X, thus [22] X ¼ AS þ E
161
ð5Þ
where A is the mixing matrix of the ICA model, S is the data matrix of the independent components, E is the residual matrix. The number of independent components k can be determined by the negentropy method,
Based on the developed subspace ICR models, the property value of the new spectra data can be calculated by combing the results obtained in different subspaces. Therefore, when the new data sample xnew ∈ Rm is available, each subspace ICR prediction result is calculated in the first step, given as T b ŷnew ¼ RICR;b xnew ðSIb Þ
ð11Þ
where b = 1 , 2 , ⋯ , k, RICR , b the regression matrix of the b-th subspace ICR model, SIb represents the variable index of each subspace in the original variable space. When all of the subspace prediction results have been generated, the next step is to combine them in a certain
162
J. Zheng, Z. Song / Chemometrics and Intelligent Laboratory Systems 155 (2016) 160–169
Fig. 1. Diagram of two-level subspace ICR modeling method.
way. Based on the average rule, the final prediction of the property value is calculated as follows ŷnew ¼
k k h i T 1X 1X b ŷnew ¼ R ICR;b xnew ðSIb Þ : k b¼1 k b¼1
ð12Þ
probability of the new data sample in each subspace can be calculated by Bayesian inference, which are given as follows P I2 ðbjxnew Þ ¼
P I2 ðb; xnew Þ P I2 ðxnew jbÞP ðbÞ ¼ k P I2 ðxnew Þ X
P I2 ðxnew jbÞP ðbÞ
ð17Þ
b¼1
However, a more appropriate way is to use the weighed combination form. Here, a probabilistic method is proposed, which is based on ICA monitoring statistics and Bayesian posterior probability inference. In the ICA-based monitoring method, two monitoring statistics are usually constructed: the I2 index for measurement of normality distance in the independent component plane and the squared prediction error index SPE for consistency measurement of the model. For the new data sample xnew, the value of I2 and SPE monitoring statistics through different subspace ICR model can be calculated as follows [25] T
I2new;b ¼ ŝnew;b ŝnew;b
ŝnew;b ¼ Wb xnew ;
enew;b ¼ xnew −Ab ŝnew;b ;
SPEnew;b ¼ eTnew;b enew;b
ð13Þ ð14Þ
where b = 1 , 2 , ⋯ , k. Before the Bayesian inference of the posterior probability for the new data sample, the conditional probability of the data sample in different subspaces should be calculated. This can be made by transferring the monitoring statistical value to the probability distribution value, which are given as follows ( P I2 ðxnew jbÞ ¼ exp −
I2new;b
)
I 2lim;b
SPEnew;b P SPE ðxnew jbÞ ¼ exp − SPE lim;b
P SPE ðbjxnew Þ ¼
P SPE ðb; xnew Þ P SPE ðxnew jbÞP ðbÞ ¼ k P SPE ðxnew Þ X ½P SPE ðxnew jbÞP ðbÞ b¼1
where P(b) are prior probabilities, which can be assumed as equal values for different subspaces, thus P(b) = 1/k , b = 1 , 2 , ⋯ , k. When the posterior probabilities of the new data sample in different subspaces have been determined, we are in the position to combine local prediction results in different subspaces to form the final result. Suppose the prediction results in different subspaces have been obtained through Eq. (11), the final prediction result can be calculated as ŷnew ¼
k X
b
P ðbjxnew Þŷnew ¼
b¼1
k X
T P ðbjxnew Þ RICR;b xnew ðSIb Þ
ð16Þ
where I2lim ,b and SPElim,b are confidence limits of the I2 and SPE monitoring statistics in different subspaces, which can be determined through the kernel density estimation method [25]. Then the posterior
ð19Þ
b¼1
where the posterior probability P(b | x new ) is a weighted value of PI2(b | xnew) and PSPE(b | xnew), thus P ðbjxnew Þ ¼ αP I2 ðbjxnew Þ þ βP SPE ðbjxnew Þ
ð15Þ
ð18Þ
ð20Þ
where α and β are weighted parameters, and α + β = 1. For simplicity, the values of α and β can be selected as 0.5. 4. Experimental results In this section, two commonly used benchmark NIR spectra datasets are employed for performance evaluations of the two-level ICR modeling approach, which are related to the transmittance spectra of wheat
J. Zheng, Z. Song / Chemometrics and Intelligent Laboratory Systems 155 (2016) 160–169
Fig. 2. Wavelengths and the protein contents of the wheat kernels.
Fig. 3. Selected variables in the first four subspaces.
163
164
J. Zheng, Z. Song / Chemometrics and Intelligent Laboratory Systems 155 (2016) 160–169
model and the random subspace based ICR model have both been developed.
Table 1 RMSE values of different methods for model calibration. Methods
ICR
Random subspace ICR
Two-level ICR
RMSE values
1.6992
1.4368
1.0575
4.1. Wheat kernels data
kernels and corn. To evaluate the performance of the two-level ICR model, the root mean square error (RMSE) criterion can be used, which is defined as follow
RMSE ¼
vffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi u n te uX u y −ŷ 2 j j u t j¼1 n te
ð21Þ
where j = 1 , 2 , ⋯ , n_te, yj and ŷ j are real and predicted values, respectively, n_te is the total number of test data samples. Similarly, the RMSE value for each subspace ICR can be defined as follows
RMSEb ¼
vffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi u n te uX u y j;b −ŷ j;b 2 u t j¼1 n te
ð22Þ
where b = 1 , 2 , ⋯ , k, yj , b and ŷ j;b are real and predicted values by the b-th subspace ICR model. For comparison, the conventional ICR
In this dataset, a total of 523 data samples were collected from three different locations of the process. Each of the data samples was analyzed at 100 wavelengths in the range of 850–1050 nm, in order to predict the protein content of wheat kernels. This dataset is public available for research study, one can refer to the following website http://www. models.life.ku.dk/research/data/wheat_kernels/. Both of the 100 wavelengths and the protein content of the wheat kernels are shown in Fig. 2. For calibration modeling and online prediction, the whole dataset has been divided into two parts: the training dataset which contains 415 data samples, and the testing dataset consisting of the rest 108 data samples. To develop the two-level ICR model, an initial ICA model decomposition should be carried out in the first step, in which 10 independent components have been selected. As a result, a total of 10 subspaces can be formulated through different independent component directions. Then, based on the defined independent component related index, the most related variables in each subspace can be determined. Here, the number of related variables in each subspace has been selected as 30. After all of the 10 subspaces have been constructed, an ICR model can be developed for each subspace. Detailed information of the selected wavelength variables in the first four subspaces are shown in Fig. 3. It can be found that different ranges
Fig. 4. Calibration results of two-level ICR model and traditional ICR model.
Fig. 5. RMSE values obtained in different subspaces.
J. Zheng, Z. Song / Chemometrics and Intelligent Laboratory Systems 155 (2016) 160–169
165
Fig. 6. Posterior probability values of the testing samples under the first four subspaces.
of wavelength have been specified in different subspaces, which may provide quite different calibration results for the property value. The RMSE value of each method for the testing dataset is tabulated in Table 1, which are the mean values of 100 running times. It can be seen from this table that the two-level ICR model has obtained the best prediction results for the testing dataset. With the incorporation of the random subspace modeling approach, the prediction accuracy can be slightly improved, compared to the single ICR model. According to those results, it can be inferred that the construction of subspaces is important to the subspace-based method. In some particularly case, a
random subspace construction method may not improve the calibration performance. [Fig. 4 shows calibration results of both two-level ICR and conventional ICR methods. Obviously, a better calibration result can be observed by the two-level ICR method. This is in consistence with the RMSE values of the two methods provided in Table 1. To examine the calibration performance in each subspace, the RMSE values obtained through the 10 subspace ICR models are shown together in Fig. 5, in which the black line represents the final RMSE value obtained by the two-level ICR model. It can be seen that the RMSE value obtained
Fig. 7. Two monitoring statistics of the testing samples under the first four subspaces.
166
J. Zheng, Z. Song / Chemometrics and Intelligent Laboratory Systems 155 (2016) 160–169
Fig. 8. RMSE values under different numbers of selected variables.
through the Bayesian inference strategy is smaller than all of the RMSE values in single subspaces. This is because the advantages of all subspace models in specific ranges have been combined together and complemented for each other. The posterior probability values of testing data samples related to different subspaces are shown in Fig. 6. Correspondingly, the values of the two statistics I2 and SPE in the same subspaces are given in Fig. 7. In the two-level ICR subspace modeling method, the number of variables in each subspace has been initially selected as 30. However, how to determine this parameter still needs investigation. In order to
examine the effect of this parameter on the calibration performance of the two-level subspace ICR model, different numbers of variables have been selected for evaluation. Fig. 8 shows the RMSE values of the twolevel ICR model under different numbers of selected variables for subspace construction. It can be seen from this figure that the best result has been obtained when the number of selected variables has been set as 30. Actually, the number of selected variables in each subspace may have high impacts on the accuracy of the data model. If only a small number of variables are selected in each subspace, the combination of all subspaces may not well cover the whole variable space. On the
Fig. 9. Data characteristics of spectra data and corresponding moisture values.
J. Zheng, Z. Song / Chemometrics and Intelligent Laboratory Systems 155 (2016) 160–169 Table 2 RMSE values of different methods for model calibration. Methods
ICR
Random subspace ICR
Two-level ICR
RMSE values
0.3768
0.3511
0.3127
other hand, if we set a large number of variables for each subspace, it will result in a severe overlap among different subspaces, which may deteriorate the complement effect among each other. Maybe a more appropriate way for variable selection in each subspace is to set different variable numbers for different subspaces. 4.2. Corn data This dataset consists of spectra from 80 samples of corn with wavelength ranging from 1100 nm to 2498 nm at 2 nm intervals (700 channels). As a result, a spectra data matrix of 80× 700 has been collected for study. The corresponding concentration values of the moisture for those collected spectra data have also been measured in lab, which form a response matrix of 80× 1. Both of the spectra and concentration data are shown in Fig. 9. Those data are available at the Eigenvector Research homepage: http://software.eigenvector.com/Data/Corn/. For model training and testing purposes, the whole dataset has been divided into two different parts: the training dataset contains 50 samples, and the testing dataset contains the rest 30 samples.
167
To develop the two-level ICR model, an initial ICA model decomposition is carried out in the first step, in which 10 independent components have also been selected. As a result, a total of 10 subspaces can be formulated through different independent component directions. Then, based on the defined independent component related index, the most related variables in each subspace can be determined. Here, the number of related variables in each subspace has also been selected as 30. After all of the 10 subspaces have been constructed, an ICR model can be developed for each subspace. Similarly, both of the conventional ICR model and the random subspace ICR model have been constructed for performance comparisons. The RMSE values of the three different methods for the testing dataset are tabulated in Table 2, which are the mean values of 100 running times. Again, the two-level ICR model has obtained the best prediction results for the testing dataset in this example. Detailed calibration errors of the 30 testing data samples by the two-level ICR model and the conventional ICR model are provided in Fig. 10, in which it can be seen that the prediction quality has been improved by the new method. Similarly, the RMSE values obtained through the 10 subspace ICR models are shown together in Fig. 11, in which the black line represents the final RMSE value obtained by the two-level ICR model. Again, it can be seen that the RMSE value obtained through the Bayesian inference strategy is smaller than all of the RMSE values in single subspaces. The posterior probability values of testing data samples related to the first four subspaces are shown in Fig. 12.
Fig. 10. Prediction errors of the two methods for testing data samples.
Fig. 11. RMSE values obtained in different subspaces.
168
J. Zheng, Z. Song / Chemometrics and Intelligent Laboratory Systems 155 (2016) 160–169
Fig. 12. Posterior probability values of the testing samples under the first four subspaces.
5. Conclusions In the present work, a two-level subspace ICR model based calibration model has been developed for multivariate spectral data. In order to improve the calibration performance of the random subspace calibration model, the individual subspace is constructed through each independent direction obtained through the ICA model. For online calibration purpose, a Bayesian inference strategy has been developed for probabilistic combination of calibration results obtained from different subspaces. According to the experimental results, the calibration performance has been improved by the proposed method, compared to both of the random subspace ICR model and the basic ICR model. Therefore, it can be inferred that the ensemble model is useful for calibration modeling and how to construct subspaces does have impacts on the ensemble performance of the model. It should be noted that although the experimental case studies have been made on the NIR datasets, applications to other spectra dataset such as IR and Raman are also possible. Due to the similar nature among those spectra data types, the proposed two-level model structure should be effective as long as the subspace calibration models can be successfully constructed. Acknowledgments This work was supported in part by the National Natural Science Foundation of China (NSFC) (61370029). References [1] J. Workman Jr., B. Lavine, R. Chrisman, M. Koch, Proc. Anal. Chem. Analy. Chem. 83 (2011) 4557–4578. [2] R.J. Pell, M.B. Seasholtz, K.R. Beebe, M.V. Koch, Process analytical chemistry and chemometrics, J. Chemometrics 28 (2014) 321–331. [3] B.K. Lavine, J. Workman Jr., Chemometrics, Anal. Chem. 85 (2012) 705–714. [4] R.G. Brereton, A short histoy of chemometrics: a personal view, J. Chemometrics 28 (2014) 749–760. [5] R.G. Brereton, Pattern recognition in chemometrics, Chemom. Intell. Lab. Syst. 149 (2015) 90–96. [6] J.A. Westerhuis, S.P. Gurden, A.K. Smilde, Spectroscopic monitoring of batch reactions for on-line fault detection and diagnosis, Anal. Chem. 72 (2000) 5322–5330. [7] M.M. Reis, P.H.H. Araujo, C. Sayer, R. Giudici, Spectroscopic on-line monitoring of reactions in dispersed medium: chemometric challenges, Anal. Chim. Acta 595 (2007) 257–265.
[8] T. Chen, E. Martin, Bayesian linear regression and variable selection for spectroscopic calibration, Anal. Chim. Acta 631 (2009) 13–21. [9] R. Ergon, Reduced PCA/PLSR models by subspace projections, Chemom. Intell. Lab. Syst. 81 (2006) 68–73. [10] J. Luts, F. Ojeda, R. Van de Plas, B. De Moor, S. Van Huffel, J.A.K. Suykens, A tutorial on support vector machine-based methods for classification problems in chemometrics, Anal. Chim. Acta 665 (2010) 129–145. [11] X.G. Shao, X.H. Bian, W.S. Cai, An improved boosting partial least squares method for near-infrared spectroscopic quantitative analysis, Anal. Chim. Acta 666 (2010) 32–37. [12] A. Martins, L.A. Daffner, A. Fenech, C. McGlinchey, M. Strlic, Non-destructive dating of fiber-based gelatin silver prints using near-infrared spectroscopy and multivariate analysis, Anal. Bioanal. Chem. 402 (2012) 1459–1469. [13] V.R. Nadadoor, H.D. Siegler, S.L. Shah, W.C. McCaffrey, A. Ben-Zvi, Online sensor for monitoring a microalgal bioreactor system using support vector regression, Chemom. Intell. Lab. Syst. 110 (2012) 38–48. [14] M.F. Abdelkader, J.B. Cooper, C.M. Larkin, Calibration transfer of partial least squares jet fuel property models using a segmented virtual standards slope-bias correction method, Chemom. Intell. Lab. Syst. 110 (2012) 64–73. [15] D. Lee, W. Lee, Y. Lee, Y. Pawitan, Sparse partial least-squares regression and its applications to high-throughput data analysis, Chemom. Intell. Lab. Syst. 109 (2011) 1–8. [16] F. Vogt, F. Gritti, G. Guiochon, Polynomial multivariate least-squares regression for modeling nonlinear data applied to in-depth characterization of chromatographic resolution, J. Chemometrics 25 (2011) 575–585. [17] P. Shahbazikhah, J.H. Kalivas, A consensus modeling approach to update a spectroscopic calibration, Chemom. Intell. Lab. Syst. 120 (2013) 142–153. [18] P.S. Gromski, Y. Xu, E. Correa, D. Ellis, M.L. Turner, R. Goodacre, A comparative investigation of modern feature selection and classification approaches for the analysis of mass spectrometry data, Anal. Chim. Acta 829 (2014) 1–8. [19] M. Lopatka, G. Vivo-Truyols, M.J. Sjerps, Probabilistic peak detection for first-order chromatographic data, Anal. Chim. Acta 817 (2014) 9–16. [20] M. Chen, S. Khare, B. Huang, A unified recursive just-in-time approach with industrial near infrared spectroscopy, Chemom. Intell. Lab. Syst. 135 (2014) 133–140. [21] M.G. Gustafsson, Independent component analysis yields chemically interpretable latent variables in multivariate regression, J. Chem. Inform. model 45 (2005) 1244–1255. [22] A. Hyvarinen, E. Oja, Independent component analysis: algorithms and applications, Neural Netw. 13 (2000) 411–430. [23] R.F. Li, X.Z. Wang, Dimension reduction of process dynamic trends using independent component analysis, Comput. Chem. Eng. 26 (2002) 467–473. [24] M. Kano, S. Tanaka, S. Hasebe, I. Hashimoto, H. Ohno, Monitoring independent components for fault detection, AICHE J. 49 (2003) 969–976. [25] J.M. Lee, C.K. Yoo, I.B. Lee, Statistical process monitoring with independent component analysis, J. Process Control 14 (2004) 467–485. [26] X. Liu, L. Xie, U. Kruger, T. Littler, S. Wang, Statistical-based monitoring of multivariate non-Gaussian systems, AICHE J. 54 (2008) 2379–2391. [27] J. Chen, X.Z. Wang, A new approach to near-infrared spectral data analysis using independent component analysis, J. Chem. Inf. Comput. Sci. 41 (2001) 992–1001. [28] F. Westad, Independent component analysis and regression applied on sensory data, J. Chemometrics 19 (2005) 171–179.
J. Zheng, Z. Song / Chemometrics and Intelligent Laboratory Systems 155 (2016) 160–169 [29] H. Ahn, E. Choi, I. Han, Extracting underlying meaningful features and canceling noise using independent component analysis for direct marketing, Expert Syst. Appl. 33 (2007) 181–191. [30] H. Kaneko, M. Arakawa, K. Funatsu, Development of a new regression analysis method using independent component analysis, J. Chem. Inf. Model. 48 (2008) 534–541. [31] L.J. Kao, C.C. Chiu, C.J. Lu, J.L. Yang, Integration of nonlinear independent component analysis and support vector regression for stock price forecasting, Neurocomputing 99 (2013) 534–542. [32] O. Forni, S. Maurice, O. Gasnault, R.C. Wiens, A. Cousin, S.M. Clegg, J.B. Sirven, J. Lasue, Independent component analysis classification of laser induced breakdown spectroscopy spectra, Spectrochim. Acta B At. Spectrosc. 86 (2013) 31–41. [33] Y.K. Chuang, Y.P. Hu, I.C. Yang, S.R. Delwiche, Y.M. Lo, C.Y. Tsai, S. Chen, Integration of independent component analysis with near infrared spectroscopy for evaluation of rice freshness, J. Cereal Sci. 60 (2014) 238–242.
169
[34] L. Breiman, Bagging predictors, Mach. Learn. 24 (1996) 123–140. [35] T.K. Ho, The random subspace method for constructing decision forests, IEEE Trans. Pattern Anal. Mach. Intell. 20 (1998) 832–844. [36] D.C. Tao, X.O. Tang, X.L. Li, X.D. Wu, Asymmetric bagging and random subspace for support vector machines-based relevance feedback in image retrieval, IEEE Trans. Pattern Anal. Mach. Intell. 28 (2006) 1088–1099. [37] D.S. Cao, Q.S. Xu, Y.Z. Liang, The boosting: a new idea of building models, Chemom. Intell. Lab. Syst. 100 (2010) 1–11. [38] Z.Q. Ge, Z.H. Song, Bagging support vector data description model for batch process monitoring, J. Process Control 23 (2013) 1090–1096. [39] K. Wang, T. Chen, R. Lau, Bagging for robust non-linear multivariate calibration of spectroscopy, Chemom. Intell. Lab. Syst. 105 (2011) 1–6. [40] G. Seni, Ensemble methods in data mining, Morgan and Claypool Publishers, 2010.
Contents lists available at ScienceDirect
Chemometrics and Intelligent Laboratory Systems journal homepage: www.elsevier.com/locate/chemolab
Two-level independent component regression model for multivariate spectroscopic calibration Junhua Zheng, Zhihuan Song ⁎ State Key Laboratory of Industrial Control Technology, Institute of Industrial Process Control, Department of Control Science and Engineering, Zhejiang University, Hangzhou, 310027, China
a r t i c l e
i n f o
Article history: Received 21 August 2015 Received in revised form 2 April 2016 Accepted 5 April 2016 Available online 12 April 2016 Keywords: Multivariate calibration Independent component regression Two-level modeling Ensemble learning Bayesian inference
a b s t r a c t In this paper, a two-level independent component regression (ICR) model is developed for multivariate spectroscopic calibration. Compared to the traditionally used principal component regression and partial least squared regression model, the ICR model is more efficient to extract high order statistical information from the spectra data. To improve the calibration performance, an ensemble form of the ICR model is proposed. In the first level of the method, various subspaces are constructed based on the independent component decomposition of the original data space. Meanwhile, by defining a related index, the most important variables in each subspace are selected for ICR modeling, which form the second level of the proposed method. A Bayesian inference strategy is further developed for probabilistic combination of calibration results obtained from different subspaces. For performance evaluation, two case studies are carried out on a benchmark spectra dataset. © 2016 Elsevier B.V. All rights reserved.
1. Introduction In the past several decades, spectroscopic calibration modeling has become an effective tool for fast and non-invasive analysis in chemistry/biochemistry related areas, such as petrochemical and food industries and pharmaceutical and biological sectors [1–5]. In order to guarantee a high calibration performance for the spectroscopic device, various chemometrics methods have been incorporated. Commonly used ones include principal component regression (PCR), partial least squares (PLS), artificial neural networks (ANN), support vector regression (SVR), Gaussian process regression (GPR), etc. [6–20]. Among all those developed chemometrics modeling methods, the linear calibration model PCR and PLS may be two of the most widely used and accepted ones, which are efficient to provide fast and linear relationship analyses between the spectra and properties of the products. Though successful studies have demonstrated the efficiency of PCR and PLS based calibration methods, as Gustafsson pointed out, neither of these two methods can generally recover a true underlying linear latent model from the data [21]. In addition, PCR/PLS can only extract the first and second order statistics from the data, which means higher order statistical information has been ignored. For non-Gaussian data, high order statistics are necessary for information extraction and interpretation. As an emergent data analysis technique in recent years, independent component analysis (ICA) aims to decompose the original signals into different directions, which are independent to each other [22]. The extracted component by the ICA model is assumed to be ⁎ Corresponding author. E-mail addresses: [email protected] (J. Zheng), [email protected] (Z. Song).
http://dx.doi.org/10.1016/j.chemolab.2016.04.002 0169-7439/© 2016 Elsevier B.V. All rights reserved.
mutually independent instead of merely uncorrelated. Through probability interpretation, independence is a much stronger condition than uncorrelatedness, which can make use of higher order statistical information. Compared to PCR/PLS, it has been demonstrated that the ICA regression method (ICR) can recover the true underlying sources much better, depending on which an improved statistical interpretation of the data can be obtained. Recent works on ICA or ICA regression (ICR) have been done for blind source signal separation, image processing, process monitoring, spectra data analysis and quality prediction [23–33]. However, most ICR model based spectroscopic calibration works have been carried out on constructing a single model, no matter how complicated the spectra data performed. This may cause unsatisfactory performance, especially when the number of training data samples is relatively small, compared to the number of spectra data variables. Inspired by the idea of ensemble learning from the area of machine learning, the calibration performance could probably be improved through constructing multiple regression models for the same purpose. Typical ensemble learning strategies include bagging, random subspace, random forest, etc. [34–40]. In this paper, the random subspace method is employed and combined with ICR for model calibration purpose. The main idea of the random subspace method is to build various individual models based on different variable subsets which are randomly selected from the original variables space. Then, the final calibration result is obtained by combining the results of different individual models. However, a critical shortcoming of this method is that the diversity among different individual models cannot be well guaranteed through a random selection manner, which is a quite important issue in the ensemble learning method.
J. Zheng, Z. Song / Chemometrics and Intelligent Laboratory Systems 155 (2016) 160–169
To improve the calibration performance of the random subspace ICR model, a two-level ICR model is developed in this paper. In the first level of the model, an ICA model is constructed on the original spectra data. Based on this model, various modeling directions can be determined, which are independent with each other. The diversity property of the random subspace method can be greatly improved if we build individual models along those independent directions. For construction of those subspaces, a related index is defined for variable selection in each subspace, which is based on the absolute values of the separating matrix in the ICA model. In the second level, an ICR model can be developed for each subspace. For online calibration purpose, an important issue is how to combine the results from different individual models. While one can resort to a simple average combination strategy, a more effective Bayesian based probabilistic combination strategy is proposed for results combination in this paper. The rest of this paper is organized as follows. In Section 2, the basic ICR model is briefly introduced. Detailed demonstration of the twolevel ICR model is provided in Section 3, followed by illustrations of two benchmark spectra data examples in the Section 4. Finally, conclusions are made. 2. Independent component regression (ICR) Based on the ICA modeling method, the ICR model can be built between the extracted independent components and quality variables. In the ICA algorithm, it is assumed that the measured process variables x ∈ Rm× 1 can be expressed as linear combinations of r(≤m) unknown independent components s ∈ Rr × 1, the relationship between them is given by [22] x ¼ As þ e
ð1Þ
where A ∈ Rm × r is the mixing matrix, e ∈ Rm × 1 is the residual vector. The basic problem of ICA is to estimate the original component s and the mixing matrix A from x. Therefore, the objective of ICA is to calculate a separating matrix W so that the components of the reconstructed data matrix ŝ become as independent of each other as possible, given as ŝ ¼ Wx:
ð2Þ
After the independent components have been estimated from the process data, the linear regression can be carried out between two datasets: the independent component dataset Ŝ ¼ ½ŝ1 ; ŝ2 ; ⋯; ŝn T ∈ Rnr and the quality variable dataset Y = [y1, y2, ⋯ , yn]T ∈ Rn × p. Therefore, the linear regression matrix can be calculated as T −1 T Q ¼ Ŝ Ŝ Ŝ Y:
ð3Þ
If we denote the dataset of process variables as X = [x1, x2, ⋯ , xn]T ∈ Rn × m, and combine the two steps of ICR modeling procedures, the ICR regression matrix can be determined as RICR ¼ Q T W:
non-Gaussianity measurement, etc. Based on the independent behavior of the extracted components, a subspace can be defined through each independent component direction, which are orthogonal to each other. Therefore, to build the ICR model in each subspace, the importance of each variable in different subspaces should be measured, depending on which the most informational ones should be retained in their corresponding subspace. To this end, an independent component related index (RI) is defined as follows RIði; jÞ ¼
w ij jwi1 j þ ⋯ þ wij þ ⋯ þ jwim j
ð6Þ
where i = 1 , 2 , ⋯ , k , j = 1 , 2 , ⋯ , m, wij is the j-th element of the i-th independent component direction in the separating matrix W. Therefore, the larger the value of the j-th element, the more significant contribution it has provided through the i-th independent component direction. Based on this defined index, the importance values of different process variables through each independent component direction can be measured and ranked from the most important one to the least important one. An appropriate number of variables can be selected to form each subspace, depending on the selection scheme. The ICR model-based subspaces can be represented as follows 8 X ¼ XðS1 Þ→subspace #1 > > < 1 X2 ¼ XðS2 Þ→subspace #2 X→ ⋮ > > : Xk ¼ XðSk Þ→subspace #k
ð7Þ
where Si , i = 1 , 2 , ⋯ , k is the column vector, which related to each subspace along the corresponding IC direction. A diagram of the proposed subspace modeling approach is given in Fig. 1. 3.1. ICR modeling in each subspace Suppose the whole variable set has been divided into k subspaces, and mb variables have been selected in each subspace, where b = 1 , 2 , ⋯ , k, the corresponding dataset for each subspace can be represented as {Xb}b = 1 , 2 , ⋯ , k. Denote the quality variable dataset as Y ∈ Rn×p, the subspace ICR model can be constructed as follows Xb ¼ Ab Sb þ Eb
ð8Þ
−1 STb Yb : Q b ¼ STb Sb
ð9Þ
Therefore, the regression form of the subspace ICR model can be developed as Yb ¼ Q Tb Ŝb ¼ Q Tb Wb Xb ¼ RICR;b Xb :
ð10Þ
3.2. Online calibration based on two-level ICR model ð4Þ
3. Two-level ICR for multivariate calibration Denote the whole variable dataset as X ∈ Rn× m, where m is the number of process variables, and n is the sample number for each variable. An initial ICA decomposition can be carried out on X, thus [22] X ¼ AS þ E
161
ð5Þ
where A is the mixing matrix of the ICA model, S is the data matrix of the independent components, E is the residual matrix. The number of independent components k can be determined by the negentropy method,
Based on the developed subspace ICR models, the property value of the new spectra data can be calculated by combing the results obtained in different subspaces. Therefore, when the new data sample xnew ∈ Rm is available, each subspace ICR prediction result is calculated in the first step, given as T b ŷnew ¼ RICR;b xnew ðSIb Þ
ð11Þ
where b = 1 , 2 , ⋯ , k, RICR , b the regression matrix of the b-th subspace ICR model, SIb represents the variable index of each subspace in the original variable space. When all of the subspace prediction results have been generated, the next step is to combine them in a certain
162
J. Zheng, Z. Song / Chemometrics and Intelligent Laboratory Systems 155 (2016) 160–169
Fig. 1. Diagram of two-level subspace ICR modeling method.
way. Based on the average rule, the final prediction of the property value is calculated as follows ŷnew ¼
k k h i T 1X 1X b ŷnew ¼ R ICR;b xnew ðSIb Þ : k b¼1 k b¼1
ð12Þ
probability of the new data sample in each subspace can be calculated by Bayesian inference, which are given as follows P I2 ðbjxnew Þ ¼
P I2 ðb; xnew Þ P I2 ðxnew jbÞP ðbÞ ¼ k P I2 ðxnew Þ X
P I2 ðxnew jbÞP ðbÞ
ð17Þ
b¼1
However, a more appropriate way is to use the weighed combination form. Here, a probabilistic method is proposed, which is based on ICA monitoring statistics and Bayesian posterior probability inference. In the ICA-based monitoring method, two monitoring statistics are usually constructed: the I2 index for measurement of normality distance in the independent component plane and the squared prediction error index SPE for consistency measurement of the model. For the new data sample xnew, the value of I2 and SPE monitoring statistics through different subspace ICR model can be calculated as follows [25] T
I2new;b ¼ ŝnew;b ŝnew;b
ŝnew;b ¼ Wb xnew ;
enew;b ¼ xnew −Ab ŝnew;b ;
SPEnew;b ¼ eTnew;b enew;b
ð13Þ ð14Þ
where b = 1 , 2 , ⋯ , k. Before the Bayesian inference of the posterior probability for the new data sample, the conditional probability of the data sample in different subspaces should be calculated. This can be made by transferring the monitoring statistical value to the probability distribution value, which are given as follows ( P I2 ðxnew jbÞ ¼ exp −
I2new;b
)
I 2lim;b
SPEnew;b P SPE ðxnew jbÞ ¼ exp − SPE lim;b
P SPE ðbjxnew Þ ¼
P SPE ðb; xnew Þ P SPE ðxnew jbÞP ðbÞ ¼ k P SPE ðxnew Þ X ½P SPE ðxnew jbÞP ðbÞ b¼1
where P(b) are prior probabilities, which can be assumed as equal values for different subspaces, thus P(b) = 1/k , b = 1 , 2 , ⋯ , k. When the posterior probabilities of the new data sample in different subspaces have been determined, we are in the position to combine local prediction results in different subspaces to form the final result. Suppose the prediction results in different subspaces have been obtained through Eq. (11), the final prediction result can be calculated as ŷnew ¼
k X
b
P ðbjxnew Þŷnew ¼
b¼1
k X
T P ðbjxnew Þ RICR;b xnew ðSIb Þ
ð16Þ
where I2lim ,b and SPElim,b are confidence limits of the I2 and SPE monitoring statistics in different subspaces, which can be determined through the kernel density estimation method [25]. Then the posterior
ð19Þ
b¼1
where the posterior probability P(b | x new ) is a weighted value of PI2(b | xnew) and PSPE(b | xnew), thus P ðbjxnew Þ ¼ αP I2 ðbjxnew Þ þ βP SPE ðbjxnew Þ
ð15Þ
ð18Þ
ð20Þ
where α and β are weighted parameters, and α + β = 1. For simplicity, the values of α and β can be selected as 0.5. 4. Experimental results In this section, two commonly used benchmark NIR spectra datasets are employed for performance evaluations of the two-level ICR modeling approach, which are related to the transmittance spectra of wheat
J. Zheng, Z. Song / Chemometrics and Intelligent Laboratory Systems 155 (2016) 160–169
Fig. 2. Wavelengths and the protein contents of the wheat kernels.
Fig. 3. Selected variables in the first four subspaces.
163
164
J. Zheng, Z. Song / Chemometrics and Intelligent Laboratory Systems 155 (2016) 160–169
model and the random subspace based ICR model have both been developed.
Table 1 RMSE values of different methods for model calibration. Methods
ICR
Random subspace ICR
Two-level ICR
RMSE values
1.6992
1.4368
1.0575
4.1. Wheat kernels data
kernels and corn. To evaluate the performance of the two-level ICR model, the root mean square error (RMSE) criterion can be used, which is defined as follow
RMSE ¼
vffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi u n te uX u y −ŷ 2 j j u t j¼1 n te
ð21Þ
where j = 1 , 2 , ⋯ , n_te, yj and ŷ j are real and predicted values, respectively, n_te is the total number of test data samples. Similarly, the RMSE value for each subspace ICR can be defined as follows
RMSEb ¼
vffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi u n te uX u y j;b −ŷ j;b 2 u t j¼1 n te
ð22Þ
where b = 1 , 2 , ⋯ , k, yj , b and ŷ j;b are real and predicted values by the b-th subspace ICR model. For comparison, the conventional ICR
In this dataset, a total of 523 data samples were collected from three different locations of the process. Each of the data samples was analyzed at 100 wavelengths in the range of 850–1050 nm, in order to predict the protein content of wheat kernels. This dataset is public available for research study, one can refer to the following website http://www. models.life.ku.dk/research/data/wheat_kernels/. Both of the 100 wavelengths and the protein content of the wheat kernels are shown in Fig. 2. For calibration modeling and online prediction, the whole dataset has been divided into two parts: the training dataset which contains 415 data samples, and the testing dataset consisting of the rest 108 data samples. To develop the two-level ICR model, an initial ICA model decomposition should be carried out in the first step, in which 10 independent components have been selected. As a result, a total of 10 subspaces can be formulated through different independent component directions. Then, based on the defined independent component related index, the most related variables in each subspace can be determined. Here, the number of related variables in each subspace has been selected as 30. After all of the 10 subspaces have been constructed, an ICR model can be developed for each subspace. Detailed information of the selected wavelength variables in the first four subspaces are shown in Fig. 3. It can be found that different ranges
Fig. 4. Calibration results of two-level ICR model and traditional ICR model.
Fig. 5. RMSE values obtained in different subspaces.
J. Zheng, Z. Song / Chemometrics and Intelligent Laboratory Systems 155 (2016) 160–169
165
Fig. 6. Posterior probability values of the testing samples under the first four subspaces.
of wavelength have been specified in different subspaces, which may provide quite different calibration results for the property value. The RMSE value of each method for the testing dataset is tabulated in Table 1, which are the mean values of 100 running times. It can be seen from this table that the two-level ICR model has obtained the best prediction results for the testing dataset. With the incorporation of the random subspace modeling approach, the prediction accuracy can be slightly improved, compared to the single ICR model. According to those results, it can be inferred that the construction of subspaces is important to the subspace-based method. In some particularly case, a
random subspace construction method may not improve the calibration performance. [Fig. 4 shows calibration results of both two-level ICR and conventional ICR methods. Obviously, a better calibration result can be observed by the two-level ICR method. This is in consistence with the RMSE values of the two methods provided in Table 1. To examine the calibration performance in each subspace, the RMSE values obtained through the 10 subspace ICR models are shown together in Fig. 5, in which the black line represents the final RMSE value obtained by the two-level ICR model. It can be seen that the RMSE value obtained
Fig. 7. Two monitoring statistics of the testing samples under the first four subspaces.
166
J. Zheng, Z. Song / Chemometrics and Intelligent Laboratory Systems 155 (2016) 160–169
Fig. 8. RMSE values under different numbers of selected variables.
through the Bayesian inference strategy is smaller than all of the RMSE values in single subspaces. This is because the advantages of all subspace models in specific ranges have been combined together and complemented for each other. The posterior probability values of testing data samples related to different subspaces are shown in Fig. 6. Correspondingly, the values of the two statistics I2 and SPE in the same subspaces are given in Fig. 7. In the two-level ICR subspace modeling method, the number of variables in each subspace has been initially selected as 30. However, how to determine this parameter still needs investigation. In order to
examine the effect of this parameter on the calibration performance of the two-level subspace ICR model, different numbers of variables have been selected for evaluation. Fig. 8 shows the RMSE values of the twolevel ICR model under different numbers of selected variables for subspace construction. It can be seen from this figure that the best result has been obtained when the number of selected variables has been set as 30. Actually, the number of selected variables in each subspace may have high impacts on the accuracy of the data model. If only a small number of variables are selected in each subspace, the combination of all subspaces may not well cover the whole variable space. On the
Fig. 9. Data characteristics of spectra data and corresponding moisture values.
J. Zheng, Z. Song / Chemometrics and Intelligent Laboratory Systems 155 (2016) 160–169 Table 2 RMSE values of different methods for model calibration. Methods
ICR
Random subspace ICR
Two-level ICR
RMSE values
0.3768
0.3511
0.3127
other hand, if we set a large number of variables for each subspace, it will result in a severe overlap among different subspaces, which may deteriorate the complement effect among each other. Maybe a more appropriate way for variable selection in each subspace is to set different variable numbers for different subspaces. 4.2. Corn data This dataset consists of spectra from 80 samples of corn with wavelength ranging from 1100 nm to 2498 nm at 2 nm intervals (700 channels). As a result, a spectra data matrix of 80× 700 has been collected for study. The corresponding concentration values of the moisture for those collected spectra data have also been measured in lab, which form a response matrix of 80× 1. Both of the spectra and concentration data are shown in Fig. 9. Those data are available at the Eigenvector Research homepage: http://software.eigenvector.com/Data/Corn/. For model training and testing purposes, the whole dataset has been divided into two different parts: the training dataset contains 50 samples, and the testing dataset contains the rest 30 samples.
167
To develop the two-level ICR model, an initial ICA model decomposition is carried out in the first step, in which 10 independent components have also been selected. As a result, a total of 10 subspaces can be formulated through different independent component directions. Then, based on the defined independent component related index, the most related variables in each subspace can be determined. Here, the number of related variables in each subspace has also been selected as 30. After all of the 10 subspaces have been constructed, an ICR model can be developed for each subspace. Similarly, both of the conventional ICR model and the random subspace ICR model have been constructed for performance comparisons. The RMSE values of the three different methods for the testing dataset are tabulated in Table 2, which are the mean values of 100 running times. Again, the two-level ICR model has obtained the best prediction results for the testing dataset in this example. Detailed calibration errors of the 30 testing data samples by the two-level ICR model and the conventional ICR model are provided in Fig. 10, in which it can be seen that the prediction quality has been improved by the new method. Similarly, the RMSE values obtained through the 10 subspace ICR models are shown together in Fig. 11, in which the black line represents the final RMSE value obtained by the two-level ICR model. Again, it can be seen that the RMSE value obtained through the Bayesian inference strategy is smaller than all of the RMSE values in single subspaces. The posterior probability values of testing data samples related to the first four subspaces are shown in Fig. 12.
Fig. 10. Prediction errors of the two methods for testing data samples.
Fig. 11. RMSE values obtained in different subspaces.
168
J. Zheng, Z. Song / Chemometrics and Intelligent Laboratory Systems 155 (2016) 160–169
Fig. 12. Posterior probability values of the testing samples under the first four subspaces.
5. Conclusions In the present work, a two-level subspace ICR model based calibration model has been developed for multivariate spectral data. In order to improve the calibration performance of the random subspace calibration model, the individual subspace is constructed through each independent direction obtained through the ICA model. For online calibration purpose, a Bayesian inference strategy has been developed for probabilistic combination of calibration results obtained from different subspaces. According to the experimental results, the calibration performance has been improved by the proposed method, compared to both of the random subspace ICR model and the basic ICR model. Therefore, it can be inferred that the ensemble model is useful for calibration modeling and how to construct subspaces does have impacts on the ensemble performance of the model. It should be noted that although the experimental case studies have been made on the NIR datasets, applications to other spectra dataset such as IR and Raman are also possible. Due to the similar nature among those spectra data types, the proposed two-level model structure should be effective as long as the subspace calibration models can be successfully constructed. Acknowledgments This work was supported in part by the National Natural Science Foundation of China (NSFC) (61370029). References [1] J. Workman Jr., B. Lavine, R. Chrisman, M. Koch, Proc. Anal. Chem. Analy. Chem. 83 (2011) 4557–4578. [2] R.J. Pell, M.B. Seasholtz, K.R. Beebe, M.V. Koch, Process analytical chemistry and chemometrics, J. Chemometrics 28 (2014) 321–331. [3] B.K. Lavine, J. Workman Jr., Chemometrics, Anal. Chem. 85 (2012) 705–714. [4] R.G. Brereton, A short histoy of chemometrics: a personal view, J. Chemometrics 28 (2014) 749–760. [5] R.G. Brereton, Pattern recognition in chemometrics, Chemom. Intell. Lab. Syst. 149 (2015) 90–96. [6] J.A. Westerhuis, S.P. Gurden, A.K. Smilde, Spectroscopic monitoring of batch reactions for on-line fault detection and diagnosis, Anal. Chem. 72 (2000) 5322–5330. [7] M.M. Reis, P.H.H. Araujo, C. Sayer, R. Giudici, Spectroscopic on-line monitoring of reactions in dispersed medium: chemometric challenges, Anal. Chim. Acta 595 (2007) 257–265.
[8] T. Chen, E. Martin, Bayesian linear regression and variable selection for spectroscopic calibration, Anal. Chim. Acta 631 (2009) 13–21. [9] R. Ergon, Reduced PCA/PLSR models by subspace projections, Chemom. Intell. Lab. Syst. 81 (2006) 68–73. [10] J. Luts, F. Ojeda, R. Van de Plas, B. De Moor, S. Van Huffel, J.A.K. Suykens, A tutorial on support vector machine-based methods for classification problems in chemometrics, Anal. Chim. Acta 665 (2010) 129–145. [11] X.G. Shao, X.H. Bian, W.S. Cai, An improved boosting partial least squares method for near-infrared spectroscopic quantitative analysis, Anal. Chim. Acta 666 (2010) 32–37. [12] A. Martins, L.A. Daffner, A. Fenech, C. McGlinchey, M. Strlic, Non-destructive dating of fiber-based gelatin silver prints using near-infrared spectroscopy and multivariate analysis, Anal. Bioanal. Chem. 402 (2012) 1459–1469. [13] V.R. Nadadoor, H.D. Siegler, S.L. Shah, W.C. McCaffrey, A. Ben-Zvi, Online sensor for monitoring a microalgal bioreactor system using support vector regression, Chemom. Intell. Lab. Syst. 110 (2012) 38–48. [14] M.F. Abdelkader, J.B. Cooper, C.M. Larkin, Calibration transfer of partial least squares jet fuel property models using a segmented virtual standards slope-bias correction method, Chemom. Intell. Lab. Syst. 110 (2012) 64–73. [15] D. Lee, W. Lee, Y. Lee, Y. Pawitan, Sparse partial least-squares regression and its applications to high-throughput data analysis, Chemom. Intell. Lab. Syst. 109 (2011) 1–8. [16] F. Vogt, F. Gritti, G. Guiochon, Polynomial multivariate least-squares regression for modeling nonlinear data applied to in-depth characterization of chromatographic resolution, J. Chemometrics 25 (2011) 575–585. [17] P. Shahbazikhah, J.H. Kalivas, A consensus modeling approach to update a spectroscopic calibration, Chemom. Intell. Lab. Syst. 120 (2013) 142–153. [18] P.S. Gromski, Y. Xu, E. Correa, D. Ellis, M.L. Turner, R. Goodacre, A comparative investigation of modern feature selection and classification approaches for the analysis of mass spectrometry data, Anal. Chim. Acta 829 (2014) 1–8. [19] M. Lopatka, G. Vivo-Truyols, M.J. Sjerps, Probabilistic peak detection for first-order chromatographic data, Anal. Chim. Acta 817 (2014) 9–16. [20] M. Chen, S. Khare, B. Huang, A unified recursive just-in-time approach with industrial near infrared spectroscopy, Chemom. Intell. Lab. Syst. 135 (2014) 133–140. [21] M.G. Gustafsson, Independent component analysis yields chemically interpretable latent variables in multivariate regression, J. Chem. Inform. model 45 (2005) 1244–1255. [22] A. Hyvarinen, E. Oja, Independent component analysis: algorithms and applications, Neural Netw. 13 (2000) 411–430. [23] R.F. Li, X.Z. Wang, Dimension reduction of process dynamic trends using independent component analysis, Comput. Chem. Eng. 26 (2002) 467–473. [24] M. Kano, S. Tanaka, S. Hasebe, I. Hashimoto, H. Ohno, Monitoring independent components for fault detection, AICHE J. 49 (2003) 969–976. [25] J.M. Lee, C.K. Yoo, I.B. Lee, Statistical process monitoring with independent component analysis, J. Process Control 14 (2004) 467–485. [26] X. Liu, L. Xie, U. Kruger, T. Littler, S. Wang, Statistical-based monitoring of multivariate non-Gaussian systems, AICHE J. 54 (2008) 2379–2391. [27] J. Chen, X.Z. Wang, A new approach to near-infrared spectral data analysis using independent component analysis, J. Chem. Inf. Comput. Sci. 41 (2001) 992–1001. [28] F. Westad, Independent component analysis and regression applied on sensory data, J. Chemometrics 19 (2005) 171–179.
J. Zheng, Z. Song / Chemometrics and Intelligent Laboratory Systems 155 (2016) 160–169 [29] H. Ahn, E. Choi, I. Han, Extracting underlying meaningful features and canceling noise using independent component analysis for direct marketing, Expert Syst. Appl. 33 (2007) 181–191. [30] H. Kaneko, M. Arakawa, K. Funatsu, Development of a new regression analysis method using independent component analysis, J. Chem. Inf. Model. 48 (2008) 534–541. [31] L.J. Kao, C.C. Chiu, C.J. Lu, J.L. Yang, Integration of nonlinear independent component analysis and support vector regression for stock price forecasting, Neurocomputing 99 (2013) 534–542. [32] O. Forni, S. Maurice, O. Gasnault, R.C. Wiens, A. Cousin, S.M. Clegg, J.B. Sirven, J. Lasue, Independent component analysis classification of laser induced breakdown spectroscopy spectra, Spectrochim. Acta B At. Spectrosc. 86 (2013) 31–41. [33] Y.K. Chuang, Y.P. Hu, I.C. Yang, S.R. Delwiche, Y.M. Lo, C.Y. Tsai, S. Chen, Integration of independent component analysis with near infrared spectroscopy for evaluation of rice freshness, J. Cereal Sci. 60 (2014) 238–242.
169
[34] L. Breiman, Bagging predictors, Mach. Learn. 24 (1996) 123–140. [35] T.K. Ho, The random subspace method for constructing decision forests, IEEE Trans. Pattern Anal. Mach. Intell. 20 (1998) 832–844. [36] D.C. Tao, X.O. Tang, X.L. Li, X.D. Wu, Asymmetric bagging and random subspace for support vector machines-based relevance feedback in image retrieval, IEEE Trans. Pattern Anal. Mach. Intell. 28 (2006) 1088–1099. [37] D.S. Cao, Q.S. Xu, Y.Z. Liang, The boosting: a new idea of building models, Chemom. Intell. Lab. Syst. 100 (2010) 1–11. [38] Z.Q. Ge, Z.H. Song, Bagging support vector data description model for batch process monitoring, J. Process Control 23 (2013) 1090–1096. [39] K. Wang, T. Chen, R. Lau, Bagging for robust non-linear multivariate calibration of spectroscopy, Chemom. Intell. Lab. Syst. 105 (2011) 1–6. [40] G. Seni, Ensemble methods in data mining, Morgan and Claypool Publishers, 2010.