- Email: [email protected]
Hybrid modeling of penicillin fermentation process based on least square support vector machine
chemical engineering research and design 8 8 ( 2 0 1 0 ) 415–420
Contents lists available at ScienceDirect
Chemical Engineering Research and Design journal homepage: www.elsevier.com/locate/cherd
Hybrid modeling of penicillin fermentation process based on least square support vector machine Xianfang Wang a,b,∗ , Jindong Chen a , Chunbo Liu a , Feng Pan a a b
School of Communications and Control Engineering, Jiangnan University, 1800 Lihu Rd, Wuxi, Jiangsu 214122, PR China Henan Institute of Science and Technology, Henan 453003, PR China
a b s t r a c t According to the problem of the pre-estimation with least square support vector machine (LSSVM) modeling is not ideal in the initial stages of penicillin fermentation process, two hybrid models are designed by utilizing the advantage of LSSVM and kinetics model. Through selecting the appropriate state variables and adopting these methods for penicillin fermentation, the mycelial concentration can be pre-estimated. Experiment results show that these hybrid modeling methods not only improve the above problem, but also have higher predicting accuracy and more powerful generalization ability than the single LSSVM method. © 2009 The Institution of Chemical Engineers. Published by Elsevier B.V. All rights reserved. Keywords: Least square support vector machine; Kinetics model; Fermentation; Hybrid model
1.
Introduction
Penicillin fermentation process is a strong nonlinear, time varying and correlating process (Bailey and Ollis, 1986). To carry out further optimizing and control strategy, one ideal model must be gained firstly. However, only those parameters such as physical and chemical parameters can be measured on-line, and the other parameters, such as bio-parameters which are more complex and important, can not be measured on-line yet in actual fermentation process, further more, sampling data are often limited, these situation made the modeling and controlling more complexity (Zhongping and Feng, 2005; Zhihua et al., 2005). The traditional kinetics modeling reflects the mechanism of the process, which belongs to “white-box modeling”, but it has some errors between the traditional kinetics model’s predicting result and the real output (Naigong and Xiaogang, 2007). The soft-sensor methods for biomass on-line estimation in fermentation process based on support vector machine (SVM) becomes significant in theory and valuable in practice (Xuejin et al., 2006; Rui et al., 2004; Yi and Haiqing, 2006). Xuejin et al. (2006) and Rui et al. (2004) had built a class of model for penicillin fermentation process based on SVM, which belongs to a “black-box moedling”, but the calcu-
lated speed was very slower with the increasing of sampling data, this method could not used for on-line pre-estimating. Yi and Haiqing (2006) adopted least squares support vector machines (LSSVM) modeling for a penicillin fed-batch fermentation, which improved the calculated speed. However, the problem of this method is that the pre-estimation result is not ideal in the initial stages of the fermentation process. In order to solve the problem, two hybrid modeling methods are designed by selecting the appropriate state variables and utilizing mechanism of fermentation and advantages of LSSVM (such as, higher generalization ability, faster calculated speed, etc.). Adopting the hybrid methods for a penicillin fermentation modeling, the mycelial concentration could be pre-estimated by using limited on-line data. Some results show that the hybrid modeling methods have higher predicting accuracy and more powerful generalization ability than the single LSSVM method. This paper is organized as follows: Section 2 describes the principles of SVM and LSSVM. The kinetics model of fermentation process is presented in Section 3. Section 4 introduces a fermentation process modeling based on LSSVM. Two hybrid models are designed in Section 5. Experiment results are given in Section 6. Finally, conclusions are given in Section 7.
∗ Corresponding author at: School of Communications and Control Engineering, Jiangnan University, 1800 Lihu Rd, Wuxi, Jiangsu 214122, PR China. Tel.: +86 13961868919. E-mail address: [email protected] (X. Wang). Received 29 April 2008; Received in revised form 8 August 2009; Accepted 20 August 2009 0263-8762/$ – see front matter © 2009 The Institution of Chemical Engineers. Published by Elsevier B.V. All rights reserved. doi:10.1016/j.cherd.2009.08.010
416
chemical engineering research and design 8 8 ( 2 0 1 0 ) 415–420
where is the regularization parameter, k is the error between the real output and estimated value at the k th sample point. This problem can be solved by using the optimization theory. The Lagrangian function for this problem can be define as follows:
Nomenclature X V t P p
biomass concentration (g/l) specific growth rate (h−1 ) culture volume (l) current fermentation time (h) penicillin concentration (g/l) the maximum specific penicillin production rate (h−1 ) S substrate concentration (g/l) Kp inhibition constant (g/h) K penicillin hydrolysis constant (h−1 ) yield coefficient (g biomass/g substrate) Yx/s Yp/s yield coefficient (g penicillin/g substrate) saturation constant (g/l) Kx m maintenance coefficient on substrate (h−1 ) Ki inhibition constant for product information (g/l) F feed flow rate of substrate (l/h) (x, y) the original values of a sampling point (xscale , yscale ) the normalization values of a sampling point xmin , xmax , ymin and ymax the minimum and maximum values in the original dataset
L(ω, b, ; ˛) = J(ω, ) −
N
˛k [ωT (xk ) + b + k − yk ]
(4)
k=1
where ˛k are Lagrange multipliers, which can be due to the equality constraints as follows from the Kuhn–Tucker conditions (Fletcher, 1987). The conditions for optimality
⎧ ⎪ ∂L ⎪ ⎪ =0 ⎪ ⎪ ∂ω ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎨ ∂L =0
∂b
⇒
ω=
N
˛k (xk )
k=1
⇒
ω=
N
˛k = 0
⎪ k=1 ⎪ ⎪ ⎪ ∂L ⎪ ⎪ = 0 ⇒ ˛k = k , k = 1, . . . , N ⎪ ⎪ ∂k ⎪ ⎪ ⎪ ⎪ ⎩ ∂L = 0 ⇒ ωT (xk ) + b + k − yk = 0, k = 1, . . . , N
(5)
∂˛k
2.
Basic principles of SVM and LSSVM
The solution is also given by
SVM is an excellent method aiming to finite data points based on statistical learning theory (SLT) (Vapnik, 1998, 1999) advanced by Vapnik in 1990s. It adopts the structure risk minimization (SRM) principle and avoids the complex computing kernel function of low dimensions instead of dot-matrix of high dimensions space. The main idea of SVM is described as follows. Given a training data set of N points {xk , yk }N with the k=1 input data xk ∈ Rn and the corresponding target yk ∈ Rm . In feature space SVM models take the form y(x) = ωT (x) + b
(1)
where the nonlinear function (·) : Rn → Rm maps the input space into higher character dimension, which is not explicitly constructed (Vapnik et al., 1997; Suykens and Vandewalle, 1999), b is the bias and ω is a weight vector of the same dimension as the feature space. The standard SVM is solved by quadratic programming methods, however these methods are often time consuming and are difficult to implement adaptively, and suffer from the problem of large memory requirement and CPU time when trained in batch model (Kuh, 2002). LSSVM is a modified version of SVM (Suykens and Vandewalle, 1999), which utilizes the equality constraints to replace the original convex quadratic programming problem. In LSSVM for function estimate, the following optimization problem is considered 1 T 1 2 k , ω ω+ 2 2 N
minJ(ω, ) = ω,b,
>0
(2)
k=1
subject to the equality constraints yk = ωT · (xk ) + b + k ,
k = 1, . . . , N
(3)
a b
=
0 − →T I
− →T I ˝
+ −1 I
0 y
(6)
− → where y = [y1 , y2 , . . . , yn ]T , I = [1, . . . , 1]T , ˛ = [˛1 , . . . , ˛N ]T , ˝ is a square matrix, the elements of its k row l column T are ˝kl = (xk ) (xl ) = K(xk , xl ), K(·, ·) is kernel function, which mapped the input vector into a high-dimension feature space (Fletcher, 1987). The most usual kernel functions are polynomial, Gaussian-like or some particular sigmoids (Chi and Ersoy, 2003; Qiong et al., 2007). Hence, the Eq. (1) is rebuilt by solving the linear set of Eqs. (5) and (6) instead of quadratic programming. The resulting LSSVM model for function estimation becomes:
y(x) =
N
˛k K(x, xk ) + b
(7)
k=1
where ˛, b are the solutions of Eq. (6). Some detail can be found in Suykens and Vandewalle (1999). According to the above description, it can be found easily that the difference between LSSVM and SVM is that they choose different lost functions and restraining conditions in optimization problems when using the SRM principle. The standard SVM chooses error i and inequality constraints, while LSSVM chooses the square of error i and equality constraints. Because of above characteristics, LSSVM not only accelerates the solve speed and reduce the training time, but also expediently ascertains parameters and reduces the complexity of the algorithm greatly, therefore, LSSVM is suitable for online modeling than SVM (Morris et al., 1989; Cristianini, 2005).
417
chemical engineering research and design 8 8 ( 2 0 1 0 ) 415–420
3.
Kinetics model of fermentation process
Hypothesis the process is a penicillin fermentation process, its kinetics model can be expressed with formulas from (8) to (11), which was built by Bajpai and Reuss (1980): dX X dV = X − dt V dt
(8)
dP SX P dV = p − KP − dt V dt Kp + S(1 + S/Ki )
(9)
−
p SX SX S dV dS · · + mX + = + dt Yx/s Kx X + S Yp/s Kp + S(1 + S/Ki ) V dt (10)
dV =F dt
(11)
where X is mycelial concentration, is specific growth rate, V is culture volume, t is current fermentation time, P is penicillin concentration, p is the maximum specific penicillin production rate, S is substrate concentration, Kp is the inhibition constant, K is the penicillin hydrolysis constant, Yx/s is the yield coefficient, Yp/s is the yield coefficient, Kx is the substrate saturation constant, m is the maintenance coefficient, Ki is the inhibition constant for product information and F is the feed flow rate of substrate. More detail expressions can be found in Ahmad et al. (2003) or in the Nomenclature. In fact, Eqs. (8)–(11) only provide part of the mechanism knowledge of the process and the kinetics character is mainly contained in specific growth rate (t), here = (t), which is a complex biology and physical chemistry function that could be described as Eq. (12) (James et al., 2002; Ahmad et al., 2003): (t) =
max S(t) KIP · Ks + S(t) KIP + P(t)
(12)
where max is the maximum specific growth rate, Ks is the saturation constant, KIP is the inhibition constant of product. Solve the differential equation (8) and disperse the result, the following equation can be got X(t + t) = X(t) exp
−
F V
t
Fig. 1 – Basic block diagram of penicillin fermentation process based on LSSVM. When establishing soft measurement model, the 2nd group variables are chosen from the input variables U as the input variables of the LSSVM model, and unmeasured variables are chosen as output Y of the soft model. Then use Eqs. (7) and (8) to achieve the non-linear relation between the input variables U and the output variables Y.
4.1.
Determination of input and output variables
During the process of building a biomass concentration model for penicillin fermentation, we choose some higher effective variables as input variables of LSSVM, such as dissolved oxygen concentration DO(t), mycelial concentration X(t), penicillin concentration P(t), substrate concentration S(t) and fermentation time T. As a matter of convenience, we only select the unable online measured variables mycelial concentration as the output Y of the model, that is Y(t) = X(t + 1).
4.2.
Preparation of training dataset
There are 12 batches data, each batch includs 150 sample points, these batches data are divided randomly into two subsets, one used exclusively for training and the other exclusively for testing. All the input variables are rescaled to be included within the interval [−1, 1] by using the following equations: xscale = 2
x − xmin −1 xmax − xmin
(14)
yscale = 2
y − ymin −1 ymax − ymin
(15)
(13)
where t is the sample time, its unit is hour. F, V are approximately constants due to V is much larger than F. If is solved, the result of X(t + t) can be obtained.
4. Fermentation process modeling based on LSSVM
where (x, y), (xscale , yscale ) are the old and new value of a sampling point respectively. xmin , xmax , ymin and ymax are the minimum and maximum values of that in the original dataset.
4.3. LSSVM has a high generalization, more efficient than the standard SVM, which has strong generalization and is suitable for modeling with limited data. The basic structure of modeling for penicillin fermentation based on LSSVM as shows in Fig. 1, the online state values (the enable measured online variables) usually include the pH value, dissolved oxygen (DO) concentration and CO2 concentration, temperature, revolution per minute (RPM) and other variables. The input variables u include air flow, hot water flow speed, cold water flow speed, output variables include CO2 concentration, reaction quantity of heat, etc. The offline state values (unable measured online variables) include biomass concentration, penicillin concentration, substrate concentration, and the other variables (Zhongping and Feng, 2005).
LSSVM training
In this study, we take commonly radial basis function (RBF) as the kernels function of LSSVM
K(xk , xl ) = exp
−
||x − xk ||2 2
(16)
It should be noted that predetermined parameters in LSSVM algorithms with RBF kernel are the punishment coefficient and the RBF coefficient . the training aim is to determine the two parameters. Here, we adopt the MSE (mean square error) as the parameters about training quality. At the end of the training, these parameters would be obtained: = 15, 500, = 1.50.
418
chemical engineering research and design 8 8 ( 2 0 1 0 ) 415–420
Fig. 2 – Predicting error of biomass concentration.
Fig. 3 – The block modeling diagram with serial hybrid LSSVM.
4.4.
Testing mycelial concentration
Define the predicting error e(t) e(t) = yp (t) − y(t)
(17)
where the y(t) is the real data, yp (t) is the predicting result of mycelial concentration at time t. Use one leave batch data to
Fig. 6 – The prediction result of mycelial with SPHLSSVM. test, the predicting error of mycelial concentration is shown in Fig. 2. Fig. 2 shows that the predicting results of mycelial concentration are not very satisfied with actual process near 50 h, while the others are better. To solve this problem, it is necessary to improve the above modeling method.
Fig. 4 – The block modeling diagram with serial and parallel hybrid LSSVM.
4.5. Comparison of modeling between LSSVM and RBFNN In order to illuminate the advantage of the modeling based on LSSVM, we adopt radial basis function neural network (RBFNN) (Thompson and Kramer, 1994) to model for the fermentation process, the input and output variables are set same as Section 4.1, the largest neuron number of hidden layer is 35, mean square error E is 0.0001, the distribution of radial basis function density is 0.5, change the number of training and testing samples, some performance indicators are presented in Table 1.
Table 1 – Comparison of MSE between LSSVM and RBFNN. Training number
Fig. 5 – The prediction result of mycelial with SHLSSVM model.
50 70 100 200
Testing number 200 200 200 200
LSSVM testing MSE 0.0588 0.0356 0.0095 0.0026
RBFNN testing MSE 0.3170 0.1683 0.1070 0.0027
419
chemical engineering research and design 8 8 ( 2 0 1 0 ) 415–420
Fig. 7 – The comparison of predicting error for mycelial among SHLSSVM, SPHLSSVM and LSSVM.
From Table 1, it can be easily found that the testing MSE of RBFNN is bigger than the LSSVM when the training number is less than 70, which indicates that the LSSVM is maintain relatively high prediction performance in the case of small samples. With the increasing of the training samples, the MSE of two methods are closed gradually. This phenomenon illuminates that LSSVM is less dependent on the size of the training dataset and has better generalization ability than that of the RBF neural network.
5.
Hybrid modeling
Modeling only with the mechanism of fermentation process is very difficult, because of the process complexity and the sample data often limited. Modeling only based on LSSVM is belong to “black-box modeling”, which only focus the input and output data of the process, and need not think of the material structure. This section has presented two hybrid modeling method by utilizing the advantages of LSSVM model and kinetics model to improve the shortcoming of single modeling. One method is showed in Fig. 3. At first, the middle variable is predicted by Eq. (7) of LSSVM in Section 4, i.e. = y(x), then the output mycelial concentration X(t + 1) is calculated by the Eq. (13). This method is called serial hybrid LSSVM (SHLSSVM). Fig. 4 shows the other method, which is same as Morris et al., 1989. is estimated by prior knowledge expression Eqs. (8)–(11) and LSSVM model (7), the output X(t + 1) can be solved by the following kinetics model (13). This method is called serial-parallel hybrid LSSVM (SPHLSSVM).
6.
Experiment results
This section focuses on the growth period of penicillin fermentation process, i.e. the first 50 h, adopts the above two hybrid methods in Section 5, the input variables, training method and testing method are chosen as same as in Section 4.1, the output variables is , some common parameters of the penicillin fermentation kinetics model in (8)–(12) are set as follows according to Ahmad et al. (2003): p = 0.005 h−1 , Kx = 0.15 g/l, Kp = 0.0002 g/h, Ki = 0.010 g/l, Yx/s = 0.45, Yp/s = 0.90, m = 0.014 h−1 , K = 0.04 h−1 . When adopting SPHLSSVM, some parameters in Eq. (12) are set as max = 0.092 h−1 , Ks = 508, 583 g/l, KIP = 7.5721 g/l. After the training, the two parameter of LSSVM can be obtained, = 15, 000, = 1.51 (SHLSSVM); = 16, 000, = 1.55 (SPHLSSVM). Predicting results are shown in Figs. 5 and 6.
Table 2 – Comparison of predicting quality for fermentation growth period with different methods. Method of models
SHLSSVM
SPHLSSVM
LSSVM
MSE
0.0002265
0.0002947
0.0041
From Figs. 5 and 6, it can be seen clearly that the SHLSVM model and SPHLSSVM model are better in the initial stages of the fermentation process, which illustrates that the two methods improved the shortcoming of single method with LSSVM availably. The predicting comparison in the growth period of SPHLSSVM, SHLSVM and LSSVM are shown in the following Fig. 7 and Table 2. From Fig. 7, it shows clearly that the predicting error of SPHLSSVM is the same as SHLSSVM in 0–22 h, which is nearly 0, and the estimated value of LSSVM is less than real data. The predicting error of three methods are steadily kept nearly 0 in 23–28 h. From 29 to 50 h, the predicting error of LSSVM is largest than the others. These results illustrate that the two hybrid models are efficiency for modeling of fermentation process. The parameter mean square error (MSE) is one important parameter of predicting quality. Table 2 shows the comparison of MSE during fermentation process with three different methods. Under the same conditions, the MSE value of SHLSSVM is the smallest, the biggest MSE is the LSSVM method (up to 0.0041, and is about 18 times of SHLSSVM). These results illustrate that the predicting quality of SHLSSVM is better than others, and the predicting accuracy of hybrid model is much better than the single model method (LSSVM modeling).
7.
Conclusions
This paper has introduced the principles of SVM and LSSVM firstly, then the kinetics model of penicillin fermentation process is presented briefly. According to the problem that the pre-estimation result of bio-parameters base on LSSVM is not ideal in the initial stages of the fermentation process, two hybrid predicting models have been built. These hybrid modeling methods have utilized the strong generalization advantages of LSSVM and the mechanism of kinetics model, and have been applied to develop a soft sensing model for a penicillin fermentation process. Experiment results show that the hybrid models are very effective in fermentation process.
420
chemical engineering research and design 8 8 ( 2 0 1 0 ) 415–420
Acknowledgment This work is supported by the 863 project of China (no. 2006AA020301). Therefore, it is necessary for the stability conditions to be investigated in the multi-regions.
References Ahmad, A., Samad, N.A.F.A. and Wei, C.A., 2003, Mathematical modeling and analysis of dynamic behavior of a fed-batch Penicillin G fermentation process, In Proceedings of International Conference on Chemical and Bioprocess Engineering , pp. 387–394. Bailey, J. and Ollis, D., (1986). Biochemical Fundamental. ((McGraw-Hill Inc., New York)), pp. 1–30 Bajpai, R. and Reuss, M., 1980, A mechanistic model for penicillin production. Journal of Chemical Technology and Biotechnology, 30: 330–344. Chi, H. and Ersoy, O.K., 2003, Recursive update algorithm for least squares support vector machines. Neural Processing Letters, 17: 165–173. Cristianini, N., 2005, An Introduction to Support Vector Machines, in John Shawe-Taylor (China Machine Press), pp. 1–189. Fletcher, R., (1987). Practical Methods of Optimization. ((John Wiley and Sons, Chichester/New York)). James, S., Legge, R. and Budman, H., 2002, Comparative study of black-box and hybrid estimation methods in fed-batch fermentation. Journal of Process Control, 1(1): 113–121. Kuh, A., 2002, Adaptive least square kernel algorithms and applications, In Proceedings of International Joint Conference on Neural Networks , pp. 2104–2107. Morris, A.J., Montague, G.A., Tham, M.T., 1989, Soft-sensors in industrial process control, in Applied Developments in Process Control (London, UK), pp. 1–3. Naigong, Y. and Xiaogang, R., 2007, Penicillin fermentation process optimal control strategy based on cellular automata
model, Journal of Beijing University of Technology, (02): 150–154 (in Chinese). Qiong, W., Wenying, L. and Yihan., 2007, Time series online prediction algorithm based on least squares support vector machine. Journal of Central South University of Technology, 14(3): 442–449. Rui, F., YueJie, Z., Yanzhu, Z., et al., 2004, Drifting modeling method using weighted support vector machines with application to soft sensor, Acta Automatica Sinca, 3(30): 436–441. Suykens, J.A.K. and Vandewalle, J., 1999, Least squares support vector machine classifiers. Neural Processing Letters, 9(3): 293–300. Thompson, M.L. and Kramer, M.A., 1994, Modeling chemical processes using prior knowledge and neural networks. AIChE Journal, 40(8): 1328–1340. Vapnik, V.N., (1998). Statistical Learning Theory. ((Wiley, New York)). Vapnik, V.N., (1999). The Nature of Statistical Learning Theory (2nd ed.). ((Springer-Verlag, New York)). Vapnik, V., Golowich, S. and Smola, A., 1997, Support vector machine for function approximation, regression estimation, and signal processing, in Neural Information Processing Systems, vol. 9, Mozer, M. and Petsche, T. (eds) (MIT Press), pp. 1–200. Xuejin, G., Pu, W., Chongzheng, S., et al., 2006, Modeling for penicillin fermentation process based on support vector machine, Journal of System Simulation, (07): 2052–2055 (in Chinese). Yi, L. and Haiqing, W., 2006, Modeling a penicillin fed-batch fermentation using least squares support vector machines, Chinese Journal of Biotechnology, (01): 144–148 (in Chinese). Zhihua, X., Guohong, H. and Huihe, S., 2005, On-line estimation of concentration parameters in fermentation process. Journal of Zhejiang University Science, 6B(6): 530–534 (in Chinese) Zhongping, S. and Feng, P., (2005). Fermentation Process Resolution, Control and Detection. ((Chemical Industry Press, Beijing)), pp. 1–10 (in Chinese)
Contents lists available at ScienceDirect
Chemical Engineering Research and Design journal homepage: www.elsevier.com/locate/cherd
Hybrid modeling of penicillin fermentation process based on least square support vector machine Xianfang Wang a,b,∗ , Jindong Chen a , Chunbo Liu a , Feng Pan a a b
School of Communications and Control Engineering, Jiangnan University, 1800 Lihu Rd, Wuxi, Jiangsu 214122, PR China Henan Institute of Science and Technology, Henan 453003, PR China
a b s t r a c t According to the problem of the pre-estimation with least square support vector machine (LSSVM) modeling is not ideal in the initial stages of penicillin fermentation process, two hybrid models are designed by utilizing the advantage of LSSVM and kinetics model. Through selecting the appropriate state variables and adopting these methods for penicillin fermentation, the mycelial concentration can be pre-estimated. Experiment results show that these hybrid modeling methods not only improve the above problem, but also have higher predicting accuracy and more powerful generalization ability than the single LSSVM method. © 2009 The Institution of Chemical Engineers. Published by Elsevier B.V. All rights reserved. Keywords: Least square support vector machine; Kinetics model; Fermentation; Hybrid model
1.
Introduction
Penicillin fermentation process is a strong nonlinear, time varying and correlating process (Bailey and Ollis, 1986). To carry out further optimizing and control strategy, one ideal model must be gained firstly. However, only those parameters such as physical and chemical parameters can be measured on-line, and the other parameters, such as bio-parameters which are more complex and important, can not be measured on-line yet in actual fermentation process, further more, sampling data are often limited, these situation made the modeling and controlling more complexity (Zhongping and Feng, 2005; Zhihua et al., 2005). The traditional kinetics modeling reflects the mechanism of the process, which belongs to “white-box modeling”, but it has some errors between the traditional kinetics model’s predicting result and the real output (Naigong and Xiaogang, 2007). The soft-sensor methods for biomass on-line estimation in fermentation process based on support vector machine (SVM) becomes significant in theory and valuable in practice (Xuejin et al., 2006; Rui et al., 2004; Yi and Haiqing, 2006). Xuejin et al. (2006) and Rui et al. (2004) had built a class of model for penicillin fermentation process based on SVM, which belongs to a “black-box moedling”, but the calcu-
lated speed was very slower with the increasing of sampling data, this method could not used for on-line pre-estimating. Yi and Haiqing (2006) adopted least squares support vector machines (LSSVM) modeling for a penicillin fed-batch fermentation, which improved the calculated speed. However, the problem of this method is that the pre-estimation result is not ideal in the initial stages of the fermentation process. In order to solve the problem, two hybrid modeling methods are designed by selecting the appropriate state variables and utilizing mechanism of fermentation and advantages of LSSVM (such as, higher generalization ability, faster calculated speed, etc.). Adopting the hybrid methods for a penicillin fermentation modeling, the mycelial concentration could be pre-estimated by using limited on-line data. Some results show that the hybrid modeling methods have higher predicting accuracy and more powerful generalization ability than the single LSSVM method. This paper is organized as follows: Section 2 describes the principles of SVM and LSSVM. The kinetics model of fermentation process is presented in Section 3. Section 4 introduces a fermentation process modeling based on LSSVM. Two hybrid models are designed in Section 5. Experiment results are given in Section 6. Finally, conclusions are given in Section 7.
∗ Corresponding author at: School of Communications and Control Engineering, Jiangnan University, 1800 Lihu Rd, Wuxi, Jiangsu 214122, PR China. Tel.: +86 13961868919. E-mail address: [email protected] (X. Wang). Received 29 April 2008; Received in revised form 8 August 2009; Accepted 20 August 2009 0263-8762/$ – see front matter © 2009 The Institution of Chemical Engineers. Published by Elsevier B.V. All rights reserved. doi:10.1016/j.cherd.2009.08.010
416
chemical engineering research and design 8 8 ( 2 0 1 0 ) 415–420
where is the regularization parameter, k is the error between the real output and estimated value at the k th sample point. This problem can be solved by using the optimization theory. The Lagrangian function for this problem can be define as follows:
Nomenclature X V t P p
biomass concentration (g/l) specific growth rate (h−1 ) culture volume (l) current fermentation time (h) penicillin concentration (g/l) the maximum specific penicillin production rate (h−1 ) S substrate concentration (g/l) Kp inhibition constant (g/h) K penicillin hydrolysis constant (h−1 ) yield coefficient (g biomass/g substrate) Yx/s Yp/s yield coefficient (g penicillin/g substrate) saturation constant (g/l) Kx m maintenance coefficient on substrate (h−1 ) Ki inhibition constant for product information (g/l) F feed flow rate of substrate (l/h) (x, y) the original values of a sampling point (xscale , yscale ) the normalization values of a sampling point xmin , xmax , ymin and ymax the minimum and maximum values in the original dataset
L(ω, b, ; ˛) = J(ω, ) −
N
˛k [ωT (xk ) + b + k − yk ]
(4)
k=1
where ˛k are Lagrange multipliers, which can be due to the equality constraints as follows from the Kuhn–Tucker conditions (Fletcher, 1987). The conditions for optimality
⎧ ⎪ ∂L ⎪ ⎪ =0 ⎪ ⎪ ∂ω ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎨ ∂L =0
∂b
⇒
ω=
N
˛k (xk )
k=1
⇒
ω=
N
˛k = 0
⎪ k=1 ⎪ ⎪ ⎪ ∂L ⎪ ⎪ = 0 ⇒ ˛k = k , k = 1, . . . , N ⎪ ⎪ ∂k ⎪ ⎪ ⎪ ⎪ ⎩ ∂L = 0 ⇒ ωT (xk ) + b + k − yk = 0, k = 1, . . . , N
(5)
∂˛k
2.
Basic principles of SVM and LSSVM
The solution is also given by
SVM is an excellent method aiming to finite data points based on statistical learning theory (SLT) (Vapnik, 1998, 1999) advanced by Vapnik in 1990s. It adopts the structure risk minimization (SRM) principle and avoids the complex computing kernel function of low dimensions instead of dot-matrix of high dimensions space. The main idea of SVM is described as follows. Given a training data set of N points {xk , yk }N with the k=1 input data xk ∈ Rn and the corresponding target yk ∈ Rm . In feature space SVM models take the form y(x) = ωT (x) + b
(1)
where the nonlinear function (·) : Rn → Rm maps the input space into higher character dimension, which is not explicitly constructed (Vapnik et al., 1997; Suykens and Vandewalle, 1999), b is the bias and ω is a weight vector of the same dimension as the feature space. The standard SVM is solved by quadratic programming methods, however these methods are often time consuming and are difficult to implement adaptively, and suffer from the problem of large memory requirement and CPU time when trained in batch model (Kuh, 2002). LSSVM is a modified version of SVM (Suykens and Vandewalle, 1999), which utilizes the equality constraints to replace the original convex quadratic programming problem. In LSSVM for function estimate, the following optimization problem is considered 1 T 1 2 k , ω ω+ 2 2 N
minJ(ω, ) = ω,b,
>0
(2)
k=1
subject to the equality constraints yk = ωT · (xk ) + b + k ,
k = 1, . . . , N
(3)
a b
=
0 − →T I
− →T I ˝
+ −1 I
0 y
(6)
− → where y = [y1 , y2 , . . . , yn ]T , I = [1, . . . , 1]T , ˛ = [˛1 , . . . , ˛N ]T , ˝ is a square matrix, the elements of its k row l column T are ˝kl = (xk ) (xl ) = K(xk , xl ), K(·, ·) is kernel function, which mapped the input vector into a high-dimension feature space (Fletcher, 1987). The most usual kernel functions are polynomial, Gaussian-like or some particular sigmoids (Chi and Ersoy, 2003; Qiong et al., 2007). Hence, the Eq. (1) is rebuilt by solving the linear set of Eqs. (5) and (6) instead of quadratic programming. The resulting LSSVM model for function estimation becomes:
y(x) =
N
˛k K(x, xk ) + b
(7)
k=1
where ˛, b are the solutions of Eq. (6). Some detail can be found in Suykens and Vandewalle (1999). According to the above description, it can be found easily that the difference between LSSVM and SVM is that they choose different lost functions and restraining conditions in optimization problems when using the SRM principle. The standard SVM chooses error i and inequality constraints, while LSSVM chooses the square of error i and equality constraints. Because of above characteristics, LSSVM not only accelerates the solve speed and reduce the training time, but also expediently ascertains parameters and reduces the complexity of the algorithm greatly, therefore, LSSVM is suitable for online modeling than SVM (Morris et al., 1989; Cristianini, 2005).
417
chemical engineering research and design 8 8 ( 2 0 1 0 ) 415–420
3.
Kinetics model of fermentation process
Hypothesis the process is a penicillin fermentation process, its kinetics model can be expressed with formulas from (8) to (11), which was built by Bajpai and Reuss (1980): dX X dV = X − dt V dt
(8)
dP SX P dV = p − KP − dt V dt Kp + S(1 + S/Ki )
(9)
−
p SX SX S dV dS · · + mX + = + dt Yx/s Kx X + S Yp/s Kp + S(1 + S/Ki ) V dt (10)
dV =F dt
(11)
where X is mycelial concentration, is specific growth rate, V is culture volume, t is current fermentation time, P is penicillin concentration, p is the maximum specific penicillin production rate, S is substrate concentration, Kp is the inhibition constant, K is the penicillin hydrolysis constant, Yx/s is the yield coefficient, Yp/s is the yield coefficient, Kx is the substrate saturation constant, m is the maintenance coefficient, Ki is the inhibition constant for product information and F is the feed flow rate of substrate. More detail expressions can be found in Ahmad et al. (2003) or in the Nomenclature. In fact, Eqs. (8)–(11) only provide part of the mechanism knowledge of the process and the kinetics character is mainly contained in specific growth rate (t), here = (t), which is a complex biology and physical chemistry function that could be described as Eq. (12) (James et al., 2002; Ahmad et al., 2003): (t) =
max S(t) KIP · Ks + S(t) KIP + P(t)
(12)
where max is the maximum specific growth rate, Ks is the saturation constant, KIP is the inhibition constant of product. Solve the differential equation (8) and disperse the result, the following equation can be got X(t + t) = X(t) exp
−
F V
t
Fig. 1 – Basic block diagram of penicillin fermentation process based on LSSVM. When establishing soft measurement model, the 2nd group variables are chosen from the input variables U as the input variables of the LSSVM model, and unmeasured variables are chosen as output Y of the soft model. Then use Eqs. (7) and (8) to achieve the non-linear relation between the input variables U and the output variables Y.
4.1.
Determination of input and output variables
During the process of building a biomass concentration model for penicillin fermentation, we choose some higher effective variables as input variables of LSSVM, such as dissolved oxygen concentration DO(t), mycelial concentration X(t), penicillin concentration P(t), substrate concentration S(t) and fermentation time T. As a matter of convenience, we only select the unable online measured variables mycelial concentration as the output Y of the model, that is Y(t) = X(t + 1).
4.2.
Preparation of training dataset
There are 12 batches data, each batch includs 150 sample points, these batches data are divided randomly into two subsets, one used exclusively for training and the other exclusively for testing. All the input variables are rescaled to be included within the interval [−1, 1] by using the following equations: xscale = 2
x − xmin −1 xmax − xmin
(14)
yscale = 2
y − ymin −1 ymax − ymin
(15)
(13)
where t is the sample time, its unit is hour. F, V are approximately constants due to V is much larger than F. If is solved, the result of X(t + t) can be obtained.
4. Fermentation process modeling based on LSSVM
where (x, y), (xscale , yscale ) are the old and new value of a sampling point respectively. xmin , xmax , ymin and ymax are the minimum and maximum values of that in the original dataset.
4.3. LSSVM has a high generalization, more efficient than the standard SVM, which has strong generalization and is suitable for modeling with limited data. The basic structure of modeling for penicillin fermentation based on LSSVM as shows in Fig. 1, the online state values (the enable measured online variables) usually include the pH value, dissolved oxygen (DO) concentration and CO2 concentration, temperature, revolution per minute (RPM) and other variables. The input variables u include air flow, hot water flow speed, cold water flow speed, output variables include CO2 concentration, reaction quantity of heat, etc. The offline state values (unable measured online variables) include biomass concentration, penicillin concentration, substrate concentration, and the other variables (Zhongping and Feng, 2005).
LSSVM training
In this study, we take commonly radial basis function (RBF) as the kernels function of LSSVM
K(xk , xl ) = exp
−
||x − xk ||2 2
(16)
It should be noted that predetermined parameters in LSSVM algorithms with RBF kernel are the punishment coefficient and the RBF coefficient . the training aim is to determine the two parameters. Here, we adopt the MSE (mean square error) as the parameters about training quality. At the end of the training, these parameters would be obtained: = 15, 500, = 1.50.
418
chemical engineering research and design 8 8 ( 2 0 1 0 ) 415–420
Fig. 2 – Predicting error of biomass concentration.
Fig. 3 – The block modeling diagram with serial hybrid LSSVM.
4.4.
Testing mycelial concentration
Define the predicting error e(t) e(t) = yp (t) − y(t)
(17)
where the y(t) is the real data, yp (t) is the predicting result of mycelial concentration at time t. Use one leave batch data to
Fig. 6 – The prediction result of mycelial with SPHLSSVM. test, the predicting error of mycelial concentration is shown in Fig. 2. Fig. 2 shows that the predicting results of mycelial concentration are not very satisfied with actual process near 50 h, while the others are better. To solve this problem, it is necessary to improve the above modeling method.
Fig. 4 – The block modeling diagram with serial and parallel hybrid LSSVM.
4.5. Comparison of modeling between LSSVM and RBFNN In order to illuminate the advantage of the modeling based on LSSVM, we adopt radial basis function neural network (RBFNN) (Thompson and Kramer, 1994) to model for the fermentation process, the input and output variables are set same as Section 4.1, the largest neuron number of hidden layer is 35, mean square error E is 0.0001, the distribution of radial basis function density is 0.5, change the number of training and testing samples, some performance indicators are presented in Table 1.
Table 1 – Comparison of MSE between LSSVM and RBFNN. Training number
Fig. 5 – The prediction result of mycelial with SHLSSVM model.
50 70 100 200
Testing number 200 200 200 200
LSSVM testing MSE 0.0588 0.0356 0.0095 0.0026
RBFNN testing MSE 0.3170 0.1683 0.1070 0.0027
419
chemical engineering research and design 8 8 ( 2 0 1 0 ) 415–420
Fig. 7 – The comparison of predicting error for mycelial among SHLSSVM, SPHLSSVM and LSSVM.
From Table 1, it can be easily found that the testing MSE of RBFNN is bigger than the LSSVM when the training number is less than 70, which indicates that the LSSVM is maintain relatively high prediction performance in the case of small samples. With the increasing of the training samples, the MSE of two methods are closed gradually. This phenomenon illuminates that LSSVM is less dependent on the size of the training dataset and has better generalization ability than that of the RBF neural network.
5.
Hybrid modeling
Modeling only with the mechanism of fermentation process is very difficult, because of the process complexity and the sample data often limited. Modeling only based on LSSVM is belong to “black-box modeling”, which only focus the input and output data of the process, and need not think of the material structure. This section has presented two hybrid modeling method by utilizing the advantages of LSSVM model and kinetics model to improve the shortcoming of single modeling. One method is showed in Fig. 3. At first, the middle variable is predicted by Eq. (7) of LSSVM in Section 4, i.e. = y(x), then the output mycelial concentration X(t + 1) is calculated by the Eq. (13). This method is called serial hybrid LSSVM (SHLSSVM). Fig. 4 shows the other method, which is same as Morris et al., 1989. is estimated by prior knowledge expression Eqs. (8)–(11) and LSSVM model (7), the output X(t + 1) can be solved by the following kinetics model (13). This method is called serial-parallel hybrid LSSVM (SPHLSSVM).
6.
Experiment results
This section focuses on the growth period of penicillin fermentation process, i.e. the first 50 h, adopts the above two hybrid methods in Section 5, the input variables, training method and testing method are chosen as same as in Section 4.1, the output variables is , some common parameters of the penicillin fermentation kinetics model in (8)–(12) are set as follows according to Ahmad et al. (2003): p = 0.005 h−1 , Kx = 0.15 g/l, Kp = 0.0002 g/h, Ki = 0.010 g/l, Yx/s = 0.45, Yp/s = 0.90, m = 0.014 h−1 , K = 0.04 h−1 . When adopting SPHLSSVM, some parameters in Eq. (12) are set as max = 0.092 h−1 , Ks = 508, 583 g/l, KIP = 7.5721 g/l. After the training, the two parameter of LSSVM can be obtained, = 15, 000, = 1.51 (SHLSSVM); = 16, 000, = 1.55 (SPHLSSVM). Predicting results are shown in Figs. 5 and 6.
Table 2 – Comparison of predicting quality for fermentation growth period with different methods. Method of models
SHLSSVM
SPHLSSVM
LSSVM
MSE
0.0002265
0.0002947
0.0041
From Figs. 5 and 6, it can be seen clearly that the SHLSVM model and SPHLSSVM model are better in the initial stages of the fermentation process, which illustrates that the two methods improved the shortcoming of single method with LSSVM availably. The predicting comparison in the growth period of SPHLSSVM, SHLSVM and LSSVM are shown in the following Fig. 7 and Table 2. From Fig. 7, it shows clearly that the predicting error of SPHLSSVM is the same as SHLSSVM in 0–22 h, which is nearly 0, and the estimated value of LSSVM is less than real data. The predicting error of three methods are steadily kept nearly 0 in 23–28 h. From 29 to 50 h, the predicting error of LSSVM is largest than the others. These results illustrate that the two hybrid models are efficiency for modeling of fermentation process. The parameter mean square error (MSE) is one important parameter of predicting quality. Table 2 shows the comparison of MSE during fermentation process with three different methods. Under the same conditions, the MSE value of SHLSSVM is the smallest, the biggest MSE is the LSSVM method (up to 0.0041, and is about 18 times of SHLSSVM). These results illustrate that the predicting quality of SHLSSVM is better than others, and the predicting accuracy of hybrid model is much better than the single model method (LSSVM modeling).
7.
Conclusions
This paper has introduced the principles of SVM and LSSVM firstly, then the kinetics model of penicillin fermentation process is presented briefly. According to the problem that the pre-estimation result of bio-parameters base on LSSVM is not ideal in the initial stages of the fermentation process, two hybrid predicting models have been built. These hybrid modeling methods have utilized the strong generalization advantages of LSSVM and the mechanism of kinetics model, and have been applied to develop a soft sensing model for a penicillin fermentation process. Experiment results show that the hybrid models are very effective in fermentation process.
420
chemical engineering research and design 8 8 ( 2 0 1 0 ) 415–420
Acknowledgment This work is supported by the 863 project of China (no. 2006AA020301). Therefore, it is necessary for the stability conditions to be investigated in the multi-regions.
References Ahmad, A., Samad, N.A.F.A. and Wei, C.A., 2003, Mathematical modeling and analysis of dynamic behavior of a fed-batch Penicillin G fermentation process, In Proceedings of International Conference on Chemical and Bioprocess Engineering , pp. 387–394. Bailey, J. and Ollis, D., (1986). Biochemical Fundamental. ((McGraw-Hill Inc., New York)), pp. 1–30 Bajpai, R. and Reuss, M., 1980, A mechanistic model for penicillin production. Journal of Chemical Technology and Biotechnology, 30: 330–344. Chi, H. and Ersoy, O.K., 2003, Recursive update algorithm for least squares support vector machines. Neural Processing Letters, 17: 165–173. Cristianini, N., 2005, An Introduction to Support Vector Machines, in John Shawe-Taylor (China Machine Press), pp. 1–189. Fletcher, R., (1987). Practical Methods of Optimization. ((John Wiley and Sons, Chichester/New York)). James, S., Legge, R. and Budman, H., 2002, Comparative study of black-box and hybrid estimation methods in fed-batch fermentation. Journal of Process Control, 1(1): 113–121. Kuh, A., 2002, Adaptive least square kernel algorithms and applications, In Proceedings of International Joint Conference on Neural Networks , pp. 2104–2107. Morris, A.J., Montague, G.A., Tham, M.T., 1989, Soft-sensors in industrial process control, in Applied Developments in Process Control (London, UK), pp. 1–3. Naigong, Y. and Xiaogang, R., 2007, Penicillin fermentation process optimal control strategy based on cellular automata
model, Journal of Beijing University of Technology, (02): 150–154 (in Chinese). Qiong, W., Wenying, L. and Yihan., 2007, Time series online prediction algorithm based on least squares support vector machine. Journal of Central South University of Technology, 14(3): 442–449. Rui, F., YueJie, Z., Yanzhu, Z., et al., 2004, Drifting modeling method using weighted support vector machines with application to soft sensor, Acta Automatica Sinca, 3(30): 436–441. Suykens, J.A.K. and Vandewalle, J., 1999, Least squares support vector machine classifiers. Neural Processing Letters, 9(3): 293–300. Thompson, M.L. and Kramer, M.A., 1994, Modeling chemical processes using prior knowledge and neural networks. AIChE Journal, 40(8): 1328–1340. Vapnik, V.N., (1998). Statistical Learning Theory. ((Wiley, New York)). Vapnik, V.N., (1999). The Nature of Statistical Learning Theory (2nd ed.). ((Springer-Verlag, New York)). Vapnik, V., Golowich, S. and Smola, A., 1997, Support vector machine for function approximation, regression estimation, and signal processing, in Neural Information Processing Systems, vol. 9, Mozer, M. and Petsche, T. (eds) (MIT Press), pp. 1–200. Xuejin, G., Pu, W., Chongzheng, S., et al., 2006, Modeling for penicillin fermentation process based on support vector machine, Journal of System Simulation, (07): 2052–2055 (in Chinese). Yi, L. and Haiqing, W., 2006, Modeling a penicillin fed-batch fermentation using least squares support vector machines, Chinese Journal of Biotechnology, (01): 144–148 (in Chinese). Zhihua, X., Guohong, H. and Huihe, S., 2005, On-line estimation of concentration parameters in fermentation process. Journal of Zhejiang University Science, 6B(6): 530–534 (in Chinese) Zhongping, S. and Feng, P., (2005). Fermentation Process Resolution, Control and Detection. ((Chemical Industry Press, Beijing)), pp. 1–10 (in Chinese)