- Email: [email protected]
Hybrid modeling for the prediction of leaching rate in leaching process based on negative correlation learning bagging ensemble algorithm
Computers and Chemical Engineering 35 (2011) 2611–2617
Contents lists available at ScienceDirect
Computers and Chemical Engineering journal homepage: www.elsevier.com/locate/compchemeng
Hybrid modeling for the prediction of leaching rate in leaching process based on negative correlation learning bagging ensemble algorithm Guanghao Hu ∗ , Zhizhong Mao, Dakuo He, Fei Yang School of Information Science & Engineering, Northeastern University, 11 WenHua Road, HePing District, Shenyang, 11004, PR China
a r t i c l e
i n f o
Article history: Received 20 August 2010 Received in revised form 24 January 2011 Accepted 11 February 2011 Available online 21 February 2011 Keywords: Leaching process Hybrid model Negative correlation learning Support vector regression Simulation
a b s t r a c t For predicting the leaching rate in hydrometallurgical process, it is very necessary to use an accurate mathematical model in leaching process. In this paper, a mechanism model is proposed for description and analysis of heat-stirring-acid leaching process. Due to some modeling errors existed between mechanism model and actual system, a hybrid model composed of mechanism model and error compensation model is established. A new support vector regression (SVR) bagging ensemble algorithm based on negative correlation learning (NCL) is investigated for solving the problem of error compensation. The sample of the next component learner is rebuilt continuously with this algorithm to improve the ensemble errors, and the optimum ensemble result also can be obtained. Simulation results indicate that the proposed hybrid model with the new algorithm has a better prediction performance in leaching process than other models. © 2011 Elsevier Ltd. All rights reserved.
1. Introduction Leaching process always play an important role in hydrometallurgical processes as the central unit operations. To reach further optimization and control strategy, an ideal model must be designed firstly. The leaching reaction with acid has been investigated in recent years (Crundwell, 2000; Liu, 2005; Ma & Ek, 1991; Schapire, 1990; Veglio, Trifoni, & Toro, 2001). A kinetic study of manganiferous ore leaching in acid media by glucose as the reducing agent also is reported (Veglio et al., 2001). A general model of the bacterial leaching process of ferrous and ferric ore is formulated (Crundwell, 2000). Several sulfuric acid leaching tests of manganese carbonate ore are carried out (Ma & Ek, 1991). The overall mass balance model, the population-balance model and the segregated-flow model in acid media are all reported in Breed and Hansford (1999) and Crundwell (1995). In this paper, the acid leaching of cobalt compound ore which takes sulfur dioxide as the reducing agent is studied. A mathematical mechanism model for leaching process is proposed via analysis of the relationship between the leaching rate and controlled variables. Most parameters of the model are estimated by nonlinear regression of experimental data. Because the mechanism model is based on some assumptions, there are some modeling errors between the mechanism model and actual system. Both two major approaches in modeling process are mechanism approach and data approach. Generally, there
∗ Corresponding author. Tel.: +86 24 83687758; fax: +86 24 83687758. E-mail address: [email protected] (G. Hu). 0098-1354/$ – see front matter © 2011 Elsevier Ltd. All rights reserved. doi:10.1016/j.compchemeng.2011.02.012
are some limitations in their individual use in practice. On the other hand, due to the lack of process data and integrated process knowledge, a completely pure data model maybe not very useful. Further, it needs to directly measure some process internal variables, which cannot be obtained with a data model. The best compromise between these approaches is to use a hybrid model consisting of part mechanism model and data model. The most knowledge of the process can go to the mechanistic part to develop accurate models for the process while provide sufficient access to internal variables of the process. Data model can be developed for those parts of the process which are hardly formulated or lead to very complicated models. In our work, we have developed a hybrid modeling approach combined with a dynamical mathematical mechanism model, which represented by a set of nonlinear ordinary differential equations with an error compensation model. An ensemble is a multi-learner system in which a series of component learners are generated to solve the same task. The component learners are combined with a certain fusion strategy to form the final prediction (Kittler, Hatef, Duin, & Matas, 1998; Wu, He, Man, & Arribas, 2004). In other words, the ensemble fuses the knowledge acquired by local learners, and make a consensus decision which is supposed to be superior to the one attained by individual learner work. The most popular ensemble algorithms are boosting (Schapire, 1990) and bagging (Breiman, 1996), which have already been widely used to improve accuracy for solving classification and regression problems (Opitz & Maclin, 1999). Bagging technology as an effective multi-learner method has received many important achievements. Bagging introduces the bootstrap sampling technique (Efron & Tibshirani, 1993) into the procedure of
2612
G. Hu et al. / Computers and Chemical Engineering 35 (2011) 2611–2617
constructing component learners, and expects to generate enough independent variance among them (Breiman, 1996). Support vector regression (SVR) is a type of learning machines that is paid wide attention in recent years (Vapnik, Golowich, & Smola, 1996). Based on statistical learning theory, SVR possesses many merits such as concise mathematical form, standard fast training algorithm and excellent generalization performance, so it has been widely applied in data mining problems such as pattern recognition, function estimation and time series prediction. If the ensemble learning technology can be introduced to SVR, the generalization performance of SVR may be improved efficiently. Therefore, research on SVR ensemble learning becomes an important research issue. Theoretical research proves that the more differences between the individual learners are, the better effect ensemble is. It is not any significant for integration algorithm that generating a batch of similar individual learners (Zhang & Xu, 2003). Recently, Liu and Yao (Liu & Yao, 1999a; Liu, Yao, & Higuchi, 2000) proposed negative correlation learning (NCL) algorithm that simultaneously trained neural networks (NNs) in the ensemble. While bagging and boosting explicitly create different training sets for different NNs by probabilistic change of the original training data distribution, negative correlation learning implicitly creates different training sets by encouraging different NNs to learn different parts or aspects of the training data. Leaching process includes reactions of ore with other procedures, so a whole leaching process always needs more than 30 h. Leaching rate calculated by chemical experiment is not only difficult, but also the cost is high, and the cycle is long. For these reasons, the collected data of leaching process is rare, the small sample modeling problem is difficult to regress using ordinary regression modeling methods. Due to various factors, such as hypothesis, result in that it is difficult to realize the prediction with high precision in mechanism models, the purpose of improving precision cannot be achieved completely only through adjusting the parameters of mechanism model (sometimes, the structure of mechanism model also has some flaws). In this paper, we have presented a new SVR bagging ensemble algorithm which adopts negative correlation learning method to build error compensation model. Take advantage of data model learning mechanism model error, and then compensate the mechanism model to set up a mixture model. In the iteration process, the method rebuilds sample of the next component learner continuously based on NCL algorithm to revise the ensemble errors before. The idea behind bagged ensembles of SVMs based on negative correlation learning is to encourage different individual SVMs in an ensemble to learn different parts or aspects of a training data so that the ensemble can learn the whole training data better. The leaching process prediction system which concludes the mathematical model and the improved algorithm is applied in advanced factory.
Fig. 1. Field device of heat-stirring-acid leaching process.
lower-priced ore is reacted with sulfuric acid to generate sulfate, and higher-priced ore is reacted with sulfur dioxide to generate sulfate. 2.2. Mechanism model of leaching process Before modeling some assumptions for the leaching process is proposed as follows: (1) (2) (3) (4)
The solution is stirred uniformly in leaching tank. The temperature of solution is uniform in leaching tank. Reaction is isothermal. The average specific heat of mixtures has nothing to do with the temperature, and does not change along with composition of reactant. (5) The sulfur dioxide injected into solution is reacted completely. (6) Classifying ore as two parts, one part is higher-priced ore, and the other is lower-priced ore.
In this section, the kinetic study of the leaching process is researched. In the leaching process, the reaction rate is supposed to be controlled by the chemical reaction (Levenspiel, 1972; Tong, 2005; Zhan, 2006), then: −dN = kSC dt
where N is the number of moles of unleached ore at time t, N = (4/3)rk 3 /M, t is the reaction time, S is the surface area of a ore particle, S = 4rk 2 , k is a rate constant, C is the concentration of acid, rk is the radius of ore particle, is the ore density, and M is the average molar mass of ore, then:
2. Mathematical mechanism model of the leaching process − 2.1. Brief introduction of leaching process Leaching is defined as removal of ore by dissolving them from the solid matrix. The reagent is injected into a leaching tank and dissolves ore to separate the valuable components and impurities. In our research, cobalt compound is prepared in the hydrometallurgical factory, and the reagents are sulfuric acid and sulfur dioxide. The field device is shown in Fig. 1. The temperature, pH value, flow rate of sulfur dioxide and flow rate of sulfuric acid all can be monitored online. The whole process is a batch process. The ore is slurried in the slurry tank and transported to the leaching tank. And then, the ore slurry is stirred and heated, and reacted with the sulfuric acid and sulfur dioxide in the leaching tank. In the reaction,
(1)
drk kMC = dt
(2)
Because the radius of ore particle is not convenient to be detected, the kinetic equation is usually expressed by the relationship between leaching rate y and reaction time t. N0 is supposed to be the number of moles of unleached ore at beginning, then: y=
N0 − N r 3 =1− k 3 N0 rk0
(3)
where rk0 is the initial radius of ore particles. Substituting Eq. (3) into Eq. (2) may result in: dy 3kCM = (1 − y)2/3 rk0 dt
(4)
G. Hu et al. / Computers and Chemical Engineering 35 (2011) 2611–2617
According to Arrhenius equation, k is defined as: k = Ae−E/RT
(5)
where A is the preexponential factor, E is the activation energy, R is the gas constant, and T is the absolute temperature. Basing on the kinetic analysis on the leaching process above, combining with the actual field production, the material balance equation of leaching process is established as followed: Lower-priced ore :
3ACH2 SO4 M −E /RT dy1 · (1 − y1 )2/3 = ·e 1 rk0 dt
(6)
Higher-priced ore :
3ACH2 SO3 M −E /RT dy2 · (1 − y2 )2/3 = ·e 2 rk0 dt
(7)
Total leaching rate :
y = y1 + ϕy2
(8)
where y1 is the leaching rate of lower-priced ore, y2 is the leaching rate of higher-priced ore, CH2 SO4 is the concentration of sulfuric acid, CH2 SO3 is the concentration of sulfurous acid, T is the temperature of leaching solution, y is the total leaching rate, is the ratio of lower-priced ore in reactant, and ϕ is the ratio of higher-priced ore in reactant. In the reaction of leaching process, the concentration of sulfuric acid and the concentration of sulfurous acid are changing. The change of the concentration of sulfuric acid is dependent on two factors: one is the consumption of sulfuric acid during the reaction; the other is the supplement of sulfuric acid during the reaction. Considering these two factors, the material balance of sulfuric acid can be formulated as the following differential equation: CH2 SO4 · QH2 SO4 · H2 SO4 /MH2 SO4 − K1 · (dy1 /dt) · ˛ dy3 = V dt
(9)
where K1 = · G/M, QH2 SO4 is the flow rate of sulfuric acid, H2 SO4 is the density of sulfuric acid, MH2 SO4 is the molar mass of sulfuric acid, is the content of lower-priced ore, G is the weight of ore, ˛ is the consumption of number of moles of sulfuric acid when 1 mol lower-priced ore is leached, and V is the volume of leaching tank. The change of concentration of sulfurous acid is dependent on four factors: consumption of sulfurous acid during the reaction, flow rate of sulfur dioxide, solubility of sulfur dioxide, contact area between sulfur dioxide and leaching solution. The dissolution rate (Parker, 1978) and the contact area between sulfur dioxide and leaching solution (Moore, 1981) can be defined as: Dissolution rate : Contact area :
vg =
Sg =
Dg (CSO2 − CH2 SO3 ) ıg
3QSO2 H
vr
(10)
2.3. Parameters of mechanism model Considering the multitudinousness of the parameters in mechanism model, an accurate setting of the parameters is important to the application of the model. The main parameters of the mechanism model are tested by nonlinear regression of experimental data. 2.3.1. Known parameters in field V = 12,000 dm3 , H = 500 cm, M = 73.31 g/mol, = 8.9 g/cm3 , CH2 SO4 = 0.95, H2 SO4 = 1.84 kg/dm3 , MH2 SO4 = 98 g/mol, ˛ = 1, ˇ = 4.19, U = 27,509 W/(m2 K), a = 14.13 m2 , r = 0.5 cm. 2.3.2. Known parameters in the literature v = 10 cm/s, R = 8.314 J/(mol K) (Li, ıg = 0.74 cm (Moore, 1981).
1998),
Dg = 1.21 cm2 ,
2.3.3. Parameters obtained from experiment and identification 2.3.3.1. Parameters obtained from experiment. The solubility of sulfur dioxide is a variable parameter about temperature. The solubility of sulfur dioxide drops along with the ascension of temperature. The function of solubility of sulfur dioxide is expressed by the regression analyzing of the experimental data: Cp = 0.289 · 10−8 · T 4 − 0.434 · 10−5 · T 3 + 0.244 · 10−2 · T 2 − 0.607 · T + 56.5
(14)
Similarly, the average heat capacity of leaching solution is: c¯ = 2298.365 J/(kg K). 2.3.3.2. Parameters obtained from identification. Parameters A, E1 , and E2 are identified by PSO optimization (Prata, Schwaab, Lima, & Pinto, 2009) algorithm with the field actual data. The result of the identification is: A = 68.36, E1 = 6889 J/mol, and E2 = 66,529 J/mol. 3. Bagging ensemble of SVR based on NCL (negative SVR bagging, NSB) 3.1. SVR algorithm
(12)
where K2 = · G/M, is the content of higher-priced ore, and ˇ is the consumption of number of moles of sulfur dioxide when 1 mol higher-priced ore is leached. The leaching process is a heat-acid-stirring reaction. It is also an energy transfer process in which energy is delivered from the heating pipe in leaching tank to the leaching solution, and it can be expressed by the following equation: T −T dy5 =U·a· h m · c¯ dt
where U is the heat transfer coefficient of leaching tank, a is the heat transfer area of leaching tank, Th is the temperature of heating, m is the mass of leaching solution, and c¯ is the average specific heat of leaching solution. In summary, mechanism model of the leaching process may be expressed by Eqs. (6)–(9), (12) and (13).
(11)
where Dg is the diffusivity of sulfur dioxide in the water, CSO2 is the solubility of sulfur dioxide, ıg is the thickness of gas-liquid contact area, QSO2 is the flow rate of sulfur dioxide, H is the height of leaching tank, v is the rising velocity of bubble, r is the radius of bubble. Considering these factors, the material balance of sulfurous acid can be formulated as the following differential equation: vg · Sg − K2 · (dy2 /dt) · ˇ dy4 = V dt
2613
(13)
SVR is an adaptation of a recent statistical learning theory based classification paradigm, namely support vector machines (Vapnik et al., 1996). The SVR formulation follows structural risk minimization (SRM) principle, as opposed to the empirical risk minimization (ERM) approach which is commonly employed within statistical machine learning methods. The principal and methodology of SVM based regression technique is briefly provided below. Consider a general regression model that relates the predictors vi to the streamflow yi in year i: yi = f (vi ) + e
(15)
and denote by fˆ (v) the estimate of the regression function f (v). The function f (v) can be any nonlinear function and in the SVM literature it is typically considered to be estimated through the sum of a set of kernel functions. Using these kernel functions, the input vector vi is mapped into a new feature space in which linear regression
2614
G. Hu et al. / Computers and Chemical Engineering 35 (2011) 2611–2617
is performed rather than nonlinear regression. Given this mapping, the estimate of the regression model could be expressed as follows: fˆ (v) = ω, v + b
(16)
where ω represents the support vector weights (basis functions), angle brackets denote a dot product and b is a bias term, similar to an intercept in linear regression. The parameters of fˆ (v) are estimated by minimizing the following regularized risk function.
1 ||ω||2 + c ( i + i∗ ) 2 n
min ·
(17)
i=1
Subject to f (v) − ω, v − b ≤ ε + i ω, v + b − f (v) ≤ ε + i∗
(19)
where i and i∗ are slack variables that determine the degree to which state space samples with error more than ε be penalized; and ε is the degree to which one would like to tolerate errors in constructing the predictor fˆ (v). The above formulation is referred as the ε-insensitive approach. The optimization problem of last equation can be solved though changing into the dual problem by Lagrange optimization method, Lagrangian function is established according to target function and constraints:
1 L(ω, b, , , a, a , , ∗) = ωT ω + c ( j + i∗ ) − ai [ j 2 n
n
i=1
i=1
∗
n
+εi − yi + (ωϕ(xi ) + b)] −
a∗i [ j + ε∗i + yi − (ωϕ(xi ) + b)]
i=1
n
−
( i j + i i∗ )
(20)
i=1
where ai , a∗i , i , ∗i are Lagrange multipliers. Request partial derivative to b, ω, i∗ and order these partial derivatives to zero, respectively. And get: ∂L = ∂b
n
f (x) = (ω · x) + b =
n
(ai − a∗i )(xi · x) + b
(26)
i=1
where corresponding xi of ai − a∗i = / 0 is support vectors, and b can be obtained by Karush–Kuhn–Tucker (KKT) calculation. For a nonlinear problem, training data is mapped by kernel function K(xi , xj ) to a high-dimensional feature space, and then in this space a linear regression function is established, so the model of nonlinear regression function can be obtained according to the above derivation process:
(18)
i , i∗ ≥ 0
∗
Furthermore, the Lagrange multiplier ai , a∗i which maximize the above target function can be obtained. After the value ω was obtained from Eq. (22), the regression function:
f (x, ˛, ˛∗ ) =
P
(˛i − ˛∗i )K(x, xi ) + b
(27)
i=1
In this paper a commonly used kernel functions, namely, radial basis function (RBF) is used. It is defined below in Eq. (17):
K(xi , xj ) = exp
−||xi − xj ||2
2 2
(28)
3.2. The basic idea of bagging For bagging algorithm, each training subset contains M learning samples, drawn randomly with replacement from the original training set of size M. Such a training subset is called a bootstrap (Efron & Tibshirani, 1993) replicate of the original set. Each bootstrap replicate contains, on average, 63.2% of the original training set, with many examples appearing some times. Because of the randomness characteristic existing in the process of distilling the training subsets, bagging algorithm has the flaw of blindness to some extent. Let T ={(xp , yp ), p = 1, . . ., N} denotes a regression type training set, and the SVR algorithm uses T to construct a regression predictor FR (x,T) for future y values. Let F be, an bagging ensemble obtained as a simple averaging combination of M predictors, that is 1 fi (xp ) M M
F(xp ) =
(29)
i=1
(a∗i − ai ) = 0
(21)
∂L =ω− (a∗i − ai )ϕ(xi ) = 0 ∂ω
(22)
where M is the number of the individual SVR in the ensemble, fi (xp ) is the output of SVR ion the pth testing pattern, and F(xp ) is the output of the ensemble on the pth testing pattern. The generalization errors of fi and F on the training set T are expressed as follows:
(23)
Ei =
i=1 n
i=1
∂L = c − a∗i − ∗i = 0 ∂ i∗
L(a, a∗ ) = −ε
(ai + a∗i ) +
i=1
n
N
(ai − a∗i ) = 0 (0 ≤ ai , a∗i ≤ c,
(yp − F(xp ))2
(31)
1 Ei M M
(24)
E¯ =
(32)
i=1
The diversity between fi and F is defined as:
Constraint condition:
i=1
(30)
The simple averaging of Ei is defined as:
i=1
i,j=1
n
2
p=1
yi (a − a∗i )
1 (ai − a∗i )(aj − a∗j )(xi , xj ) 2
E=
n
−
(yp − fi (xp ))
p=1
Let Eqs. (21)–(23) substitute into Eq. (20), and then Eq. (20) is changed into: n
N
i = 1, . . . , n)
(25)
Ai =
N p=1
(fi (xp ) − F(xp ))
2
(33)
G. Hu et al. / Computers and Chemical Engineering 35 (2011) 2611–2617
2615
Fig. 2. Hybrid model structure.
The simple averaging of Ai is defined as:
To set the partial derivative as zero, the desired output is calculated by minimizing E as shown in the following equation:
1 A¯ = Ai M M
(34)
i=1
From the conclusion of Krogh (Krogh & Vedelsby, 1995), we get: E = E¯ − A¯
(35)
This result implies that increasing ensemble diversity while decreasing or maintaining the simple average generalization error of ensemble members, should lead to a decrease in generalization error. 3.3. Negative correlation learning Negative correlation learning introduces a correlation penalty term Ci into the error function of each individual SVR in the ensemble so that all the networks can be trained simultaneously and interactively on the same training data set T (Liu & Yao, 1999b). The correlation function Ci is defined as:
Ci = (fi (xp ) − F(xp ))
(fj (xp ) − F(xp )) = −(fi (xp ) − F(xp ))
2
(36)
j= / i
The error function Ei for SVR i in negative correlation learning is defined as:
Ei =
N 1 p=1
2
2
(yp − fi (xp )) + Ci
(37)
The parameter 0 ≤ ≤ 1 is used to adjust the strength of the penalty. The simple averaging of for the ensemble in negative correlation learning is defined as: 1 M M
Eˆ =
N
i=1 p=1
1 2
2
(yp − fi (xp )) + Ci
(40)
The minimization of the generalization error of the ensemble is achieved by replacing yp with f¯ of the individual SVR. The proposed NSB algorithm rebuilds sample of the next component learner continuously to revise the ensemble errors before, which makes the ensemble result best. The algorithm consists of two parts: training the individual SVR by NCL algorithm and combining the individual SVR by an advanced strategy. The proposed NSB algorithm is described as follows: Initialization parameters: the maximal number of individuals of ensemble is M, the maximal generalization error of individuals SVR is e1 , the maximal generalization error of ensemble is e2, , i = 1. Step 1. Generate training subsets T1 from T by using Bootstrap sampling algorithm (Efron & Tibshirani, 1993). Train an individual model f1 on the training subset T1 by using SVR algorithm. Step 2. Calculate the training error E1 of f1 by Eq. (30). If E1 < e1, set i = i + 1, otherwise repeat step 1. Step 3. Generate training subsets Ti from T by using Bootstrap sampling algorithm. Train an individual model fi on the training subset Ti by using SVR algorithm. Get the desired output f¯i using Eq. (29). Replace Ti with f¯i . Finally, retrain fi on the new training subset Ti by SVR algorithm. Step 4. Calculate the training error Ei of fi by Eq. (30). If Ei < e1 , set i = i + 1, otherwise repeat step 3. Step 5. Calculate the training error E of this ensemble by Eq. (31). If E < e2 , or i = M, output: F(xp ) =
1 M
M
fi (xp ) Otherwise repeat step
i=1
3.
(38)
The partial derivative of E with respect to the output of SVR i on the nth training pattern is: 1 ∂Eˆ = [(f (xp ) − yp ) − 2(fi (xp ) − F(xp ))] M i ∂fi
yp − 2F(xp ) f¯i = 1 − 2
(39)
4. Hybrid model based on NSB (NSBH) Based on NSB algorithm, a hybrid model which compensates the mechanism model with error compensation model is proposed in this paper. The foundational model structure of our approach is shown in Fig. 2. In this paper, the input variables of the hybrid model are determined. There may be some redundant information if we
2616
G. Hu et al. / Computers and Chemical Engineering 35 (2011) 2611–2617
take all the measurable variables for the training. The problem of “over-fitting” is always encountered due to the redundant information. It can be solved using both statistical analysis and preferable deep knowledge about the causal links in the particular process. Based on the latter, three on-line measured variables: heating temperature (u1 ), flow rate of concentrated sulfuric acid (u2 ) and flow rate of sulfur dioxide (u3 ) have been selected as inputs of the hybrid model. The output is leaching rate (y) in the hybrid model. The error compensation model is used to predict the p-step value in future. The predictive control algorithm can be formally described as follows:
y(k + P) = f (ˆy(k − 1), u(k − 1), u(k), u(k + 1), . . . , u(k + P − 1)) (41) where u = [u1 , u2 , u3 ], yˆ (k) is the error between the leaching rate predicted by mechanism model and the one tested by laboratory in the k-step. P is the key parameter of the hybrid model. With the increasing of the value of P, the prediction precision is improved and the model is more complex. With the decreasing of the value of P, the dynamic response rate is increased and the stability and the robustness of the model are worse. For the foregoing, we choose the value P = 2 in our model.
Fig. 4. Dynamic prediction results of MM model, BBH model, SVRH model and NSBH model for test 2.
5. Practical application and experiments 5.1. Leaching process prediction system The leaching process prediction system, which is based on the study of the proposed models and the improved algorithm, is developed and put into operation in an advanced factory. The prediction system is developed by C++ and runs on an Intel P4 2.0 GHz PC with 1 G RAM. 5.2. Experiment and analysis In the practical applications, process data always carry various measurement errors inevitably, because of measuring instrumentation precision, reliability and environment factors of field measurement. With low precise measurement data, the deviation of online optimum results maybe caused, even unsuccessful production. Therefore, the pretreatment of measurement data is very important to ensure the normal operation of online optimization. The main of this paper is that eliminate abnormal data, with the Pauta criteria of statistics discriminance method. Its principle is as follows: set sample date is X = (x1 , x2 , . . ., xn )T ,
Fig. 5. Prediction result of final leaching rate of MM model, BBH model, SVRH model and NSBH model.
¯ deviation is i = xi − X¯ (i = 1, 2, . . . , n). The average value is X, standard deviation is calculated according to the Bayesian formula:
=
n v2i i=1
1/2
(42)
n−1
If the deviation value vi of sample date xi corresponds with |vi | > 3 , the sample date is abnormal data, which should be deleted and then replaced by average value of samples. Experiments have been conducted 36 times to test the effect of varying values of three operating condition variables (u1 ,u2 ,u3 ) on leaching rate (y). After preprocessing and standardizing, a data set with 36 groups of data can be obtained and each group has 12 samples. The data set is divided into two sets averagely. One is used as training set, and the other is used as testing set. The number of individual SVR is 25 (Breiman, 1996). The main parameters of Table 1 Comparison of four models.
Fig. 3. Dynamic prediction results of MM model, BBH model, SVRH model and NSBH model for test 1.
MODEL
RMSE
MAXE
MM BBH SVRH NSBH
0.0153 0.0139 0.0156 0.0105
0.0312 0.0326 0.0258 0.0231
G. Hu et al. / Computers and Chemical Engineering 35 (2011) 2611–2617
2617
Table 2 Relative error of two tests. Relative error
RE(3) RE(4) RE(5) RE(6) RE(7) RE(8) RE(9) RE(10) RE(11) RE(12)
Test1
Test2
MM
BBH
SVRH
NSBH
MM
BBH
SVRH
NSBH
0.0667 0.1768 0.0487 0.1086 0.0864 0.0481 0.0048 0.0057 0.0091 0.0112
0.0415 0.0854 0.0598 0.0774 0.0412 0.0285 0.0311 0.0397 0.0305 0.0114
0.0877 0.1306 0.0051 0.0914 0.0580 0.0331 0.0377 0.0418 0.0315 0.0094
0.0198 0.0010 0.0155 0.0404 0.0082 0.0087 0.0193 0.0042 0.0069 0.0017
0.0762 0.0207 0.0722 0.1124 0.0386 0.0287 0.0495 0.0484 0.0228 0.0266
0.0844 0.0730 0.0574 0.0385 0.0527 0.0365 0.0153 0.0130 0.0246 0.0206
0.0945 0.0317 0.0527 0.0875 0.0825 0.0466 0.0669 0.0615 0.0142 0.0378
0.0399 0.0383 0.0357 0.0177 0.0183 0.0184 0.0199 0.0046 0.0100 0.0122
NSB algorithm is that: fitting error ε = 0.0126, c = 103, = 5.3, P = 2, e1 = 0.005, e2 = 0.001, = 0.2. Comprising with NSBH model, mechanism model (MM), basic bagging hybrid model (BBH) and single SVR hybrid model (SVRH) are also applied in this experiment. Two dynamic prediction results of these tests are shown in Figs. 3 and 4. Eighteen prediction results of final leaching rate are shown in Fig. 5. Eighteen independent experiments are carried out for each method and the average root mean squared error (RMSE) and maximal absolute error (MAXE) are shown in Table 1. The relative error of two tests is shown in Table 2. Figs. 3–5 show that the predicted precision of leaching rate based on the proposed NSBH model is higher than the other models and it has better generalization ability. Figs. 3 and 4 show that for the effect of predicted leaching rate following the actual value, the NSBH model is better than the other models. Table 1 shows that the NSBH model performs not only better than MM model but also better than BBH model and SVRH model in generalization ability. 6. Conclusions In this paper, a mathematical mechanism model of the heatstirring-acid leaching process has been developed, in which cobalt compound ores particles dissolve in acidic aqueous solution. Also, a hybrid model is established based on the proposed NSB algorithm. The simulation results show that the NSBH model does consistently improve the predicted precision versus MM model, BBH model and SVRH model for leaching process. Acknowledgements The authors are grateful for the financial support of the National High Technology Research and Development Program of China (863 Program, No. 2006AA060201). References Breed, A. W., & Hansford, G. S. (1999). Modeling continuous bioleach reactors. Biotechnology and Bioengineering, 64(6), 671–677. Breiman, L. (1996). Bagging predictors. Machine Learning, 24(2), 123–140.
Crundwell, F. K. (1995). Progress in the mathematical modelling of leaching reactors. Hydrometallurgy, 39(1–3), 321–335. Crundwell, F. K. (2000). Modeling, simulation, and optimization of bacterial leaching reactors. Biotechnology and Bioengineering, 71(4), 255–265. Efron, B., & Tibshirani, R. (1993). An introduction to the bootstrap. New York: Chapman & Hall. Kittler, J., Hatef, M., Duin, R. P. W., & Matas, J. (1998). On combining classifiers. IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(3), 226–239. Krogh, A., & Vedelsby, J. (1995). Neural network ensembles, cross validation, and active learning. In Advances in Neural Information Processing Systems 7: Proceedings conference Cambridge, MA, (pp. 231–238). Levenspiel, O. (1972). Chemical reaction engineering. New York: John Wiley& Sons. Li, H. (1998). Hydrometallurgy. Changsha: Central South University Press. Liu, Y. (2005). Generate different neural networks by negative correlation learning. Lecture Notes in Computer Science, 3610, 149–156. Liu, Y., & Yao, X. (1999a). Simultaneous training of negatively correlated neural networks in an ensemble. IEEE Transactions on Systems, Man, and Cybernetics B, 29(6), 716–725. Liu, Y., & Yao, X. (1999b). Ensemble learning via negative correlation. Neural Networks, 12(10), 1399–1404. Liu, Y., Yao, X., & Higuchi, T. (2000). Evolutionary ensembles with negative correlation learning. IEEE Transactions on Evolutionary Computation, 4(4), 380–387. Ma, Z., & Ek, C. (1991). Rate processes and mathematical modelling of the acid leaching of a manganese carbonate ore. Hydrometallurgy, 27(2), 125–139. Moore, J. J. (1981). Chemical metallurgy. London: Butterworths Press. Opitz, D., & Maclin, R. (1999). Popular ensemble methods: An empirical study. Journal of Artificial Intelligence Research, 11, 169–198. Parker, R. H. (1978). An introduction to chemical metallurgy. New York: Pergamon Press Ltd. Prata, D. M., Schwaab, M., Lima, E. L., & Pinto, J. C. (2009). Nonlinear dynamic data reconciliation and parameter estimation through particle swarm optimization: Application for an industrial polypropylene reactor. Chemical Engineering Science, 64(18), 3953–3967. Schapire, R. (1990). The strength of weak learnability. Machine Learning, 5(2), 197–227. Tong, Z. (2005). Leaching kinetics of calcium aluminate slag. The Chinese Journal of Process Engineering, 5(4), 399–402. Vapnik, V., Golowich, S. E., & Smola, A. G. (1996). Support vector method for function approximation, regression estimation and signal processing. In Proceedings of Neural Information Process System (NIPS’1996) London, UK, (pp. 281–287). Veglio, F., Trifoni, M., & Toro, L. (2001). Leaching of manganiferous ores by glucose in a sulfuric acid solution: Kinetic modeling and related statistical analysis. Industrial & Engineering Chemistry Research, 40(18), 3895–3901. Wu, Y., He, J., Man, Y., & Arribas, J. I. (2004). Neural network fusion strategies for identifying breast masses. 2004 IEEE International Joint Conference on Neural Networks, 3, 2437–2442. Zhan, S. (2006). Study on leaching kinetics of pyrite cinder. Chemical Engineering, 34(11), 36–39 (Chinese). Zhang, X. Y., & Xu, L. B. (2003). The stock market forecast model based on neural network ensemble. Systems Engineering—Theory & Practice, 9(9), 67–70.
Contents lists available at ScienceDirect
Computers and Chemical Engineering journal homepage: www.elsevier.com/locate/compchemeng
Hybrid modeling for the prediction of leaching rate in leaching process based on negative correlation learning bagging ensemble algorithm Guanghao Hu ∗ , Zhizhong Mao, Dakuo He, Fei Yang School of Information Science & Engineering, Northeastern University, 11 WenHua Road, HePing District, Shenyang, 11004, PR China
a r t i c l e
i n f o
Article history: Received 20 August 2010 Received in revised form 24 January 2011 Accepted 11 February 2011 Available online 21 February 2011 Keywords: Leaching process Hybrid model Negative correlation learning Support vector regression Simulation
a b s t r a c t For predicting the leaching rate in hydrometallurgical process, it is very necessary to use an accurate mathematical model in leaching process. In this paper, a mechanism model is proposed for description and analysis of heat-stirring-acid leaching process. Due to some modeling errors existed between mechanism model and actual system, a hybrid model composed of mechanism model and error compensation model is established. A new support vector regression (SVR) bagging ensemble algorithm based on negative correlation learning (NCL) is investigated for solving the problem of error compensation. The sample of the next component learner is rebuilt continuously with this algorithm to improve the ensemble errors, and the optimum ensemble result also can be obtained. Simulation results indicate that the proposed hybrid model with the new algorithm has a better prediction performance in leaching process than other models. © 2011 Elsevier Ltd. All rights reserved.
1. Introduction Leaching process always play an important role in hydrometallurgical processes as the central unit operations. To reach further optimization and control strategy, an ideal model must be designed firstly. The leaching reaction with acid has been investigated in recent years (Crundwell, 2000; Liu, 2005; Ma & Ek, 1991; Schapire, 1990; Veglio, Trifoni, & Toro, 2001). A kinetic study of manganiferous ore leaching in acid media by glucose as the reducing agent also is reported (Veglio et al., 2001). A general model of the bacterial leaching process of ferrous and ferric ore is formulated (Crundwell, 2000). Several sulfuric acid leaching tests of manganese carbonate ore are carried out (Ma & Ek, 1991). The overall mass balance model, the population-balance model and the segregated-flow model in acid media are all reported in Breed and Hansford (1999) and Crundwell (1995). In this paper, the acid leaching of cobalt compound ore which takes sulfur dioxide as the reducing agent is studied. A mathematical mechanism model for leaching process is proposed via analysis of the relationship between the leaching rate and controlled variables. Most parameters of the model are estimated by nonlinear regression of experimental data. Because the mechanism model is based on some assumptions, there are some modeling errors between the mechanism model and actual system. Both two major approaches in modeling process are mechanism approach and data approach. Generally, there
∗ Corresponding author. Tel.: +86 24 83687758; fax: +86 24 83687758. E-mail address: [email protected] (G. Hu). 0098-1354/$ – see front matter © 2011 Elsevier Ltd. All rights reserved. doi:10.1016/j.compchemeng.2011.02.012
are some limitations in their individual use in practice. On the other hand, due to the lack of process data and integrated process knowledge, a completely pure data model maybe not very useful. Further, it needs to directly measure some process internal variables, which cannot be obtained with a data model. The best compromise between these approaches is to use a hybrid model consisting of part mechanism model and data model. The most knowledge of the process can go to the mechanistic part to develop accurate models for the process while provide sufficient access to internal variables of the process. Data model can be developed for those parts of the process which are hardly formulated or lead to very complicated models. In our work, we have developed a hybrid modeling approach combined with a dynamical mathematical mechanism model, which represented by a set of nonlinear ordinary differential equations with an error compensation model. An ensemble is a multi-learner system in which a series of component learners are generated to solve the same task. The component learners are combined with a certain fusion strategy to form the final prediction (Kittler, Hatef, Duin, & Matas, 1998; Wu, He, Man, & Arribas, 2004). In other words, the ensemble fuses the knowledge acquired by local learners, and make a consensus decision which is supposed to be superior to the one attained by individual learner work. The most popular ensemble algorithms are boosting (Schapire, 1990) and bagging (Breiman, 1996), which have already been widely used to improve accuracy for solving classification and regression problems (Opitz & Maclin, 1999). Bagging technology as an effective multi-learner method has received many important achievements. Bagging introduces the bootstrap sampling technique (Efron & Tibshirani, 1993) into the procedure of
2612
G. Hu et al. / Computers and Chemical Engineering 35 (2011) 2611–2617
constructing component learners, and expects to generate enough independent variance among them (Breiman, 1996). Support vector regression (SVR) is a type of learning machines that is paid wide attention in recent years (Vapnik, Golowich, & Smola, 1996). Based on statistical learning theory, SVR possesses many merits such as concise mathematical form, standard fast training algorithm and excellent generalization performance, so it has been widely applied in data mining problems such as pattern recognition, function estimation and time series prediction. If the ensemble learning technology can be introduced to SVR, the generalization performance of SVR may be improved efficiently. Therefore, research on SVR ensemble learning becomes an important research issue. Theoretical research proves that the more differences between the individual learners are, the better effect ensemble is. It is not any significant for integration algorithm that generating a batch of similar individual learners (Zhang & Xu, 2003). Recently, Liu and Yao (Liu & Yao, 1999a; Liu, Yao, & Higuchi, 2000) proposed negative correlation learning (NCL) algorithm that simultaneously trained neural networks (NNs) in the ensemble. While bagging and boosting explicitly create different training sets for different NNs by probabilistic change of the original training data distribution, negative correlation learning implicitly creates different training sets by encouraging different NNs to learn different parts or aspects of the training data. Leaching process includes reactions of ore with other procedures, so a whole leaching process always needs more than 30 h. Leaching rate calculated by chemical experiment is not only difficult, but also the cost is high, and the cycle is long. For these reasons, the collected data of leaching process is rare, the small sample modeling problem is difficult to regress using ordinary regression modeling methods. Due to various factors, such as hypothesis, result in that it is difficult to realize the prediction with high precision in mechanism models, the purpose of improving precision cannot be achieved completely only through adjusting the parameters of mechanism model (sometimes, the structure of mechanism model also has some flaws). In this paper, we have presented a new SVR bagging ensemble algorithm which adopts negative correlation learning method to build error compensation model. Take advantage of data model learning mechanism model error, and then compensate the mechanism model to set up a mixture model. In the iteration process, the method rebuilds sample of the next component learner continuously based on NCL algorithm to revise the ensemble errors before. The idea behind bagged ensembles of SVMs based on negative correlation learning is to encourage different individual SVMs in an ensemble to learn different parts or aspects of a training data so that the ensemble can learn the whole training data better. The leaching process prediction system which concludes the mathematical model and the improved algorithm is applied in advanced factory.
Fig. 1. Field device of heat-stirring-acid leaching process.
lower-priced ore is reacted with sulfuric acid to generate sulfate, and higher-priced ore is reacted with sulfur dioxide to generate sulfate. 2.2. Mechanism model of leaching process Before modeling some assumptions for the leaching process is proposed as follows: (1) (2) (3) (4)
The solution is stirred uniformly in leaching tank. The temperature of solution is uniform in leaching tank. Reaction is isothermal. The average specific heat of mixtures has nothing to do with the temperature, and does not change along with composition of reactant. (5) The sulfur dioxide injected into solution is reacted completely. (6) Classifying ore as two parts, one part is higher-priced ore, and the other is lower-priced ore.
In this section, the kinetic study of the leaching process is researched. In the leaching process, the reaction rate is supposed to be controlled by the chemical reaction (Levenspiel, 1972; Tong, 2005; Zhan, 2006), then: −dN = kSC dt
where N is the number of moles of unleached ore at time t, N = (4/3)rk 3 /M, t is the reaction time, S is the surface area of a ore particle, S = 4rk 2 , k is a rate constant, C is the concentration of acid, rk is the radius of ore particle, is the ore density, and M is the average molar mass of ore, then:
2. Mathematical mechanism model of the leaching process − 2.1. Brief introduction of leaching process Leaching is defined as removal of ore by dissolving them from the solid matrix. The reagent is injected into a leaching tank and dissolves ore to separate the valuable components and impurities. In our research, cobalt compound is prepared in the hydrometallurgical factory, and the reagents are sulfuric acid and sulfur dioxide. The field device is shown in Fig. 1. The temperature, pH value, flow rate of sulfur dioxide and flow rate of sulfuric acid all can be monitored online. The whole process is a batch process. The ore is slurried in the slurry tank and transported to the leaching tank. And then, the ore slurry is stirred and heated, and reacted with the sulfuric acid and sulfur dioxide in the leaching tank. In the reaction,
(1)
drk kMC = dt
(2)
Because the radius of ore particle is not convenient to be detected, the kinetic equation is usually expressed by the relationship between leaching rate y and reaction time t. N0 is supposed to be the number of moles of unleached ore at beginning, then: y=
N0 − N r 3 =1− k 3 N0 rk0
(3)
where rk0 is the initial radius of ore particles. Substituting Eq. (3) into Eq. (2) may result in: dy 3kCM = (1 − y)2/3 rk0 dt
(4)
G. Hu et al. / Computers and Chemical Engineering 35 (2011) 2611–2617
According to Arrhenius equation, k is defined as: k = Ae−E/RT
(5)
where A is the preexponential factor, E is the activation energy, R is the gas constant, and T is the absolute temperature. Basing on the kinetic analysis on the leaching process above, combining with the actual field production, the material balance equation of leaching process is established as followed: Lower-priced ore :
3ACH2 SO4 M −E /RT dy1 · (1 − y1 )2/3 = ·e 1 rk0 dt
(6)
Higher-priced ore :
3ACH2 SO3 M −E /RT dy2 · (1 − y2 )2/3 = ·e 2 rk0 dt
(7)
Total leaching rate :
y = y1 + ϕy2
(8)
where y1 is the leaching rate of lower-priced ore, y2 is the leaching rate of higher-priced ore, CH2 SO4 is the concentration of sulfuric acid, CH2 SO3 is the concentration of sulfurous acid, T is the temperature of leaching solution, y is the total leaching rate, is the ratio of lower-priced ore in reactant, and ϕ is the ratio of higher-priced ore in reactant. In the reaction of leaching process, the concentration of sulfuric acid and the concentration of sulfurous acid are changing. The change of the concentration of sulfuric acid is dependent on two factors: one is the consumption of sulfuric acid during the reaction; the other is the supplement of sulfuric acid during the reaction. Considering these two factors, the material balance of sulfuric acid can be formulated as the following differential equation: CH2 SO4 · QH2 SO4 · H2 SO4 /MH2 SO4 − K1 · (dy1 /dt) · ˛ dy3 = V dt
(9)
where K1 = · G/M, QH2 SO4 is the flow rate of sulfuric acid, H2 SO4 is the density of sulfuric acid, MH2 SO4 is the molar mass of sulfuric acid, is the content of lower-priced ore, G is the weight of ore, ˛ is the consumption of number of moles of sulfuric acid when 1 mol lower-priced ore is leached, and V is the volume of leaching tank. The change of concentration of sulfurous acid is dependent on four factors: consumption of sulfurous acid during the reaction, flow rate of sulfur dioxide, solubility of sulfur dioxide, contact area between sulfur dioxide and leaching solution. The dissolution rate (Parker, 1978) and the contact area between sulfur dioxide and leaching solution (Moore, 1981) can be defined as: Dissolution rate : Contact area :
vg =
Sg =
Dg (CSO2 − CH2 SO3 ) ıg
3QSO2 H
vr
(10)
2.3. Parameters of mechanism model Considering the multitudinousness of the parameters in mechanism model, an accurate setting of the parameters is important to the application of the model. The main parameters of the mechanism model are tested by nonlinear regression of experimental data. 2.3.1. Known parameters in field V = 12,000 dm3 , H = 500 cm, M = 73.31 g/mol, = 8.9 g/cm3 , CH2 SO4 = 0.95, H2 SO4 = 1.84 kg/dm3 , MH2 SO4 = 98 g/mol, ˛ = 1, ˇ = 4.19, U = 27,509 W/(m2 K), a = 14.13 m2 , r = 0.5 cm. 2.3.2. Known parameters in the literature v = 10 cm/s, R = 8.314 J/(mol K) (Li, ıg = 0.74 cm (Moore, 1981).
1998),
Dg = 1.21 cm2 ,
2.3.3. Parameters obtained from experiment and identification 2.3.3.1. Parameters obtained from experiment. The solubility of sulfur dioxide is a variable parameter about temperature. The solubility of sulfur dioxide drops along with the ascension of temperature. The function of solubility of sulfur dioxide is expressed by the regression analyzing of the experimental data: Cp = 0.289 · 10−8 · T 4 − 0.434 · 10−5 · T 3 + 0.244 · 10−2 · T 2 − 0.607 · T + 56.5
(14)
Similarly, the average heat capacity of leaching solution is: c¯ = 2298.365 J/(kg K). 2.3.3.2. Parameters obtained from identification. Parameters A, E1 , and E2 are identified by PSO optimization (Prata, Schwaab, Lima, & Pinto, 2009) algorithm with the field actual data. The result of the identification is: A = 68.36, E1 = 6889 J/mol, and E2 = 66,529 J/mol. 3. Bagging ensemble of SVR based on NCL (negative SVR bagging, NSB) 3.1. SVR algorithm
(12)
where K2 = · G/M, is the content of higher-priced ore, and ˇ is the consumption of number of moles of sulfur dioxide when 1 mol higher-priced ore is leached. The leaching process is a heat-acid-stirring reaction. It is also an energy transfer process in which energy is delivered from the heating pipe in leaching tank to the leaching solution, and it can be expressed by the following equation: T −T dy5 =U·a· h m · c¯ dt
where U is the heat transfer coefficient of leaching tank, a is the heat transfer area of leaching tank, Th is the temperature of heating, m is the mass of leaching solution, and c¯ is the average specific heat of leaching solution. In summary, mechanism model of the leaching process may be expressed by Eqs. (6)–(9), (12) and (13).
(11)
where Dg is the diffusivity of sulfur dioxide in the water, CSO2 is the solubility of sulfur dioxide, ıg is the thickness of gas-liquid contact area, QSO2 is the flow rate of sulfur dioxide, H is the height of leaching tank, v is the rising velocity of bubble, r is the radius of bubble. Considering these factors, the material balance of sulfurous acid can be formulated as the following differential equation: vg · Sg − K2 · (dy2 /dt) · ˇ dy4 = V dt
2613
(13)
SVR is an adaptation of a recent statistical learning theory based classification paradigm, namely support vector machines (Vapnik et al., 1996). The SVR formulation follows structural risk minimization (SRM) principle, as opposed to the empirical risk minimization (ERM) approach which is commonly employed within statistical machine learning methods. The principal and methodology of SVM based regression technique is briefly provided below. Consider a general regression model that relates the predictors vi to the streamflow yi in year i: yi = f (vi ) + e
(15)
and denote by fˆ (v) the estimate of the regression function f (v). The function f (v) can be any nonlinear function and in the SVM literature it is typically considered to be estimated through the sum of a set of kernel functions. Using these kernel functions, the input vector vi is mapped into a new feature space in which linear regression
2614
G. Hu et al. / Computers and Chemical Engineering 35 (2011) 2611–2617
is performed rather than nonlinear regression. Given this mapping, the estimate of the regression model could be expressed as follows: fˆ (v) = ω, v + b
(16)
where ω represents the support vector weights (basis functions), angle brackets denote a dot product and b is a bias term, similar to an intercept in linear regression. The parameters of fˆ (v) are estimated by minimizing the following regularized risk function.
1 ||ω||2 + c ( i + i∗ ) 2 n
min ·
(17)
i=1
Subject to f (v) − ω, v − b ≤ ε + i ω, v + b − f (v) ≤ ε + i∗
(19)
where i and i∗ are slack variables that determine the degree to which state space samples with error more than ε be penalized; and ε is the degree to which one would like to tolerate errors in constructing the predictor fˆ (v). The above formulation is referred as the ε-insensitive approach. The optimization problem of last equation can be solved though changing into the dual problem by Lagrange optimization method, Lagrangian function is established according to target function and constraints:
1 L(ω, b, , , a, a , , ∗) = ωT ω + c ( j + i∗ ) − ai [ j 2 n
n
i=1
i=1
∗
n
+εi − yi + (ωϕ(xi ) + b)] −
a∗i [ j + ε∗i + yi − (ωϕ(xi ) + b)]
i=1
n
−
( i j + i i∗ )
(20)
i=1
where ai , a∗i , i , ∗i are Lagrange multipliers. Request partial derivative to b, ω, i∗ and order these partial derivatives to zero, respectively. And get: ∂L = ∂b
n
f (x) = (ω · x) + b =
n
(ai − a∗i )(xi · x) + b
(26)
i=1
where corresponding xi of ai − a∗i = / 0 is support vectors, and b can be obtained by Karush–Kuhn–Tucker (KKT) calculation. For a nonlinear problem, training data is mapped by kernel function K(xi , xj ) to a high-dimensional feature space, and then in this space a linear regression function is established, so the model of nonlinear regression function can be obtained according to the above derivation process:
(18)
i , i∗ ≥ 0
∗
Furthermore, the Lagrange multiplier ai , a∗i which maximize the above target function can be obtained. After the value ω was obtained from Eq. (22), the regression function:
f (x, ˛, ˛∗ ) =
P
(˛i − ˛∗i )K(x, xi ) + b
(27)
i=1
In this paper a commonly used kernel functions, namely, radial basis function (RBF) is used. It is defined below in Eq. (17):
K(xi , xj ) = exp
−||xi − xj ||2
2 2
(28)
3.2. The basic idea of bagging For bagging algorithm, each training subset contains M learning samples, drawn randomly with replacement from the original training set of size M. Such a training subset is called a bootstrap (Efron & Tibshirani, 1993) replicate of the original set. Each bootstrap replicate contains, on average, 63.2% of the original training set, with many examples appearing some times. Because of the randomness characteristic existing in the process of distilling the training subsets, bagging algorithm has the flaw of blindness to some extent. Let T ={(xp , yp ), p = 1, . . ., N} denotes a regression type training set, and the SVR algorithm uses T to construct a regression predictor FR (x,T) for future y values. Let F be, an bagging ensemble obtained as a simple averaging combination of M predictors, that is 1 fi (xp ) M M
F(xp ) =
(29)
i=1
(a∗i − ai ) = 0
(21)
∂L =ω− (a∗i − ai )ϕ(xi ) = 0 ∂ω
(22)
where M is the number of the individual SVR in the ensemble, fi (xp ) is the output of SVR ion the pth testing pattern, and F(xp ) is the output of the ensemble on the pth testing pattern. The generalization errors of fi and F on the training set T are expressed as follows:
(23)
Ei =
i=1 n
i=1
∂L = c − a∗i − ∗i = 0 ∂ i∗
L(a, a∗ ) = −ε
(ai + a∗i ) +
i=1
n
N
(ai − a∗i ) = 0 (0 ≤ ai , a∗i ≤ c,
(yp − F(xp ))2
(31)
1 Ei M M
(24)
E¯ =
(32)
i=1
The diversity between fi and F is defined as:
Constraint condition:
i=1
(30)
The simple averaging of Ei is defined as:
i=1
i,j=1
n
2
p=1
yi (a − a∗i )
1 (ai − a∗i )(aj − a∗j )(xi , xj ) 2
E=
n
−
(yp − fi (xp ))
p=1
Let Eqs. (21)–(23) substitute into Eq. (20), and then Eq. (20) is changed into: n
N
i = 1, . . . , n)
(25)
Ai =
N p=1
(fi (xp ) − F(xp ))
2
(33)
G. Hu et al. / Computers and Chemical Engineering 35 (2011) 2611–2617
2615
Fig. 2. Hybrid model structure.
The simple averaging of Ai is defined as:
To set the partial derivative as zero, the desired output is calculated by minimizing E as shown in the following equation:
1 A¯ = Ai M M
(34)
i=1
From the conclusion of Krogh (Krogh & Vedelsby, 1995), we get: E = E¯ − A¯
(35)
This result implies that increasing ensemble diversity while decreasing or maintaining the simple average generalization error of ensemble members, should lead to a decrease in generalization error. 3.3. Negative correlation learning Negative correlation learning introduces a correlation penalty term Ci into the error function of each individual SVR in the ensemble so that all the networks can be trained simultaneously and interactively on the same training data set T (Liu & Yao, 1999b). The correlation function Ci is defined as:
Ci = (fi (xp ) − F(xp ))
(fj (xp ) − F(xp )) = −(fi (xp ) − F(xp ))
2
(36)
j= / i
The error function Ei for SVR i in negative correlation learning is defined as:
Ei =
N 1 p=1
2
2
(yp − fi (xp )) + Ci
(37)
The parameter 0 ≤ ≤ 1 is used to adjust the strength of the penalty. The simple averaging of for the ensemble in negative correlation learning is defined as: 1 M M
Eˆ =
N
i=1 p=1
1 2
2
(yp − fi (xp )) + Ci
(40)
The minimization of the generalization error of the ensemble is achieved by replacing yp with f¯ of the individual SVR. The proposed NSB algorithm rebuilds sample of the next component learner continuously to revise the ensemble errors before, which makes the ensemble result best. The algorithm consists of two parts: training the individual SVR by NCL algorithm and combining the individual SVR by an advanced strategy. The proposed NSB algorithm is described as follows: Initialization parameters: the maximal number of individuals of ensemble is M, the maximal generalization error of individuals SVR is e1 , the maximal generalization error of ensemble is e2, , i = 1. Step 1. Generate training subsets T1 from T by using Bootstrap sampling algorithm (Efron & Tibshirani, 1993). Train an individual model f1 on the training subset T1 by using SVR algorithm. Step 2. Calculate the training error E1 of f1 by Eq. (30). If E1 < e1, set i = i + 1, otherwise repeat step 1. Step 3. Generate training subsets Ti from T by using Bootstrap sampling algorithm. Train an individual model fi on the training subset Ti by using SVR algorithm. Get the desired output f¯i using Eq. (29). Replace Ti with f¯i . Finally, retrain fi on the new training subset Ti by SVR algorithm. Step 4. Calculate the training error Ei of fi by Eq. (30). If Ei < e1 , set i = i + 1, otherwise repeat step 3. Step 5. Calculate the training error E of this ensemble by Eq. (31). If E < e2 , or i = M, output: F(xp ) =
1 M
M
fi (xp ) Otherwise repeat step
i=1
3.
(38)
The partial derivative of E with respect to the output of SVR i on the nth training pattern is: 1 ∂Eˆ = [(f (xp ) − yp ) − 2(fi (xp ) − F(xp ))] M i ∂fi
yp − 2F(xp ) f¯i = 1 − 2
(39)
4. Hybrid model based on NSB (NSBH) Based on NSB algorithm, a hybrid model which compensates the mechanism model with error compensation model is proposed in this paper. The foundational model structure of our approach is shown in Fig. 2. In this paper, the input variables of the hybrid model are determined. There may be some redundant information if we
2616
G. Hu et al. / Computers and Chemical Engineering 35 (2011) 2611–2617
take all the measurable variables for the training. The problem of “over-fitting” is always encountered due to the redundant information. It can be solved using both statistical analysis and preferable deep knowledge about the causal links in the particular process. Based on the latter, three on-line measured variables: heating temperature (u1 ), flow rate of concentrated sulfuric acid (u2 ) and flow rate of sulfur dioxide (u3 ) have been selected as inputs of the hybrid model. The output is leaching rate (y) in the hybrid model. The error compensation model is used to predict the p-step value in future. The predictive control algorithm can be formally described as follows:
y(k + P) = f (ˆy(k − 1), u(k − 1), u(k), u(k + 1), . . . , u(k + P − 1)) (41) where u = [u1 , u2 , u3 ], yˆ (k) is the error between the leaching rate predicted by mechanism model and the one tested by laboratory in the k-step. P is the key parameter of the hybrid model. With the increasing of the value of P, the prediction precision is improved and the model is more complex. With the decreasing of the value of P, the dynamic response rate is increased and the stability and the robustness of the model are worse. For the foregoing, we choose the value P = 2 in our model.
Fig. 4. Dynamic prediction results of MM model, BBH model, SVRH model and NSBH model for test 2.
5. Practical application and experiments 5.1. Leaching process prediction system The leaching process prediction system, which is based on the study of the proposed models and the improved algorithm, is developed and put into operation in an advanced factory. The prediction system is developed by C++ and runs on an Intel P4 2.0 GHz PC with 1 G RAM. 5.2. Experiment and analysis In the practical applications, process data always carry various measurement errors inevitably, because of measuring instrumentation precision, reliability and environment factors of field measurement. With low precise measurement data, the deviation of online optimum results maybe caused, even unsuccessful production. Therefore, the pretreatment of measurement data is very important to ensure the normal operation of online optimization. The main of this paper is that eliminate abnormal data, with the Pauta criteria of statistics discriminance method. Its principle is as follows: set sample date is X = (x1 , x2 , . . ., xn )T ,
Fig. 5. Prediction result of final leaching rate of MM model, BBH model, SVRH model and NSBH model.
¯ deviation is i = xi − X¯ (i = 1, 2, . . . , n). The average value is X, standard deviation is calculated according to the Bayesian formula:
=
n v2i i=1
1/2
(42)
n−1
If the deviation value vi of sample date xi corresponds with |vi | > 3 , the sample date is abnormal data, which should be deleted and then replaced by average value of samples. Experiments have been conducted 36 times to test the effect of varying values of three operating condition variables (u1 ,u2 ,u3 ) on leaching rate (y). After preprocessing and standardizing, a data set with 36 groups of data can be obtained and each group has 12 samples. The data set is divided into two sets averagely. One is used as training set, and the other is used as testing set. The number of individual SVR is 25 (Breiman, 1996). The main parameters of Table 1 Comparison of four models.
Fig. 3. Dynamic prediction results of MM model, BBH model, SVRH model and NSBH model for test 1.
MODEL
RMSE
MAXE
MM BBH SVRH NSBH
0.0153 0.0139 0.0156 0.0105
0.0312 0.0326 0.0258 0.0231
G. Hu et al. / Computers and Chemical Engineering 35 (2011) 2611–2617
2617
Table 2 Relative error of two tests. Relative error
RE(3) RE(4) RE(5) RE(6) RE(7) RE(8) RE(9) RE(10) RE(11) RE(12)
Test1
Test2
MM
BBH
SVRH
NSBH
MM
BBH
SVRH
NSBH
0.0667 0.1768 0.0487 0.1086 0.0864 0.0481 0.0048 0.0057 0.0091 0.0112
0.0415 0.0854 0.0598 0.0774 0.0412 0.0285 0.0311 0.0397 0.0305 0.0114
0.0877 0.1306 0.0051 0.0914 0.0580 0.0331 0.0377 0.0418 0.0315 0.0094
0.0198 0.0010 0.0155 0.0404 0.0082 0.0087 0.0193 0.0042 0.0069 0.0017
0.0762 0.0207 0.0722 0.1124 0.0386 0.0287 0.0495 0.0484 0.0228 0.0266
0.0844 0.0730 0.0574 0.0385 0.0527 0.0365 0.0153 0.0130 0.0246 0.0206
0.0945 0.0317 0.0527 0.0875 0.0825 0.0466 0.0669 0.0615 0.0142 0.0378
0.0399 0.0383 0.0357 0.0177 0.0183 0.0184 0.0199 0.0046 0.0100 0.0122
NSB algorithm is that: fitting error ε = 0.0126, c = 103, = 5.3, P = 2, e1 = 0.005, e2 = 0.001, = 0.2. Comprising with NSBH model, mechanism model (MM), basic bagging hybrid model (BBH) and single SVR hybrid model (SVRH) are also applied in this experiment. Two dynamic prediction results of these tests are shown in Figs. 3 and 4. Eighteen prediction results of final leaching rate are shown in Fig. 5. Eighteen independent experiments are carried out for each method and the average root mean squared error (RMSE) and maximal absolute error (MAXE) are shown in Table 1. The relative error of two tests is shown in Table 2. Figs. 3–5 show that the predicted precision of leaching rate based on the proposed NSBH model is higher than the other models and it has better generalization ability. Figs. 3 and 4 show that for the effect of predicted leaching rate following the actual value, the NSBH model is better than the other models. Table 1 shows that the NSBH model performs not only better than MM model but also better than BBH model and SVRH model in generalization ability. 6. Conclusions In this paper, a mathematical mechanism model of the heatstirring-acid leaching process has been developed, in which cobalt compound ores particles dissolve in acidic aqueous solution. Also, a hybrid model is established based on the proposed NSB algorithm. The simulation results show that the NSBH model does consistently improve the predicted precision versus MM model, BBH model and SVRH model for leaching process. Acknowledgements The authors are grateful for the financial support of the National High Technology Research and Development Program of China (863 Program, No. 2006AA060201). References Breed, A. W., & Hansford, G. S. (1999). Modeling continuous bioleach reactors. Biotechnology and Bioengineering, 64(6), 671–677. Breiman, L. (1996). Bagging predictors. Machine Learning, 24(2), 123–140.
Crundwell, F. K. (1995). Progress in the mathematical modelling of leaching reactors. Hydrometallurgy, 39(1–3), 321–335. Crundwell, F. K. (2000). Modeling, simulation, and optimization of bacterial leaching reactors. Biotechnology and Bioengineering, 71(4), 255–265. Efron, B., & Tibshirani, R. (1993). An introduction to the bootstrap. New York: Chapman & Hall. Kittler, J., Hatef, M., Duin, R. P. W., & Matas, J. (1998). On combining classifiers. IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(3), 226–239. Krogh, A., & Vedelsby, J. (1995). Neural network ensembles, cross validation, and active learning. In Advances in Neural Information Processing Systems 7: Proceedings conference Cambridge, MA, (pp. 231–238). Levenspiel, O. (1972). Chemical reaction engineering. New York: John Wiley& Sons. Li, H. (1998). Hydrometallurgy. Changsha: Central South University Press. Liu, Y. (2005). Generate different neural networks by negative correlation learning. Lecture Notes in Computer Science, 3610, 149–156. Liu, Y., & Yao, X. (1999a). Simultaneous training of negatively correlated neural networks in an ensemble. IEEE Transactions on Systems, Man, and Cybernetics B, 29(6), 716–725. Liu, Y., & Yao, X. (1999b). Ensemble learning via negative correlation. Neural Networks, 12(10), 1399–1404. Liu, Y., Yao, X., & Higuchi, T. (2000). Evolutionary ensembles with negative correlation learning. IEEE Transactions on Evolutionary Computation, 4(4), 380–387. Ma, Z., & Ek, C. (1991). Rate processes and mathematical modelling of the acid leaching of a manganese carbonate ore. Hydrometallurgy, 27(2), 125–139. Moore, J. J. (1981). Chemical metallurgy. London: Butterworths Press. Opitz, D., & Maclin, R. (1999). Popular ensemble methods: An empirical study. Journal of Artificial Intelligence Research, 11, 169–198. Parker, R. H. (1978). An introduction to chemical metallurgy. New York: Pergamon Press Ltd. Prata, D. M., Schwaab, M., Lima, E. L., & Pinto, J. C. (2009). Nonlinear dynamic data reconciliation and parameter estimation through particle swarm optimization: Application for an industrial polypropylene reactor. Chemical Engineering Science, 64(18), 3953–3967. Schapire, R. (1990). The strength of weak learnability. Machine Learning, 5(2), 197–227. Tong, Z. (2005). Leaching kinetics of calcium aluminate slag. The Chinese Journal of Process Engineering, 5(4), 399–402. Vapnik, V., Golowich, S. E., & Smola, A. G. (1996). Support vector method for function approximation, regression estimation and signal processing. In Proceedings of Neural Information Process System (NIPS’1996) London, UK, (pp. 281–287). Veglio, F., Trifoni, M., & Toro, L. (2001). Leaching of manganiferous ores by glucose in a sulfuric acid solution: Kinetic modeling and related statistical analysis. Industrial & Engineering Chemistry Research, 40(18), 3895–3901. Wu, Y., He, J., Man, Y., & Arribas, J. I. (2004). Neural network fusion strategies for identifying breast masses. 2004 IEEE International Joint Conference on Neural Networks, 3, 2437–2442. Zhan, S. (2006). Study on leaching kinetics of pyrite cinder. Chemical Engineering, 34(11), 36–39 (Chinese). Zhang, X. Y., & Xu, L. B. (2003). The stock market forecast model based on neural network ensemble. Systems Engineering—Theory & Practice, 9(9), 67–70.