- Email: [email protected]
USNFIS: Uniform stable neuro fuzzy inference system
Accepted Manuscript
USNFIS: Uniform Stable Neuro Fuzzy Inference System Jose´ de Jesus ´ Rubio PII: DOI: Reference:
S0925-2312(17)30980-3 10.1016/j.neucom.2016.08.150 NEUCOM 18510
To appear in:
Neurocomputing
Received date: Revised date: Accepted date:
1 June 2016 5 August 2016 6 August 2016
Please cite this article as: Jose´ de Jesus ´ Rubio, USNFIS: Uniform Stable Neuro Fuzzy Inference System, Neurocomputing (2017), doi: 10.1016/j.neucom.2016.08.150
This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
ACCEPTED MANUSCRIPT
USNFIS: Uniform Stable Neuro Fuzzy Inference System
CR IP T
Jos´e de Jes´ us Rubio Secci´on de Estudios de Posgrado e Investigaci´on, ESIME Azcapotzalco, Instituto Polit´ecnico Nacional Av. de las Granjas no.682, Col. Santa Catarina, M´exico D.F., 02250, M´exico
AN US
(phone:(+52)55-57296000-64497) (email: [email protected]; [email protected]) Abstract
Introduction
AC
1
CE
PT
ED
M
The algorithms utilized for the big data learning must satisfy three conditions to improve the performance in the processing of big quantity of data: 1) they need to be compact, 2) they need to be effective, and 3) they need to be stable. In this paper, a stable neuro fuzzy inference system is designed from the multilayer neural network and fuzzy inference system to satisfy the three conditions for the big data learning: 1) it utilizes the numerator of the average defuzzifier instead of the average defuzzifier to be compact, 2) it employs gaussian functions instead of sigmoid functions to be effective, and 3) it uses a time varying learning speed instead of the constant learning speed to be stable. The suggested technique is applied for the modeling of the crude oil blending process and the beetle population process. Keywords: Neuro fuzzy system, fuzzy inference system, multilayer neural network, big data learning, crude oil blending, beetle population.
The neuro fuzzy intelligent systems are the combination of the neural networks and the fuzzy systems which are applied for the learning of nonlinear behaviors. Some interesting investigations are detailed as follows. In [4], [5], [12], [19], novel algorithms used in the modelling are detailed, in [6], [7], self learning algorithms for the classification are designed, in [13], [24], [25], interesting algorithms used in the prediction are studied, in [15], [16], [50] 1
ACCEPTED MANUSCRIPT
AC
CE
PT
ED
M
AN US
CR IP T
new clustering algorithms utilized in the fault detection are proposed, in [29], [30], [31], [32], novel algorithms employed in the classification are described, and in [48], [49], [33], [34], [35], metacognitive learning algorithms are introduced. Few researches have been carried out in the past to introduce this kind of algorithms to be utilized for the big data learning. Therefore, new efforts to increase the knowledge in this interesting issue would be of great interest. Big data learning is the learning ability to solve via intelligent systems the problems where huge amounts of data are generated and updated during a short time, in this kind of systems the processing and analysis of data are important challenges. Some interesting works of this topic are described as follows. In [10], [38], [39], [40], the classification of big data is focused, in [14], [18], [41], the modeling of big data is analyzed, and in [52], [54], the pattern recognition of big data is studied. In this research, there are two kind of big data issues described as follows: 1) the systems with many inputs and outputs such as the considered in [10], [18], [38], [39], [52], [54], or 2) the systems with high changing data during a short time such as the considered in [14], [40], [41]; it is because in both cases huge amounts of data are generated and updated during a short time. This study is focused in the case 2). On the other hand, in the aforementioned research, the stability of the algorithms is not analyzed, and the stability of the algorithms should be assured to avoid the damage of the devices due to the processing of big quantity of data. The stable intelligent systems are the algorithms where the inputs, outputs, and parameters remain bounded through the time and where the overfitting is avoided. An algorithm with overfit has many parameters relative to the number of data; therefore, it has poor learning performance because it overreacts to minor fluctuations in the data. There is some research about the stable intelligent systems. In [1], [2], [3], the stability of continuous-time fuzzy neural networks is studied, in [8], [9], [21], [23], [42], [44] the stability of the gradient algorithm is introduced, in [36], [55], [56], the stability of continuous time neural networks is described, in [22], [57], the stability of discrete-time fuzzy systems is analyzed, in [26], [46], [28], the stability of continuous-time control systems is assured, and in [27], [47], the stability of controlled robotic systems is guaranteed. The stability should be assured in big data learning to guarantee a satisfactory behavior through the time for this kind of systems. Most of the stable algorithms use a time varying learning speed such as the mentioned in [8], [17], and [42] for the learning of a multilayer neural network, or the mentioned in [9], [21], and [44] for the learning of a fuzzy inference system. It is an efficient algorithm; therefore, it would be interesting to modify this algorithm to be applied in a neuro fuzzy system. 2
ACCEPTED MANUSCRIPT
In this research, a neuro fuzzy inference system with a structure different with the multilayer neural network and the fuzzy inference system is suggested. Three differences between the introduced algorithm and the multilayer neural network and fuzzy inference system are described in function of the compactness, effectiveness, and stability as follows.
CR IP T
1. The suggested algorithm is different with the fuzzy inference system because the first only uses the numerator of the average defuzzifier while the second utilizes the average defuzzifier. The numerator of the average defuzzifier is better for the learning in big data than the average defuzzifier because the first is more compact than the second. Consequently, the suggested algorithm is compact.
M
AN US
2. The introduced algorithm is different with the multilayer neural network because the first employs gaussian functions while the second utilizes sigmoid functions. The gaussian function can be adapted better to the changing behavior of the systems than the sigmoid function because the first has three kind of parameters while the second only uses two kind of parameters, and because the first considers positive and negative values, while the second only considers positive values. Therefore, the proposed algorithm is effective.
PT
ED
3. The proposed algorithm is different with both the multilayer neural network and fuzzy inference system because the first uses a time varying learning speed while the other uses a constant learning speed. The time varying learning speed is better for the learning in big data than the constant learning speed because the first reaches the stability and the boundedness of the parameters while the other does not. Thus, the introduced algorithm is stable.
AC
CE
The paper is organized as follows. In section 2, the neuro fuzzy inference system is presented. In section 3, the closed loop dynamics of the neuro fuzzy inference system and the nonlinear system are obtained. In section 4, the introduced algorithm for the big data learning of the neuro fuzzy inference system is designed. In section 5, the stability, convergence, and boundedness of parameters for the aforementioned technique are guaranteed. In section 6, the suggested strategy is summarized. In section 7, the recommended algorithm is compared with other two algorithms for two processes. Section 8 presents the conclusions and suggests the future research directions.
3
ACCEPTED MANUSCRIPT
2
Neuro fuzzy inference system
CR IP T
In this section, first, the big data nonlinear systems studied in this research are described, and second, the neuro fuzzy inference system for the big data learning of the nonlinear system behavior is introduced. Consider the big data unknown discrete-time multiple inputs multiple outputs nonlinear system as follows: yl∗ (k) = fl [Zk ] (1)
AN US
where i = 1, . . . , N , l = 1, . . . , O, Zk = [z1 (k) . . . , zi (k), . . . , zN (k)]T ∈
2
M
αj (uj (k)) = e−uj (k) N X uj (k) = bij (k) [zi (k) − ci (k)]
(2)
ED
i=1
CE
PT
where i = 1, . . . , N , j = 1, . . . , M , l = 1, . . . , O, zi (k) ∈ < and yl (k) ∈ < are the inputs and outputs of the neuro fuzzy inference system, ajl (k) ∈ <, bij (k) ∈ <, ci (k) ∈ < are the parameters of the output layer, hidden layer, and centers, αj (uj (k)) ∈ < is a nonlinear function, uj (k) ∈ < is the addition function, M is the neurons number in the hidden layer and O is the outputs number. Figure 1 shows the architecture of the neuro fuzzy inference system where the input layer, hidden layer, and output layer are observed.
AC
Remark 1 In [14], [20], [51], [53], and [58], the interesting radial basis function neural networks are considered. A radial basis function neural network can not be seen as a multilayer neural network because the first utilizes the gaussian functions while the second employs the sigmoid functions. From [37], [45], a radial basis function neural network can be seen as a fuzzy inference system because both use the gaussian functions.
4
AN US
CR IP T
ACCEPTED MANUSCRIPT
Figure 1: Architecture of the neuro fuzzy inference system
M
Remark 2 The fuzzy inference system is given as follows:
ED
yl (k) = dl (k) =
ajl (k)αj (uj (k))
j=1 M X
αj (uj (k))
j=1 −u2j (k)
(3)
αj (uj (k)) = e N X uj (k) = bij (k) [zi (k) − ci (k)]
PT CE
M X
i=1
AC
the numerator of the average defuzzifier described by the first equation of (2) is more compact than the average defuzzifier described by the first equation of (3) because the first utilizes less number of operations than the second.
5
ACCEPTED MANUSCRIPT
Remark 3 The multilayer neural network is given as follows: yl (k) = dl (k) =
M X ajl (k)αj (uj (k)) j=1
i=1
(4)
CR IP T
αj (uj (k)) = sig [uj (k)] N X uj (k) = bij (k)zi (k)
3
AN US
the gaussian function explained by the second and third equations of (2) is more effective than the sigmoid function explained by the second and third equations of (4) because the first utilizes two kind of parameters while the second employs only one kind of parameter.
Closed loop dynamics of the neuro fuzzy inference system
PT
ED
M
In this section, the closed loop dynamics of the neuro fuzzy system applied for the big data learning of the nonlinear system behavior are obtained via the linearization technique. The closed loop dynamics of the neuro fuzzy inference system are required for the algorithm design which described in the next section, and for the stability analysis which is explained two sections after. According to the Stone-Weierstrass theorem [41], [42], the unknown nonlinear function f of (1) is approximated as follows:
AC
CE
yl∗ (k) = dl∗ + ∈lf =
M X ajl∗ αj∗ + ∈lf j=1 2
αj∗ = e−uj∗ N X uj∗ = bij∗ [zi (k) − ci∗ ]
(5)
i=1
where ∈lf = yl∗ (k) − dl∗ ∈ < is the modeling error, αj∗ ∈ <, ajl∗ ∈ <, bij∗ ∈ <, and ci∗ ∈ < are the optimal parameters that can minimize the modeling error ∈lf . In the case of three independent variables, a function has a Taylor series as follows: 1 ,ω2 ,ω3 ) fl (ω1 , ω2 , ω3 ) = fl (ω10 , ω20 , ω30 ) + (ω1 − ω10 ) ∂fl (ω∂ω 1 ∂fl (ω1 ,ω2 ,ω3 ) 1 ,ω2 ,ω3 ) 0 + (ω2 − ω20 ) ∂fl (ω∂ω + (ω − ω ) + ξlf 3 3 ∂ω3 2
6
(6)
ACCEPTED MANUSCRIPT
dl (k) = dl∗ +
N X M M X X ∂dl (k) ebij (k) ∂dl (k) + e ajl (k) ∂a ∂bij (k) jl (k) i=1 j=1
j=1
N X l (k) + e ci (k) ∂d + ξlf ∂ci (k) i=1
∂dl (k) ∂ajl (k)
∈ <,
∂dl (k) ∂bij (k)
∈ <, and
∂dl (k) ∂ci (k)
∈ <, please note that dl (k) =
AN US
where
CR IP T
where ξlf ∈ < is the remainder of the Taylor series. ω1 , ω2 , and ω3 correspond to ajl (k) ∈ <, bij (k) ∈ <, and ci (k) ∈ <, ω10 , ω20 , and ω30 correspond to ajl∗ ∈ <, bij∗ ∈ <, and ci∗ ∈ <, define e ajl (k) = ajl (k) − ajl∗ ∈ <, ebij (k) = bij (k) − bij∗ ∈ <, and e ci (k) = ci (k) − ci∗ ∈ <; therefore, the Taylor series is applied to obtain the closed loop dynamics of the neuro fuzzy inference system (2) and the nonlinear system behavior (1) as follows:
(7)
M X ajl (k)αj (uj (k)) ∈ j=1
<. Since all the parameters are scalars, the Taylor series is fully applicable. Considering (2) and using the chain rule, it gives: (8)
M
∂dl (k) = Fj (k) = αj (uj (k)) ∂ajl (k) 2
where αj (uj (k)) = e−uj (k) and uj (k) are given in (2). Utilizing the same process gives:
ED
∂dl (k) ∂bij (k)
= Gijl (k)
= γj (uj (k))ajl (k) [ci (k) − zi (k)]
(9)
CE
PT
where γj (uj (k)) = 2uj (k)αj (uj (k)) ∈ <. Again employing the same process gives:
AC
Substituting
∂dl (k) ∂ajl (k)
∂dl (k) ∂ci (k)
= Hil (k)
= bij (k)γj (uj (k))ajl (k) of (8),
∂dl (k) ∂bij (k)
dl (k) = dl∗ +
of (9), and
∂dl (k) ∂ci (k)
of (10) into (7), it gives:
M M N X X X ebij (k)Gijl (k) e ajl (k)Fj (k) + j=1
i=1 j=1
N X + e ci (k)Hil (k) + ξlf i=1
7
(10)
(11)
ACCEPTED MANUSCRIPT
Define the learning error yel (k) ∈ < as follows:
yel (k) = yl (k) − yl∗ (k)
(12)
yel (k) =
M N X M X X ebij (k)Gijl (k) e ajl (k)Fj (k) + j=1
i=1 j=1
N X + e ci (k)Hil (k) + µl (k) i=1
4
(13)
AN US
where µl (k) = ξlf − ∈lf .
CR IP T
where yl∗ (k) and yl (k) are defined in (1) and (2), respectively. Substituting (2), (5), and (12) into (11) gives closed loop dynamics:
Design of the recommended algorithm
M
In this section, the recommended algorithm utilized in a neuro fuzzy inference system is designed for the big data learning of the nonlinear system behavior. In this part, the adapting law of the proposed algorithm is obtained.
ED
Theorem 1 The introduced algorithm which is the updating function of the neuro fuzzy inference system (2) for the big data learning of the nonlinear system (1) is given as follows:
(14)
CE
PT
ajl (k + 1) = ajl (k) − η(k)Fj (k)e yl (k) bij (k + 1) = bij (k) − η(k)Gijl (k)e yl (k) ci (k + 1) = ci (k) − η(k)Hil (k)e yl (k)
AC
where Fj (k), Gijl (k), Hil (k) are given in (8), (9), and (10), respectively, yel (k) is the learning error of (12). Proof. Define an error sum as follows: O
1X 2 ye (k) ε(k) = 2 l=1 l
8
(15)
ACCEPTED MANUSCRIPT
The recommended algorithm is obtained using the following equations: ∂ε(k) 4ajl (k) = ajl (k + 1) − ajl (k) = −η(k) ∂a jl (k)
4bij (k) = bij (k + 1) − bij (k) = −η(k) ∂b∂ε(k) ij (k)
CR IP T
4ci (k) = ci (k + 1) − ci (k) =
(16)
∂ε(k) −η(k) ∂c i (k)
where η(k) is the learning speed which will be defined in the next section. Using the chain ∂ε(k) rule to obtain ∂a gives: jl (k) ∂ε(k) ∂ yel (k) ∂yl (k) ∂ yel (k) ∂yl (k) ∂ajl (k) α (uj (k)) = Fj (k)e yl (k) yel (k)(1) j g(k)
= where
∂yl (k) ∂ajl (k)
= Fj (k) is defined in (8). Using the chain rule to obtain ∂ε(k) ∂bij (k)
∂yl (k) ∂bij (k)
=
∂ε(k) ∂bij (k)
∂ε(k) ∂ yel (k) ∂yl (k) ∂ yel (k) ∂yl (k) ∂bij (k)
= yel (k)(1)Gijl (k) = Gijl (k)e yl (k)
= Gijl (k) is defined in (9). Using the chain rule to obtain ∂ε(k) ∂ci (k)
=
(18) ∂ε(k) ∂ci (k)
gives:
∂ε(k) ∂ yel (k) ∂yl (k) ∂ yel (k) ∂yl (k) ∂ci (k)
= yel (k)(1)Hil (k) = Hil (k)e yl (k)
ED
(17)
gives:
M
where
=
AN US
∂ε(k) ∂ajl (k)
(19)
Stability analysis of the introduced algorithm
CE
5
PT
l (k) where ∂y = Hil (k) is defined in (10). By substituting (17), (18) and (19) into (16), the ∂ci (k) equation of (14) is established.
AC
In this section, the recommended stable algorithm utilized in a neuro fuzzy inference system is designed for the big data learning of the nonlinear system behavior. In this part, the time varying learning speed used in the adapting law of the proposed algorithm is suggested; furthermore, the stability and convergence of the introduced algorithm are assured. The introduced algorithm is given in (14) with a time varying learning speed as follows: ajl (k + 1) = ajl (k) − η(k)Fj (k)e yl (k) bij (k + 1) = bij (k) − η(k)Gijl (k)e yl (k) ci (k + 1) = ci (k) − η(k)Hil (k)e yl (k) 9
(20)
ACCEPTED MANUSCRIPT
where the new time varying learning speed η(k) is as follows: η(k) = 2
1 2
+
M X
Fj2 (k) +
j=1
η0 N M XX
G2ijl (k) +
i=1 j=1
N X
!
Hil2 (k)
i=1
CR IP T
where i = 1, . . . , N , j = 1, . . . , M , l = 1, . . . , O, Fj (k) ∈ < is defined in (8), Gijl (k) ∈ < is defined in (9), Hil (k) ∈ < is defined in (10), yel (k) is defined in (12), 0 < η0 ≤ 1 ∈ <, consequently 0 < η(k) ∈ <, µl is its upper bound of the uncertainty µl (k), |µl (k)| < µl .
AN US
Remark 4 η(k) is the one main part of the recommended algorithm and it is selected by the designer as an average and bounded function such as the stability of the algorithm (20) can be assured. The following theorem gives the stability of the suggested algorithm.
M
Theorem 2 The algorithm (2), (12), and (20) applied for the big data learning of the non2 linear system (1) is uniformly stable and the upper bound of the average learning error yelp (k) satisfies: T 1X 2 lim sup yelp (k) ≤ α0 µ2l (21) T →∞ T k=2
PT
ED
2 yel2 (k − 1), 0 < η0 ≤ 1 ∈ < and 0 < η(k) ∈ < are defined in (20), yel (k) (k) = η(k−1) where yelp 2 is defined in (12), µl is the upper bound of the uncertainty µl (k), |µl (k)| < µl .
AC
CE
Proof. Define the following Lyapunov function: Vl (k) = 21 η(k − 1)e yl2 (k − 1) M M X N N X X X 2 2 e + e ajl (k) + bij (k) + e c2i (k) j=1
j=1 i=1
10
i=1
(22)
ACCEPTED MANUSCRIPT
where e ajl (k), ebij (k), and e ci (k) are defined in (6). Then ∆Vl (k) is as follows: 1 η(k)e yl2 (k) 2
∆Vl (k) =
M X + e a2jl (k + 1) j=1
j=1 i=1
− 21 η(k
−
i=1
1)e yl2 (k
CR IP T
M X N N X X 2 e + bij (k + 1) + e c2i (k + 1) M X − 1) − e a2jl (k) j=1
M X N N X X 2 e − bij (k) − e c2i (k) j=1 i=1
i=1
AN US
Now, the parameters error can be rewritten as follows:
(23)
M M X X 2 e ajl (k + 1) = e a2jl (k) j=1
j=1
M M X X −2η(k)e yl (k) e ajl (k)Fj (k) + η 2 (k)e yl2 (k) Fj2 (k) j=1
j=1
M
M X N M X N X X eb2 (k) eb2 (k + 1) = ij ij j=1 i=1
j=1 i=1
ED
M X N M X N X X 2 2 e bij (k)Gijl (k) + η (k)e yl (k) G2ijl (k) −2η(k)e yl (k) j=1 i=1
PT
j=1 i=1 N X i=1
N X 2 e ci (k + 1) = e c2i (k) i=1
N N X X 2 2 −2η(k)e yl (k) e ci (k)Hil (k) + η (k)e yl (k) Hil2 (k)
CE
(24)
i=1
i=1
AC
It will be proven that (24) is true. Consider the e ajl (k) case, substituting e ajl (k + 1) of (20)
11
ACCEPTED MANUSCRIPT
into
M X e a2jl (k + 1) gives: j=1
M M X X e a2jl (k + 1) = [e ajl (k) − η(k)Fj (k)e yl (k)]2 j=1
j=1
j=1
j=1
M X 2 2 +η (k)e yl (k) Fj2 (k) j=1
CR IP T
M M X X 2 = yl (k) e e ajl (k) − 2η(k)e ajl (k)Fj (k)
AN US
Then for e ajl (k) of (24) is true. Consider the ebij (k) case, substituting ebij (k + 1) of (20) into M N XX eb2 (k + 1) gives: ij j=1 i=1
M X N h M X N i2 X X ebij (k) − η(k)Gijl (k)e eb2 (k) = yl (k) ij j=1 i=1 M X N X
j=1 i=1 M N XX +η 2 (k)e yl2 (k) G2ijl (k) j=1 i=1
ED
j=1 i=1
M X N X ebij (k)Gijl (k) eb2 (k) − 2η(k)e yl (k) ij
M
=
j=1 i=1
PT
Then for ebij (k) of (24) is true. Consider the e ci (k) case, substituting e ci (k + 1) of (20) into N X e c2i (k + 1) gives:
AC
CE
i=1
N N X X 2 e ci (k + 1) = [e ci (k) − η(k)Hil (k)e yl (k)]2 i=1
i=1
N N X X 2 = e ci (k) − 2η(k)e yl (k) e ci (k)Hil (k) i=1
i=1
N X 2 2 +η (k)e yl (k) Hil2 (k) i=1
12
ACCEPTED MANUSCRIPT
Then for e ci (k) of (24) is true. Thus, (24) is true. Substituting (24) into (23) gives: M M X X 2 2 ∆Vl (k) = −2η(k)e yl (k) e ajl (k)Fj (k) + η (k)e yl (k) Fj2 (k) j=1
j=1
CR IP T
M X N M X N X X 2 2 e −2η(k)e yl (k) bij (k)Gijl (k) + η (k)e yl (k) G2ijl (k) j=1 i=1 N X
j=1 i=1 N X −2η(k)e yl (k) e ci (k)Hil (k) + η 2 (k)e yl2 (k) Hil2 (k) i=1 i=1 yl2 (k) − 21 η(k − 1)e yl2 (k − 1) + 21 η(k)e
AN US
(25) can be rewritten as follows:
∆Vl (k) = 21 η(k)e yl2 (k) − 12 η(k − 1)e yl2 (k − 1) − 2η(k)e yl (k) # "M N M N X X XX ebij (k)Gijl (k) + e ci (k)Hil (k) ∗ e ajl (k)Fj (k) + i=1 j=1 j=1 i=1 "M # M X N N X X X +η 2 (k)e yl2 (k) Fj2 (k) + G2ijl (k) + Hil2 (k) j=1 i=1
(26)
i=1
M
j=1
(25)
Substituting (13) in the second term of the equation (26) gives:
PT
ED
∆Vl (k) = 21 η(k)e yl2 (k) − 12 η(k − 1)e yl2 (k − 1) −2η(k)e yl (k) [e yl (k) − µl (k)] "M # M X N N X X X +η 2 (k)e yl2 (k) G2ijl (k) + Hil2 (k) Fj2 (k) + j=1
j=1 i=1
i=1
AC
CE
yl2 (k) − 12 η(k − 1)e yl2 (k − 1) ∆Vl (k) = 21 η(k)e −2η(k)e y 2 (k) + 2η(k)e yl (k)µl (k) "M l # M N N X XX X +η 2 (k)e yl2 (k) Fj2 (k) + G2ijl (k) + Hil2 (k) j=1
j=1 i=1
"
(27)
i=1
M M X N N X X X 2 2 2 2 Substituting η(k) of (20) into the term η (k)e yl (k) Fj (k) + Gijl (k) + Hil2 (k)
13
j=1
j=1 i=1
i=1
#
ACCEPTED MANUSCRIPT
η 2 (k)e yl2 (k)
j=1
M X
η0
=
"M X
j=1 i=1
Fj2 (k)+
j=1 M X
1 2 2+
N M X N X X Hil2 (k) G2ijl (k) + Fj2 (k) + M X N X
j=1 i=1 N X M X
Fj2 (k)+
j=1
≤
G2ijl (k)+
N X
i=1 j=1 η0 η(k)e yl2 (k) 2
≤
2 (k) Hil
i=1 N X
G2ijl (k)+
i=1
#
CR IP T
and considering η0 ≤ 1 gives:
η(k)e yl2 (k) 2 (k) Hil
i=1 1 η(k)e yl2 (k) 2
(28)
AN US
Considering that 2η(k)e yl (k)µl (k) ≤ η(k)e yl2 (k)+η(k)µ2l (k), and considering (28) in (27) gives: ∆Vl (k) ≤ 21 η(k)e yl2 (k) − 12 η(k − 1)e yl2 (k − 1) −2η(k)e yl2 (k) + η(k)e yl2 (k) + η(k)µ2l (k) + 21 η(k)e yl2 (k)
µl in (29) gives:
1 2 2+
Fj2 (k)+
j=1
M
G2ijl (k)+
i=1 j=1
i=1
ED
From (20), η(k) =
1 ∆Vl (k) ≤ − η(k − 1)e yl2 (k − 1) + η(k)µ2l (k) (29) 2 η0 ≤ η0 , and considering that |µl (k)| ≤ M N M N X XX X 2 (k) Hil
PT
1 ∆Vl (k) ≤ − η(k − 1)e yl2 (k − 1) + η0 µ2l 2 It is known that there exits K∞ functions γ1 (·) and γ2 (·) such that: 2 2 e γ1 − − 1), bij (k), e ci (k) ≤ Vl (k) Vl (k) ≤ γ2 21 η(k − 1)e yl2 (k − 1), e a2jl (k), eb2ij (k), e c2i (k)
CE
1 η(k 2
1)e yl2 (k
e a2jl (k),
(30)
(31)
AC
From [42] and using the equations (30) and (31), the learning error of the neuro fuzzy inference system applied for the big data learning of a nonlinear system behavior is uniformly 2 stable. Therefore, Vl (k) is bounded. Using (30) and using yelp (k) of (21), it gives: 2 ∆Vl (k) ≤ −e ylp (k) + η0 µ2l
14
(32)
ACCEPTED MANUSCRIPT
Summarizing (32) from 2 to T , gives: T X k=2
Since Vl (T ) > 0 is bounded:
1 T
CR IP T
2 (k) − η0 µ2l ≤ Vl (1) − Vl (T ) yelp T X 2 (k) ≤ η0 µ2l + T1 Vl (1) yelp k=2
lim sup T1 T →∞
k=2
2 (k) ≤ η0 µ2l yelp
(34)
AN US
(21) is established.
T X
(33)
Remark 5 There are two requirements to apply this algorithm for the big data learning of the nonlinear system behavior, the first is that the uncertainty µl (k) should be bounded, the second is that the nonlinear system should have the structure described by the equation (1).
M
Remark 6 The bound of µl (k) denoted as µl is not utilized in the suggested algorithm (2), (12), (20) because it is only considered to assure its stability.
ED
The following theorem proves that the parameters of the introduced algorithm are bounded.
AC
CE
PT
2 Theorem 3 When the average learning error yelp (k + 1) is bigger than the uncertainty η0 µ2l , the parameters error is bounded by the initial parameters error as follows: 2 yelp (k + 1) ≥ η0 µ2 M M X N N X X X 2 2 e =⇒ e ajl (k) + bij (k) + e c2i (k)
≤
j=1 M X j=1
e a2jl (1) +
j=1 i=1 M X N X j=1 i=1
eb2 (1) + ij
i=1 N X i=1
e c2i (1)
(35)
where i = 1, . . . , N , j = 1, . . . , M , l = 1, . . . , O, e ajl (k), ebij (k), and e ci (k) are defined in 2 e yl2 (k), ajl (k + (6), e ajl (1), bij (1), and e ci (1) are the initial parameters errors, yelp (k +1) = 12 η(k)e 1), bij (k + 1), ci (k + 1), 0 < η0 ≤ 1 ∈ <, and 0 < η(k) ∈ < are defined in (20), yel (k) is defined in (12), µl is the upper bound of the uncertainty µl (k), |µl (k)| < µl . 15
ACCEPTED MANUSCRIPT
Proof. From (24), the parameters are written as follows: M M X X 2 e ajl (k + 1) = e a2jl (k) j=1
j=1
j=1
CR IP T
M M X X 2 2 −2η(k)e yl (k) e ajl (k)Fj (k) + η (k)e yl (k) Fj2 (k) j=1
M X N M X N X X eb2 (k + 1) = eb2 (k) ij ij j=1 i=1
j=1 i=1
M X N M X N X X ebij (k)Gijl (k) + η 2 (k)e −2η(k)e yl (k) yl2 (k) G2ijl (k) j=1 i=1 N X
j=1 i=1
N X 2 e ci (k + 1) = e c2i (k)
AN US
i=1
(36)
i=1
N N X X 2 2 −2η(k)e yl (k) e ci (k)Hil (k) + η (k)e yl (k) Hil2 (k) i=1
M M X N N X X X eb2 (k + 1) and e a2jl (k + 1) with e c2i (k + 1) of (36) gives: ij j=1
j=1 i=1
M
Adding
i=1
i=1
ED
N M M X N X X X eb2 (k + 1) + e c2i (k + 1) e a2jl (k + 1) + ij j=1
i=1
j=1 i=1
N M M X N X X X 2 2 e bij (k) + e c2i (k) = e ajl (k) +
PT
j=1
j=1 i=1
i=1
CE
M M X X 2 2 −2η(k)e yl (k) e ajl (k)Fj (k) + η (k)e yl (k) Fj2 (k) j=1 M X N X
AC
−2η(k)e yl (k)
j=1 M X N X
ebij (k)Gijl (k) + η 2 (k)e yl2 (k)
j=1 i=1 N X
−2η(k)e yl (k)
i=1
e ci (k)Hil (k) +
16
G2ijl (k)
j=1 i=1 N X η 2 (k)e yl2 (k) Hil2 (k) i=1
(37)
ACCEPTED MANUSCRIPT
(37) can be rewritten as follows:
j=1 M X
j=1 i=1
i=1
M X N N X X 2 2 e e ajl (k) + yl (k) bij (k) + e c2i (k) − 2η(k)e
CR IP T
=
M M X N N X X X 2 2 e e ajl (k + 1) + bij (k + 1) + e c2i (k + 1)
j=1 j=1 i=1 i=1 # M M X N N X X X ebij (k)Gijl (k) + ∗ e ajl (k)Fj (k) + e ci (k)Hil (k) j=1 i=1 " M j=1 i=1 M N # N X XX X +η 2 (k)e yl2 (k) Fj2 (k) + G2ijl (k) + Hil2 (k)
"
j=1
j=1 i=1
(38)
i=1
AN US
Substituting (13) in the second term of the equation (38) gives:
N M M X N X X X 2 2 e e ajl (k + 1) + bij (k + 1) + e c2i (k + 1) j=1
=
M X j=1
j=1 i=1 M X N X
e a2jl (k) +
j=1 i=1
eb2 (k) + ij
i=1 N X i=1
e c2i (k)
M
−2η(k)e yl (k) [e yl (k) − µl (k)] "M # M X N N X X X +η 2 (k)e yl2 (k) Fj2 (k) + G2ijl (k) + Hil2 (k) j=1 i=1
ED
j=1
i=1
PT
N M M X N X X X 2 2 e e c2i (k + 1) bij (k + 1) + e ajl (k + 1) +
AC
CE
j=1
i=1
j=1 i=1
N M M X N X X X eb2 (k) + e c2i (k) = e a2jl (k) + ij j=1
j=1 i=1
i=1
−2η(k)e y 2 (k) + 2η(k)e yl (k)µl (k) "M l # M N N X XX X +η 2 (k)e yl2 (k) Fj2 (k) + G2ijl (k) + Hil2 (k) j=1
j=1 i=1
17
i=1
(39)
ACCEPTED MANUSCRIPT
Substituting η(k) of (20) into the last term of (39) and considering η0 ≤ 1 gives:
j=1
M X
η0
=
j=1 i=1
Fj2 (k)+
j=1 M X
1 2 2+
M X N N X X 2 2 Fj (k) + Gijl (k) + Hil2 (k) M X N X
G2ijl (k)+
j=1 i=1 N X M X
Fj2 (k)+
j=1
≤
N X
≤
2 (k) Hil
i=1 N X
G2ijl (k)+
i=1 j=1 η0 η(k)e yl2 (k) 2
i=1
#
CR IP T
η 2 (k)e yl2 (k)
"M X
η(k)e yl2 (k) 2 (k) Hil
i=1 1 η(k)e yl2 (k) 2
(40)
AN US
Considering that 2η(k)e yl (k)µl (k) ≤ η(k)e yl2 (k) + η(k)µ2l (k), and considering (40) in (39) gives: M M X N N X X X eb2 (k + 1) + e a2jl (k + 1) + e c2i (k + 1) ij j=1
j=1 i=1
i=1
N M X N M X X X 2 2 e e c2i (k) bij (k) + ≤ e ajl (k) +
+
i=1 j=1 i=1 2 2 η(k)e yl (k) + η(k)µl (k) + 21 η(k)e yl2 (k)
M
j=1 2 −2η(k)e yl (k)
N M X N M X X X eb2 (k + 1) + e c2i (k + 1) e a2jl (k + 1) + ij i=1
j=1 i=1
ED
j=1
N M M X N X X X 2 2 e e c2i (k) bij (k) + ≤ e ajl (k) +
PT
j=1
AC
µl in (41) gives:
M X
1 2 2+
CE
From (20), η(k) =
Fj2 (k)+
j=1
j=1 i=1 1 − 2 η(k)e yl2 (k) +
i=1 2 η(k)µl (k)
η0 N X M X
G2ijl (k)+
i=1 j=1
N X i=1
2 (k) Hil
≤ η0 and considering that |µl (k)| ≤
M M X N N X X X 2 2 e e ajl (k + 1) + bij (k + 1) + e c2i (k + 1) j=1
j=1 i=1
i=1
M M X N N X X X 2 2 e ≤ e ajl (k) + bij (k) + e c2i (k) j=1
(41)
j=1 i=1 1 yl2 (k) − 2 η(k)e
18
i=1
+
η0 µ2l
(42)
ACCEPTED MANUSCRIPT
η(k) 2 yel (k) 2
2 (k + 1) = Considering yelp
gives:
2 yelp (k + 1) ≥ η0 µ2l M M X N N X X X eb2 (k + 1) + ⇒ e a2jl (k + 1) + e c2i (k + 1) ij j=1
j=1 i=1
i=1
j=1
j=1 i=1
i=1
CR IP T
M M X N N X X X 2 2 e ≤ e ajl (k) + bij (k) + e c2i (k)
2 Considering that yelp (k + 1) ≥ η0 µ2l for k ∈ [1, k] is true, consequently: N M M X N X X X 2 2 e e c2i (k + 1) bij (k + 1) + e ajl (k + 1) + i=1
j=1 i=1
AN US
j=1
N M M X N X X X 2 2 e e c2i (k) ≤ e ajl (k) + bij (k) + j=1
j=1 i=1
j=1 i=1
i=1
M
j=1
Thus (35) is true.
i=1
N M M X N X X X eb2 (1) + e c2i (1) ≤ ... ≤ e a2jl (1) + ij
The suggested algorithm
CE
6
PT
ED
2 Remark 7 From the theorem 2 the average learning error yelp (k + 1) of the suggested method 2 c2i (k) are bounded, is bounded and from the theorem 3 the parameters errors e ajl (k), eb2ij (k), and e i.e., the introduced technique for the learning of a neuro fuzzy inference system is uniform stable in the presence of unmodelled dynamics and the overfitting is avoided. Furthermore, the learning error converges to a small zone bounded by the unmodelled dynamics µl .
AC
In this section, the steps of the application for suggested algorithm are explained. 1) Obtain the outputs of the nonlinear system yl∗ (k) with equation (1). Note, that the nonlinear system may have the structure represented by equation (1); the parameters N and O are selected according to the inputs and outputs number of this nonlinear system. 2) Select the following parameters; ajl (1), bij (1), and ci (1) as random numbers between 0 and 1; M as an integer number, and η0 as a positive value smaller or equal to 1; obtain the outputs of the neuro fuzzy inference system yl (1) with equation (2). 3) For each iteration k, obtain the outputs of the neuro fuzzy inference system yl (k) with 19
ACCEPTED MANUSCRIPT
equation (2), obtain the learning error yel (k) with equation (12), and update the parameters ajl (k + 1), bij (k + 1), and ci (k + 1) with equation (20). 4) Note that the behavior of the algorithm could be improved by selecting other values for M or η0 .
7
CR IP T
Remark 8 The focused neuro fuzzy inference system has one hidden layer. A neuro fuzzy inference system with one hidden layer is sufficient to approximate any nonlinear system.
Results
ED
M
AN US
In this section, two examples are considered. In the examples, the suggested algorithm is applied for the big data learning of the crude oil blending process and the beetle population process. In all cases, the recommended algorithm denoted as USNFIS is compared with the fuzzy inference system of [44] denoted as FIS, and with the gradient algorithm of [42] denoted as G. It is important to note that the three mentioned algorithms are stable with the difference that the G is a multilayer neural network which utilizes sigmoid functions, the FIS is a fuzzy inference system which employs the average defuzzifier, while the USNFIS is a uniform stable fuzzy inference system which considers gaussian functions and the numerator of the average defuzzifier. The root mean square error (RMSE) is used for the algorithms comparison and it is given as follows:
PT
RM SE =
! 21 T O 1 XX 2 ye (k) T k=1 l=1 l
(43)
Crude oil blending process
AC
7.1
CE
where yel (k) is the learning error of (12), T is the iterations number, and O is the outputs number.
In this example, the studied algorithm is applied for the modeling of the crude oil blending process [41]. One neuro fuzzy inference system is utilized for the training and the same system is utilized for the testing. The crude oil blending process has 6 inputs and 3 outputs which are high changing data during a short time. The inputs are z1 (k) = L3 , z2 (k) = P uerto Ceiba, the output is y1∗ (k) = Qa for the first blending process, the inputs are z3 (k) = Qb , z4 (k) = M aya, the output is y2∗ (k) = Qc for the second blending process, and the inputs are z5 (k) = Qc , z6 (k) = El Golpe, the output is y3∗ (k) = In = International for the third 20
AN US
CR IP T
ACCEPTED MANUSCRIPT
M
Figure 2: Average learning errors for the oil blending process
AC
CE
PT
ED
blending process. In all the cases the o API is considered. The data of 7875 iterations of operation is used for the training and the data of the least 525 iterations is used for the testing. G is given by [42] with parameters N = 6, O = 3, M = 10, α0 = 1, Vj1 and Wij1 are random numbers between 0 and 1. FIS is given by [44] with parameters N = 6, O = 3, M = 10, η0 = 1, ajl (1), bij (1), and ci (1) are random numbers between 0 and 1. USNFIS is given as (2), (12), and (20) with parameters N = 6, O = 3, M = 10, η0 = 1, ajl (1), bij (1), and ci (1) are random numbers between 0 and 1. Figure 2 shows the comparison results for the average learning error where in USNFIS the final average error is 7.6086 × 10−4 , and in FIS of [44] the final average error is 0.0020, and in G of [42] the final average error is 0.0016. Figure 3 shows the training results and Figure 4 shows the testing results. Table 1 shows the training RMSE results and Table 2 shows the testing RMSE results for many intermediate iterations termed with th via the equation (43).
21
AN US
CR IP T
ACCEPTED MANUSCRIPT
AC
CE
PT
ED
M
Figure 3: Training results for the oil blending process
Figure 4: Testing results for the oil blending process
22
ACCEPTED MANUSCRIPT
the oil blending 4000th 5000th 0.1409 0.1404 0.1547 0.1540 0.0946 0.0882
process 6000th 0.1399 0.1534 0.0833
7000th 0.1389 0.1530 0.0794
7875th 0.1389 0.1528 0.0766
CR IP T
Table 1: RMSE training results for Techniques 1000th 2000th 3000th G 0.1588 0.1473 0.1419 FIS 0.1624 0.1580 0.1560 USNFIS 0.1473 0.1183 0.1037
AN US
Table 2: RMSE testing results for the oil blending process Techniques 8075th 8275th 8400th G 0.0353 0.0311 0.0228 FIS 0.0375 0.0331 0.0240 USNFIS 0.0122 0.0107 0.0077
Beetle population process
ED
7.2
M
From Figures 2, 3, and 4, it is observed that USNFIS is better than both the G and FIS because the signal of the first follows better the signal of the plant than the signal of the other. From Table 1 and Table 2, it can be observed that the USNFIS obtained better accuracy when is compared with both G and FIS because the RMSE is smaller for the first. Thus, the USNFIS is preferable for the big data learning of the oil blending process.
AC
CE
PT
In this example, the introduced algorithm is applied for the modeling on the flour beetle population [43]. The beetle population process has 6 inputs and 3 outputs which are high changing data during a short time. The model of experimental population studies of a model of flour beetle population dynamics describes an age-structured population: L(k + 1) = bf A(k)e−cea A(k)−cel L(k) P (k + 1) = [1 − µl ] L(k) A(k + 1) = P (k)e−cpa A(k) + [1 − µa ] A(k)
where: bf = Larvae recruits per adult = 11.98 numbers cea = Susceptibility of eggs to cannibalism by adults = 0.011 unitless cel = Susceptibility of eggs to cannibalism by larvae = 0.013 unitless cpa = Susceptibility of pupae to cannibalism by adults = 0.017 unitless µl = Fraction of larvae dying (not cannibalism) = 0.513 unitless 23
(44)
ACCEPTED MANUSCRIPT
ED
M
AN US
CR IP T
µa = Fraction of adults dying = 0.96 unitless L(k) are the Larvae which starts with 250 numbers, P (k) are the Pupae which starts with 5 numbers, and A(k) are the Adults which starts with 100 numbers. The data of 7800 iterations of operation is used for the training and the data of the least 200 iterations is used for the testing. One neuro fuzzy inference system is used for the training and the same system is used for the testing. z1 (k) = P (k) and z2 (k) = A(k) are the inputs and y1∗ (k) = L(k + 1) is the output for the training of the first population process. z3 (k) = L(k) and z4 (k) = A(k) are the inputs and y2∗ (k) = P (k + 1) is the output for the training of the second population process. Finally, z5 (k) = L(k) and z6 (k) = P (k) are the inputs and y3∗ (k) = A(k + 1) is the output for the training of the third population process. G is given by [42] with parameters N = 6, O = 3, M = 10, α0 = 1, Vj1 and Wij1 are random numbers between 0 and 1. FIS is given by [44] with parameters N = 6, O = 3, M = 10, η0 = 1, ajl (1), bij (1), and ci (1) are random numbers between 0 and 1. USNFIS is given as (2), (12), and (20) with parameters N = 6, O = 3, M = 10, η0 = 1, ajl (1), bij (1), and ci (1) are random numbers between 0 and 1. Figure 5 shows the comparison results of the average learning error where in USNFIS the final average error is 8.5516 × 10−4 , in FIS of [44] the final average error is 0.0011, and in G of [42] the final average error is 0.0382. Figure 6 shows the training results and Figure 7 shows the testing results. Table 3 shows the training RMSE results and Table 4 shows the testing RMSE results for many intermediate iterations termed with th via the equation (43).
the beetle population process 4000th 5000th 6000th 7000th 0.4254 0.4252 0.4250 0.4248 0.1788 0.1652 0.1536 0.1437 0.0951 0.0856 0.0786 0.0732
AC
CE
PT
Table 3: RMSE training results for Techniques 1000th 2000th 3000th G 0.4267 0.4263 0.4257 FIS 0.3092 0.2291 0.1967 USNFIS 0.1841 0.1321 0.1089
7800th 0.4247 0.1369 0.0696
Table 4: RMSE testing results for the beetle population process Techniques 7900th 8000th G 0.0644 0.0460 FIS 0.0066 0.0047 USNFIS 0.0027 0.0019
24
AN US
CR IP T
ACCEPTED MANUSCRIPT
AC
CE
PT
ED
M
Figure 5: Average learning errors for the beetle population process
Figure 6: Training results for the beetle population process
25
AN US
CR IP T
ACCEPTED MANUSCRIPT
M
Figure 7: Testing results for the beetle population process
Conclusion
CE
8
PT
ED
From Figures 5, 6, and 7, it can be observed that the USNFIS is better than both G and FIS because the signal of the first follows better the signal of the plant than the signal of the other. From Table 3 and Table 4 , it is observed that the USNFIS achieves better accuracy when is compared with both G and FIS because the RMSE is smaller for the first. Thus, the USNFIS is preferable for the big data learning of the beetle population process.
AC
In this paper, a novel algorithm is designed for the big data learning of a neuro fuzzy inference system, the stability, convergence, and boundedness of parameters for the studied technique are guaranteed. From the results, it is shown that the introduced approach achieves better accuracy for the big data learning of nonlinear system behaviors when is compared with both the gradient and fuzzy inference system methods. The suggested technique could be used to train a neuro fuzzy inference system such as is applied in this paper, or it could be used as the parameters updating of an evolving intelligent system. As a future research, the focused method will be applied in the control or in the evolving intelligent systems, or other new algorithms will be designed. 26
ACCEPTED MANUSCRIPT
9
Acknowledgements
CR IP T
The author is grateful with the editors and the reviewers for their valuable comments. The author thanks the Secretar´ıa de Investigaci´on y Posgrado, the Comisi´on de Operaci´on y Fomento de Actividades Acad´emicas del IPN, and Consejo Nacional de Ciencia y Tecnolog´ıa for their help in this research.
References
[1] C.-K. Ahn, A new solution to the induced l∞ finite impulse response filtering problem based on two matrix inequalities, International Journal of Control 87 (2) (2014) 404-409.
AN US
[2] C.-K. Ahn, M. T. Lim, Model predictive stabilizer for T-S fuzzy recurrent multilayer neural network models with general terminal weighting matrix, Neural Computing and Applications 23 (suppl 1) (2013) S271-S277.
M
[3] C.-K. Ahn, An error passivation approach to filtering for switched neural networks with noise disturbance, Neural Computing and Applications 21 (5) (2012) 853-861.
ED
[4] R. Ballini, R.-R. Yager, OWA filters and forecasting models applied to electric power load time series, Evolving Systems 5 (2014) 159-173. [5] F. Bordignon, F. Gomide, Uninorm based evolving neural networks and approximation capabilities, Neurocomputing 127 (2014) 13-20.
PT
[6] A. Bouchachia, C. Vanaret, GT2FC: An nnline growing interval type-2 self-learning fuzzy classifier, IEEE Transactions on Fuzzy Systems 22 (4) (2014) 999-1018.
CE
[7] A. Bouchachia, A. Ortner, Self-learning recursive neural networks for structured data classification, International Joint Conference on Neural Networks (2014) 1-8.
AC
[8] J. Cheng-Lv, Z. Yi, Y. Li, Non-divergence of stochastic discrete time algorithms for PCA neural networks, IEEE Transactions on Neural Networks and Learning Systems 26 (2) (2015) 394-399. [9] T. Hernandez-Cortes, J.-A. Meda-Campa˜ na, L.-A. Paramo-Carranza, J.-C. GomezMancilla, A simplified output regulator for a class of Takagi-Sugeno fuzzy models, Mathematical Problems in Engineering 2015 (2015) 1-18.
27
ACCEPTED MANUSCRIPT
[10] D. Kangin, P. Angelov, J. A. Iglesias, A. Sanchis, Evolving classifier tedaclass for big data, Procedia Computer Science, 53 (2015) 9-18.
CR IP T
[11] D. Leite, R. M. Palhares, V.-C.-S. Campos, F. Gomide, Evolving granular fuzzy modelbased control of nonlinear dynamic systems, IEEE Transactions on Fuzzy Systems 23 (4) (2015) 923-938. [12] D. Leite, P. Costa, F. Gomide, Evolving granular neural networks from fuzzy data streams, Neural Networks 38 (2013) 1-16.
AN US
[13] X. Li, W.-F. Lu, L. Zhai, M.-J. Er, Y. Pan, Remaining life prediction of cores based on data-driven and physical modeling methods, Handbook of Manufacturing Engineering and Technology, A. Y. C. Nee, Ed., London, UK: Springer, (2014) 1-22. [14] Y. Li, Q. Liu, S.-R. Tan, R.-H.-M. Chan, High-resolution time-frequency analysis of EEG signals using multiscale radial basis functions, Neurocomputing, 195 (2016) 96103.
M
[15] E. Lughofer, M. Sayed-Mouchaweh, Autonomous data stream clustering implementing split-and-merge concepts-towards a plug-and-play approach, Information Sciences, 304 (2015) 54-79.
ED
[16] E. Lughofer, Hybrid active learning for reducing the annotation effort of operators in classification systems, Pattern Recognition, 45 (2012) 884-896.
PT
[17] E. Lughofer, Evolving fuzzy systems-methodologies, advanced concepts and applications, Springer, Berlin Heidelberg, ISBN: 978-3-642-18086-6, (2011).
AC
CE
[18] B. Luitel, G. K. Venayagamoorthy, Cellular computational networks-a scalable architecture for learning the dynamics of large networked systems, Neural Networks, 50 (2014) 120-123. [19] L. Maciel, F. Gomide, R. Ballini, Enhanced evolving participatory learning fuzzy modeling: an application for asset returns volatility forecasting, Evolving Systems, 5 (2014) 75-88. [20] C.-J.-B. Macnab, Using RBFs in a CMAC to prevent parameter drift in adaptive control, Neurocomputing, 205 (2016) 45-52.
28
ACCEPTED MANUSCRIPT
[21] J. A. Meda-Campa˜ na, J. Rodriguez-Valdez, T. Hernandez-Cortes, R. Tapia-Herrera, V. Nosov, Analysis of the fuzzy controllability property and stabilization for a class of T-S fuzzy models, IEEE Transactions on Fuzzy Systems, 23 (2) (2015) 291-301.
CR IP T
[22] Z. Ning, L. Zhang, J.-J. Rubio, X. Yin, Asynchronous filtering for discrete-time fuzzy affine systems with variable quantization density, IEEE Transactions on Cybernetics, (2016). DOI 10.1109/TCYB.2015.2509170
AN US
[23] V. Nosov, J.-A. Meda-Campa˜ na, J.-C. Gomez-Mancilla, J.-O. Escobedo-Alva, R.G. Hernandez-Garcia, Stability analysis for autonomous dynamical switched systems through nonconventional Lyapunov functions, Mathematical Problems in Engineering, 2015 (2015) 1-18. [24] Y. Pan, M.-J. Er, X. Li, H. Yu, R. Gouriveau, Machine health condition prediction via online dynamic fuzzy neural networks, Engineering Applications of Artificial Intelligence, 35 (2014) 105-113.
M
[25] Y. Pan, X. Hu, M.-J. Er, X. Li, R. Gouriveau, Bearing condition prediction using enhanced online learning fuzzy neural networks, Re-engineering Manufacturing for Sustainability, A. Y. C. Nee, B. Song, and S. K. Ong, Eds., Singapore: Springer, (2013) 175-182.
PT
ED
[26] Y. Pan, M.-J. Er, D. Huang, Q. Wang, Adaptive fuzzy control with guaranteed convergence of optimal approximation error, IEEE Transactions on Fuzzy Systems, 19 (5) (2011) 807-818.
CE
[27] Y. Pan, Y. Liu, B. Xu, H. Yu, Hybrid feedback feedforward: An efficient design of adaptive neural network control, Neural Networks, 76 (2016) 122-134.
AC
[28] Y. Pan, H. Yu, Biomimetic hybrid feedback feedforward neural-network learning control, IEEE Transactions on Neural Networks and Learning Systems, (2016). DOI 10.1109/TNNLS.2016.2527501 [29] M. Pratama, S.-G. Anavatti, E. Lughofer, GENEFIS: Toward an effective localist network, IEEE Transactions on Fuzzy Systems, 22 (3) (2014) 547-562. [30] M. Pratama, S.-G. Anavatti, M.-J. Er, E. Lughofer, pClass: An effective classifier for streaming examples, IEEE Transactions on Fuzzy Systems, 23 (2) (2015) 369-386.
29
ACCEPTED MANUSCRIPT
[31] M. Pratama, S.-G. Anavatti, J. Lu, Recurrent Classifier based on An incremental metacognitive-based scaffolding algorithm, IEEE Transactions on Fuzzy Systems, 23 (6) (2015) 2048-2066.
CR IP T
[32] M. Pratama, J. Lu, G. Zhang, Evolving type-2 fuzzy classifier, IEEE Transactions on Fuzzy Systems, (2015). DOI: 10.1109/TFUZZ.2015.2463732 [33] M. Pratama, G. Zhang, M-J. Er, S. Anavatti, An incremental type-2 metacognitive extreme learning machine, IEEE Transactions on Cybernetics, (2016). DOI: 10.1109/TCYB.2016.2514537
AN US
[34] M. Pratama, J. Lu, S. Anavatti, E. Lughofer, C.-P. Lim, An incremental meta-cognitivebased scaffolding fuzzy neural network, Neurocomputing, 171 (2016) 89-105. [35] M. Pratama, J. Lu, E. Lughofer, G. Zhang, S. Anavatti, Scaffolding type-2 classifier for incremental learning under concept drifts, Neurocomputing, 191 (2016) 304-329.
M
[36] R. Rakkiyappan, R.Sasirekha, Y. Zhu, L. Zhang, H∞ state estimator design for discretetime switched neural networks with multiple missing measurements and sojourn probabilities, Journal of the Franklin Institute, 353 (2016) 1358-1385.
ED
[37] J.-S. Roger-Jang, C.-T. Sun, E. Mitzutani, Neuro-fuzzy and soft computing, a computational approach to learning and machine intelligence, (1997). ISBN: 0-13-261066-3
PT
[38] A. Roy, A classification algorithm for high-dimensional data, Procedia Computer Science, 53 (2015) 345-355.
CE
[39] A. Roy, P. D. Mackin, S. Mukhopadhyay, Methods for pattern selection, class-specific feature selection and classification for automated learning, Neural Networks, 41 (2013) 113-129.
AC
[40] A. Roy, On findings of category and other concept cells in the brain: some theoretical perspectives on mental representation, Cognitive Computation, 7 (2015) 279-284. [41] J.-J. Rubio, Least square neural network model of the crude oil blending process, Neural Networks, 78 (2016) 88-96. [42] J.-J. Rubio, P. Angelov, J. Pacheco, An uniformly stable backpropagation algorithm to train a feedforward neural network, IEEE Transactions on Neural Networks, 22 (3) (2011) 356-366. 30
ACCEPTED MANUSCRIPT
[43] J.-J. Rubio, Stability analysis for an on-line evolving neuro-fuzzy recurrent network, Evolving Intelligent Systems: Methodology and Applications, ISBN: 978-0-470-287194, John Willey y Sons - IEEE Press, Chapter 8 (2010) 173-199.
CR IP T
[44] J.-J. Rubio, Fuzzy slopes model of nonlinear systems with sparse data, Soft Computing, 19 (12) (2015) 3507-3514. [45] J.-J. Rubio, A method with neural networks for the classification of fruits and vegetables, Soft Computing, 2016. DOI: 10.1007/s00500-016-2263-2
AN US
[46] T. Sun, H. Pei, Y. Pan, C. Zhang, Robust wavelet network control for a class of autonomous vehicles to track environmental contour line, Neurocomputing, 74 (2011) 2886-2892. [47] T. Sun, H. Pei, Y. Pan, C. Zhang, Robust adaptive neural network control for environmental boundary tracking by mobile robots, International Journal of Robust and Nonlinear Control, 23 (2013) 123-136.
M
[48] S. Suresh, K. Dong, H.-J. Kim, A sequential learning algorithm for self-adaptive resource allocation network classifier, Neurocomputing, 73 (16-18) (2010) 3012-3019.
ED
[49] S. Suresh, R. Savitha, N. Sundararajan, A sequential learning algorithm for complexvalued self-regulatory resource allocation network, IEEE Transaction on Neural Networks, 22 (7) (2011) 1061-1072.
CE
PT
[50] H. Toubakh, M. Sayed-Mouchaweh, Hybrid dynamic classifier for drift-like fault diagnosis in a class of hybrid dynamic systems: application to wind turbine converters, Neurocomputing, 171 (2016) 1496-1516.
AC
[51] Q. Wu, X. Wang, Q. Shen, Research on dynamic modeling and simulation of axial-flow pumping system based on RBF neural network, Neurocomputing, 186 (2016) 200-206. [52] B. Xu, K. Huang, I. King, C.-L. Liu, J. Sun, N. Satoshi, Graphical lasso quadratic discriminant function and its application to character recognition, Neurocomputing, 129 (2014) 33-40. [53] R. Yang, P.-V. Er, Z. Wang, K.-K. Tan, An RBF neural network approach towards precision motion system with selective sensor fusion, Neurocomputing, 199 (2016) 3139.
31
ACCEPTED MANUSCRIPT
[54] M.-C. Yuen, I. King, K.-S. Leung, Taskrec: a task recommendation framework in crowdsourcing systems, Neural Processing Letters, 41 (2) (2015) 223-238.
CR IP T
[55] L. Zhang, Y. Zhu, W.-X. Zheng, Energy-to-peak state estimation for Markov jump RNNs with time-varying delays via nonsynchronous filter with nonstationary mode transitions, IEEE Transactions on Neural Networks and Learning Systems, 26 (10) (2015) 2346-2356. [56] L. Zhang, Y. Zhu, W.-X. Zheng, Synchronization and state estimation of a class of hierarchical hybrid neural networks with time-varying delays, IEEE Transactions on Neural Networks and Learning Systems, 27 (2) (2016) 459-470.
AN US
[57] L. Zhang, T. Yang, P. Shi, M. Liu, Stability and stabilization of a class of discretetime fuzzy systems with semi-Markov stochastic uncertainties, IEEE Transactions on Systems, Man, and Cybernetics: Systems, (2016). DOI 10.1109/TSMC.2016.2527627
AC
CE
PT
ED
M
[58] C. Zhang, H. Wei, L. Xie, Y. Shen, K. Zhang, Direct interval forecasting of winds peed using radial basis function neural networks in a multi-objective optimization framework, Neurocomputing, 205 (2016) 53-63.
32
ACCEPTED MANUSCRIPT
CR IP T
Biography of the author
AC
CE
PT
ED
M
AN US
Jos´ e de Jes´ us Rubio is a full time professor of the Secci´on de Estudios de Posgrado e Investigaci´on, ESIME Azcapotzalco, Instituto Polit´ecnico Nacional. He has published 93 papers in International Journals, 1 International Book, 8 chapters in International Books, and he has presented 31 papers in International Conferences with 800 citations. He is a member of the IEEE AFS Adaptive Fuzzy Systems. He is member of the National Systems of Researchers with level II. He has been the tutor of 4 P.Ph.D. students, 5 Ph.D. students, 30 M.S. students, 4 S. students, and 17 B.S. students.
33
USNFIS: Uniform Stable Neuro Fuzzy Inference System Jose´ de Jesus ´ Rubio PII: DOI: Reference:
S0925-2312(17)30980-3 10.1016/j.neucom.2016.08.150 NEUCOM 18510
To appear in:
Neurocomputing
Received date: Revised date: Accepted date:
1 June 2016 5 August 2016 6 August 2016
Please cite this article as: Jose´ de Jesus ´ Rubio, USNFIS: Uniform Stable Neuro Fuzzy Inference System, Neurocomputing (2017), doi: 10.1016/j.neucom.2016.08.150
This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
ACCEPTED MANUSCRIPT
USNFIS: Uniform Stable Neuro Fuzzy Inference System
CR IP T
Jos´e de Jes´ us Rubio Secci´on de Estudios de Posgrado e Investigaci´on, ESIME Azcapotzalco, Instituto Polit´ecnico Nacional Av. de las Granjas no.682, Col. Santa Catarina, M´exico D.F., 02250, M´exico
AN US
(phone:(+52)55-57296000-64497) (email: [email protected]; [email protected]) Abstract
Introduction
AC
1
CE
PT
ED
M
The algorithms utilized for the big data learning must satisfy three conditions to improve the performance in the processing of big quantity of data: 1) they need to be compact, 2) they need to be effective, and 3) they need to be stable. In this paper, a stable neuro fuzzy inference system is designed from the multilayer neural network and fuzzy inference system to satisfy the three conditions for the big data learning: 1) it utilizes the numerator of the average defuzzifier instead of the average defuzzifier to be compact, 2) it employs gaussian functions instead of sigmoid functions to be effective, and 3) it uses a time varying learning speed instead of the constant learning speed to be stable. The suggested technique is applied for the modeling of the crude oil blending process and the beetle population process. Keywords: Neuro fuzzy system, fuzzy inference system, multilayer neural network, big data learning, crude oil blending, beetle population.
The neuro fuzzy intelligent systems are the combination of the neural networks and the fuzzy systems which are applied for the learning of nonlinear behaviors. Some interesting investigations are detailed as follows. In [4], [5], [12], [19], novel algorithms used in the modelling are detailed, in [6], [7], self learning algorithms for the classification are designed, in [13], [24], [25], interesting algorithms used in the prediction are studied, in [15], [16], [50] 1
ACCEPTED MANUSCRIPT
AC
CE
PT
ED
M
AN US
CR IP T
new clustering algorithms utilized in the fault detection are proposed, in [29], [30], [31], [32], novel algorithms employed in the classification are described, and in [48], [49], [33], [34], [35], metacognitive learning algorithms are introduced. Few researches have been carried out in the past to introduce this kind of algorithms to be utilized for the big data learning. Therefore, new efforts to increase the knowledge in this interesting issue would be of great interest. Big data learning is the learning ability to solve via intelligent systems the problems where huge amounts of data are generated and updated during a short time, in this kind of systems the processing and analysis of data are important challenges. Some interesting works of this topic are described as follows. In [10], [38], [39], [40], the classification of big data is focused, in [14], [18], [41], the modeling of big data is analyzed, and in [52], [54], the pattern recognition of big data is studied. In this research, there are two kind of big data issues described as follows: 1) the systems with many inputs and outputs such as the considered in [10], [18], [38], [39], [52], [54], or 2) the systems with high changing data during a short time such as the considered in [14], [40], [41]; it is because in both cases huge amounts of data are generated and updated during a short time. This study is focused in the case 2). On the other hand, in the aforementioned research, the stability of the algorithms is not analyzed, and the stability of the algorithms should be assured to avoid the damage of the devices due to the processing of big quantity of data. The stable intelligent systems are the algorithms where the inputs, outputs, and parameters remain bounded through the time and where the overfitting is avoided. An algorithm with overfit has many parameters relative to the number of data; therefore, it has poor learning performance because it overreacts to minor fluctuations in the data. There is some research about the stable intelligent systems. In [1], [2], [3], the stability of continuous-time fuzzy neural networks is studied, in [8], [9], [21], [23], [42], [44] the stability of the gradient algorithm is introduced, in [36], [55], [56], the stability of continuous time neural networks is described, in [22], [57], the stability of discrete-time fuzzy systems is analyzed, in [26], [46], [28], the stability of continuous-time control systems is assured, and in [27], [47], the stability of controlled robotic systems is guaranteed. The stability should be assured in big data learning to guarantee a satisfactory behavior through the time for this kind of systems. Most of the stable algorithms use a time varying learning speed such as the mentioned in [8], [17], and [42] for the learning of a multilayer neural network, or the mentioned in [9], [21], and [44] for the learning of a fuzzy inference system. It is an efficient algorithm; therefore, it would be interesting to modify this algorithm to be applied in a neuro fuzzy system. 2
ACCEPTED MANUSCRIPT
In this research, a neuro fuzzy inference system with a structure different with the multilayer neural network and the fuzzy inference system is suggested. Three differences between the introduced algorithm and the multilayer neural network and fuzzy inference system are described in function of the compactness, effectiveness, and stability as follows.
CR IP T
1. The suggested algorithm is different with the fuzzy inference system because the first only uses the numerator of the average defuzzifier while the second utilizes the average defuzzifier. The numerator of the average defuzzifier is better for the learning in big data than the average defuzzifier because the first is more compact than the second. Consequently, the suggested algorithm is compact.
M
AN US
2. The introduced algorithm is different with the multilayer neural network because the first employs gaussian functions while the second utilizes sigmoid functions. The gaussian function can be adapted better to the changing behavior of the systems than the sigmoid function because the first has three kind of parameters while the second only uses two kind of parameters, and because the first considers positive and negative values, while the second only considers positive values. Therefore, the proposed algorithm is effective.
PT
ED
3. The proposed algorithm is different with both the multilayer neural network and fuzzy inference system because the first uses a time varying learning speed while the other uses a constant learning speed. The time varying learning speed is better for the learning in big data than the constant learning speed because the first reaches the stability and the boundedness of the parameters while the other does not. Thus, the introduced algorithm is stable.
AC
CE
The paper is organized as follows. In section 2, the neuro fuzzy inference system is presented. In section 3, the closed loop dynamics of the neuro fuzzy inference system and the nonlinear system are obtained. In section 4, the introduced algorithm for the big data learning of the neuro fuzzy inference system is designed. In section 5, the stability, convergence, and boundedness of parameters for the aforementioned technique are guaranteed. In section 6, the suggested strategy is summarized. In section 7, the recommended algorithm is compared with other two algorithms for two processes. Section 8 presents the conclusions and suggests the future research directions.
3
ACCEPTED MANUSCRIPT
2
Neuro fuzzy inference system
CR IP T
In this section, first, the big data nonlinear systems studied in this research are described, and second, the neuro fuzzy inference system for the big data learning of the nonlinear system behavior is introduced. Consider the big data unknown discrete-time multiple inputs multiple outputs nonlinear system as follows: yl∗ (k) = fl [Zk ] (1)
AN US
where i = 1, . . . , N , l = 1, . . . , O, Zk = [z1 (k) . . . , zi (k), . . . , zN (k)]T ∈
2
M
αj (uj (k)) = e−uj (k) N X uj (k) = bij (k) [zi (k) − ci (k)]
(2)
ED
i=1
CE
PT
where i = 1, . . . , N , j = 1, . . . , M , l = 1, . . . , O, zi (k) ∈ < and yl (k) ∈ < are the inputs and outputs of the neuro fuzzy inference system, ajl (k) ∈ <, bij (k) ∈ <, ci (k) ∈ < are the parameters of the output layer, hidden layer, and centers, αj (uj (k)) ∈ < is a nonlinear function, uj (k) ∈ < is the addition function, M is the neurons number in the hidden layer and O is the outputs number. Figure 1 shows the architecture of the neuro fuzzy inference system where the input layer, hidden layer, and output layer are observed.
AC
Remark 1 In [14], [20], [51], [53], and [58], the interesting radial basis function neural networks are considered. A radial basis function neural network can not be seen as a multilayer neural network because the first utilizes the gaussian functions while the second employs the sigmoid functions. From [37], [45], a radial basis function neural network can be seen as a fuzzy inference system because both use the gaussian functions.
4
AN US
CR IP T
ACCEPTED MANUSCRIPT
Figure 1: Architecture of the neuro fuzzy inference system
M
Remark 2 The fuzzy inference system is given as follows:
ED
yl (k) = dl (k) =
ajl (k)αj (uj (k))
j=1 M X
αj (uj (k))
j=1 −u2j (k)
(3)
αj (uj (k)) = e N X uj (k) = bij (k) [zi (k) − ci (k)]
PT CE
M X
i=1
AC
the numerator of the average defuzzifier described by the first equation of (2) is more compact than the average defuzzifier described by the first equation of (3) because the first utilizes less number of operations than the second.
5
ACCEPTED MANUSCRIPT
Remark 3 The multilayer neural network is given as follows: yl (k) = dl (k) =
M X ajl (k)αj (uj (k)) j=1
i=1
(4)
CR IP T
αj (uj (k)) = sig [uj (k)] N X uj (k) = bij (k)zi (k)
3
AN US
the gaussian function explained by the second and third equations of (2) is more effective than the sigmoid function explained by the second and third equations of (4) because the first utilizes two kind of parameters while the second employs only one kind of parameter.
Closed loop dynamics of the neuro fuzzy inference system
PT
ED
M
In this section, the closed loop dynamics of the neuro fuzzy system applied for the big data learning of the nonlinear system behavior are obtained via the linearization technique. The closed loop dynamics of the neuro fuzzy inference system are required for the algorithm design which described in the next section, and for the stability analysis which is explained two sections after. According to the Stone-Weierstrass theorem [41], [42], the unknown nonlinear function f of (1) is approximated as follows:
AC
CE
yl∗ (k) = dl∗ + ∈lf =
M X ajl∗ αj∗ + ∈lf j=1 2
αj∗ = e−uj∗ N X uj∗ = bij∗ [zi (k) − ci∗ ]
(5)
i=1
where ∈lf = yl∗ (k) − dl∗ ∈ < is the modeling error, αj∗ ∈ <, ajl∗ ∈ <, bij∗ ∈ <, and ci∗ ∈ < are the optimal parameters that can minimize the modeling error ∈lf . In the case of three independent variables, a function has a Taylor series as follows: 1 ,ω2 ,ω3 ) fl (ω1 , ω2 , ω3 ) = fl (ω10 , ω20 , ω30 ) + (ω1 − ω10 ) ∂fl (ω∂ω 1 ∂fl (ω1 ,ω2 ,ω3 ) 1 ,ω2 ,ω3 ) 0 + (ω2 − ω20 ) ∂fl (ω∂ω + (ω − ω ) + ξlf 3 3 ∂ω3 2
6
(6)
ACCEPTED MANUSCRIPT
dl (k) = dl∗ +
N X M M X X ∂dl (k) ebij (k) ∂dl (k) + e ajl (k) ∂a ∂bij (k) jl (k) i=1 j=1
j=1
N X l (k) + e ci (k) ∂d + ξlf ∂ci (k) i=1
∂dl (k) ∂ajl (k)
∈ <,
∂dl (k) ∂bij (k)
∈ <, and
∂dl (k) ∂ci (k)
∈ <, please note that dl (k) =
AN US
where
CR IP T
where ξlf ∈ < is the remainder of the Taylor series. ω1 , ω2 , and ω3 correspond to ajl (k) ∈ <, bij (k) ∈ <, and ci (k) ∈ <, ω10 , ω20 , and ω30 correspond to ajl∗ ∈ <, bij∗ ∈ <, and ci∗ ∈ <, define e ajl (k) = ajl (k) − ajl∗ ∈ <, ebij (k) = bij (k) − bij∗ ∈ <, and e ci (k) = ci (k) − ci∗ ∈ <; therefore, the Taylor series is applied to obtain the closed loop dynamics of the neuro fuzzy inference system (2) and the nonlinear system behavior (1) as follows:
(7)
M X ajl (k)αj (uj (k)) ∈ j=1
<. Since all the parameters are scalars, the Taylor series is fully applicable. Considering (2) and using the chain rule, it gives: (8)
M
∂dl (k) = Fj (k) = αj (uj (k)) ∂ajl (k) 2
where αj (uj (k)) = e−uj (k) and uj (k) are given in (2). Utilizing the same process gives:
ED
∂dl (k) ∂bij (k)
= Gijl (k)
= γj (uj (k))ajl (k) [ci (k) − zi (k)]
(9)
CE
PT
where γj (uj (k)) = 2uj (k)αj (uj (k)) ∈ <. Again employing the same process gives:
AC
Substituting
∂dl (k) ∂ajl (k)
∂dl (k) ∂ci (k)
= Hil (k)
= bij (k)γj (uj (k))ajl (k) of (8),
∂dl (k) ∂bij (k)
dl (k) = dl∗ +
of (9), and
∂dl (k) ∂ci (k)
of (10) into (7), it gives:
M M N X X X ebij (k)Gijl (k) e ajl (k)Fj (k) + j=1
i=1 j=1
N X + e ci (k)Hil (k) + ξlf i=1
7
(10)
(11)
ACCEPTED MANUSCRIPT
Define the learning error yel (k) ∈ < as follows:
yel (k) = yl (k) − yl∗ (k)
(12)
yel (k) =
M N X M X X ebij (k)Gijl (k) e ajl (k)Fj (k) + j=1
i=1 j=1
N X + e ci (k)Hil (k) + µl (k) i=1
4
(13)
AN US
where µl (k) = ξlf − ∈lf .
CR IP T
where yl∗ (k) and yl (k) are defined in (1) and (2), respectively. Substituting (2), (5), and (12) into (11) gives closed loop dynamics:
Design of the recommended algorithm
M
In this section, the recommended algorithm utilized in a neuro fuzzy inference system is designed for the big data learning of the nonlinear system behavior. In this part, the adapting law of the proposed algorithm is obtained.
ED
Theorem 1 The introduced algorithm which is the updating function of the neuro fuzzy inference system (2) for the big data learning of the nonlinear system (1) is given as follows:
(14)
CE
PT
ajl (k + 1) = ajl (k) − η(k)Fj (k)e yl (k) bij (k + 1) = bij (k) − η(k)Gijl (k)e yl (k) ci (k + 1) = ci (k) − η(k)Hil (k)e yl (k)
AC
where Fj (k), Gijl (k), Hil (k) are given in (8), (9), and (10), respectively, yel (k) is the learning error of (12). Proof. Define an error sum as follows: O
1X 2 ye (k) ε(k) = 2 l=1 l
8
(15)
ACCEPTED MANUSCRIPT
The recommended algorithm is obtained using the following equations: ∂ε(k) 4ajl (k) = ajl (k + 1) − ajl (k) = −η(k) ∂a jl (k)
4bij (k) = bij (k + 1) − bij (k) = −η(k) ∂b∂ε(k) ij (k)
CR IP T
4ci (k) = ci (k + 1) − ci (k) =
(16)
∂ε(k) −η(k) ∂c i (k)
where η(k) is the learning speed which will be defined in the next section. Using the chain ∂ε(k) rule to obtain ∂a gives: jl (k) ∂ε(k) ∂ yel (k) ∂yl (k) ∂ yel (k) ∂yl (k) ∂ajl (k) α (uj (k)) = Fj (k)e yl (k) yel (k)(1) j g(k)
= where
∂yl (k) ∂ajl (k)
= Fj (k) is defined in (8). Using the chain rule to obtain ∂ε(k) ∂bij (k)
∂yl (k) ∂bij (k)
=
∂ε(k) ∂bij (k)
∂ε(k) ∂ yel (k) ∂yl (k) ∂ yel (k) ∂yl (k) ∂bij (k)
= yel (k)(1)Gijl (k) = Gijl (k)e yl (k)
= Gijl (k) is defined in (9). Using the chain rule to obtain ∂ε(k) ∂ci (k)
=
(18) ∂ε(k) ∂ci (k)
gives:
∂ε(k) ∂ yel (k) ∂yl (k) ∂ yel (k) ∂yl (k) ∂ci (k)
= yel (k)(1)Hil (k) = Hil (k)e yl (k)
ED
(17)
gives:
M
where
=
AN US
∂ε(k) ∂ajl (k)
(19)
Stability analysis of the introduced algorithm
CE
5
PT
l (k) where ∂y = Hil (k) is defined in (10). By substituting (17), (18) and (19) into (16), the ∂ci (k) equation of (14) is established.
AC
In this section, the recommended stable algorithm utilized in a neuro fuzzy inference system is designed for the big data learning of the nonlinear system behavior. In this part, the time varying learning speed used in the adapting law of the proposed algorithm is suggested; furthermore, the stability and convergence of the introduced algorithm are assured. The introduced algorithm is given in (14) with a time varying learning speed as follows: ajl (k + 1) = ajl (k) − η(k)Fj (k)e yl (k) bij (k + 1) = bij (k) − η(k)Gijl (k)e yl (k) ci (k + 1) = ci (k) − η(k)Hil (k)e yl (k) 9
(20)
ACCEPTED MANUSCRIPT
where the new time varying learning speed η(k) is as follows: η(k) = 2
1 2
+
M X
Fj2 (k) +
j=1
η0 N M XX
G2ijl (k) +
i=1 j=1
N X
!
Hil2 (k)
i=1
CR IP T
where i = 1, . . . , N , j = 1, . . . , M , l = 1, . . . , O, Fj (k) ∈ < is defined in (8), Gijl (k) ∈ < is defined in (9), Hil (k) ∈ < is defined in (10), yel (k) is defined in (12), 0 < η0 ≤ 1 ∈ <, consequently 0 < η(k) ∈ <, µl is its upper bound of the uncertainty µl (k), |µl (k)| < µl .
AN US
Remark 4 η(k) is the one main part of the recommended algorithm and it is selected by the designer as an average and bounded function such as the stability of the algorithm (20) can be assured. The following theorem gives the stability of the suggested algorithm.
M
Theorem 2 The algorithm (2), (12), and (20) applied for the big data learning of the non2 linear system (1) is uniformly stable and the upper bound of the average learning error yelp (k) satisfies: T 1X 2 lim sup yelp (k) ≤ α0 µ2l (21) T →∞ T k=2
PT
ED
2 yel2 (k − 1), 0 < η0 ≤ 1 ∈ < and 0 < η(k) ∈ < are defined in (20), yel (k) (k) = η(k−1) where yelp 2 is defined in (12), µl is the upper bound of the uncertainty µl (k), |µl (k)| < µl .
AC
CE
Proof. Define the following Lyapunov function: Vl (k) = 21 η(k − 1)e yl2 (k − 1) M M X N N X X X 2 2 e + e ajl (k) + bij (k) + e c2i (k) j=1
j=1 i=1
10
i=1
(22)
ACCEPTED MANUSCRIPT
where e ajl (k), ebij (k), and e ci (k) are defined in (6). Then ∆Vl (k) is as follows: 1 η(k)e yl2 (k) 2
∆Vl (k) =
M X + e a2jl (k + 1) j=1
j=1 i=1
− 21 η(k
−
i=1
1)e yl2 (k
CR IP T
M X N N X X 2 e + bij (k + 1) + e c2i (k + 1) M X − 1) − e a2jl (k) j=1
M X N N X X 2 e − bij (k) − e c2i (k) j=1 i=1
i=1
AN US
Now, the parameters error can be rewritten as follows:
(23)
M M X X 2 e ajl (k + 1) = e a2jl (k) j=1
j=1
M M X X −2η(k)e yl (k) e ajl (k)Fj (k) + η 2 (k)e yl2 (k) Fj2 (k) j=1
j=1
M
M X N M X N X X eb2 (k) eb2 (k + 1) = ij ij j=1 i=1
j=1 i=1
ED
M X N M X N X X 2 2 e bij (k)Gijl (k) + η (k)e yl (k) G2ijl (k) −2η(k)e yl (k) j=1 i=1
PT
j=1 i=1 N X i=1
N X 2 e ci (k + 1) = e c2i (k) i=1
N N X X 2 2 −2η(k)e yl (k) e ci (k)Hil (k) + η (k)e yl (k) Hil2 (k)
CE
(24)
i=1
i=1
AC
It will be proven that (24) is true. Consider the e ajl (k) case, substituting e ajl (k + 1) of (20)
11
ACCEPTED MANUSCRIPT
into
M X e a2jl (k + 1) gives: j=1
M M X X e a2jl (k + 1) = [e ajl (k) − η(k)Fj (k)e yl (k)]2 j=1
j=1
j=1
j=1
M X 2 2 +η (k)e yl (k) Fj2 (k) j=1
CR IP T
M M X X 2 = yl (k) e e ajl (k) − 2η(k)e ajl (k)Fj (k)
AN US
Then for e ajl (k) of (24) is true. Consider the ebij (k) case, substituting ebij (k + 1) of (20) into M N XX eb2 (k + 1) gives: ij j=1 i=1
M X N h M X N i2 X X ebij (k) − η(k)Gijl (k)e eb2 (k) = yl (k) ij j=1 i=1 M X N X
j=1 i=1 M N XX +η 2 (k)e yl2 (k) G2ijl (k) j=1 i=1
ED
j=1 i=1
M X N X ebij (k)Gijl (k) eb2 (k) − 2η(k)e yl (k) ij
M
=
j=1 i=1
PT
Then for ebij (k) of (24) is true. Consider the e ci (k) case, substituting e ci (k + 1) of (20) into N X e c2i (k + 1) gives:
AC
CE
i=1
N N X X 2 e ci (k + 1) = [e ci (k) − η(k)Hil (k)e yl (k)]2 i=1
i=1
N N X X 2 = e ci (k) − 2η(k)e yl (k) e ci (k)Hil (k) i=1
i=1
N X 2 2 +η (k)e yl (k) Hil2 (k) i=1
12
ACCEPTED MANUSCRIPT
Then for e ci (k) of (24) is true. Thus, (24) is true. Substituting (24) into (23) gives: M M X X 2 2 ∆Vl (k) = −2η(k)e yl (k) e ajl (k)Fj (k) + η (k)e yl (k) Fj2 (k) j=1
j=1
CR IP T
M X N M X N X X 2 2 e −2η(k)e yl (k) bij (k)Gijl (k) + η (k)e yl (k) G2ijl (k) j=1 i=1 N X
j=1 i=1 N X −2η(k)e yl (k) e ci (k)Hil (k) + η 2 (k)e yl2 (k) Hil2 (k) i=1 i=1 yl2 (k) − 21 η(k − 1)e yl2 (k − 1) + 21 η(k)e
AN US
(25) can be rewritten as follows:
∆Vl (k) = 21 η(k)e yl2 (k) − 12 η(k − 1)e yl2 (k − 1) − 2η(k)e yl (k) # "M N M N X X XX ebij (k)Gijl (k) + e ci (k)Hil (k) ∗ e ajl (k)Fj (k) + i=1 j=1 j=1 i=1 "M # M X N N X X X +η 2 (k)e yl2 (k) Fj2 (k) + G2ijl (k) + Hil2 (k) j=1 i=1
(26)
i=1
M
j=1
(25)
Substituting (13) in the second term of the equation (26) gives:
PT
ED
∆Vl (k) = 21 η(k)e yl2 (k) − 12 η(k − 1)e yl2 (k − 1) −2η(k)e yl (k) [e yl (k) − µl (k)] "M # M X N N X X X +η 2 (k)e yl2 (k) G2ijl (k) + Hil2 (k) Fj2 (k) + j=1
j=1 i=1
i=1
AC
CE
yl2 (k) − 12 η(k − 1)e yl2 (k − 1) ∆Vl (k) = 21 η(k)e −2η(k)e y 2 (k) + 2η(k)e yl (k)µl (k) "M l # M N N X XX X +η 2 (k)e yl2 (k) Fj2 (k) + G2ijl (k) + Hil2 (k) j=1
j=1 i=1
"
(27)
i=1
M M X N N X X X 2 2 2 2 Substituting η(k) of (20) into the term η (k)e yl (k) Fj (k) + Gijl (k) + Hil2 (k)
13
j=1
j=1 i=1
i=1
#
ACCEPTED MANUSCRIPT
η 2 (k)e yl2 (k)
j=1
M X
η0
=
"M X
j=1 i=1
Fj2 (k)+
j=1 M X
1 2 2+
N M X N X X Hil2 (k) G2ijl (k) + Fj2 (k) + M X N X
j=1 i=1 N X M X
Fj2 (k)+
j=1
≤
G2ijl (k)+
N X
i=1 j=1 η0 η(k)e yl2 (k) 2
≤
2 (k) Hil
i=1 N X
G2ijl (k)+
i=1
#
CR IP T
and considering η0 ≤ 1 gives:
η(k)e yl2 (k) 2 (k) Hil
i=1 1 η(k)e yl2 (k) 2
(28)
AN US
Considering that 2η(k)e yl (k)µl (k) ≤ η(k)e yl2 (k)+η(k)µ2l (k), and considering (28) in (27) gives: ∆Vl (k) ≤ 21 η(k)e yl2 (k) − 12 η(k − 1)e yl2 (k − 1) −2η(k)e yl2 (k) + η(k)e yl2 (k) + η(k)µ2l (k) + 21 η(k)e yl2 (k)
µl in (29) gives:
1 2 2+
Fj2 (k)+
j=1
M
G2ijl (k)+
i=1 j=1
i=1
ED
From (20), η(k) =
1 ∆Vl (k) ≤ − η(k − 1)e yl2 (k − 1) + η(k)µ2l (k) (29) 2 η0 ≤ η0 , and considering that |µl (k)| ≤ M N M N X XX X 2 (k) Hil
PT
1 ∆Vl (k) ≤ − η(k − 1)e yl2 (k − 1) + η0 µ2l 2 It is known that there exits K∞ functions γ1 (·) and γ2 (·) such that: 2 2 e γ1 − − 1), bij (k), e ci (k) ≤ Vl (k) Vl (k) ≤ γ2 21 η(k − 1)e yl2 (k − 1), e a2jl (k), eb2ij (k), e c2i (k)
CE
1 η(k 2
1)e yl2 (k
e a2jl (k),
(30)
(31)
AC
From [42] and using the equations (30) and (31), the learning error of the neuro fuzzy inference system applied for the big data learning of a nonlinear system behavior is uniformly 2 stable. Therefore, Vl (k) is bounded. Using (30) and using yelp (k) of (21), it gives: 2 ∆Vl (k) ≤ −e ylp (k) + η0 µ2l
14
(32)
ACCEPTED MANUSCRIPT
Summarizing (32) from 2 to T , gives: T X k=2
Since Vl (T ) > 0 is bounded:
1 T
CR IP T
2 (k) − η0 µ2l ≤ Vl (1) − Vl (T ) yelp T X 2 (k) ≤ η0 µ2l + T1 Vl (1) yelp k=2
lim sup T1 T →∞
k=2
2 (k) ≤ η0 µ2l yelp
(34)
AN US
(21) is established.
T X
(33)
Remark 5 There are two requirements to apply this algorithm for the big data learning of the nonlinear system behavior, the first is that the uncertainty µl (k) should be bounded, the second is that the nonlinear system should have the structure described by the equation (1).
M
Remark 6 The bound of µl (k) denoted as µl is not utilized in the suggested algorithm (2), (12), (20) because it is only considered to assure its stability.
ED
The following theorem proves that the parameters of the introduced algorithm are bounded.
AC
CE
PT
2 Theorem 3 When the average learning error yelp (k + 1) is bigger than the uncertainty η0 µ2l , the parameters error is bounded by the initial parameters error as follows: 2 yelp (k + 1) ≥ η0 µ2 M M X N N X X X 2 2 e =⇒ e ajl (k) + bij (k) + e c2i (k)
≤
j=1 M X j=1
e a2jl (1) +
j=1 i=1 M X N X j=1 i=1
eb2 (1) + ij
i=1 N X i=1
e c2i (1)
(35)
where i = 1, . . . , N , j = 1, . . . , M , l = 1, . . . , O, e ajl (k), ebij (k), and e ci (k) are defined in 2 e yl2 (k), ajl (k + (6), e ajl (1), bij (1), and e ci (1) are the initial parameters errors, yelp (k +1) = 12 η(k)e 1), bij (k + 1), ci (k + 1), 0 < η0 ≤ 1 ∈ <, and 0 < η(k) ∈ < are defined in (20), yel (k) is defined in (12), µl is the upper bound of the uncertainty µl (k), |µl (k)| < µl . 15
ACCEPTED MANUSCRIPT
Proof. From (24), the parameters are written as follows: M M X X 2 e ajl (k + 1) = e a2jl (k) j=1
j=1
j=1
CR IP T
M M X X 2 2 −2η(k)e yl (k) e ajl (k)Fj (k) + η (k)e yl (k) Fj2 (k) j=1
M X N M X N X X eb2 (k + 1) = eb2 (k) ij ij j=1 i=1
j=1 i=1
M X N M X N X X ebij (k)Gijl (k) + η 2 (k)e −2η(k)e yl (k) yl2 (k) G2ijl (k) j=1 i=1 N X
j=1 i=1
N X 2 e ci (k + 1) = e c2i (k)
AN US
i=1
(36)
i=1
N N X X 2 2 −2η(k)e yl (k) e ci (k)Hil (k) + η (k)e yl (k) Hil2 (k) i=1
M M X N N X X X eb2 (k + 1) and e a2jl (k + 1) with e c2i (k + 1) of (36) gives: ij j=1
j=1 i=1
M
Adding
i=1
i=1
ED
N M M X N X X X eb2 (k + 1) + e c2i (k + 1) e a2jl (k + 1) + ij j=1
i=1
j=1 i=1
N M M X N X X X 2 2 e bij (k) + e c2i (k) = e ajl (k) +
PT
j=1
j=1 i=1
i=1
CE
M M X X 2 2 −2η(k)e yl (k) e ajl (k)Fj (k) + η (k)e yl (k) Fj2 (k) j=1 M X N X
AC
−2η(k)e yl (k)
j=1 M X N X
ebij (k)Gijl (k) + η 2 (k)e yl2 (k)
j=1 i=1 N X
−2η(k)e yl (k)
i=1
e ci (k)Hil (k) +
16
G2ijl (k)
j=1 i=1 N X η 2 (k)e yl2 (k) Hil2 (k) i=1
(37)
ACCEPTED MANUSCRIPT
(37) can be rewritten as follows:
j=1 M X
j=1 i=1
i=1
M X N N X X 2 2 e e ajl (k) + yl (k) bij (k) + e c2i (k) − 2η(k)e
CR IP T
=
M M X N N X X X 2 2 e e ajl (k + 1) + bij (k + 1) + e c2i (k + 1)
j=1 j=1 i=1 i=1 # M M X N N X X X ebij (k)Gijl (k) + ∗ e ajl (k)Fj (k) + e ci (k)Hil (k) j=1 i=1 " M j=1 i=1 M N # N X XX X +η 2 (k)e yl2 (k) Fj2 (k) + G2ijl (k) + Hil2 (k)
"
j=1
j=1 i=1
(38)
i=1
AN US
Substituting (13) in the second term of the equation (38) gives:
N M M X N X X X 2 2 e e ajl (k + 1) + bij (k + 1) + e c2i (k + 1) j=1
=
M X j=1
j=1 i=1 M X N X
e a2jl (k) +
j=1 i=1
eb2 (k) + ij
i=1 N X i=1
e c2i (k)
M
−2η(k)e yl (k) [e yl (k) − µl (k)] "M # M X N N X X X +η 2 (k)e yl2 (k) Fj2 (k) + G2ijl (k) + Hil2 (k) j=1 i=1
ED
j=1
i=1
PT
N M M X N X X X 2 2 e e c2i (k + 1) bij (k + 1) + e ajl (k + 1) +
AC
CE
j=1
i=1
j=1 i=1
N M M X N X X X eb2 (k) + e c2i (k) = e a2jl (k) + ij j=1
j=1 i=1
i=1
−2η(k)e y 2 (k) + 2η(k)e yl (k)µl (k) "M l # M N N X XX X +η 2 (k)e yl2 (k) Fj2 (k) + G2ijl (k) + Hil2 (k) j=1
j=1 i=1
17
i=1
(39)
ACCEPTED MANUSCRIPT
Substituting η(k) of (20) into the last term of (39) and considering η0 ≤ 1 gives:
j=1
M X
η0
=
j=1 i=1
Fj2 (k)+
j=1 M X
1 2 2+
M X N N X X 2 2 Fj (k) + Gijl (k) + Hil2 (k) M X N X
G2ijl (k)+
j=1 i=1 N X M X
Fj2 (k)+
j=1
≤
N X
≤
2 (k) Hil
i=1 N X
G2ijl (k)+
i=1 j=1 η0 η(k)e yl2 (k) 2
i=1
#
CR IP T
η 2 (k)e yl2 (k)
"M X
η(k)e yl2 (k) 2 (k) Hil
i=1 1 η(k)e yl2 (k) 2
(40)
AN US
Considering that 2η(k)e yl (k)µl (k) ≤ η(k)e yl2 (k) + η(k)µ2l (k), and considering (40) in (39) gives: M M X N N X X X eb2 (k + 1) + e a2jl (k + 1) + e c2i (k + 1) ij j=1
j=1 i=1
i=1
N M X N M X X X 2 2 e e c2i (k) bij (k) + ≤ e ajl (k) +
+
i=1 j=1 i=1 2 2 η(k)e yl (k) + η(k)µl (k) + 21 η(k)e yl2 (k)
M
j=1 2 −2η(k)e yl (k)
N M X N M X X X eb2 (k + 1) + e c2i (k + 1) e a2jl (k + 1) + ij i=1
j=1 i=1
ED
j=1
N M M X N X X X 2 2 e e c2i (k) bij (k) + ≤ e ajl (k) +
PT
j=1
AC
µl in (41) gives:
M X
1 2 2+
CE
From (20), η(k) =
Fj2 (k)+
j=1
j=1 i=1 1 − 2 η(k)e yl2 (k) +
i=1 2 η(k)µl (k)
η0 N X M X
G2ijl (k)+
i=1 j=1
N X i=1
2 (k) Hil
≤ η0 and considering that |µl (k)| ≤
M M X N N X X X 2 2 e e ajl (k + 1) + bij (k + 1) + e c2i (k + 1) j=1
j=1 i=1
i=1
M M X N N X X X 2 2 e ≤ e ajl (k) + bij (k) + e c2i (k) j=1
(41)
j=1 i=1 1 yl2 (k) − 2 η(k)e
18
i=1
+
η0 µ2l
(42)
ACCEPTED MANUSCRIPT
η(k) 2 yel (k) 2
2 (k + 1) = Considering yelp
gives:
2 yelp (k + 1) ≥ η0 µ2l M M X N N X X X eb2 (k + 1) + ⇒ e a2jl (k + 1) + e c2i (k + 1) ij j=1
j=1 i=1
i=1
j=1
j=1 i=1
i=1
CR IP T
M M X N N X X X 2 2 e ≤ e ajl (k) + bij (k) + e c2i (k)
2 Considering that yelp (k + 1) ≥ η0 µ2l for k ∈ [1, k] is true, consequently: N M M X N X X X 2 2 e e c2i (k + 1) bij (k + 1) + e ajl (k + 1) + i=1
j=1 i=1
AN US
j=1
N M M X N X X X 2 2 e e c2i (k) ≤ e ajl (k) + bij (k) + j=1
j=1 i=1
j=1 i=1
i=1
M
j=1
Thus (35) is true.
i=1
N M M X N X X X eb2 (1) + e c2i (1) ≤ ... ≤ e a2jl (1) + ij
The suggested algorithm
CE
6
PT
ED
2 Remark 7 From the theorem 2 the average learning error yelp (k + 1) of the suggested method 2 c2i (k) are bounded, is bounded and from the theorem 3 the parameters errors e ajl (k), eb2ij (k), and e i.e., the introduced technique for the learning of a neuro fuzzy inference system is uniform stable in the presence of unmodelled dynamics and the overfitting is avoided. Furthermore, the learning error converges to a small zone bounded by the unmodelled dynamics µl .
AC
In this section, the steps of the application for suggested algorithm are explained. 1) Obtain the outputs of the nonlinear system yl∗ (k) with equation (1). Note, that the nonlinear system may have the structure represented by equation (1); the parameters N and O are selected according to the inputs and outputs number of this nonlinear system. 2) Select the following parameters; ajl (1), bij (1), and ci (1) as random numbers between 0 and 1; M as an integer number, and η0 as a positive value smaller or equal to 1; obtain the outputs of the neuro fuzzy inference system yl (1) with equation (2). 3) For each iteration k, obtain the outputs of the neuro fuzzy inference system yl (k) with 19
ACCEPTED MANUSCRIPT
equation (2), obtain the learning error yel (k) with equation (12), and update the parameters ajl (k + 1), bij (k + 1), and ci (k + 1) with equation (20). 4) Note that the behavior of the algorithm could be improved by selecting other values for M or η0 .
7
CR IP T
Remark 8 The focused neuro fuzzy inference system has one hidden layer. A neuro fuzzy inference system with one hidden layer is sufficient to approximate any nonlinear system.
Results
ED
M
AN US
In this section, two examples are considered. In the examples, the suggested algorithm is applied for the big data learning of the crude oil blending process and the beetle population process. In all cases, the recommended algorithm denoted as USNFIS is compared with the fuzzy inference system of [44] denoted as FIS, and with the gradient algorithm of [42] denoted as G. It is important to note that the three mentioned algorithms are stable with the difference that the G is a multilayer neural network which utilizes sigmoid functions, the FIS is a fuzzy inference system which employs the average defuzzifier, while the USNFIS is a uniform stable fuzzy inference system which considers gaussian functions and the numerator of the average defuzzifier. The root mean square error (RMSE) is used for the algorithms comparison and it is given as follows:
PT
RM SE =
! 21 T O 1 XX 2 ye (k) T k=1 l=1 l
(43)
Crude oil blending process
AC
7.1
CE
where yel (k) is the learning error of (12), T is the iterations number, and O is the outputs number.
In this example, the studied algorithm is applied for the modeling of the crude oil blending process [41]. One neuro fuzzy inference system is utilized for the training and the same system is utilized for the testing. The crude oil blending process has 6 inputs and 3 outputs which are high changing data during a short time. The inputs are z1 (k) = L3 , z2 (k) = P uerto Ceiba, the output is y1∗ (k) = Qa for the first blending process, the inputs are z3 (k) = Qb , z4 (k) = M aya, the output is y2∗ (k) = Qc for the second blending process, and the inputs are z5 (k) = Qc , z6 (k) = El Golpe, the output is y3∗ (k) = In = International for the third 20
AN US
CR IP T
ACCEPTED MANUSCRIPT
M
Figure 2: Average learning errors for the oil blending process
AC
CE
PT
ED
blending process. In all the cases the o API is considered. The data of 7875 iterations of operation is used for the training and the data of the least 525 iterations is used for the testing. G is given by [42] with parameters N = 6, O = 3, M = 10, α0 = 1, Vj1 and Wij1 are random numbers between 0 and 1. FIS is given by [44] with parameters N = 6, O = 3, M = 10, η0 = 1, ajl (1), bij (1), and ci (1) are random numbers between 0 and 1. USNFIS is given as (2), (12), and (20) with parameters N = 6, O = 3, M = 10, η0 = 1, ajl (1), bij (1), and ci (1) are random numbers between 0 and 1. Figure 2 shows the comparison results for the average learning error where in USNFIS the final average error is 7.6086 × 10−4 , and in FIS of [44] the final average error is 0.0020, and in G of [42] the final average error is 0.0016. Figure 3 shows the training results and Figure 4 shows the testing results. Table 1 shows the training RMSE results and Table 2 shows the testing RMSE results for many intermediate iterations termed with th via the equation (43).
21
AN US
CR IP T
ACCEPTED MANUSCRIPT
AC
CE
PT
ED
M
Figure 3: Training results for the oil blending process
Figure 4: Testing results for the oil blending process
22
ACCEPTED MANUSCRIPT
the oil blending 4000th 5000th 0.1409 0.1404 0.1547 0.1540 0.0946 0.0882
process 6000th 0.1399 0.1534 0.0833
7000th 0.1389 0.1530 0.0794
7875th 0.1389 0.1528 0.0766
CR IP T
Table 1: RMSE training results for Techniques 1000th 2000th 3000th G 0.1588 0.1473 0.1419 FIS 0.1624 0.1580 0.1560 USNFIS 0.1473 0.1183 0.1037
AN US
Table 2: RMSE testing results for the oil blending process Techniques 8075th 8275th 8400th G 0.0353 0.0311 0.0228 FIS 0.0375 0.0331 0.0240 USNFIS 0.0122 0.0107 0.0077
Beetle population process
ED
7.2
M
From Figures 2, 3, and 4, it is observed that USNFIS is better than both the G and FIS because the signal of the first follows better the signal of the plant than the signal of the other. From Table 1 and Table 2, it can be observed that the USNFIS obtained better accuracy when is compared with both G and FIS because the RMSE is smaller for the first. Thus, the USNFIS is preferable for the big data learning of the oil blending process.
AC
CE
PT
In this example, the introduced algorithm is applied for the modeling on the flour beetle population [43]. The beetle population process has 6 inputs and 3 outputs which are high changing data during a short time. The model of experimental population studies of a model of flour beetle population dynamics describes an age-structured population: L(k + 1) = bf A(k)e−cea A(k)−cel L(k) P (k + 1) = [1 − µl ] L(k) A(k + 1) = P (k)e−cpa A(k) + [1 − µa ] A(k)
where: bf = Larvae recruits per adult = 11.98 numbers cea = Susceptibility of eggs to cannibalism by adults = 0.011 unitless cel = Susceptibility of eggs to cannibalism by larvae = 0.013 unitless cpa = Susceptibility of pupae to cannibalism by adults = 0.017 unitless µl = Fraction of larvae dying (not cannibalism) = 0.513 unitless 23
(44)
ACCEPTED MANUSCRIPT
ED
M
AN US
CR IP T
µa = Fraction of adults dying = 0.96 unitless L(k) are the Larvae which starts with 250 numbers, P (k) are the Pupae which starts with 5 numbers, and A(k) are the Adults which starts with 100 numbers. The data of 7800 iterations of operation is used for the training and the data of the least 200 iterations is used for the testing. One neuro fuzzy inference system is used for the training and the same system is used for the testing. z1 (k) = P (k) and z2 (k) = A(k) are the inputs and y1∗ (k) = L(k + 1) is the output for the training of the first population process. z3 (k) = L(k) and z4 (k) = A(k) are the inputs and y2∗ (k) = P (k + 1) is the output for the training of the second population process. Finally, z5 (k) = L(k) and z6 (k) = P (k) are the inputs and y3∗ (k) = A(k + 1) is the output for the training of the third population process. G is given by [42] with parameters N = 6, O = 3, M = 10, α0 = 1, Vj1 and Wij1 are random numbers between 0 and 1. FIS is given by [44] with parameters N = 6, O = 3, M = 10, η0 = 1, ajl (1), bij (1), and ci (1) are random numbers between 0 and 1. USNFIS is given as (2), (12), and (20) with parameters N = 6, O = 3, M = 10, η0 = 1, ajl (1), bij (1), and ci (1) are random numbers between 0 and 1. Figure 5 shows the comparison results of the average learning error where in USNFIS the final average error is 8.5516 × 10−4 , in FIS of [44] the final average error is 0.0011, and in G of [42] the final average error is 0.0382. Figure 6 shows the training results and Figure 7 shows the testing results. Table 3 shows the training RMSE results and Table 4 shows the testing RMSE results for many intermediate iterations termed with th via the equation (43).
the beetle population process 4000th 5000th 6000th 7000th 0.4254 0.4252 0.4250 0.4248 0.1788 0.1652 0.1536 0.1437 0.0951 0.0856 0.0786 0.0732
AC
CE
PT
Table 3: RMSE training results for Techniques 1000th 2000th 3000th G 0.4267 0.4263 0.4257 FIS 0.3092 0.2291 0.1967 USNFIS 0.1841 0.1321 0.1089
7800th 0.4247 0.1369 0.0696
Table 4: RMSE testing results for the beetle population process Techniques 7900th 8000th G 0.0644 0.0460 FIS 0.0066 0.0047 USNFIS 0.0027 0.0019
24
AN US
CR IP T
ACCEPTED MANUSCRIPT
AC
CE
PT
ED
M
Figure 5: Average learning errors for the beetle population process
Figure 6: Training results for the beetle population process
25
AN US
CR IP T
ACCEPTED MANUSCRIPT
M
Figure 7: Testing results for the beetle population process
Conclusion
CE
8
PT
ED
From Figures 5, 6, and 7, it can be observed that the USNFIS is better than both G and FIS because the signal of the first follows better the signal of the plant than the signal of the other. From Table 3 and Table 4 , it is observed that the USNFIS achieves better accuracy when is compared with both G and FIS because the RMSE is smaller for the first. Thus, the USNFIS is preferable for the big data learning of the beetle population process.
AC
In this paper, a novel algorithm is designed for the big data learning of a neuro fuzzy inference system, the stability, convergence, and boundedness of parameters for the studied technique are guaranteed. From the results, it is shown that the introduced approach achieves better accuracy for the big data learning of nonlinear system behaviors when is compared with both the gradient and fuzzy inference system methods. The suggested technique could be used to train a neuro fuzzy inference system such as is applied in this paper, or it could be used as the parameters updating of an evolving intelligent system. As a future research, the focused method will be applied in the control or in the evolving intelligent systems, or other new algorithms will be designed. 26
ACCEPTED MANUSCRIPT
9
Acknowledgements
CR IP T
The author is grateful with the editors and the reviewers for their valuable comments. The author thanks the Secretar´ıa de Investigaci´on y Posgrado, the Comisi´on de Operaci´on y Fomento de Actividades Acad´emicas del IPN, and Consejo Nacional de Ciencia y Tecnolog´ıa for their help in this research.
References
[1] C.-K. Ahn, A new solution to the induced l∞ finite impulse response filtering problem based on two matrix inequalities, International Journal of Control 87 (2) (2014) 404-409.
AN US
[2] C.-K. Ahn, M. T. Lim, Model predictive stabilizer for T-S fuzzy recurrent multilayer neural network models with general terminal weighting matrix, Neural Computing and Applications 23 (suppl 1) (2013) S271-S277.
M
[3] C.-K. Ahn, An error passivation approach to filtering for switched neural networks with noise disturbance, Neural Computing and Applications 21 (5) (2012) 853-861.
ED
[4] R. Ballini, R.-R. Yager, OWA filters and forecasting models applied to electric power load time series, Evolving Systems 5 (2014) 159-173. [5] F. Bordignon, F. Gomide, Uninorm based evolving neural networks and approximation capabilities, Neurocomputing 127 (2014) 13-20.
PT
[6] A. Bouchachia, C. Vanaret, GT2FC: An nnline growing interval type-2 self-learning fuzzy classifier, IEEE Transactions on Fuzzy Systems 22 (4) (2014) 999-1018.
CE
[7] A. Bouchachia, A. Ortner, Self-learning recursive neural networks for structured data classification, International Joint Conference on Neural Networks (2014) 1-8.
AC
[8] J. Cheng-Lv, Z. Yi, Y. Li, Non-divergence of stochastic discrete time algorithms for PCA neural networks, IEEE Transactions on Neural Networks and Learning Systems 26 (2) (2015) 394-399. [9] T. Hernandez-Cortes, J.-A. Meda-Campa˜ na, L.-A. Paramo-Carranza, J.-C. GomezMancilla, A simplified output regulator for a class of Takagi-Sugeno fuzzy models, Mathematical Problems in Engineering 2015 (2015) 1-18.
27
ACCEPTED MANUSCRIPT
[10] D. Kangin, P. Angelov, J. A. Iglesias, A. Sanchis, Evolving classifier tedaclass for big data, Procedia Computer Science, 53 (2015) 9-18.
CR IP T
[11] D. Leite, R. M. Palhares, V.-C.-S. Campos, F. Gomide, Evolving granular fuzzy modelbased control of nonlinear dynamic systems, IEEE Transactions on Fuzzy Systems 23 (4) (2015) 923-938. [12] D. Leite, P. Costa, F. Gomide, Evolving granular neural networks from fuzzy data streams, Neural Networks 38 (2013) 1-16.
AN US
[13] X. Li, W.-F. Lu, L. Zhai, M.-J. Er, Y. Pan, Remaining life prediction of cores based on data-driven and physical modeling methods, Handbook of Manufacturing Engineering and Technology, A. Y. C. Nee, Ed., London, UK: Springer, (2014) 1-22. [14] Y. Li, Q. Liu, S.-R. Tan, R.-H.-M. Chan, High-resolution time-frequency analysis of EEG signals using multiscale radial basis functions, Neurocomputing, 195 (2016) 96103.
M
[15] E. Lughofer, M. Sayed-Mouchaweh, Autonomous data stream clustering implementing split-and-merge concepts-towards a plug-and-play approach, Information Sciences, 304 (2015) 54-79.
ED
[16] E. Lughofer, Hybrid active learning for reducing the annotation effort of operators in classification systems, Pattern Recognition, 45 (2012) 884-896.
PT
[17] E. Lughofer, Evolving fuzzy systems-methodologies, advanced concepts and applications, Springer, Berlin Heidelberg, ISBN: 978-3-642-18086-6, (2011).
AC
CE
[18] B. Luitel, G. K. Venayagamoorthy, Cellular computational networks-a scalable architecture for learning the dynamics of large networked systems, Neural Networks, 50 (2014) 120-123. [19] L. Maciel, F. Gomide, R. Ballini, Enhanced evolving participatory learning fuzzy modeling: an application for asset returns volatility forecasting, Evolving Systems, 5 (2014) 75-88. [20] C.-J.-B. Macnab, Using RBFs in a CMAC to prevent parameter drift in adaptive control, Neurocomputing, 205 (2016) 45-52.
28
ACCEPTED MANUSCRIPT
[21] J. A. Meda-Campa˜ na, J. Rodriguez-Valdez, T. Hernandez-Cortes, R. Tapia-Herrera, V. Nosov, Analysis of the fuzzy controllability property and stabilization for a class of T-S fuzzy models, IEEE Transactions on Fuzzy Systems, 23 (2) (2015) 291-301.
CR IP T
[22] Z. Ning, L. Zhang, J.-J. Rubio, X. Yin, Asynchronous filtering for discrete-time fuzzy affine systems with variable quantization density, IEEE Transactions on Cybernetics, (2016). DOI 10.1109/TCYB.2015.2509170
AN US
[23] V. Nosov, J.-A. Meda-Campa˜ na, J.-C. Gomez-Mancilla, J.-O. Escobedo-Alva, R.G. Hernandez-Garcia, Stability analysis for autonomous dynamical switched systems through nonconventional Lyapunov functions, Mathematical Problems in Engineering, 2015 (2015) 1-18. [24] Y. Pan, M.-J. Er, X. Li, H. Yu, R. Gouriveau, Machine health condition prediction via online dynamic fuzzy neural networks, Engineering Applications of Artificial Intelligence, 35 (2014) 105-113.
M
[25] Y. Pan, X. Hu, M.-J. Er, X. Li, R. Gouriveau, Bearing condition prediction using enhanced online learning fuzzy neural networks, Re-engineering Manufacturing for Sustainability, A. Y. C. Nee, B. Song, and S. K. Ong, Eds., Singapore: Springer, (2013) 175-182.
PT
ED
[26] Y. Pan, M.-J. Er, D. Huang, Q. Wang, Adaptive fuzzy control with guaranteed convergence of optimal approximation error, IEEE Transactions on Fuzzy Systems, 19 (5) (2011) 807-818.
CE
[27] Y. Pan, Y. Liu, B. Xu, H. Yu, Hybrid feedback feedforward: An efficient design of adaptive neural network control, Neural Networks, 76 (2016) 122-134.
AC
[28] Y. Pan, H. Yu, Biomimetic hybrid feedback feedforward neural-network learning control, IEEE Transactions on Neural Networks and Learning Systems, (2016). DOI 10.1109/TNNLS.2016.2527501 [29] M. Pratama, S.-G. Anavatti, E. Lughofer, GENEFIS: Toward an effective localist network, IEEE Transactions on Fuzzy Systems, 22 (3) (2014) 547-562. [30] M. Pratama, S.-G. Anavatti, M.-J. Er, E. Lughofer, pClass: An effective classifier for streaming examples, IEEE Transactions on Fuzzy Systems, 23 (2) (2015) 369-386.
29
ACCEPTED MANUSCRIPT
[31] M. Pratama, S.-G. Anavatti, J. Lu, Recurrent Classifier based on An incremental metacognitive-based scaffolding algorithm, IEEE Transactions on Fuzzy Systems, 23 (6) (2015) 2048-2066.
CR IP T
[32] M. Pratama, J. Lu, G. Zhang, Evolving type-2 fuzzy classifier, IEEE Transactions on Fuzzy Systems, (2015). DOI: 10.1109/TFUZZ.2015.2463732 [33] M. Pratama, G. Zhang, M-J. Er, S. Anavatti, An incremental type-2 metacognitive extreme learning machine, IEEE Transactions on Cybernetics, (2016). DOI: 10.1109/TCYB.2016.2514537
AN US
[34] M. Pratama, J. Lu, S. Anavatti, E. Lughofer, C.-P. Lim, An incremental meta-cognitivebased scaffolding fuzzy neural network, Neurocomputing, 171 (2016) 89-105. [35] M. Pratama, J. Lu, E. Lughofer, G. Zhang, S. Anavatti, Scaffolding type-2 classifier for incremental learning under concept drifts, Neurocomputing, 191 (2016) 304-329.
M
[36] R. Rakkiyappan, R.Sasirekha, Y. Zhu, L. Zhang, H∞ state estimator design for discretetime switched neural networks with multiple missing measurements and sojourn probabilities, Journal of the Franklin Institute, 353 (2016) 1358-1385.
ED
[37] J.-S. Roger-Jang, C.-T. Sun, E. Mitzutani, Neuro-fuzzy and soft computing, a computational approach to learning and machine intelligence, (1997). ISBN: 0-13-261066-3
PT
[38] A. Roy, A classification algorithm for high-dimensional data, Procedia Computer Science, 53 (2015) 345-355.
CE
[39] A. Roy, P. D. Mackin, S. Mukhopadhyay, Methods for pattern selection, class-specific feature selection and classification for automated learning, Neural Networks, 41 (2013) 113-129.
AC
[40] A. Roy, On findings of category and other concept cells in the brain: some theoretical perspectives on mental representation, Cognitive Computation, 7 (2015) 279-284. [41] J.-J. Rubio, Least square neural network model of the crude oil blending process, Neural Networks, 78 (2016) 88-96. [42] J.-J. Rubio, P. Angelov, J. Pacheco, An uniformly stable backpropagation algorithm to train a feedforward neural network, IEEE Transactions on Neural Networks, 22 (3) (2011) 356-366. 30
ACCEPTED MANUSCRIPT
[43] J.-J. Rubio, Stability analysis for an on-line evolving neuro-fuzzy recurrent network, Evolving Intelligent Systems: Methodology and Applications, ISBN: 978-0-470-287194, John Willey y Sons - IEEE Press, Chapter 8 (2010) 173-199.
CR IP T
[44] J.-J. Rubio, Fuzzy slopes model of nonlinear systems with sparse data, Soft Computing, 19 (12) (2015) 3507-3514. [45] J.-J. Rubio, A method with neural networks for the classification of fruits and vegetables, Soft Computing, 2016. DOI: 10.1007/s00500-016-2263-2
AN US
[46] T. Sun, H. Pei, Y. Pan, C. Zhang, Robust wavelet network control for a class of autonomous vehicles to track environmental contour line, Neurocomputing, 74 (2011) 2886-2892. [47] T. Sun, H. Pei, Y. Pan, C. Zhang, Robust adaptive neural network control for environmental boundary tracking by mobile robots, International Journal of Robust and Nonlinear Control, 23 (2013) 123-136.
M
[48] S. Suresh, K. Dong, H.-J. Kim, A sequential learning algorithm for self-adaptive resource allocation network classifier, Neurocomputing, 73 (16-18) (2010) 3012-3019.
ED
[49] S. Suresh, R. Savitha, N. Sundararajan, A sequential learning algorithm for complexvalued self-regulatory resource allocation network, IEEE Transaction on Neural Networks, 22 (7) (2011) 1061-1072.
CE
PT
[50] H. Toubakh, M. Sayed-Mouchaweh, Hybrid dynamic classifier for drift-like fault diagnosis in a class of hybrid dynamic systems: application to wind turbine converters, Neurocomputing, 171 (2016) 1496-1516.
AC
[51] Q. Wu, X. Wang, Q. Shen, Research on dynamic modeling and simulation of axial-flow pumping system based on RBF neural network, Neurocomputing, 186 (2016) 200-206. [52] B. Xu, K. Huang, I. King, C.-L. Liu, J. Sun, N. Satoshi, Graphical lasso quadratic discriminant function and its application to character recognition, Neurocomputing, 129 (2014) 33-40. [53] R. Yang, P.-V. Er, Z. Wang, K.-K. Tan, An RBF neural network approach towards precision motion system with selective sensor fusion, Neurocomputing, 199 (2016) 3139.
31
ACCEPTED MANUSCRIPT
[54] M.-C. Yuen, I. King, K.-S. Leung, Taskrec: a task recommendation framework in crowdsourcing systems, Neural Processing Letters, 41 (2) (2015) 223-238.
CR IP T
[55] L. Zhang, Y. Zhu, W.-X. Zheng, Energy-to-peak state estimation for Markov jump RNNs with time-varying delays via nonsynchronous filter with nonstationary mode transitions, IEEE Transactions on Neural Networks and Learning Systems, 26 (10) (2015) 2346-2356. [56] L. Zhang, Y. Zhu, W.-X. Zheng, Synchronization and state estimation of a class of hierarchical hybrid neural networks with time-varying delays, IEEE Transactions on Neural Networks and Learning Systems, 27 (2) (2016) 459-470.
AN US
[57] L. Zhang, T. Yang, P. Shi, M. Liu, Stability and stabilization of a class of discretetime fuzzy systems with semi-Markov stochastic uncertainties, IEEE Transactions on Systems, Man, and Cybernetics: Systems, (2016). DOI 10.1109/TSMC.2016.2527627
AC
CE
PT
ED
M
[58] C. Zhang, H. Wei, L. Xie, Y. Shen, K. Zhang, Direct interval forecasting of winds peed using radial basis function neural networks in a multi-objective optimization framework, Neurocomputing, 205 (2016) 53-63.
32
ACCEPTED MANUSCRIPT
CR IP T
Biography of the author
AC
CE
PT
ED
M
AN US
Jos´ e de Jes´ us Rubio is a full time professor of the Secci´on de Estudios de Posgrado e Investigaci´on, ESIME Azcapotzalco, Instituto Polit´ecnico Nacional. He has published 93 papers in International Journals, 1 International Book, 8 chapters in International Books, and he has presented 31 papers in International Conferences with 800 citations. He is a member of the IEEE AFS Adaptive Fuzzy Systems. He is member of the National Systems of Researchers with level II. He has been the tutor of 4 P.Ph.D. students, 5 Ph.D. students, 30 M.S. students, 4 S. students, and 17 B.S. students.
33