- Email: [email protected]
Relation-based neurofuzzy networks with evolutionary data granulation
MATHEMATICAL AND COMPUTER MODELLING
Available online at www.sciencedirect.com
ac,z•ce ~ D m e c T ELgEVIER
Mathematical and Computer Modelling 40 (2004) 891-921 www.elsevier.com/locate/recto
R e l a t i o n - B a s e d Neurofuzzy Networks with Evolutionary D a t a Granulation SUNG-KWUN OH Department of Electrical Electronic and Information Engineering Wonkwang University, 344-2, Shinyong-Dong, Iksan, Chon-Buk 570-749, South Korea
W. PEDRYCZ Department of Electrical and Computer Engineering University of Alberta, Edmonton, AB T6G 2G6 Canada and Systems Research Institute Polish Academy of Sciences, Warsaw, Poland
BYOUNG-JUN PARK Department of Electrical Electronic and Information Engineering Wonkwang University, 344-2, Shinyong-Dong, Iksan, Chon-Buk 570-749, South Korea
(Received October 2002; revised and accepted May 2003)
Abstract--In
this study, we introduce a concept of self-organizing neurofuzzy networks (SONFN), a hybrid modeling architecture combining relation-based neurofuzzy networks (NFN) and self-organizing polynomial neural networks (PNN). For such networks we develop a comprehensive design methodology and carry out a series of numeric experiments using data coming from the area of software engineering. The construction of SONFNs exploits fundamental technologies of computational intelligence (CI), namely fuzzy sets, neural networks, and genetic algorithms. The architecture of the SONFN results from a synergistic usage of NFN and PNN. NFN contributes to the formation of the premise part of the rule-based structure of the SONFN. The consequence part of the SONFN is designed using PNNs. We discuss two types of SONFN architectures with the taxonomy based on the NFN scheme being applied to the premise part of SONFN and propose a comprehensive learning algorithm. It is shown that this network exhibits a dynamic structure as the number of its layers as well as the number of nodes in each layer of the SONFN are not predetermined (as this is usually the case for a popular topology of a multilayer perceptron). The experimental results deal with well-known software data such as the NASA dataset concerning software cost estimation and the one describing software modules of the medical imaging system (MIS). In comparison with the previously discussed approaches, the self-organizing networks are more accurate and exhibit superb generalization capabilities. (~) 2004 Elsevier Ltd. All rights reserved. K e y w o r d s - - S e l f - o r g a n i z i n g neurofuzzy networks, Neurofuzzy networks, Fuzzy relation-based fuzzy inference, Polynomial neural networks, Computational intelligence, Genetic algorithms, Design methodology.
This paper was supported by Wonkwang University in 2004.
0895..7177/04/$ - see front matter (~) 2004 Elsevier Ltd. All rights reserved. doi:10.1016/j.mcm.2004.10.019
Typeset by A A 4 , S - ~
892
S.-K. Ou et al. 1. I N T R O D U C T I O N
It is expected that efficient modeling techniques should allow for a selection of pertinent variables and a formation of highly representative datasets. Furthermore, the resulting models should be able to take advantage of the existing domain knowledge (such as a prior experience of human observers or operators) and augment it by available numeric data to form a coherent data-knowledge modeling entity. Most recently, the omnipresent trends in system modeling are concerned with a broad range of techniques of computational intelligence (CI) that dwell on the paradigm of fuzzy modeling, neurocomputing, and genetic optimization [1-3]. The list of evident landmarks in the area of fuzzy and neurofuzzy modeling [4-9] is impressive. While the accomplishments are profound, there are still a number of open issues regarding structure problems of the models along with their comprehensive development and testing. Empirical studies in software engineering employ experimental or historical data to gain insight into the software development process. Data concerning software products and software processes are crucial to their better understanding and, in the sequel, establishing more effective ways of enhancing software quality. However, data have no meaning in themselves, they have meaning only in relation to a conceptual model of the phenomenon studied [10]. Yet, the mechanism underlying the software development process is not understood sufficiently well or is too complicated to allow an exact model to be postulated from theory. Accordingly bearing these in mind, we are vitally interested in the development of adaptive and highly nonlinear models that are capable of handling efficacies of software processes. In this study, we develop a hybrid modeling architecture, called self-organizing neurofuzzy networks (SONFN). In a nutshell, SONFN is composed of two main substructures, namely neurofuzzy networks (NFN) and polynomial neural networks (PNN). From a standpoint of rule-based architectures, one can regard a NFN as an implementation of the antecedent part of the rules while the consequents (conclusion parts of the rules) are realized with the aid of PNNs. The role of the NFN based on fuzzy relation-based fuzzy inference and back-propagation (BP) algorithm is to interact with input data, granulate the corresponding input spaces (viz. convert the numeric data into representations realized at the level of fuzzy sets). Fuzzy granulation of the input space is based on fuzzy relation-based approach where all variables are considered en block (and give rise to fuzzy relations). The role of the PNN is to carry out nonlinear transformation at the level of the fuzzy sets (and corresponding membership grades) formed at the level of NFN. The PNN which has a flexible and versatile structure [7,8] is constructed on a basis of a group method of data handling (GMDH [11]) method. In this network, the number of layers and number of nodes in each layer are not predetermined (unlike this occurs in most neural networks) but can be generated dynamically through a growth process. The number of the input variables used in a partial description (PD) is extended and the order of regression polynomial is also made higher to represent other types of nonlinearities. Especially, the number of nodes in each layer of the PNN architecture can be modified with new nodes added, if required. To assess the performance of the proposed model, we experiment with well-known NASA dataset [12] and medical imaging system (MIS) [13] widely used in software engineering. 2. T H E
ARCHITECTURE
AND
DEVELOPMENT
OF
SONFN
In this section, we elaborate on the architecture and a design process of the SONFN. These networks result as a synergy between two other general constructs such as NFN [14] and PNN [7]. First, we briefly discuss these two classes of models by underlining their profound features and afterwards show how a synergy between them develops. 2.1 F u z z y R e l a t i o n - B a s e d N e u r o f u z z y N e t w o r k s We use fuzzy spaces partitioned by fuzzy relation-based approach in the premise part of the SONFN and its premise part is constructed with the aid of neurofuzzy networks (NFN).
Relation-Based Neurofuzzy Networks
(/4J Bs [
893
L(x,x2) A (x,,xJ
B2
f l(X , A1
A2
A3
( 11)
q 13)
Figure 1, Fuzzy granulation of the input space: fuzzy relation-based approach where all variables are considered en block. As visualized in Figure 1, N F N can be designed by using fuzzy space partitioning using all variables simultaneously• T h e situation visualized in Figure 1 involves partitioning of several variables one at a time so we end up with fuzzy relations defined in the Cartesian product of the spaces of the input variables. Let us consider an extension of the network by considering the fuzzy partition realized in terms of fuzzy relations. T h e networks are classified into the two main categories according to the type of fuzzy inference. We distinguish between a simplified and linear fuzzy inference. The fuzzy partitions are formed for the all variables and two different fuzzy inference methods lead us to the topologies visualized in Figure 2. Figure 2 shows architectures of such N F N in case of two inputs and single output. (a) is NFN structure based on simplified fuzzy inference where each input assumes three membership functions and (b) shows linear fuzzy inference based NFN with two membership functions for each input variable. In Figure 2, the "circles" denote units of the NFN. The node denoted by l--[ realizes a Cartesian product• T h e o u t p u t of the node is taken as a product of all the incoming signals. The "N" identifies a normalization procedure applied to the membership grades of the input variable x~. The " ~ " nodes realize a sum of the arguments. Finally, the output of the N F N ~) is governed by the following expression n
Y=fl +f2 +"'+In
= Efj,
(1)
j=l
with n being the number of the fuzzy rules. Considering the language of the rule-based systems, the NFN structure based on simplified fuzzy inference translates into the following collection of rules R 1 : if xl is A n and .. • xi is AI~, then Yl = wl,
R j : if Xl is Ajl and • • • xi is Aji, then yj = wj,
R n : if Xl is Anl and • - xi is Ani, then yn = Wn.
(2)
894
S,-K. OH Layerl 3
e t aI.
Layerl--4
Layer5
xt
A.
Y
)ci
(a) Simplified fuzzy inference based NFN. Layer 1
Layer 2
Layer 3
Layer d
Layer 5
Connection point
(b) Linear fuzzy inference based NFN. Figure 2. NFN structures realized by fuzzy relations-based fuzzy inferences.
Relation-Based Neurofuzzy Networks
895
The fuzzy rules in (2) constitute an overall network of the NFN as shown in Figure 2a. The output f j of each node generates a final output ?) of the form 7%
~%
]=1
j=l
n
•
j=l E ]Aj j=l
Linear fuzzy inference based NFN is equivalent to the following collection of the rules R 1 : if xl is All and ...x~ is Ali, then C y l = w01 4- Wll Xl 4"
R j : if xl is A j l and •
"'"
4- Wil
"
Xi,
xi is Aji, then C y j = woj + w l j • Xl + . ' . + wij • xi,
(4)
R ~ : if xl is A,~I and • • x~ is A,~i, then Cyn = won + w i n • x l + . . . 4- win • x~. .As shown in Figure 2b, the overall network of the NFN based on linear fuzzy inference is constructed by the fuzzy rules described by (4)• A final output ~) results from as following 9 = ~fj j=l
= ~fz. j=l
Cy j
= ~ j=l
#j .Cyj E
,j.
(5)
j = l
The NFN structure shows one possible connection point with the rest of the model for combination with PNN. The location of this point implies the character of the network (both in terms of its flexibility and learning capabilities). Note that the connection point allows perceiving each linguistic manifestation of the original variables (viz. these variables are transformed by fuzzy sets and normalized). The learning of the NFN is realized by adjusting connection weights wj or w~j of the nodes and as such it follows a standard back-propagation (BP) algorithm. In this study, we use two measures (performance indexes) that is the Euclidean error and the mean magnitude of relative error (MMRE). (i) The Euclidean error Ep = (yp - ~)p)2,
(6)
N
E=
I
(yp _ p)2 .
(7)
p=l
(ii) The mean magnitude of relative error (8)
Zp = lYv - flp[, Yp 1 N _ E=
-N ~= I
(9)
YP
where, Ep is an error measure for the pth data, yp is the pth target output data and ~)p stands for the pth actual output of the model for this specific data point. N is total input-output data pairs and E is an overall (global) performance index defined as a sum of the errors for the N. The new connection weight is generated by addition previous connection weight to variation Aw such as (10). w(new) = w(old) + Aw,
where Aw = 77. k.
Ow] "
(10)
896
S.-K. OH
et al.
Here, let us consider a connection weight wit of the NFN structure based on linear fuzzy inference. In order to minimize error Ep, the modification of the connection A w i j is computed through a gradient descent method that is ( _ OEp
(11)
By the chain rule, a partial differential of the right side in (11) can be expressed as follows
OE,
OE. Of~, Oft
OCyt
Ow~t
O~)p
Ow~t
Of i
8Cy t
(12)
Each part of right side in (12) for the (8) is replaced by the detailed expressions
(i) yp > 9v
ofj O~)p
O~)p
yp '
Of j = 1,
ocyj
OCyt = #J'
-
Owit
=-
Xi;
(13)
(ii) yp < ~)p
OEp
O lyp-yp)
1,
Oyp
Ofj
OCyj
--
Xi"
(14)
Therefore, Awij can be simplified as shown below Awi~ . .~ . /2j . x~, Yp
^ for yp >_ yp,
Aw,j = - ~ . #_L. xi, Yp
for yp < ~p,
(15)
where V is a positive learning rate. If i=0, then x o = l . The final adjustment of the connections including momentum terms reads as A w i j = O" fz-Lj ' xi + a (wij(t) - w i j ( t - 1)), Yp
for yp > ~)p,
A w i j = -~l" fz--Zj " xi + c~ (wij(t) - w i j ( t - 1)),
for yp < ~)p,
Yp
(16)
(here the momentum coefficient, a is constrained to the unit interval). The fuzzy relation-based fuzzy inference rules in (2) or (4) are constructed based on all the combinations of Aji, that is, each membership function is not independent of the corresponding fuzzy inference rule. As we can note in Figure 2, the number of input variables and their partition realized by membership functions for the input variable may not always be equal. This implies that we can produce a rather reasonable fuzzy space partition for each input variable based on some of characters between input variable and the output, such as the nonlinearity, the complexity a n d so on.
2.2. A G e n e t i c O p t i m i z a t i o n o f N F N Genetic algorithms (GAs [15]) have proven to be useful in optimization problems because of their ability to efficiently use historical information to produce new solutions with enhanced performance. Likewise, they support a global nature of search supported there. GAs are also theoretically and empirically proven to support robust search in complex search spaces. Moreover they do not get trapped in local minima as opposed to gradient decent techniques being quite susceptible to this shortcoming. GAs is a stochastic search technique based on the principles
Relation-Based Neurofuzzy Networks
897
of evolution, natural selection, and genetic recombination by simulating "survival of the fittest" in a population of potential solutions (individuals) to the problem at hand. GAs are capable of globally exploring a solution space, pursuing potentially fruitful paths while also examining random points to reduce the likelihood of setting for a local optimum. The main features of genetic algorithms concern individuals viewed as strings, population-based optimization (search through the genotype space) and stochastic search mechanism (such as selection and crossover). A fitness function (or fitness, for short) used in genetic optimization is a vehicle to evaluate a performance of a given individual (string). The search of the solution space is completed with the aid of several genetic operators. There are three basic genetic operators used in any GAs-supported search, that is reproduction, crossover, and mutation. Reproduction is a process in which the mating pool for the next generation is chosen. Individual strings are copied into the mating pool according to their fitness function values. Crossover usually proceeds in two steps. First, members from the mating pool are mated at random. Second, each pair of strings undergoes crossover as follows: a position l along the string is selected uniformly at random from the interval [1, l - 1], where I is the length of the string. Two new strings are created by swapping al][ characters between the positions k and I. Mutation is a random alteration of the value of a string position. In a binary coding, mutation means changing a zero to a one or vice versa. Mutation occurs with small probability. Those operators, combined with the proper definition of the fitness function, constitute the main body of the genetic computing. A general flowchart of the genetic optimization (GA) is visualized in Figure 3. In order to enhance the learning of the SONFN and augment its performance of a SONFN, we use genetic algorithms to adjust learning rate, momentum coefficient and the parameters of Initialization
(Randomlygeneram an initialpopulation)
The parameters are encoded into a string as the concatenation of their binary representations
Evaluation
(Computeand storethe fitnessfor each individual) N~ Objectivefunction Reproduction (Generate the next generation by probabilistically selecting individuals to produce offspring via genetic operators)
__operat%._ /
,r/
/ Invert mutation operator [ R
One-point crossover operator
lol,lllOl,10111 I RmlmmM__ IollllNolllolll
I rllllll
1 1 0 0 0 I 1 0 [--~
Figure 3. A general GA flowchart.
, I 1 0 0 0
i, moll
I
S.-K. OH et al.
898
Generation
I °°n 11 °en2 I O°n 3 I Oen 4 I Oen 5 I """" I
Population (Individual)
Variable Chromosome (String) Figure 4. Data structure of genetic algorithms used in the optimization of NFNs.
the membership functions of the antecedents of the rules. Here, GAs use serial method of binary type, roulette-wheel as the selection operator, one-point crossover, and an invert operation in the mutation operator [15]. Figure 4 portrays a chromosome (string of parameters) being used in the genetic optimization of NFNs. Here successive variables xl, x 2 , . . . , and x k denote input variables of a given system, MFmink and MFmaxk (k = 1 , 2 , . . . ; variable number) denote the parameters of a membership function for each input variable (while min and max denote a minimal and a maximal value of a fuzzy space, repectively), ~ is a learning rate and a stands for a momentum coefficient. 2.3. S e l f - O r g a n i z i n g P o l y n o m i a l N e u r a l N e t w o r k s We use PNN in the consequence structure of the SONFN. Each neuron of the network realizes a polynomial type of partial description (PD) of the mapping between input and output variables. The structure of the PNN is not fixed in advance but becomes dynamically organized during a growth process of the structure. In this sense, PNN becomes a self-organizing network. The PNN Mgorithm based on the GMDH method can produce an optimal nonlinear system by selecting significant input variables among dozens of those available in data and forming various types of polynomials. The GMDH is used in selecting the best ones in PDs according to a discrimination criterion. Successive layers of the SONFN are generated until we reach a structure of the best performance. The input-output relation formed by the PNN algorithm can be described in the following way
y = / (xl, x~,..., ~n).
(17)
The estimated output ~) of actual output y is -- / ( X l , X 2 , . - - , X n ) = COq- E
kl
CklXkl "}- E Cklk2XklXk2 q- E Cklk2k3XklXk2Xk3 q-''" ' (18) klk2 klk2k3
where CkS are the coefficients of the model to be optimized. To obtain the estimate Y, we construct a PD for each pair of independent variables existing in the problem. PDs use regression polynomials, Table 1. Next, we determine the coefficients of PD through the standard least squares error (LSE) method. The optimal structure of the model is determined stepwise: we form layers of PDs operating on pairs of variables and then select the best ones and discard the others. Once the final layer of the structure has been chosen, the node characterized by the best performance is selected as the output node and all other nodes in this layer are removed. Furthermore, all the nodes at the previous layers that do not affect this output node are also removed. Tracing the dataflow back to the previous layers leads to the removal of the excessive node.
Relation-Based Neurofuzzy Networks
899
Depending on the number of input variables as introduced below, two types of generic PNN architectures are considered for performance improvement of P N N model. Also, the structure of P N N is selected on the basis of the number of input variables and the order of PD in each layer. Two kinds of P N N structures, namely a basic P N N and a modified P N N structure are distinguished. Each of t h e m comes with two cases. For details, refer to Figure 5. Table 1. T y p e s of regression polynomial. O r d e r of t h e Polynomial
2
N u m b e r of Inputs 3
1 (Type 1)
Bilinear
Trilinear
Tetralinear
2 ( T y p e 2)
Biquadratic-1
Triquadratic-1
Tetraquadratic-1
2 (Type 3)
Biquadratic-2
Triquadratic-2
Tetraquadratic-2
4
T h e following types of t h e polynomials are considered: • Bilinear = co + elXl + c o x 2 , • Biquadratic-1 = Bilinear +c3x~ + c 4 x 2 + C S X l X 2 , • Biquadratic-2 = Bilinear +¢3XlX2 .
1 s~ ia}'er It...~
2 nd laver or higher
¢,. . . . . .
! . . . . . . . . . . . . . . . ,ll . . . . . . . . . . . . . . . . . . . . . .
l
,
i
!
I
xI X2
OOI
X4~'-
i "----~ ~ !
!
!
L.
Basic
x
1
_ . Ill, xp ~ L,=I(X,X)
-
-
~
•p ~ 2
ype
z.=f(x_, x.~)-Case . ',_ v
-
P PNN I ~,"-'Yi_ PD b - - z,
Modified PN N
"
[
~~ - - ~
I
~0n e r l c
. . . . . . . . . . . . . . . . . . . . . . . . . . .
....
I rv
~.-~
=----~~----'::'-:F:"
. . . . .
p-----I~ L
i
= fix
'
'
............................................
.
~X )
..
f
'
lI l[ II xX . ---r- -Fi ~ = i~lJ -I 1iI~7_,L__j
*
i.--..i~
-
Zi=f(x.,x.,Xr)-Casel i, IJ
Z,i=fi
.
]
i
x ~
I
.
, ..........
,
......
--.~
^ Y
..;_;;.
"l
..................
Advanced Type
:~[_-~ ....
I
I
Z'i=f'(x p) -Case 2
Figure 5. Configuration of t h e P N N structure.
900
S.-K. OH et al.
(a) Combination of 2 inputs.
(b) Combination of 2 or 3 inputs.
(c) Combination of 3 inputs.
Figure 6. The case of not generation nodes (or layers) anymore through growth process.
(a) Basic PNN structure the number of input variables of PDs is same in every layer. (b) Modified PNN s t r u c t u r e - - t h e number of input variables of PDs varies from layer to layer. CASE 1. The polynomial order of PDs is the same in each layer of the network. CASE 2. The polynomial order of PDs in the 2 nd layer or higher has a different or modified type in comparison with the one of PDs in the 1st layer. The selection and use of system input variables of the networks depends upon the topology of the PNN. According to the diverse topologies of PNN selected on the basis of the number of inputs and order of polynomial, we build various architectures of the PNN. If there are less than two (or three) input variables, the generic PNN algorithm does not generate a highly versatile structure as shown in Figure 6. To alleviate the problems, the advanced type of the architecture is taken into consideration that can be treated as the modified version of the generic type of the topology of the networks as shown in Figure 5. Accordingly, we identify also two types as the following. (1) Generic type : in case that the number of system input variables is four or higher, the generic PNN is used. (2) Advanced type : in case that the number of system input variables is less than three (or four), the advanced PNN is used. The basic and modified PNN architecture are shown in Figure 5, where z~ (Case 2) in the 2nd layer or higher denotes that the polynomial order of the PD of each node has a different or modified type each other in comparison with zi of the 1st layer. In the advanced type of Figure 5, the "NOP" node means the A th node of the current layer that is the same as the node of the corresponding previous layer (NOP denotes no operation). An arrow to the NOP node is used to show that the corresponding same node moves from the previous layer to the current layer. 2.4. S O N F N Topologies: Architecture Combined with Fuzzy Relation-Based NFN and PNN The SONFN is an architecture combined with the NFN and PNN as shown in Figures 7 and 8. These networks result as a synergy between the two other general constructs such as NFNs and PNNs. The SONFNs distinguish between two kinds of architectures, namely, basic and modified architectures. Moreover, for each architecture of the SONFN we identify two more detailed cases. (a) Basic architecture the number of input variables of PDs of PNN is the same in each layer. (b) Modified architecture the number of input variables of PDs of PNN is different in each layer. CASE 1. The polynomial order of PDs of PNN is same in every layer. CASE 2. The polynomial order of PDs in the 2 nd layer or higher of PNN has a different or modified type in comparison with the one of PDs in the 1st layer. As mentioned above, the topologies of the SONFN depend on those of the PNN used for the consequence part of SONFN. The above taxonomy is also summarized in Table 2.
Relation-Based Neurofuzzy Networks
901
%-. -. -. -. -o -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -.,
'. "
,,"
Premise part(NFN) f
'," "
Consequence part(PNN)
li Layer 4 Layer3 I Layer5 Layerl .I. .L _,. . v • , ~_ _ , / ~
......
Layer 2
|
L a y e r 1~
,...
:: :
I
......................
.----.
,, Xj t|
% %
% % %
"
r....................
',
i:
%
I
,
t,. ' - I'. ' - - . " : : . . ' . ' - - . "
Xp - ~
! I -J
t ...................
Partial Description
~
Xq - - ~ (Polynomial Function)
" ............
~-'r
...............
!
GenericType
::
I
(.,'g
q "~
gi=-'J~Xp~Xq)
............
,~ -.{' . ..... .. ... .. ... .. ... .. ... .. ... .. .. . . . . .
...............
Partiai Description ~ (Polynomial Function)
%,,,,,,,
Zi=AXP'~(t) Zi'=f"(X~q)
-
case
2_~
,,,, ,,,
r,'. ". ".-. %
". ".-.-.-.
~,
I I.-. !
,
,,S i--i ". ". ". ". ". "1.-. I
, 4.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-. I
r ......
• .................... I I I I I ~-L..L.
i ~i i % % %
,
i
I I I I I
D
%% r* 1.-. ". ". ".-. I
" .........
~ It
!
~l
------ . . . .
®',,X l
~ x/X.
v
n ".-2J/~//"¢-..~I/F ¢..~
.
"
r
".-.
". ".-.-.-.
.....
PD~/.[~
".-.
". "."
-, "
-,
:
:
::
: I I :
I i I I I I
I I i I I I
D, f it .
I I
: I
I I
,I
'
~ur
~'.-.~
/'~ ~" 1.-. ,.-.-.-. I I
I
I
i
!
,.----.
ll\\~::.~L. -]
I( PD )...._
~
,,
,,
,
: ~Ill ) :
,,
:.x,-<
,,
i % "1
. . . . . .
iI lu ii
el
.i I
"|
:
It
ll
|
ii
% ~1
: I
J* I$
I I
AdvancedType
~1 In
I I
',
-m--r .........
",---, . . . . . . . .
:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: I
I l
,,,
"~'
!
I I
Function)
I I
!
I
iI
.
. . . . .
,',',,,y
<~ g'"
~l (Polynomial
i I
ii
J
Partial Description
'
-case l
z i'=f'(xp)- case 2 [I (a) Basic SONFN architecture.
Figure 7. Configuration of SONFN architecture; combination of simplified fuzzy inference-based NFN and PNN.
902
S.-K. OH et al.
I
Premise part(NFN)
'. el
', "In
i
',
,,
I
II
,
"
ii
Consequence part(PNN)
Layer 2 Layer 4 , I Layer 3 / L ayer 5 Layer 1
Layer IV
~
~
~
:: ,•"
,,
"
Jl
~I
-- . . . .
"
Layer II
ii
al
s~
,
~_~ . .~ . ~ .- ' "
Y
~1
i|
Ii
,,
~Sv
%
~
~
~
, I
[
Xq
...........................
l
," I
Partial Description L (Polynomial Function) J " "~i
xp
* N
l I
, ....................
i.
',*.
~-~--r--'\~ !
,,
.X)
Xq
r
,
Generic Tvpe.
" ',',
I_.
Partial Description (Polynomial Function)
r
~
zi=flX,Xq,X,. ) - case 1 1 zi~-f'(Xp,Xq.,X,. ) case 2
%
-C"#-, ,
"q/-"r"
/" "-"
..................,..,.......-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.,-.,-.-.-.-.-.-.-.-.-.-.-.-.-.-..-.-.-.-.-.-.-.-.-..
: '
%
! ~ i / ~ ~ :
I
', ,,
~
'- ...................
"
I.
,
ii
-
I I
h
• •
I--L--L-
I l
i~ .. jl
|l
sl
,, fl
II
'I ~. I i.
[. . . . . .
II
I I
iI
.I,
.
.
.
.
.
~
.........
i
II
I
IJ
I
A
~J
v ::
•
.
ii |a -'~,'~
II l*
|I
lJ
% °I
......
Ii el
'I I
lI
!
II If ,s
ij !
%
ii
l
',%
I.,_,I.__T. a I
f ..................
.I
1
I
I
I
Advanced Type
-,1,4'#" 7, , .
I
'
(Polynomial Function) ,,,,,
(b) Modified SONFN archtecture. Figure 7. (cont.)
t, ii ~t
~!
' X
l d'
I............... "T = 11
R e l a t i o n - B a s e d N e u r o f u z z y Networks
~
.
:"
~
[
~ ~ ~ ~ . Premise part(NFN)
~
.
~
~
~
~
Consequence
.
903
:
,,
part(PNN)
t[
Layer 2Layer 3Layer 4
I
"
ii
:: :: x: ::
II iI
1 I I I ',:
:" .
]
....................
Generic Type
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
I''I
. . . . . . . . . . . . . . . . . .
(P"" .... ,.,,~.o~,o.,
:rq
1 " ~,:J(5,'~,)
.............
J. J . . . . . . . . . . . . . . . . . . . . .
Ii I,
I l
.-.-.- . . . . .
"-.-L- .......
- ......
(,.,,.°,.,., ~..~,,..)
~
. . . . . . .
•
. I
. Z .
I j__~
.-
-.-'2
....
zi'=f'(x e,.~l ) - c a s e 2
"3. -~-". . . . . . . . . . .
• . . . . . . . . . . . . . . . . . . . .
:,,
I
.j
"~.
,
I
ii . . . . . . .
•
Q
I
,,
l
:: x ~ - - ¢
,'i i
~1
i . . . . . .
.J--J
. . . . . .
"
"l
:',
y
',',
. ................
.,
X/ -'~
,, ,i
:i :,,
:', ,, I.--flr..t--d
Ii ~ I
. . . . . . . . . . . . . . . . . .
..........
l
L . . . . . . I j I
II"
-I---I i l I I
- -
Advanced T y p e . ,II
. . . . . . . l I !
"
-
......
-
i~Z
:.p)
case2
(a) Basic S O N F N architecture. F i g u r e 8. C o n f i g u r a t i o n of S O N F N architecture; c o m b i n a t i o n of linear fuzzy inferencebased NFN and PNN.
I
904
O H et al.
S.-K.
Premise part(NFN) ::" ,,
I
:,,
J
Consequence part(PNN) it
Layer2Layer3Layer4 Layer 1~
I
.i
ii
.............................
Layer 1 % %
~. . . . .
~
,, x l - ~
o
~
[
~
•
:: " Xi - ' ~
"•:
C
i~
~
~
.............
I I! ! I I
,i
,,
|
.]~ ....................
"
i I[
*, it
'*" Me
,,,. tt i.
Generic Type ,.
!
~.~. t~ d
::
i
I I! ! | 1
!
,,
~
i
r--r ,","
.............
..
"
i
. .~--~g~,'
,.," ~
,, iI iz
Layer~
,,
ml ul
,, ::,, ,,
..
h7
uV J
I
"
"~
l?artialDescripti::'
[Xp ~
11
zi=J(xp'xq'x)-casel
~
Partial Descriptioa
%
........... .
"-'-'-'-'-'-'-'-'-'-'-'-','-.'-'-'-'-'-'-'~,
.
.
.
.
.
. . . . . . • . . . . . . . . . . . . . . . . . . . .
l
.
. .
. .
. .
. .
. .
n . . . . . . . . . . . . . . . . . . . .
D
ii
•
xI
. .
. .
. .
. . . . . . .
l= .P . . . . I' - k . . . ! ! .t__l.
. .
. .
. . . . . . . . . . .
. .
.
j. .
.
! . .
. .
. .
.
. .
. .
. .
. .
. .
, it
. . . . . .
_L__L_
m,
i
:
!
. . . . . . . . . . . . .
I
,,~
,, ,~'[ ' ~ , , " m* mt
!
0
:
X/
i.
..................
, )
".
~
..........................
~-v--r-"
~......
"-~ ....... ' !
I'. . . . . . . . .
:,
+.t [ .........,
'
!,
4
• u
. . - - - - -.-
-.-.-.-.-
(b) Modified SONFN architecture.
. .
...,
~,
I
Figure 8. (cont.)
Advanced Type
. . . . . . .
......... ,
I
" . z, =-7 tx~1 - case 2
Relation-Based Neurofuzzy Networks
905
Table 2. A family of the topologies of the SONFN. Layer of PNN
Number of Input Variables of Polynomial
Order of Ploynomial
1st Layer
p
Type P
2nd or Higher Layer
SONFN Architecture (1) p = q : Basic SONFN (a) P = Q : Case 1 (b) P ~ Q : Case 2 (2) p # q : Modified SONFN (a) P = Q : Case 1 (b) P C Q : Case2
Type Q
(p = 2 , 3 , 4 , q = 2,3,4; P = 1,2,3, Q = 1 , 2 , 3 )
The SONFN architecture combined with the fuzzy relation-based NFN and PNN under discussion is shown in the Figures 7 and 8. Let us recall that the fuzzy inference (both simplified and linear) based NFN is constructed with the aid of the space partitioning realized by fuzzy relations. We also identify two types as follows• - Generic type of SONFN; Combination of the fuzzy inference based NFN and the generic PNN. - Advanced type of SONFN; Combination of the fuzzy inference based NFN and the advanced PNN. 2.5. T h e A l g o r i t h m i c F r a m e w o r k o f S O N F N ]In this section, we elaborate on the algorithmic details of the design method by considering the functionality of the individual layers in the network architectures (refer to Figures 2, 5, 7, and 8). The design procedure for each layer in the premise and the consequence of SONFN comprises of the following steps. THE PREMISE OF S O N F N : N F N . LAYER 1. Distributing the signals to the nodes in the next layer as input layer. LAYER 2. Computing activation degrees of linguistic labels (fuzzy sets; small, large, etc.). Each node in this layer corresponds to one linguistic label (say, small, large, etc.) of each input variable positioned in the first layer. The layer produces a degree of satisfaction (activation) of this label by the input. In essence, we compute a possibility degree (measure) of the numeric input with respect to the given fuzzy set, refer to Figure 9. LAYER 3. Computing fitness of premise rule. Every node in this layer is a fixed node labeled l-I, whose o u t p u t is the product of all the incoming signals. /£j = #A (Xl)
X #B
(Xi),
A, B = small, large, etc.
(19)
~/ik+l
zik(x) r
min~
//
x. ;
I
//
mix~
Figure 9. Triangular membership functions and the resulting computations of the actibation levels.
906
S.-K. OH e t
al.
A node in this layer represents one fuzzy rule and each node output represents the firing strength of a rule. LAYER 4. Normalization of a degree of activation (firing) of the rule. pj-
,
(20)
/zj j=l
where n denotes a number of rules. LAYER 5. Multiplying a normalized activation degree of the rule by the corresponding connection weight. (i) Simplified fuzzy inference based NFN #j .wj
(21)
A., #j
j=l
(ii) Linear fuzzy inference based NFN
fJ = #5 x cys, Cy 5 = woj
+ wlj
= Ewij
(22) " xl
+ w2j
"x,
" x2 + " " + w~j " x~
(where i = 0 , 1 , 2 , . . . and x0 = 1).
(23)
i f5
is given as the input variable of the PNN which is the consequence structure of SONFN.
LAYER 6. Computing output of NFN: The output in the 6 th layer of the NFN is inferred by the center of gravity method. (i) Simplified fuzzy inference based NFN n
n
j=l
j=l
(24) j=l
E ~tj j=l
(ii) Linear fuzzy inference based NFN fi
J=~
n ij:Z n s cyj=X%: c y ` - zn" `
J=~
J=~ ~ #5
xl÷
J=~
j=l
*w'S
(25)
~ #~
5=1
THE CONSEQUENCE OF S O N F N : P N N .
STEP 1. Configuration of input variables: denotes a rule number.
xl
=
fl,
x2
=
f2,
...
, x,~
=
f~(n
=
j); where j
STEP 2. Forming a PNN structure: here we select input variables and fix an order of a PD. STEP 3. Estimate the coefficients of a PD: The vector of coefficients of the PDs (C~) in each layer is determined by the standard least-squares method that assumes a well known form Ci = ( X { X ~ -1 ) X/TY,
(26)
w h e r e / s t a n d s for a node number. This procedure is implemented repeatedly for all nodes of the layer and also for all layers of consequence part of SONFN.
Relation-Based Neurofuzzy Networks
907
STEP 4. Choosing PDs in case that the training and testing dataset are taken into consideration: each PD is constructed and evaluated using the training and testing dataset, respectively. Then, we compare the values of the performance index and select PDs using an aggregate performance index with a sound balance between approximation and prediction capabilities. We may use (i) the predetermined number W of the PDs (width of the layer) or (ii) go for all of them whose performance index is lower than a certain prespecified value. Especially, the method of (ii) uses the threshold criterion ~ to select the node with the best performance in each layer. e = Emin + 5,
(27)
where 0 is a new value of the criterion, ~ is a positive constant (increment) and Emin denotes the performance index with smallest value obtained in the each layer. STEP 5. Termination condition: we take into consideration a stopping condition (Emin _> Emin.) for better performance and the number of iterations (size of the network) predetermined by the designer. Here Emin is a minimal identification error at the current layer while Z m i n . denotes a minimal identification error at the previous layer. STEP 6. Determining new input variables for the next layer: the outputs of the preserved PDs sep/e as new inputs to the next layer. In other words, we set Xl~ = zl~, x2i = z 2 ~ , • • • , x w ~ = z w i . The consequence part of SONFN is repeated through a sequence of Steps 3-6. 2.6 M o d e l S e l e c t i o n ]~br the NASA software dataset, our model selection procedure is based on seeking a sound compromise between approximation and generalization errors. The main performance measure that we use in this paper is the MMRE (the mean magnitude of relative error) of (9). For evaluation of generalization ability, many estimates have been proposed in the literature; the most popular ones being the holdout estimate and the k-fold cross-validation estimate [16]. The holdout estimate is obtained by partitioning the dataset into two mutually exclusive subsets called training and test sets. The error estimate on the test set is used to assess generalization ability. On the other hand, the k-fold cross-validation estimate is obtained by a sample reuse technique. The dataset is divided into k mutually exclusive subsets of almost equal size, k - 1 subsets are used for training, and the k th is used for prediction. This process is repeated k times, each employing a different subset for prediction. When k is equal to data size, it is called leave-one-out cross-validation (LOOCV) estimate. In this study, we employ the LOOCV estimate of generalization error because of two reasons. Fir,st, it possesses good mathematical properties [17]. Second, it seems to be particularly suited for software engineering applications where the best available data are relatively small sets [18]. Thus, our model selection is based on the analysis of LOOCV estimate of generalization error for SONFN models in NASA dataset. 3.
EXPERIMENTAL
STUDIES
In this section, we illustrate the development of the SONFN and show its performance for well known and widely used datasets in software engineering. The first one is the NASA dataset [12]. The second one is medical imaging system (MIS) [13]. 3.1 N A S A S o f t w a r e D a t a The experimental studies are concerned with a well-known software effort dataset from NASA [12]. The dataset consists of two independent variables, viz. developed lines of code (DL) and methodology (ME), and one dependent variable, viz., effort (Y). DL is in KLOC and Y is
S.-K. OH et al.
908
Table 3. The data set of the NASA software project. Project Number
Independent
Variables
Dependendt Variable
DL
ME
Effort (Y)
90.2
115.8
46.2 46.5 54.5 31.1 67.5 12.8 10.5 21.5
30 20 19 20 35 29 26 34 31
3.1 4.2 7.8 2.1 5.0 78.6 9.7 12.5 100.8
26 19 31 28 29 35 27 27 34
10 11 12 13 14 15 16 17 18
96.0 79.0 90.8 39.6 98.4 18.9 10.3 28.5 7.0 9.0 7.3 5.0 8.4 98.7 15.6 23.9 138.3
in m a n - m o n t h s . Here, M E is a c o m p o s i t e m e a s u r e of m e t h o d o l o g i e s e m p l o y e d in this N A S A software e n v i r o n m e n t . T h e d a t a set is shown in T a b l e 3. I n t h e following, we develop software effort e s t i m a t i o n m o d e l s for t w o collections of i n d e p e n d e n t variables, i.e., DL a n d (DL, M E ) . 3.1.1. Results
of NFN
modeling
A c c o r d i n g to each S c h e m e of N F N s t r u c t u r e s , t h e i d e n t i f i c a t i o n shown in T a b l e 4. N o t e t h a t P I d e s c r i b e s a p e r f o r m a n c e i n d e x for E_PI c o n c e r n s t h e p e r f o r m a n c e i n d e x for t h e t e s t i n g d a t a s e t using ( L O O C V ) . G A s help o p t i m i z e l e a r n i n g rate, m o m e n t u m coefficient,
errors of P I a n d E_PI are t h e t r a i n i n g d a t a s e t while a leave-one-out validation a n d t h e p a r a m e t e r s of t h e
m e m b e r s h i p functions. G A s was r u n for 100 g e n e r a t i o n s w i t h a p o p u l a t i o n of 60 i n d i v i d u a l s . E a c h s t r i n g was 10 b i t s long. T h e crossover r a t e was set to 0.6. T h e p r o b a b i l i t y of m u t a t i o n was equal to 0.1. I n each N F N s t r u c t u r e , two m e m b e r s h i p f u n c t i o n s for each i n p u t v a r i a b l e are used a n d 500 i t e r a t i o n s for t h e n u m b e r of t r a i n i n g cycles are used. Table 4. Performance index of the NFN. NFN
1 System Input (DL)
2 System Inputs (DL,ME)
PI
E_PI
PI
E PI
Simplified Fuzzy Inference
0.2870
0.2990
0.2115
0.2415
Linear Fuzzy Inference
0.2094
0.2462
0.1544
0.1967
3.1.2. R e s u l t s o f S O N F N m o d e l i n g W e now p r e s e n t t h e d e t a i l s of S O N F N m o d e l i n g using t h e m e t h o d o l o g y d e s c r i b e d in t h e previous Section 2.
Relation-Based Neurofuzzy Networks 3.1.2.a. In case of 1 system
input
909
(DL)
T h e g o a l is t o seek a p a r s i m o n i o u s m o d e l , w h i c h p r o v i d e s a g o o d fit t o t h e D L a n d Y d a t a of N A S A software p r o j e c t d a t a a n d e x h i b i t s g o o d g e n e r a l i z a t i o n c a p a b i l i t y . T a b l e s 5 a n d 6 s u m m a r i z e t h e r e s u l t s of t h e p r e f e r r e d a r c h i t e c t u r e s a c c o r d i n g t o a d v a n c e d t y p e a n d a r c h i t e c t u r e (basic or m o d i f i e d ) . Table 5. Performance index of SONFN with the simplified fuzzy inference based NFN. Premise Part; NFN SONFN
Number of Number of Connection MFs Point Inputs & PolyNomial Type Case 1
2
1
Case 2
2
1
Case 1
2
1
Modified Case 2 Architecture
2
1
Case 2
2
2
Basic Architecture Advanced Type
Consequence Part; PNN PI
E_PI
1.2356 0.2775 0.3166 0.1850 1.2356 0.4720
Layer
2 inputs Type 2 2 inputs Type 3 --* 2 2 --* 3 inputs Type 2
1 2 1 2 1 2
0.2486 0.1801 0.2479 0.1748 0.2486 0.1797
2 --* 3 inputs
1 2
0.2479 .03166 0.2285 0.2866
1
0.4719 i 0.5344
2
0.2479 0.5344 i
Type3 -~ 2 1 --* 2 inputs Type 1 --* 2
Table 6. Performance index of SONFN with the linear fuzzy inference based NFN. Premise Part; NFN SONFN
PI
E_PI 0.2166 0.1572
Case 1
2
1
2 inputs Type 2
1 2
0.1349 0.1143
Case 2
2
1
2 inputs Type 3 --* 2
!Case 1
2
1
2 -* 3 inputs Type 2
1 2 1 2 3
0.1873 0.2427 0.1349 0.1582 0.1349 0.2166 0.0901 21.283 0.0822 0.1313
Case 2
2
1
2 --* 3 inputs Type 1 -* 2
1 2 3
0.2117 0.1349 0.0877
Basic Architecture Advanced Type
Consequence Part; PNN
Number of Connection Number of Inputs Layer MFs Point & Polynomial Type
Modified Architecture
0.2596 0.2166 0.1309
I n T a b l e 5, t h e basic S O N F N in Case 2 is t h e p r e f e r r e d a r c h i t e c t u r e of t h e n e t w o r k in t h e a d v a n c e d t y p e a n d its d e t a i l e d t o p o l o g y is v i s u a l i z e d in F i g u r e 10. T h e values of t h e p e r f o r m a n c e i n d e x of t h e simplified fuzzy inference b a s e d N F N o p t i m i z e d b y G A s are P I = 0.2870 a n d E_PI = 0.2990. W h e n c o n s i d e r i n g b o t h P I a n d E_PI, t h e m i n i m a l value of t h e p e r f o r m a n c e index, t h a t is P I = 0.1748, E_PI = 0.1850 are o b t a i n e d b y using T y p e 3 in t h e 1 st layer a n d T y p e 2 in t h e 2 nd layer or h i g h e r ( T y p e 3 4 2 ) w i t h 2 n o d e i n p u t s . T h e form of each p o l y n o m i a l t y p e is shown in 'Table 1. F i g u r e 10 i l l u s t r a t e s an o p t i m a l a r c h i t e c t u r e in t h e a d v a n c e d t y p e of t h e S O N F N t h a t is c o m p o s e d of simplified fuzzy inference b a s e d N F N a n d P N N w i t h 2 i n p u t s - T y p e 3--*2 topology. T h e w a y in w h i c h l e a r n i n g h a s b e e n realized is shown in F i g u r e 11 w h e r e t r a i n i n g errors ( p e r f o r m a n c e i n d e x ) a r e i l l u s t r a t e d . I n F i g u r e 5, Q is t h e A th n o d e of t h e each c o r r e s p o n d i n g layer used for t h e g e n e r a t i o n of t h e o u t p u t Y, ~ is t h e A th n o d e of t h e each c o r r e s p o n d i n g layer used for t h e g e n e r a t i o n of t h e o u t p u t a n d i n d i c a t e s t h e o p t i m a l n o d e in each layer.
910
S.-K. OH et al.
DL
Figure 10. The optima' topo'ogy of the advanced and basic SONFN in Case 2. (Simplified fuzzy inference based NFN, 2 node inputs and Type 3 ~ 2 PNN.) 1
0.9 0.8
0.7 0.6
0.4
0.3
0.10
Part :
~ - - ~ ~--------~/
Consequence Part : PNN
Premise 1oo
2°0
300
Iteration
,oo Layer
Figure 11. Learning procedure of the advanced and basic SONFN in Case 2. (Simplified fuzzy inferecne based NFN, 2 node inputs and Type 3 --* 2 PNN.)
Figure 12. The optimal topology of the advanced and modified SONFN in Case 2. (Linear fuzzy inference based NFN 2 --* 3 inputs-Type 1 ---* 2 PNN.)
In light of the values reported in Table 6, the modified SONFN (Case 2) emerges as a preferred architecture of the network in the advanced type. Figure 12 shows an optimal architecture in advanced type of the SONFN that is composed of the linear fuzzy inference based NFN and PNN with 2-*3 inputs-Type 1-.2 topology. The values of the performance index of the linear fuzzy inference based NFN optimized by GAs are PI = 0.2094 and E_PI = 0.2462. When considering both PI and E_PI, the minimal value of the performance index, that is PI = 0.0877, E_PI = 0.1309 are obtained by using 2 node inputs and Type 1 in the 1st layer and 3 node inputs and Type 2 in the 2nd layer or higher(2-*3 inputs-Type 1-.2). The training errors of SONFN are illustrated in Figure 13. In Figure 12, the "NOPA" node means the A TM node of the current layer that is the same as the node of the corresponding previous layer(NOP denotes no operation). An arrow to the NOP node is used to show that the corresponding same node moves from the previous layer to the current layer.
Relation-Based Neurofuzzy Networks I
T ~ T
T
1
911
1
r
T
0.9 0.8
0.7 0.6
Premlse Part :
Consequence Part :
/[---
PNN
0.5 :~ O.4 0.3 0.2 0.1 0
0
100
200 300 Iteration
400
5~~0
1
2 Layer
Figure 13. Learning procedure of the advanced and modified SONFN in Case 2. (Linear fuzzy inference based NFN, 2 --* 3 inputs-Type 1 --* 2 PNN.)
3.1.2.b.
In case of 2 system
inputs
(DL,ME)
No w we d e v e l o p effort e s t i m a t i o n m o d e l b a sed on t w o i n d e p e n d e n t v a r i a b l e s D L an d ME. For t h e simplified a n d linear fuzzy inference based N F N , t h e p e r f o r m a n c e i n d ex es o f S O N F N are sh o wn in T a b l e 7 a n d 8, respectively. T a b l e 7 s u m m a r i z e s t h e values o f t h e p e r f o r m a n c e i n d e x for t h e S O N F N w i t h t h e simplified fuzzy i n f eren ce b a s e d N F N . In T a b l e 7, t h e m o d i f i e d S O N F N in C a s e 2 is t h e p r e f e r r e d a r c h i t e c t u r e of t h e n e t w o r k in t h e generic or t h e a d v a n c e t y p e respectively. A n o p t i m a l a r c h i t e c t u r e of t h e
Table 7. Performance index of SONFN with the simplified fuzzy inference based
NFN.
SONFN
Premise Part; NFN
Consequence Part; PNN
Number of MFs
Number of Inputs Layer & Polynomial Type
Case 1
2×2
Case 2
2x2
Basic Architecture Generic Type Case 1
2x2
Case 2
2x2
Modified Architecture
Case 1
2x2
Basic Architecture Advanced Type
Case 2
2x2
Case 1
2x2
Case 2
2x2
Modified Architecture
[ PI !0.1369
i E PI
2 inputs
1
Type 3
2
0.1085 0.0600
0.1130
2 inputs
1
0.1369 I 0.1130
Type 3 ---*2
2
0.1048 [ 0.0621
2 -~ 3 inputs
1
[ 0.1369 0.1130
Type 3
2
0.0893 0.0527
2 --* 3 inputs
1
0.1369 0.1130
Type 3 --* 2
2
0.0493 0.0565
3 inputs
1
0.0695 0.1890 0.0420 0.0417
Type 2
2
3 inputs
1
0.0695 0.1890
Type 2 --* 3
2
0.0390 0.0429
3 ---*4 inputs
1
0.0695 0.1890
Type 2
2
0.0189 0.0397
3 --* 4 inputs
1
0.0695 0.1890
Type 2 --+ 3
2
0.0231 0.0252
912
S.-K. OH et al. Table 8. Performance index of SONFN with the linear fuzzy inference based NFN.
SONFN
Premise Part; NFN
Consequence Part; PNN
Number of MFs
Number of Inputs Layer & Polynomial Type
Case 1
2x2
Case
2×2
Basic Architecture 2
2 inputs Type 3
PI
E_PI
0.1269 0.1229 0.1166 0.1269
0.1595 0.1712 0.1550 0.1595
2 inputs
1 2 3 1
Type 3 ~ 2
2
i 0.1122 0.1695
2 ---*3 inputs
3 1
I 0.0770 0. 1220 0.1269 0.1595
Type 3
2 3
0.1040 0.1257 0.0680 0.1042
2 -+ 3 inputs Type 2 --* 3
1 2 3
10.1283 0.1949 0.0964 0.2031 0.0630 0.0736
3 inputs Type 3 3 inputs Type 2 --* 3 2 --* 4 inputs Type 2
i 2 i 2 1 2
0.1084 0.0940 0.0829 0.0603 0.1283 0.0131
2 --~ 4 inputs Type 3 --* 2
1 2
0.1269 0.1061 0.0137 0.0290
I
Generic Type Case 1
2x2
Case 2
2x 2
Case I
2x 2
Case2
2x 2
Case 1
2x 2
Case 2
2x 2
Modified Architecture
Basic Architecture Advanced Type Modified Architecture
0.1275 0.0378 0.1583 0.0452 0.1269 i 0.0205
S O N F N in t h e a d v a n c e t y p e is v i s u a l i z e d in F i g u r e 14. I n F i g u r e 14, t h e N F N p a r t of t h e n e t w o r k uses two m e m b e r s h i p f u n c t i o n s for each i n p u t variable, so t h e a r c h i t e c t u r e has 4 rules. T h e values of t h e p e r f o r m a n c e i n d e x of t h e N F N o p t i m i z e d b y G A s are 0.2115 for t h e t r a i n i n g d a t a a n d 0.2415 for t h e t e s t i n g d a t a . T h e P N N p a r t of t h e n e t w o r k s is c o n s t r u c t e d b y using T y p e 2 w i t h 3 n o d e i n p u t s in t h e 1st layer a n d T y p e 3 w i t h 4 n o d e i n p u t s in t h e 2 nd layer(3--~4 i n p u t s a n d T y p e 2 4 3 ) . T h e final r e s u l t s of t h e S O N F N t o p o l o g y are P I = 0 . 0 2 3 1 a n d E _ P I = 0 . 0 2 5 2 . F i g u r e 15 i l l u s t r a t e s t h e p e r f o r m a n c e o f t h e o b t a i n e d networks.
ME
39
Figure 14. The optimal topology of the advanced and modified SONFN in Case 2. (Simplified fuzzy inference based NFN, 3 --+ 4 node inputs an Type 2 --* 3 PNN.) F o r t h e S O N F N w i t h t h e linear fuzzy inference b a s e d N F N , t h e values of t h e p e r f o r m a n c e i n d e x a r e s u m m a r i z e d in T a b l e 8. T h e N F N p a r t of t h e S O N F N uses two m e m b e r s h i p f u n c t i o n s for each i n p u t variable. Therefore, t h e a r c h i t e c t u r e e x h i b i t s four rules as well. T h e values of t h e p e r f o r m a n c e i n d e x of t h e N F N o p t i m i z e d b y G A s is e q u a l to 0.1544 for t h e t r a i n i n g d a t a a n d 0.1967 for t h e t e s t i n g d a t a . I n T a b l e 8, t h e m o d i f i e d S O N F N in C a s e 2 or C a s e 1 is t h e p r e f e r r e d a r c h i t e c t u r e of t h e n e t w o r k in t h e generic or t h e a d v a n c e t y p e respectively. A n o p t i m a l a r c h i t e c t u r e of t h e S O N F N in t h e generic t y p e is v i s u a l i z e d in F i g u r e 16 while t h e l e a r n i n g
Relation-Based Neurofuzzy Networks
I
r
I
r
913
r
0.9
0.8 0.7
Part :
o b0
~ - - ~
ConsequencePart : PNN
Premise
0.6 0.5 0.4 [.-.
0.3
O. 0
100
200
300
400
5 0
1
Iteration
2
Layer
Figure 15. Learning procedure of the advanced and modified SONFN in Case 2. (Simplified fuzzy inference based NFN, 3 --* 4 node inputs, and Type 2 ---*3 PNN.)
f DL
ME -c
....
:
'
~
Y
Figure 16. The optimal topology of the generic and modified SONFN in Case 2. (Linear fuzzy inference based NFN, 2 --* 3 node inputs, and Type 2 ---*3 PNN,)
process is quantified in Figure 17. The shadowed nodes indicate neurons which have the optimal polynomial in each layer(the optimal being expressed from the viewpoint of PI as well as E_PI). In Figures 16 and 18, is the A th node of the each corresponding layer that is not used for the generation of the ~), and is the A th node of the each corresponding layer that is not used for the generation of the output and indicates the optimal node in each layer. Also, the solid line is used for the generation of the output and the dashed line is not used for the generation of the Z). And the values of the performance index vis-d-vis number of layers of the modified SONFN wit]h the generic or advanced type related to the preferred architectures of the network are shown in Figures 20 and 21, respectively. In Figures 20 and 21, the notation 'Type p --~ q~(p, q = 1, 2, 3) indiLcates that the polynomial order of the consequence part of the fuzzy rules in a node changes from Type "p" in the i st layer to Type "q" in the 2 nd layer and higher. And A(-), B(.), C(.),
914
S.-K. OH et al. 1 0.9 0.8 0.7 0.6
P r e m i s e Part :
C o n s e q u e n c e Part :
NFN
PNN
0.5 ....E
o.4! 0.3' 0.2 0.11 0
0
1O0
200 300 Iteration
400
tO
1
2
Layer
Figure 17. Learning procedure of the generic and modified SONFN in Case 2. (Linear fuzzy inference based NFN, 2 --* 3 node inputs, and Type 2 --* 3 PNN.)
DL
ME
Figure 18. The optimal topology of the advanced and modified SONFN in Case 1. (Linear fuzzy inference based NFN, 2 --* 4 node inputs, and Type 2 PNN.) and D(*) denote the optimal node numbers according to each type of the polynomial. Namely, the node numbers of the 1st layer represent the node numbers located in the 4 th layer of NFN, and the node numbers of each layer in the 2 nd layer or higher represent the output node numbers of the preceding layer, as the optimal node which has the best output performance in the current layer. Especially, identification error in the layer 0 of Figures 20 and 21 means the optimal output performance index of linear fuzzy inference based NFN as the premise part of SONFN. Table 9 shows a comparative analysis t h a t involves a number of previously developed models. In comparison with them, the SONFN comes with high accuracy and improved prediction (generalization) capabilities.
Relation-Based Neurofuzzy Networks
915
Table 9. Comparison of identification error with the performance obtained for some models reported in the literature. System Training Generalization Input (PI) (E_PI)
Model
DL
0.1579
DL,ME
O.O87O
0.1870 0.1907
DL
0.2870
0.2990
DL,ME
0.1444
0.1702
DL
0.2094
0.2462
DL,ME
0.1457
0.1679
DL
0.2870
0.2990
DL,ME
0.2115
0.2415
DL
0.2094
0.2462
DL,ME
0.1544
0.1967
DL
0.1748
0.1850
0.0493
0.0565
0.0231
0.0252
0.0877
0.1309
Modified Architecture Case 2 DL,ME
0.0630
0.0736
Advanced Modified Type Architecture Case 2 DL,ME
0.0131
0.0205
Shin and GoeFs RBF Model [12] Simplified Fuzzy Inference [19] Fuzzy Set-Based NFN LinearFuzzy Inference Simplified Fuzzy Inference [19] Fuzzy Relation-Based NFN Linear Fuzzy Inference Advanced Basic Type Architecture Case 2
Simplified Fuzzy Generic Basic Case 2 DL,ME Inference Based Type Architecture Our Model NFN (SONFN) Advanced Modified Type Architecture Case 2 DL,ME Advanced Basic Type Architecture Case 2 Linear Fuzzy Inference Based NFN
1
Generic
Type
,
,
DL
=
0"9f 0.8 i 0.7
\
o.o,\
Premise Part :
Consequence
Part :
PNN
'-~
0.4 0.3 0.2
OA~ 0
0
t
J
1O0 200
i
I
,
300 400
)0
I 1
I
Iteration Layer Figure 19. Learning procedure of the advanced and modified SONFN in Case 1. (Linear fuzzy inference based NFN, 2 --* 4 node inputs, and Type 2 PNN.) 3.2
Medical
Imaging
System
We consider a n m e d i c a l i m a g i n g s y s t e m (MIS [13]) subset of 390 m o d u l e s w r i t t e n in Pascal a n d F O R T R A N for modeling. T h e s e m o d u l e s consist of a p p r o x i m a t e l y 40,000 lines of code. To desiign a n o p t i m a l m o d e l from t h e MIS, we utilize 11 s y s t e m i n p u t variables such as, LOC, CL,
916
S.-K. OH et al.
(A) Type 1 -~ 3; @ (C) Type 2 --~ 3; [] (B)Type3-~ 1; O (D)Type3-~ 2; V 0.2
iiiiiiiiiiiiiiiiiiiiiiiiiii
0.1~
iii ........... i ............
0.11
..............
,~ 0,1, ..~ 0.12
~
o.1 0.08 0.06 0.04
0
1
2
3
Layer (a) Training errors (PI).
[ (A)TypeI: 3; @ (C)'rype2-~ 3; ~l (B)Type3
1; •
(D)Type3-~ 2;
l I
0
0
~
i
0
: ' :
: \
6 1
~
2
~:(9
t6 18) I
3
Layer
(b) Testing errors (E PI). Figure 20. Performance index of the generic and modified SONFN in Case 2. (Linear fuzzy inference-based NFN.)
TCHAR, TCOMN, MCHAR, CDHAR, N, NE, NF, VG, and BW. And as output variable, "NC" is used, See Table i0. Also, we use 4 system inputs [TCOMN, MCHAR, CDHAR, N] among II system inputs. Here 4 system inputs are selected from structure identification by GMDH method. When applying any modeling technique, an assessment of predictive quality is important. Data splitting is a modeling technique that is often applied to test predictive quality. Applying this technique, one randomly partitions the data set to produce two data sets. The first 60% data
Relation-Based Neurofuzzy Networks
917
(A) Type2;@ (C) Type3:[] (B) Type2;O (D) T y p e 3 ; V 0.16
I
I
I
E
0.1~ 0.12
E
0.1
.=
.~ 0.08 0.06
0.04 0.02 0
0 Layer (a) Training errors (PI). 1
(A) Type 2 ; @ (C)Type3 ; I~[ (B) Type 2 ; O (D) Type 3 ; ~'] I
0.18
-
-
i
==1
:~
~. oo
i
........ ~
~
i
-
~
!
i
~
~
i Tt'! ~
;
\.
~.x,,
,o~ i .......
1
\B:o ~ 6 |
I
oo, ........ i ........ i ........ i ........... "i ........ i ........... 0.02
-
-
'
'
L 1
L 2
Layer
(b) Testing errors (E_PI). Figure 21. Performance index of the advanced and modified SONFN in Case 1. (Linear fuzzy inference-based NFN.) set is used for fitting t h e models. T h e r e m a i n i n g 40% d a t a set, t h e t e s t i n g d a t a set, provides for q u a n t i f y i n g t h e predictive q u a l i t y of the fitted models. U s i n g MIS d a t a s e t , t h e regression e q u a t i o n is o b t a i n e d as follows. N C = - 1 . 8 3 7 1 + 0.14097- L O C - 0.14115 • CL - 0.0026487 • T C H A R +0.10312 • T C O M N + 0.0019233- M C H A R + 0.0086053. C D H A R
(28)
- 0 . 0 2 4 5 0 8 • N + 0.16458 • N E - 0.20678 • N F + 0.16353 • V G - 0.95354 • B W T h i s simple m o d e l comes with t h e value of P I = 40.056 a n d E_PI = 36.322 b y (7) for 11 system i n p u t s . W e will b e u s i n g as a reference p o i n t w h e n discussing S O N F N proposed in this paper.
S.-K. OH et aL
918
Table 10. Input and output variables of medical imaging system. length of code with comments
LOC
length of code without comments
CL TCHAR
number of characters
TCOMN
number of comments
MCHAR
number of comments characters
CDHAR
number of code characters program length
N NE
estimated program length
NF
Jensen's program length
VG
McCabe's cyclomatic number
BW
Belady's bandwidth metric
NC
number of changes
3.2.1. R e s u l t s o f N F N m o d e l i n g For t h e N F N s t r u c t u r e s , t h e i d e n t i f i c a t i o n errors of P I a n d E_PI are sh o w n in b el o w T a b l e 11. G A s help o p t i m i z e l e a r n i n g rate, m o m e n t u m coefficient, an d t h e p a r a m e t e r s of t h e m e m b e r s h i p functions. In N F N s t r u c t u r e , t w o m e m b e r s h i p f u n c ti o n s for each i n p u t v a r i a b l e are used. So N F N struct u r e is r e p r e s e n t e d by 16 fuzzy rules for t h e 4 s y s t e m inputs. For t h e l e a r n i n g of N F N , t h e n u m b e r of t r a i n i n g cycles is 500 iterations. Table 11. Performance index of the NFN. Model
Number of System Inputs
PI
E_PI
Regression
11
40.056
36.322
Simplified Fuzzy Inference Based NFN
4
46.084
38.559
Linear Fuzzy Inference Based NFN
4
41.412
34.048
Table 12. Performance index of the SONFN with 4 system inputs.
SONFN
Premise Part; NFN
Consequence Part; PNN
Number of MFs Fuzzy Inference
Number of Inputs & Polynomial Type Layer
2x2x2x2 Simplified 2x2x2x2 Generic Basic Case 1 Type Architecture
Simplified 2x2x2x2 Simplified 2x2x2x2
Simplified
E_PI
3 inputs
1
49.264 27.766
Type 2
2
40.753 17.898
3 inputs
1
50.957 26.599
Type 3
2
4 8 . 3 1 23.265
4 inputs
1
36.602 26.516
Type 2
2
42.776 16.066
4 inputs
1
48.69 25.441
Type 3
2
45.187 18.025
2x2x2x2
3 inputs
1
47.229 27.176
Linear
Type 2
2
43.595 21.375
3inputs
1
48.439 29.423
Type 3
2
44.344 22.186
4inputs
1
40.271 23.677
Linear
Type 2
2
37.373 16.679
2x 2x 2x2
4inputs
1
40.055 23.025
Linear
Type 3
2
35.748 17.807
2× 2 x 2 x Generic Basic Case 1 Type Architecture
PI
2
Linear 2x 2x2x
2
Relation-Based Neurofuzzy Networks
919
3.2.2. R e s u l t s of S O N F N modeling Table 12 summarizes the results of the preferred architectures according to SONFN. Here we select the SONFN in Case 1 with the 3 inputs-Type 2 for the simplified fuzzy inference based NFN and its detailed topology is visualized in Figure 22. Also, for the SONFN with linear fuzzy inference based NFN, the 4 inputs-Type 3 is selected and shown in Figure 23. Table 13 contains a comparative analysis including the previous model. Regression models are constructed by a linear equation. The comparative analysis reveals that the SONFN comes with high accuracy and improved prediction (generalization) capabilities. Table 13. Comparison of identification error with previous modeling methods.)
Number of System Training Inputs (PI)
Model Regression Model
Generalization (E_PI)
11
40.056
36.322
Fuzzy Relation-
Based on Simplified fuzzy inference [19]
4
46.084
38.559
based NFN
Based on linear fuzzy inference
4
41.412
34.048
Simplified Generic Basic Type Architecture Case 1
4
40.753
17.898
Generic Basic Type Architecture Case 1
4
35.748
17.807
Our Model (SONFN)
Linear
TCOMN
MCHAR Y
CDHAR
N
Figure 22. The optimal topology of the generic and basic SONFN in Case 1. (Simplified fuzzy inference based NFN, 3 inputs-Type 2 PNN.)
4. C O N C L U D I N G
REMARKS
In this study, we have introduced a class of SONFN regarded as a modeling vehicle for nonlinear and complex systems, studied its properties, came up with a detailed design procedure and used
920
S.-K. OH et al. im
TCOMN
MCHAR
CDHAR
N
Figure 23. The optimal topology of the generic and basic SONFN in Case 1. (Linear fuzzy inference based NFN, 4 inputs-Type 3 PNN.)
these networks to model a well-known NASA dataset and MIS dataset which are experimental data widely used in software engineering. SONFN is constructed by combining fuzzy relationbased NFN(Simplified or Linear fuzzy inference) with PNN. In this sense, we have constructed a coherent platform in which all components of CI are fully utilized. The model is inherently dynamic--the architecture of the PNN is not fully predetermined (as it usually happens in case of e.g., multiplayer perceptron) and can be generated (adjusted) during learning. A comprehensive design procedure was developed. The series of experiments helped compare the network with other fuzzy models--in all cases the previous models came with higher values of the performance index. REFERENCES 1. W. Pedrycz, Computational Intelligence: A n Introduction, CRC Press, Florida, (1998). 2. J.F. Peters and W. Pedrycz, Computational intelligence, In Encyclopedia of Electrical and Electronic Engineering, Volume 22, (Edited by J.G. Webster), John Wiley ~ Sons, New York, (1999). 3. W. Pedrycz and J.F. Peters (Editers), Computational Intelligence in Software Engineering, World Scientific, Singapore, (1998). 4. H. Takagi and I. Hayashi, NN-driven fuzzy reasoning, Int. J. of Approximate Reasoning 5 (3), 191-212, (1991). 5. S. Horikawa, T. Furuhashi and Y. Uchigawa, On fuzzy modeling using fuzzy neural networks with the back propagation algorithm, I E E E Trans. Neural Networks 3 (5), 801-806, (1992). 6. S.-K. Oh, T.-C. Ahn and W. Pedrycz, A study on the self-organizing polynomial neural networks, Joint 9 th IFSA World Congress, 1690-1695, (2001). 7. S.-K. Oh, D.-W. Kim and B.-J. Park, Transactions on Control, Automation and Systems Engineering, vol. 3, (2001). 8. D.-W. Kim, S.-K. Oh and H.-K. Kim, A Study on the self-organizing fuzzy polynomial neural networks, K I E E International Transactions on Systems and Control 12D (1), 12-16, (2002). 9. S.-K. Oh, D.-W. Kim and W. Pedrycz, Hybrid fuzzy polynomial neural networks, International Journal of Uncertainty Fuzziness and Knowledge-Based Systems 10 (3), 257-280, (2002).
Relation-Based Neurofuzzy Networks
921
10. G.E.P. Box, W.C. Hunter and J.S. Hunter, Statistics for Experimenters, John Wiley K: Sons, (1978). 11. A.G. Ivahnenko, The group method of data handling: A rival of method of stochastic approximation, Soviet Automatic Control 13 (3), 43-55, (1968). 12. M. Shin and A.L. Goel, Empirical data modeling in software engineering using radial basis functions, IEEE Trans on Software Engineering 26 (6), 567-579, (June 2000). 13. M.R. Lyu (Editor), Handbook of Software Reliability Engineering, pp. 510-514, McGraw-Hill, (1995). 14. T. Yamakawa, A new effective learning algorithm for a neo fuzzy neuron model, 5 TM IFSA World Conference, pp. 1017-1020, 1993. 15. D.B. Goldberg, Genetic Algorithms in Search, Optimization ~4 Machine Learning, Addison-Wesley, (1989). 16. C.M. Bishop, Neural Networks for Pattern Recognition, Oxford University, (1995). 17. M. Kearns and D. Ron, Algorithmic Stability and Sanity-Check Bounds for Leave-One-Out Cross-Validation, In Proc. 10th Ann. Conf. Computational Learning Theory, pp. 152-162, (1997). 18. C.F. IKemerer, An empirical validation of software cost estimation models, Comm. ACM 30 (5), 416-429, (May 1987). 19. S.-K. Oh, W. Pedrycz, B.-J. Park and T.-C. Ahn, A hybrid modeling architecture for software engineering, In Proceedings of the 6th IASTED International Conference ASC, pp. 309-314, (July 2002).
Available online at www.sciencedirect.com
ac,z•ce ~ D m e c T ELgEVIER
Mathematical and Computer Modelling 40 (2004) 891-921 www.elsevier.com/locate/recto
R e l a t i o n - B a s e d Neurofuzzy Networks with Evolutionary D a t a Granulation SUNG-KWUN OH Department of Electrical Electronic and Information Engineering Wonkwang University, 344-2, Shinyong-Dong, Iksan, Chon-Buk 570-749, South Korea
W. PEDRYCZ Department of Electrical and Computer Engineering University of Alberta, Edmonton, AB T6G 2G6 Canada and Systems Research Institute Polish Academy of Sciences, Warsaw, Poland
BYOUNG-JUN PARK Department of Electrical Electronic and Information Engineering Wonkwang University, 344-2, Shinyong-Dong, Iksan, Chon-Buk 570-749, South Korea
(Received October 2002; revised and accepted May 2003)
Abstract--In
this study, we introduce a concept of self-organizing neurofuzzy networks (SONFN), a hybrid modeling architecture combining relation-based neurofuzzy networks (NFN) and self-organizing polynomial neural networks (PNN). For such networks we develop a comprehensive design methodology and carry out a series of numeric experiments using data coming from the area of software engineering. The construction of SONFNs exploits fundamental technologies of computational intelligence (CI), namely fuzzy sets, neural networks, and genetic algorithms. The architecture of the SONFN results from a synergistic usage of NFN and PNN. NFN contributes to the formation of the premise part of the rule-based structure of the SONFN. The consequence part of the SONFN is designed using PNNs. We discuss two types of SONFN architectures with the taxonomy based on the NFN scheme being applied to the premise part of SONFN and propose a comprehensive learning algorithm. It is shown that this network exhibits a dynamic structure as the number of its layers as well as the number of nodes in each layer of the SONFN are not predetermined (as this is usually the case for a popular topology of a multilayer perceptron). The experimental results deal with well-known software data such as the NASA dataset concerning software cost estimation and the one describing software modules of the medical imaging system (MIS). In comparison with the previously discussed approaches, the self-organizing networks are more accurate and exhibit superb generalization capabilities. (~) 2004 Elsevier Ltd. All rights reserved. K e y w o r d s - - S e l f - o r g a n i z i n g neurofuzzy networks, Neurofuzzy networks, Fuzzy relation-based fuzzy inference, Polynomial neural networks, Computational intelligence, Genetic algorithms, Design methodology.
This paper was supported by Wonkwang University in 2004.
0895..7177/04/$ - see front matter (~) 2004 Elsevier Ltd. All rights reserved. doi:10.1016/j.mcm.2004.10.019
Typeset by A A 4 , S - ~
892
S.-K. Ou et al. 1. I N T R O D U C T I O N
It is expected that efficient modeling techniques should allow for a selection of pertinent variables and a formation of highly representative datasets. Furthermore, the resulting models should be able to take advantage of the existing domain knowledge (such as a prior experience of human observers or operators) and augment it by available numeric data to form a coherent data-knowledge modeling entity. Most recently, the omnipresent trends in system modeling are concerned with a broad range of techniques of computational intelligence (CI) that dwell on the paradigm of fuzzy modeling, neurocomputing, and genetic optimization [1-3]. The list of evident landmarks in the area of fuzzy and neurofuzzy modeling [4-9] is impressive. While the accomplishments are profound, there are still a number of open issues regarding structure problems of the models along with their comprehensive development and testing. Empirical studies in software engineering employ experimental or historical data to gain insight into the software development process. Data concerning software products and software processes are crucial to their better understanding and, in the sequel, establishing more effective ways of enhancing software quality. However, data have no meaning in themselves, they have meaning only in relation to a conceptual model of the phenomenon studied [10]. Yet, the mechanism underlying the software development process is not understood sufficiently well or is too complicated to allow an exact model to be postulated from theory. Accordingly bearing these in mind, we are vitally interested in the development of adaptive and highly nonlinear models that are capable of handling efficacies of software processes. In this study, we develop a hybrid modeling architecture, called self-organizing neurofuzzy networks (SONFN). In a nutshell, SONFN is composed of two main substructures, namely neurofuzzy networks (NFN) and polynomial neural networks (PNN). From a standpoint of rule-based architectures, one can regard a NFN as an implementation of the antecedent part of the rules while the consequents (conclusion parts of the rules) are realized with the aid of PNNs. The role of the NFN based on fuzzy relation-based fuzzy inference and back-propagation (BP) algorithm is to interact with input data, granulate the corresponding input spaces (viz. convert the numeric data into representations realized at the level of fuzzy sets). Fuzzy granulation of the input space is based on fuzzy relation-based approach where all variables are considered en block (and give rise to fuzzy relations). The role of the PNN is to carry out nonlinear transformation at the level of the fuzzy sets (and corresponding membership grades) formed at the level of NFN. The PNN which has a flexible and versatile structure [7,8] is constructed on a basis of a group method of data handling (GMDH [11]) method. In this network, the number of layers and number of nodes in each layer are not predetermined (unlike this occurs in most neural networks) but can be generated dynamically through a growth process. The number of the input variables used in a partial description (PD) is extended and the order of regression polynomial is also made higher to represent other types of nonlinearities. Especially, the number of nodes in each layer of the PNN architecture can be modified with new nodes added, if required. To assess the performance of the proposed model, we experiment with well-known NASA dataset [12] and medical imaging system (MIS) [13] widely used in software engineering. 2. T H E
ARCHITECTURE
AND
DEVELOPMENT
OF
SONFN
In this section, we elaborate on the architecture and a design process of the SONFN. These networks result as a synergy between two other general constructs such as NFN [14] and PNN [7]. First, we briefly discuss these two classes of models by underlining their profound features and afterwards show how a synergy between them develops. 2.1 F u z z y R e l a t i o n - B a s e d N e u r o f u z z y N e t w o r k s We use fuzzy spaces partitioned by fuzzy relation-based approach in the premise part of the SONFN and its premise part is constructed with the aid of neurofuzzy networks (NFN).
Relation-Based Neurofuzzy Networks
(/4J Bs [
893
L(x,x2) A (x,,xJ
B2
f l(X , A1
A2
A3
( 11)
q 13)
Figure 1, Fuzzy granulation of the input space: fuzzy relation-based approach where all variables are considered en block. As visualized in Figure 1, N F N can be designed by using fuzzy space partitioning using all variables simultaneously• T h e situation visualized in Figure 1 involves partitioning of several variables one at a time so we end up with fuzzy relations defined in the Cartesian product of the spaces of the input variables. Let us consider an extension of the network by considering the fuzzy partition realized in terms of fuzzy relations. T h e networks are classified into the two main categories according to the type of fuzzy inference. We distinguish between a simplified and linear fuzzy inference. The fuzzy partitions are formed for the all variables and two different fuzzy inference methods lead us to the topologies visualized in Figure 2. Figure 2 shows architectures of such N F N in case of two inputs and single output. (a) is NFN structure based on simplified fuzzy inference where each input assumes three membership functions and (b) shows linear fuzzy inference based NFN with two membership functions for each input variable. In Figure 2, the "circles" denote units of the NFN. The node denoted by l--[ realizes a Cartesian product• T h e o u t p u t of the node is taken as a product of all the incoming signals. The "N" identifies a normalization procedure applied to the membership grades of the input variable x~. The " ~ " nodes realize a sum of the arguments. Finally, the output of the N F N ~) is governed by the following expression n
Y=fl +f2 +"'+In
= Efj,
(1)
j=l
with n being the number of the fuzzy rules. Considering the language of the rule-based systems, the NFN structure based on simplified fuzzy inference translates into the following collection of rules R 1 : if xl is A n and .. • xi is AI~, then Yl = wl,
R j : if Xl is Ajl and • • • xi is Aji, then yj = wj,
R n : if Xl is Anl and • - xi is Ani, then yn = Wn.
(2)
894
S,-K. OH Layerl 3
e t aI.
Layerl--4
Layer5
xt
A.
Y
)ci
(a) Simplified fuzzy inference based NFN. Layer 1
Layer 2
Layer 3
Layer d
Layer 5
Connection point
(b) Linear fuzzy inference based NFN. Figure 2. NFN structures realized by fuzzy relations-based fuzzy inferences.
Relation-Based Neurofuzzy Networks
895
The fuzzy rules in (2) constitute an overall network of the NFN as shown in Figure 2a. The output f j of each node generates a final output ?) of the form 7%
~%
]=1
j=l
n
•
j=l E ]Aj j=l
Linear fuzzy inference based NFN is equivalent to the following collection of the rules R 1 : if xl is All and ...x~ is Ali, then C y l = w01 4- Wll Xl 4"
R j : if xl is A j l and •
"'"
4- Wil
"
Xi,
xi is Aji, then C y j = woj + w l j • Xl + . ' . + wij • xi,
(4)
R ~ : if xl is A,~I and • • x~ is A,~i, then Cyn = won + w i n • x l + . . . 4- win • x~. .As shown in Figure 2b, the overall network of the NFN based on linear fuzzy inference is constructed by the fuzzy rules described by (4)• A final output ~) results from as following 9 = ~fj j=l
= ~fz. j=l
Cy j
= ~ j=l
#j .Cyj E
,j.
(5)
j = l
The NFN structure shows one possible connection point with the rest of the model for combination with PNN. The location of this point implies the character of the network (both in terms of its flexibility and learning capabilities). Note that the connection point allows perceiving each linguistic manifestation of the original variables (viz. these variables are transformed by fuzzy sets and normalized). The learning of the NFN is realized by adjusting connection weights wj or w~j of the nodes and as such it follows a standard back-propagation (BP) algorithm. In this study, we use two measures (performance indexes) that is the Euclidean error and the mean magnitude of relative error (MMRE). (i) The Euclidean error Ep = (yp - ~)p)2,
(6)
N
E=
I
(yp _ p)2 .
(7)
p=l
(ii) The mean magnitude of relative error (8)
Zp = lYv - flp[, Yp 1 N _ E=
-N ~= I
(9)
YP
where, Ep is an error measure for the pth data, yp is the pth target output data and ~)p stands for the pth actual output of the model for this specific data point. N is total input-output data pairs and E is an overall (global) performance index defined as a sum of the errors for the N. The new connection weight is generated by addition previous connection weight to variation Aw such as (10). w(new) = w(old) + Aw,
where Aw = 77. k.
Ow] "
(10)
896
S.-K. OH
et al.
Here, let us consider a connection weight wit of the NFN structure based on linear fuzzy inference. In order to minimize error Ep, the modification of the connection A w i j is computed through a gradient descent method that is ( _ OEp
(11)
By the chain rule, a partial differential of the right side in (11) can be expressed as follows
OE,
OE. Of~, Oft
OCyt
Ow~t
O~)p
Ow~t
Of i
8Cy t
(12)
Each part of right side in (12) for the (8) is replaced by the detailed expressions
(i) yp > 9v
ofj O~)p
O~)p
yp '
Of j = 1,
ocyj
OCyt = #J'
-
Owit
=-
Xi;
(13)
(ii) yp < ~)p
OEp
O lyp-yp)
1,
Oyp
Ofj
OCyj
--
Xi"
(14)
Therefore, Awij can be simplified as shown below Awi~ . .~ . /2j . x~, Yp
^ for yp >_ yp,
Aw,j = - ~ . #_L. xi, Yp
for yp < ~p,
(15)
where V is a positive learning rate. If i=0, then x o = l . The final adjustment of the connections including momentum terms reads as A w i j = O" fz-Lj ' xi + a (wij(t) - w i j ( t - 1)), Yp
for yp > ~)p,
A w i j = -~l" fz--Zj " xi + c~ (wij(t) - w i j ( t - 1)),
for yp < ~)p,
Yp
(16)
(here the momentum coefficient, a is constrained to the unit interval). The fuzzy relation-based fuzzy inference rules in (2) or (4) are constructed based on all the combinations of Aji, that is, each membership function is not independent of the corresponding fuzzy inference rule. As we can note in Figure 2, the number of input variables and their partition realized by membership functions for the input variable may not always be equal. This implies that we can produce a rather reasonable fuzzy space partition for each input variable based on some of characters between input variable and the output, such as the nonlinearity, the complexity a n d so on.
2.2. A G e n e t i c O p t i m i z a t i o n o f N F N Genetic algorithms (GAs [15]) have proven to be useful in optimization problems because of their ability to efficiently use historical information to produce new solutions with enhanced performance. Likewise, they support a global nature of search supported there. GAs are also theoretically and empirically proven to support robust search in complex search spaces. Moreover they do not get trapped in local minima as opposed to gradient decent techniques being quite susceptible to this shortcoming. GAs is a stochastic search technique based on the principles
Relation-Based Neurofuzzy Networks
897
of evolution, natural selection, and genetic recombination by simulating "survival of the fittest" in a population of potential solutions (individuals) to the problem at hand. GAs are capable of globally exploring a solution space, pursuing potentially fruitful paths while also examining random points to reduce the likelihood of setting for a local optimum. The main features of genetic algorithms concern individuals viewed as strings, population-based optimization (search through the genotype space) and stochastic search mechanism (such as selection and crossover). A fitness function (or fitness, for short) used in genetic optimization is a vehicle to evaluate a performance of a given individual (string). The search of the solution space is completed with the aid of several genetic operators. There are three basic genetic operators used in any GAs-supported search, that is reproduction, crossover, and mutation. Reproduction is a process in which the mating pool for the next generation is chosen. Individual strings are copied into the mating pool according to their fitness function values. Crossover usually proceeds in two steps. First, members from the mating pool are mated at random. Second, each pair of strings undergoes crossover as follows: a position l along the string is selected uniformly at random from the interval [1, l - 1], where I is the length of the string. Two new strings are created by swapping al][ characters between the positions k and I. Mutation is a random alteration of the value of a string position. In a binary coding, mutation means changing a zero to a one or vice versa. Mutation occurs with small probability. Those operators, combined with the proper definition of the fitness function, constitute the main body of the genetic computing. A general flowchart of the genetic optimization (GA) is visualized in Figure 3. In order to enhance the learning of the SONFN and augment its performance of a SONFN, we use genetic algorithms to adjust learning rate, momentum coefficient and the parameters of Initialization
(Randomlygeneram an initialpopulation)
The parameters are encoded into a string as the concatenation of their binary representations
Evaluation
(Computeand storethe fitnessfor each individual) N~ Objectivefunction Reproduction (Generate the next generation by probabilistically selecting individuals to produce offspring via genetic operators)
__operat%._ /
,r/
/ Invert mutation operator [ R
One-point crossover operator
lol,lllOl,10111 I RmlmmM__ IollllNolllolll
I rllllll
1 1 0 0 0 I 1 0 [--~
Figure 3. A general GA flowchart.
, I 1 0 0 0
i, moll
I
S.-K. OH et al.
898
Generation
I °°n 11 °en2 I O°n 3 I Oen 4 I Oen 5 I """" I
Population (Individual)
Variable Chromosome (String) Figure 4. Data structure of genetic algorithms used in the optimization of NFNs.
the membership functions of the antecedents of the rules. Here, GAs use serial method of binary type, roulette-wheel as the selection operator, one-point crossover, and an invert operation in the mutation operator [15]. Figure 4 portrays a chromosome (string of parameters) being used in the genetic optimization of NFNs. Here successive variables xl, x 2 , . . . , and x k denote input variables of a given system, MFmink and MFmaxk (k = 1 , 2 , . . . ; variable number) denote the parameters of a membership function for each input variable (while min and max denote a minimal and a maximal value of a fuzzy space, repectively), ~ is a learning rate and a stands for a momentum coefficient. 2.3. S e l f - O r g a n i z i n g P o l y n o m i a l N e u r a l N e t w o r k s We use PNN in the consequence structure of the SONFN. Each neuron of the network realizes a polynomial type of partial description (PD) of the mapping between input and output variables. The structure of the PNN is not fixed in advance but becomes dynamically organized during a growth process of the structure. In this sense, PNN becomes a self-organizing network. The PNN Mgorithm based on the GMDH method can produce an optimal nonlinear system by selecting significant input variables among dozens of those available in data and forming various types of polynomials. The GMDH is used in selecting the best ones in PDs according to a discrimination criterion. Successive layers of the SONFN are generated until we reach a structure of the best performance. The input-output relation formed by the PNN algorithm can be described in the following way
y = / (xl, x~,..., ~n).
(17)
The estimated output ~) of actual output y is -- / ( X l , X 2 , . - - , X n ) = COq- E
kl
CklXkl "}- E Cklk2XklXk2 q- E Cklk2k3XklXk2Xk3 q-''" ' (18) klk2 klk2k3
where CkS are the coefficients of the model to be optimized. To obtain the estimate Y, we construct a PD for each pair of independent variables existing in the problem. PDs use regression polynomials, Table 1. Next, we determine the coefficients of PD through the standard least squares error (LSE) method. The optimal structure of the model is determined stepwise: we form layers of PDs operating on pairs of variables and then select the best ones and discard the others. Once the final layer of the structure has been chosen, the node characterized by the best performance is selected as the output node and all other nodes in this layer are removed. Furthermore, all the nodes at the previous layers that do not affect this output node are also removed. Tracing the dataflow back to the previous layers leads to the removal of the excessive node.
Relation-Based Neurofuzzy Networks
899
Depending on the number of input variables as introduced below, two types of generic PNN architectures are considered for performance improvement of P N N model. Also, the structure of P N N is selected on the basis of the number of input variables and the order of PD in each layer. Two kinds of P N N structures, namely a basic P N N and a modified P N N structure are distinguished. Each of t h e m comes with two cases. For details, refer to Figure 5. Table 1. T y p e s of regression polynomial. O r d e r of t h e Polynomial
2
N u m b e r of Inputs 3
1 (Type 1)
Bilinear
Trilinear
Tetralinear
2 ( T y p e 2)
Biquadratic-1
Triquadratic-1
Tetraquadratic-1
2 (Type 3)
Biquadratic-2
Triquadratic-2
Tetraquadratic-2
4
T h e following types of t h e polynomials are considered: • Bilinear = co + elXl + c o x 2 , • Biquadratic-1 = Bilinear +c3x~ + c 4 x 2 + C S X l X 2 , • Biquadratic-2 = Bilinear +¢3XlX2 .
1 s~ ia}'er It...~
2 nd laver or higher
¢,. . . . . .
! . . . . . . . . . . . . . . . ,ll . . . . . . . . . . . . . . . . . . . . . .
l
,
i
!
I
xI X2
OOI
X4~'-
i "----~ ~ !
!
!
L.
Basic
x
1
_ . Ill, xp ~ L,=I(X,X)
-
-
~
•p ~ 2
ype
z.=f(x_, x.~)-Case . ',_ v
-
P PNN I ~,"-'Yi_ PD b - - z,
Modified PN N
"
[
~~ - - ~
I
~0n e r l c
. . . . . . . . . . . . . . . . . . . . . . . . . . .
....
I rv
~.-~
=----~~----'::'-:F:"
. . . . .
p-----I~ L
i
= fix
'
'
............................................
.
~X )
..
f
'
lI l[ II xX . ---r- -Fi ~ = i~lJ -I 1iI~7_,L__j
*
i.--..i~
-
Zi=f(x.,x.,Xr)-Casel i, IJ
Z,i=fi
.
]
i
x ~
I
.
, ..........
,
......
--.~
^ Y
..;_;;.
"l
..................
Advanced Type
:~[_-~ ....
I
I
Z'i=f'(x p) -Case 2
Figure 5. Configuration of t h e P N N structure.
900
S.-K. OH et al.
(a) Combination of 2 inputs.
(b) Combination of 2 or 3 inputs.
(c) Combination of 3 inputs.
Figure 6. The case of not generation nodes (or layers) anymore through growth process.
(a) Basic PNN structure the number of input variables of PDs is same in every layer. (b) Modified PNN s t r u c t u r e - - t h e number of input variables of PDs varies from layer to layer. CASE 1. The polynomial order of PDs is the same in each layer of the network. CASE 2. The polynomial order of PDs in the 2 nd layer or higher has a different or modified type in comparison with the one of PDs in the 1st layer. The selection and use of system input variables of the networks depends upon the topology of the PNN. According to the diverse topologies of PNN selected on the basis of the number of inputs and order of polynomial, we build various architectures of the PNN. If there are less than two (or three) input variables, the generic PNN algorithm does not generate a highly versatile structure as shown in Figure 6. To alleviate the problems, the advanced type of the architecture is taken into consideration that can be treated as the modified version of the generic type of the topology of the networks as shown in Figure 5. Accordingly, we identify also two types as the following. (1) Generic type : in case that the number of system input variables is four or higher, the generic PNN is used. (2) Advanced type : in case that the number of system input variables is less than three (or four), the advanced PNN is used. The basic and modified PNN architecture are shown in Figure 5, where z~ (Case 2) in the 2nd layer or higher denotes that the polynomial order of the PD of each node has a different or modified type each other in comparison with zi of the 1st layer. In the advanced type of Figure 5, the "NOP" node means the A th node of the current layer that is the same as the node of the corresponding previous layer (NOP denotes no operation). An arrow to the NOP node is used to show that the corresponding same node moves from the previous layer to the current layer. 2.4. S O N F N Topologies: Architecture Combined with Fuzzy Relation-Based NFN and PNN The SONFN is an architecture combined with the NFN and PNN as shown in Figures 7 and 8. These networks result as a synergy between the two other general constructs such as NFNs and PNNs. The SONFNs distinguish between two kinds of architectures, namely, basic and modified architectures. Moreover, for each architecture of the SONFN we identify two more detailed cases. (a) Basic architecture the number of input variables of PDs of PNN is the same in each layer. (b) Modified architecture the number of input variables of PDs of PNN is different in each layer. CASE 1. The polynomial order of PDs of PNN is same in every layer. CASE 2. The polynomial order of PDs in the 2 nd layer or higher of PNN has a different or modified type in comparison with the one of PDs in the 1st layer. As mentioned above, the topologies of the SONFN depend on those of the PNN used for the consequence part of SONFN. The above taxonomy is also summarized in Table 2.
Relation-Based Neurofuzzy Networks
901
%-. -. -. -. -o -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -. -.,
'. "
,,"
Premise part(NFN) f
'," "
Consequence part(PNN)
li Layer 4 Layer3 I Layer5 Layerl .I. .L _,. . v • , ~_ _ , / ~
......
Layer 2
|
L a y e r 1~
,...
:: :
I
......................
.----.
,, Xj t|
% %
% % %
"
r....................
',
i:
%
I
,
t,. ' - I'. ' - - . " : : . . ' . ' - - . "
Xp - ~
! I -J
t ...................
Partial Description
~
Xq - - ~ (Polynomial Function)
" ............
~-'r
...............
!
GenericType
::
I
(.,'g
q "~
gi=-'J~Xp~Xq)
............
,~ -.{' . ..... .. ... .. ... .. ... .. ... .. ... .. .. . . . . .
...............
Partiai Description ~ (Polynomial Function)
%,,,,,,,
Zi=AXP'~(t) Zi'=f"(X~q)
-
case
2_~
,,,, ,,,
r,'. ". ".-. %
". ".-.-.-.
~,
I I.-. !
,
,,S i--i ". ". ". ". ". "1.-. I
, 4.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-. I
r ......
• .................... I I I I I ~-L..L.
i ~i i % % %
,
i
I I I I I
D
%% r* 1.-. ". ". ".-. I
" .........
~ It
!
~l
------ . . . .
®',,X l
~ x/X.
v
n ".-2J/~//"¢-..~I/F ¢..~
.
"
r
".-.
". ".-.-.-.
.....
PD~/.[~
".-.
". "."
-, "
-,
:
:
::
: I I :
I i I I I I
I I i I I I
D, f it .
I I
: I
I I
,I
'
~ur
~'.-.~
/'~ ~" 1.-. ,.-.-.-. I I
I
I
i
!
,.----.
ll\\~::.~L. -]
I( PD )...._
~
,,
,,
,
: ~Ill ) :
,,
:.x,-<
,,
i % "1
. . . . . .
iI lu ii
el
.i I
"|
:
It
ll
|
ii
% ~1
: I
J* I$
I I
AdvancedType
~1 In
I I
',
-m--r .........
",---, . . . . . . . .
:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: I
I l
,,,
"~'
!
I I
Function)
I I
!
I
iI
.
. . . . .
,',',,,y
<~ g'"
~l (Polynomial
i I
ii
J
Partial Description
'
-case l
z i'=f'(xp)- case 2 [I (a) Basic SONFN architecture.
Figure 7. Configuration of SONFN architecture; combination of simplified fuzzy inference-based NFN and PNN.
902
S.-K. OH et al.
I
Premise part(NFN)
'. el
', "In
i
',
,,
I
II
,
"
ii
Consequence part(PNN)
Layer 2 Layer 4 , I Layer 3 / L ayer 5 Layer 1
Layer IV
~
~
~
:: ,•"
,,
"
Jl
~I
-- . . . .
"
Layer II
ii
al
s~
,
~_~ . .~ . ~ .- ' "
Y
~1
i|
Ii
,,
~Sv
%
~
~
~
, I
[
Xq
...........................
l
," I
Partial Description L (Polynomial Function) J " "~i
xp
* N
l I
, ....................
i.
',*.
~-~--r--'\~ !
,,
.X)
Xq
r
,
Generic Tvpe.
" ',',
I_.
Partial Description (Polynomial Function)
r
~
zi=flX,Xq,X,. ) - case 1 1 zi~-f'(Xp,Xq.,X,. ) case 2
%
-C"#-, ,
"q/-"r"
/" "-"
..................,..,.......-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.,-.,-.-.-.-.-.-.-.-.-.-.-.-.-.-..-.-.-.-.-.-.-.-.-..
: '
%
! ~ i / ~ ~ :
I
', ,,
~
'- ...................
"
I.
,
ii
-
I I
h
• •
I--L--L-
I l
i~ .. jl
|l
sl
,, fl
II
'I ~. I i.
[. . . . . .
II
I I
iI
.I,
.
.
.
.
.
~
.........
i
II
I
IJ
I
A
~J
v ::
•
.
ii |a -'~,'~
II l*
|I
lJ
% °I
......
Ii el
'I I
lI
!
II If ,s
ij !
%
ii
l
',%
I.,_,I.__T. a I
f ..................
.I
1
I
I
I
Advanced Type
-,1,4'#" 7, , .
I
'
(Polynomial Function) ,,,,,
(b) Modified SONFN archtecture. Figure 7. (cont.)
t, ii ~t
~!
' X
l d'
I............... "T = 11
R e l a t i o n - B a s e d N e u r o f u z z y Networks
~
.
:"
~
[
~ ~ ~ ~ . Premise part(NFN)
~
.
~
~
~
~
Consequence
.
903
:
,,
part(PNN)
t[
Layer 2Layer 3Layer 4
I
"
ii
:: :: x: ::
II iI
1 I I I ',:
:" .
]
....................
Generic Type
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
I''I
. . . . . . . . . . . . . . . . . .
(P"" .... ,.,,~.o~,o.,
:rq
1 " ~,:J(5,'~,)
.............
J. J . . . . . . . . . . . . . . . . . . . . .
Ii I,
I l
.-.-.- . . . . .
"-.-L- .......
- ......
(,.,,.°,.,., ~..~,,..)
~
. . . . . . .
•
. I
. Z .
I j__~
.-
-.-'2
....
zi'=f'(x e,.~l ) - c a s e 2
"3. -~-". . . . . . . . . . .
• . . . . . . . . . . . . . . . . . . . .
:,,
I
.j
"~.
,
I
ii . . . . . . .
•
Q
I
,,
l
:: x ~ - - ¢
,'i i
~1
i . . . . . .
.J--J
. . . . . .
"
"l
:',
y
',',
. ................
.,
X/ -'~
,, ,i
:i :,,
:', ,, I.--flr..t--d
Ii ~ I
. . . . . . . . . . . . . . . . . .
..........
l
L . . . . . . I j I
II"
-I---I i l I I
- -
Advanced T y p e . ,II
. . . . . . . l I !
"
-
......
-
i~Z
:.p)
case2
(a) Basic S O N F N architecture. F i g u r e 8. C o n f i g u r a t i o n of S O N F N architecture; c o m b i n a t i o n of linear fuzzy inferencebased NFN and PNN.
I
904
O H et al.
S.-K.
Premise part(NFN) ::" ,,
I
:,,
J
Consequence part(PNN) it
Layer2Layer3Layer4 Layer 1~
I
.i
ii
.............................
Layer 1 % %
~. . . . .
~
,, x l - ~
o
~
[
~
•
:: " Xi - ' ~
"•:
C
i~
~
~
.............
I I! ! I I
,i
,,
|
.]~ ....................
"
i I[
*, it
'*" Me
,,,. tt i.
Generic Type ,.
!
~.~. t~ d
::
i
I I! ! | 1
!
,,
~
i
r--r ,","
.............
..
"
i
. .~--~g~,'
,.," ~
,, iI iz
Layer~
,,
ml ul
,, ::,, ,,
..
h7
uV J
I
"
"~
l?artialDescripti::'
[Xp ~
11
zi=J(xp'xq'x)-casel
~
Partial Descriptioa
%
........... .
"-'-'-'-'-'-'-'-'-'-'-'-','-.'-'-'-'-'-'-'~,
.
.
.
.
.
. . . . . . • . . . . . . . . . . . . . . . . . . . .
l
.
. .
. .
. .
. .
. .
n . . . . . . . . . . . . . . . . . . . .
D
ii
•
xI
. .
. .
. .
. . . . . . .
l= .P . . . . I' - k . . . ! ! .t__l.
. .
. .
. . . . . . . . . . .
. .
.
j. .
.
! . .
. .
. .
.
. .
. .
. .
. .
. .
, it
. . . . . .
_L__L_
m,
i
:
!
. . . . . . . . . . . . .
I
,,~
,, ,~'[ ' ~ , , " m* mt
!
0
:
X/
i.
..................
, )
".
~
..........................
~-v--r-"
~......
"-~ ....... ' !
I'. . . . . . . . .
:,
+.t [ .........,
'
!,
4
• u
. . - - - - -.-
-.-.-.-.-
(b) Modified SONFN architecture.
. .
...,
~,
I
Figure 8. (cont.)
Advanced Type
. . . . . . .
......... ,
I
" . z, =-7 tx~1 - case 2
Relation-Based Neurofuzzy Networks
905
Table 2. A family of the topologies of the SONFN. Layer of PNN
Number of Input Variables of Polynomial
Order of Ploynomial
1st Layer
p
Type P
2nd or Higher Layer
SONFN Architecture (1) p = q : Basic SONFN (a) P = Q : Case 1 (b) P ~ Q : Case 2 (2) p # q : Modified SONFN (a) P = Q : Case 1 (b) P C Q : Case2
Type Q
(p = 2 , 3 , 4 , q = 2,3,4; P = 1,2,3, Q = 1 , 2 , 3 )
The SONFN architecture combined with the fuzzy relation-based NFN and PNN under discussion is shown in the Figures 7 and 8. Let us recall that the fuzzy inference (both simplified and linear) based NFN is constructed with the aid of the space partitioning realized by fuzzy relations. We also identify two types as follows• - Generic type of SONFN; Combination of the fuzzy inference based NFN and the generic PNN. - Advanced type of SONFN; Combination of the fuzzy inference based NFN and the advanced PNN. 2.5. T h e A l g o r i t h m i c F r a m e w o r k o f S O N F N ]In this section, we elaborate on the algorithmic details of the design method by considering the functionality of the individual layers in the network architectures (refer to Figures 2, 5, 7, and 8). The design procedure for each layer in the premise and the consequence of SONFN comprises of the following steps. THE PREMISE OF S O N F N : N F N . LAYER 1. Distributing the signals to the nodes in the next layer as input layer. LAYER 2. Computing activation degrees of linguistic labels (fuzzy sets; small, large, etc.). Each node in this layer corresponds to one linguistic label (say, small, large, etc.) of each input variable positioned in the first layer. The layer produces a degree of satisfaction (activation) of this label by the input. In essence, we compute a possibility degree (measure) of the numeric input with respect to the given fuzzy set, refer to Figure 9. LAYER 3. Computing fitness of premise rule. Every node in this layer is a fixed node labeled l-I, whose o u t p u t is the product of all the incoming signals. /£j = #A (Xl)
X #B
(Xi),
A, B = small, large, etc.
(19)
~/ik+l
zik(x) r
min~
//
x. ;
I
//
mix~
Figure 9. Triangular membership functions and the resulting computations of the actibation levels.
906
S.-K. OH e t
al.
A node in this layer represents one fuzzy rule and each node output represents the firing strength of a rule. LAYER 4. Normalization of a degree of activation (firing) of the rule. pj-
,
(20)
/zj j=l
where n denotes a number of rules. LAYER 5. Multiplying a normalized activation degree of the rule by the corresponding connection weight. (i) Simplified fuzzy inference based NFN #j .wj
(21)
A., #j
j=l
(ii) Linear fuzzy inference based NFN
fJ = #5 x cys, Cy 5 = woj
+ wlj
= Ewij
(22) " xl
+ w2j
"x,
" x2 + " " + w~j " x~
(where i = 0 , 1 , 2 , . . . and x0 = 1).
(23)
i f5
is given as the input variable of the PNN which is the consequence structure of SONFN.
LAYER 6. Computing output of NFN: The output in the 6 th layer of the NFN is inferred by the center of gravity method. (i) Simplified fuzzy inference based NFN n
n
j=l
j=l
(24) j=l
E ~tj j=l
(ii) Linear fuzzy inference based NFN fi
J=~
n ij:Z n s cyj=X%: c y ` - zn" `
J=~
J=~ ~ #5
xl÷
J=~
j=l
*w'S
(25)
~ #~
5=1
THE CONSEQUENCE OF S O N F N : P N N .
STEP 1. Configuration of input variables: denotes a rule number.
xl
=
fl,
x2
=
f2,
...
, x,~
=
f~(n
=
j); where j
STEP 2. Forming a PNN structure: here we select input variables and fix an order of a PD. STEP 3. Estimate the coefficients of a PD: The vector of coefficients of the PDs (C~) in each layer is determined by the standard least-squares method that assumes a well known form Ci = ( X { X ~ -1 ) X/TY,
(26)
w h e r e / s t a n d s for a node number. This procedure is implemented repeatedly for all nodes of the layer and also for all layers of consequence part of SONFN.
Relation-Based Neurofuzzy Networks
907
STEP 4. Choosing PDs in case that the training and testing dataset are taken into consideration: each PD is constructed and evaluated using the training and testing dataset, respectively. Then, we compare the values of the performance index and select PDs using an aggregate performance index with a sound balance between approximation and prediction capabilities. We may use (i) the predetermined number W of the PDs (width of the layer) or (ii) go for all of them whose performance index is lower than a certain prespecified value. Especially, the method of (ii) uses the threshold criterion ~ to select the node with the best performance in each layer. e = Emin + 5,
(27)
where 0 is a new value of the criterion, ~ is a positive constant (increment) and Emin denotes the performance index with smallest value obtained in the each layer. STEP 5. Termination condition: we take into consideration a stopping condition (Emin _> Emin.) for better performance and the number of iterations (size of the network) predetermined by the designer. Here Emin is a minimal identification error at the current layer while Z m i n . denotes a minimal identification error at the previous layer. STEP 6. Determining new input variables for the next layer: the outputs of the preserved PDs sep/e as new inputs to the next layer. In other words, we set Xl~ = zl~, x2i = z 2 ~ , • • • , x w ~ = z w i . The consequence part of SONFN is repeated through a sequence of Steps 3-6. 2.6 M o d e l S e l e c t i o n ]~br the NASA software dataset, our model selection procedure is based on seeking a sound compromise between approximation and generalization errors. The main performance measure that we use in this paper is the MMRE (the mean magnitude of relative error) of (9). For evaluation of generalization ability, many estimates have been proposed in the literature; the most popular ones being the holdout estimate and the k-fold cross-validation estimate [16]. The holdout estimate is obtained by partitioning the dataset into two mutually exclusive subsets called training and test sets. The error estimate on the test set is used to assess generalization ability. On the other hand, the k-fold cross-validation estimate is obtained by a sample reuse technique. The dataset is divided into k mutually exclusive subsets of almost equal size, k - 1 subsets are used for training, and the k th is used for prediction. This process is repeated k times, each employing a different subset for prediction. When k is equal to data size, it is called leave-one-out cross-validation (LOOCV) estimate. In this study, we employ the LOOCV estimate of generalization error because of two reasons. Fir,st, it possesses good mathematical properties [17]. Second, it seems to be particularly suited for software engineering applications where the best available data are relatively small sets [18]. Thus, our model selection is based on the analysis of LOOCV estimate of generalization error for SONFN models in NASA dataset. 3.
EXPERIMENTAL
STUDIES
In this section, we illustrate the development of the SONFN and show its performance for well known and widely used datasets in software engineering. The first one is the NASA dataset [12]. The second one is medical imaging system (MIS) [13]. 3.1 N A S A S o f t w a r e D a t a The experimental studies are concerned with a well-known software effort dataset from NASA [12]. The dataset consists of two independent variables, viz. developed lines of code (DL) and methodology (ME), and one dependent variable, viz., effort (Y). DL is in KLOC and Y is
S.-K. OH et al.
908
Table 3. The data set of the NASA software project. Project Number
Independent
Variables
Dependendt Variable
DL
ME
Effort (Y)
90.2
115.8
46.2 46.5 54.5 31.1 67.5 12.8 10.5 21.5
30 20 19 20 35 29 26 34 31
3.1 4.2 7.8 2.1 5.0 78.6 9.7 12.5 100.8
26 19 31 28 29 35 27 27 34
10 11 12 13 14 15 16 17 18
96.0 79.0 90.8 39.6 98.4 18.9 10.3 28.5 7.0 9.0 7.3 5.0 8.4 98.7 15.6 23.9 138.3
in m a n - m o n t h s . Here, M E is a c o m p o s i t e m e a s u r e of m e t h o d o l o g i e s e m p l o y e d in this N A S A software e n v i r o n m e n t . T h e d a t a set is shown in T a b l e 3. I n t h e following, we develop software effort e s t i m a t i o n m o d e l s for t w o collections of i n d e p e n d e n t variables, i.e., DL a n d (DL, M E ) . 3.1.1. Results
of NFN
modeling
A c c o r d i n g to each S c h e m e of N F N s t r u c t u r e s , t h e i d e n t i f i c a t i o n shown in T a b l e 4. N o t e t h a t P I d e s c r i b e s a p e r f o r m a n c e i n d e x for E_PI c o n c e r n s t h e p e r f o r m a n c e i n d e x for t h e t e s t i n g d a t a s e t using ( L O O C V ) . G A s help o p t i m i z e l e a r n i n g rate, m o m e n t u m coefficient,
errors of P I a n d E_PI are t h e t r a i n i n g d a t a s e t while a leave-one-out validation a n d t h e p a r a m e t e r s of t h e
m e m b e r s h i p functions. G A s was r u n for 100 g e n e r a t i o n s w i t h a p o p u l a t i o n of 60 i n d i v i d u a l s . E a c h s t r i n g was 10 b i t s long. T h e crossover r a t e was set to 0.6. T h e p r o b a b i l i t y of m u t a t i o n was equal to 0.1. I n each N F N s t r u c t u r e , two m e m b e r s h i p f u n c t i o n s for each i n p u t v a r i a b l e are used a n d 500 i t e r a t i o n s for t h e n u m b e r of t r a i n i n g cycles are used. Table 4. Performance index of the NFN. NFN
1 System Input (DL)
2 System Inputs (DL,ME)
PI
E_PI
PI
E PI
Simplified Fuzzy Inference
0.2870
0.2990
0.2115
0.2415
Linear Fuzzy Inference
0.2094
0.2462
0.1544
0.1967
3.1.2. R e s u l t s o f S O N F N m o d e l i n g W e now p r e s e n t t h e d e t a i l s of S O N F N m o d e l i n g using t h e m e t h o d o l o g y d e s c r i b e d in t h e previous Section 2.
Relation-Based Neurofuzzy Networks 3.1.2.a. In case of 1 system
input
909
(DL)
T h e g o a l is t o seek a p a r s i m o n i o u s m o d e l , w h i c h p r o v i d e s a g o o d fit t o t h e D L a n d Y d a t a of N A S A software p r o j e c t d a t a a n d e x h i b i t s g o o d g e n e r a l i z a t i o n c a p a b i l i t y . T a b l e s 5 a n d 6 s u m m a r i z e t h e r e s u l t s of t h e p r e f e r r e d a r c h i t e c t u r e s a c c o r d i n g t o a d v a n c e d t y p e a n d a r c h i t e c t u r e (basic or m o d i f i e d ) . Table 5. Performance index of SONFN with the simplified fuzzy inference based NFN. Premise Part; NFN SONFN
Number of Number of Connection MFs Point Inputs & PolyNomial Type Case 1
2
1
Case 2
2
1
Case 1
2
1
Modified Case 2 Architecture
2
1
Case 2
2
2
Basic Architecture Advanced Type
Consequence Part; PNN PI
E_PI
1.2356 0.2775 0.3166 0.1850 1.2356 0.4720
Layer
2 inputs Type 2 2 inputs Type 3 --* 2 2 --* 3 inputs Type 2
1 2 1 2 1 2
0.2486 0.1801 0.2479 0.1748 0.2486 0.1797
2 --* 3 inputs
1 2
0.2479 .03166 0.2285 0.2866
1
0.4719 i 0.5344
2
0.2479 0.5344 i
Type3 -~ 2 1 --* 2 inputs Type 1 --* 2
Table 6. Performance index of SONFN with the linear fuzzy inference based NFN. Premise Part; NFN SONFN
PI
E_PI 0.2166 0.1572
Case 1
2
1
2 inputs Type 2
1 2
0.1349 0.1143
Case 2
2
1
2 inputs Type 3 --* 2
!Case 1
2
1
2 -* 3 inputs Type 2
1 2 1 2 3
0.1873 0.2427 0.1349 0.1582 0.1349 0.2166 0.0901 21.283 0.0822 0.1313
Case 2
2
1
2 --* 3 inputs Type 1 -* 2
1 2 3
0.2117 0.1349 0.0877
Basic Architecture Advanced Type
Consequence Part; PNN
Number of Connection Number of Inputs Layer MFs Point & Polynomial Type
Modified Architecture
0.2596 0.2166 0.1309
I n T a b l e 5, t h e basic S O N F N in Case 2 is t h e p r e f e r r e d a r c h i t e c t u r e of t h e n e t w o r k in t h e a d v a n c e d t y p e a n d its d e t a i l e d t o p o l o g y is v i s u a l i z e d in F i g u r e 10. T h e values of t h e p e r f o r m a n c e i n d e x of t h e simplified fuzzy inference b a s e d N F N o p t i m i z e d b y G A s are P I = 0.2870 a n d E_PI = 0.2990. W h e n c o n s i d e r i n g b o t h P I a n d E_PI, t h e m i n i m a l value of t h e p e r f o r m a n c e index, t h a t is P I = 0.1748, E_PI = 0.1850 are o b t a i n e d b y using T y p e 3 in t h e 1 st layer a n d T y p e 2 in t h e 2 nd layer or h i g h e r ( T y p e 3 4 2 ) w i t h 2 n o d e i n p u t s . T h e form of each p o l y n o m i a l t y p e is shown in 'Table 1. F i g u r e 10 i l l u s t r a t e s an o p t i m a l a r c h i t e c t u r e in t h e a d v a n c e d t y p e of t h e S O N F N t h a t is c o m p o s e d of simplified fuzzy inference b a s e d N F N a n d P N N w i t h 2 i n p u t s - T y p e 3--*2 topology. T h e w a y in w h i c h l e a r n i n g h a s b e e n realized is shown in F i g u r e 11 w h e r e t r a i n i n g errors ( p e r f o r m a n c e i n d e x ) a r e i l l u s t r a t e d . I n F i g u r e 5, Q is t h e A th n o d e of t h e each c o r r e s p o n d i n g layer used for t h e g e n e r a t i o n of t h e o u t p u t Y, ~ is t h e A th n o d e of t h e each c o r r e s p o n d i n g layer used for t h e g e n e r a t i o n of t h e o u t p u t a n d i n d i c a t e s t h e o p t i m a l n o d e in each layer.
910
S.-K. OH et al.
DL
Figure 10. The optima' topo'ogy of the advanced and basic SONFN in Case 2. (Simplified fuzzy inference based NFN, 2 node inputs and Type 3 ~ 2 PNN.) 1
0.9 0.8
0.7 0.6
0.4
0.3
0.10
Part :
~ - - ~ ~--------~/
Consequence Part : PNN
Premise 1oo
2°0
300
Iteration
,oo Layer
Figure 11. Learning procedure of the advanced and basic SONFN in Case 2. (Simplified fuzzy inferecne based NFN, 2 node inputs and Type 3 --* 2 PNN.)
Figure 12. The optimal topology of the advanced and modified SONFN in Case 2. (Linear fuzzy inference based NFN 2 --* 3 inputs-Type 1 ---* 2 PNN.)
In light of the values reported in Table 6, the modified SONFN (Case 2) emerges as a preferred architecture of the network in the advanced type. Figure 12 shows an optimal architecture in advanced type of the SONFN that is composed of the linear fuzzy inference based NFN and PNN with 2-*3 inputs-Type 1-.2 topology. The values of the performance index of the linear fuzzy inference based NFN optimized by GAs are PI = 0.2094 and E_PI = 0.2462. When considering both PI and E_PI, the minimal value of the performance index, that is PI = 0.0877, E_PI = 0.1309 are obtained by using 2 node inputs and Type 1 in the 1st layer and 3 node inputs and Type 2 in the 2nd layer or higher(2-*3 inputs-Type 1-.2). The training errors of SONFN are illustrated in Figure 13. In Figure 12, the "NOPA" node means the A TM node of the current layer that is the same as the node of the corresponding previous layer(NOP denotes no operation). An arrow to the NOP node is used to show that the corresponding same node moves from the previous layer to the current layer.
Relation-Based Neurofuzzy Networks I
T ~ T
T
1
911
1
r
T
0.9 0.8
0.7 0.6
Premlse Part :
Consequence Part :
/[---
PNN
0.5 :~ O.4 0.3 0.2 0.1 0
0
100
200 300 Iteration
400
5~~0
1
2 Layer
Figure 13. Learning procedure of the advanced and modified SONFN in Case 2. (Linear fuzzy inference based NFN, 2 --* 3 inputs-Type 1 --* 2 PNN.)
3.1.2.b.
In case of 2 system
inputs
(DL,ME)
No w we d e v e l o p effort e s t i m a t i o n m o d e l b a sed on t w o i n d e p e n d e n t v a r i a b l e s D L an d ME. For t h e simplified a n d linear fuzzy inference based N F N , t h e p e r f o r m a n c e i n d ex es o f S O N F N are sh o wn in T a b l e 7 a n d 8, respectively. T a b l e 7 s u m m a r i z e s t h e values o f t h e p e r f o r m a n c e i n d e x for t h e S O N F N w i t h t h e simplified fuzzy i n f eren ce b a s e d N F N . In T a b l e 7, t h e m o d i f i e d S O N F N in C a s e 2 is t h e p r e f e r r e d a r c h i t e c t u r e of t h e n e t w o r k in t h e generic or t h e a d v a n c e t y p e respectively. A n o p t i m a l a r c h i t e c t u r e of t h e
Table 7. Performance index of SONFN with the simplified fuzzy inference based
NFN.
SONFN
Premise Part; NFN
Consequence Part; PNN
Number of MFs
Number of Inputs Layer & Polynomial Type
Case 1
2×2
Case 2
2x2
Basic Architecture Generic Type Case 1
2x2
Case 2
2x2
Modified Architecture
Case 1
2x2
Basic Architecture Advanced Type
Case 2
2x2
Case 1
2x2
Case 2
2x2
Modified Architecture
[ PI !0.1369
i E PI
2 inputs
1
Type 3
2
0.1085 0.0600
0.1130
2 inputs
1
0.1369 I 0.1130
Type 3 ---*2
2
0.1048 [ 0.0621
2 -~ 3 inputs
1
[ 0.1369 0.1130
Type 3
2
0.0893 0.0527
2 --* 3 inputs
1
0.1369 0.1130
Type 3 --* 2
2
0.0493 0.0565
3 inputs
1
0.0695 0.1890 0.0420 0.0417
Type 2
2
3 inputs
1
0.0695 0.1890
Type 2 --* 3
2
0.0390 0.0429
3 ---*4 inputs
1
0.0695 0.1890
Type 2
2
0.0189 0.0397
3 --* 4 inputs
1
0.0695 0.1890
Type 2 --+ 3
2
0.0231 0.0252
912
S.-K. OH et al. Table 8. Performance index of SONFN with the linear fuzzy inference based NFN.
SONFN
Premise Part; NFN
Consequence Part; PNN
Number of MFs
Number of Inputs Layer & Polynomial Type
Case 1
2x2
Case
2×2
Basic Architecture 2
2 inputs Type 3
PI
E_PI
0.1269 0.1229 0.1166 0.1269
0.1595 0.1712 0.1550 0.1595
2 inputs
1 2 3 1
Type 3 ~ 2
2
i 0.1122 0.1695
2 ---*3 inputs
3 1
I 0.0770 0. 1220 0.1269 0.1595
Type 3
2 3
0.1040 0.1257 0.0680 0.1042
2 -+ 3 inputs Type 2 --* 3
1 2 3
10.1283 0.1949 0.0964 0.2031 0.0630 0.0736
3 inputs Type 3 3 inputs Type 2 --* 3 2 --* 4 inputs Type 2
i 2 i 2 1 2
0.1084 0.0940 0.0829 0.0603 0.1283 0.0131
2 --~ 4 inputs Type 3 --* 2
1 2
0.1269 0.1061 0.0137 0.0290
I
Generic Type Case 1
2x2
Case 2
2x 2
Case I
2x 2
Case2
2x 2
Case 1
2x 2
Case 2
2x 2
Modified Architecture
Basic Architecture Advanced Type Modified Architecture
0.1275 0.0378 0.1583 0.0452 0.1269 i 0.0205
S O N F N in t h e a d v a n c e t y p e is v i s u a l i z e d in F i g u r e 14. I n F i g u r e 14, t h e N F N p a r t of t h e n e t w o r k uses two m e m b e r s h i p f u n c t i o n s for each i n p u t variable, so t h e a r c h i t e c t u r e has 4 rules. T h e values of t h e p e r f o r m a n c e i n d e x of t h e N F N o p t i m i z e d b y G A s are 0.2115 for t h e t r a i n i n g d a t a a n d 0.2415 for t h e t e s t i n g d a t a . T h e P N N p a r t of t h e n e t w o r k s is c o n s t r u c t e d b y using T y p e 2 w i t h 3 n o d e i n p u t s in t h e 1st layer a n d T y p e 3 w i t h 4 n o d e i n p u t s in t h e 2 nd layer(3--~4 i n p u t s a n d T y p e 2 4 3 ) . T h e final r e s u l t s of t h e S O N F N t o p o l o g y are P I = 0 . 0 2 3 1 a n d E _ P I = 0 . 0 2 5 2 . F i g u r e 15 i l l u s t r a t e s t h e p e r f o r m a n c e o f t h e o b t a i n e d networks.
ME
39
Figure 14. The optimal topology of the advanced and modified SONFN in Case 2. (Simplified fuzzy inference based NFN, 3 --+ 4 node inputs an Type 2 --* 3 PNN.) F o r t h e S O N F N w i t h t h e linear fuzzy inference b a s e d N F N , t h e values of t h e p e r f o r m a n c e i n d e x a r e s u m m a r i z e d in T a b l e 8. T h e N F N p a r t of t h e S O N F N uses two m e m b e r s h i p f u n c t i o n s for each i n p u t variable. Therefore, t h e a r c h i t e c t u r e e x h i b i t s four rules as well. T h e values of t h e p e r f o r m a n c e i n d e x of t h e N F N o p t i m i z e d b y G A s is e q u a l to 0.1544 for t h e t r a i n i n g d a t a a n d 0.1967 for t h e t e s t i n g d a t a . I n T a b l e 8, t h e m o d i f i e d S O N F N in C a s e 2 or C a s e 1 is t h e p r e f e r r e d a r c h i t e c t u r e of t h e n e t w o r k in t h e generic or t h e a d v a n c e t y p e respectively. A n o p t i m a l a r c h i t e c t u r e of t h e S O N F N in t h e generic t y p e is v i s u a l i z e d in F i g u r e 16 while t h e l e a r n i n g
Relation-Based Neurofuzzy Networks
I
r
I
r
913
r
0.9
0.8 0.7
Part :
o b0
~ - - ~
ConsequencePart : PNN
Premise
0.6 0.5 0.4 [.-.
0.3
O. 0
100
200
300
400
5 0
1
Iteration
2
Layer
Figure 15. Learning procedure of the advanced and modified SONFN in Case 2. (Simplified fuzzy inference based NFN, 3 --* 4 node inputs, and Type 2 ---*3 PNN.)
f DL
ME -c
....
:
'
~
Y
Figure 16. The optimal topology of the generic and modified SONFN in Case 2. (Linear fuzzy inference based NFN, 2 --* 3 node inputs, and Type 2 ---*3 PNN,)
process is quantified in Figure 17. The shadowed nodes indicate neurons which have the optimal polynomial in each layer(the optimal being expressed from the viewpoint of PI as well as E_PI). In Figures 16 and 18, is the A th node of the each corresponding layer that is not used for the generation of the ~), and is the A th node of the each corresponding layer that is not used for the generation of the output and indicates the optimal node in each layer. Also, the solid line is used for the generation of the output and the dashed line is not used for the generation of the Z). And the values of the performance index vis-d-vis number of layers of the modified SONFN wit]h the generic or advanced type related to the preferred architectures of the network are shown in Figures 20 and 21, respectively. In Figures 20 and 21, the notation 'Type p --~ q~(p, q = 1, 2, 3) indiLcates that the polynomial order of the consequence part of the fuzzy rules in a node changes from Type "p" in the i st layer to Type "q" in the 2 nd layer and higher. And A(-), B(.), C(.),
914
S.-K. OH et al. 1 0.9 0.8 0.7 0.6
P r e m i s e Part :
C o n s e q u e n c e Part :
NFN
PNN
0.5 ....E
o.4! 0.3' 0.2 0.11 0
0
1O0
200 300 Iteration
400
tO
1
2
Layer
Figure 17. Learning procedure of the generic and modified SONFN in Case 2. (Linear fuzzy inference based NFN, 2 --* 3 node inputs, and Type 2 --* 3 PNN.)
DL
ME
Figure 18. The optimal topology of the advanced and modified SONFN in Case 1. (Linear fuzzy inference based NFN, 2 --* 4 node inputs, and Type 2 PNN.) and D(*) denote the optimal node numbers according to each type of the polynomial. Namely, the node numbers of the 1st layer represent the node numbers located in the 4 th layer of NFN, and the node numbers of each layer in the 2 nd layer or higher represent the output node numbers of the preceding layer, as the optimal node which has the best output performance in the current layer. Especially, identification error in the layer 0 of Figures 20 and 21 means the optimal output performance index of linear fuzzy inference based NFN as the premise part of SONFN. Table 9 shows a comparative analysis t h a t involves a number of previously developed models. In comparison with them, the SONFN comes with high accuracy and improved prediction (generalization) capabilities.
Relation-Based Neurofuzzy Networks
915
Table 9. Comparison of identification error with the performance obtained for some models reported in the literature. System Training Generalization Input (PI) (E_PI)
Model
DL
0.1579
DL,ME
O.O87O
0.1870 0.1907
DL
0.2870
0.2990
DL,ME
0.1444
0.1702
DL
0.2094
0.2462
DL,ME
0.1457
0.1679
DL
0.2870
0.2990
DL,ME
0.2115
0.2415
DL
0.2094
0.2462
DL,ME
0.1544
0.1967
DL
0.1748
0.1850
0.0493
0.0565
0.0231
0.0252
0.0877
0.1309
Modified Architecture Case 2 DL,ME
0.0630
0.0736
Advanced Modified Type Architecture Case 2 DL,ME
0.0131
0.0205
Shin and GoeFs RBF Model [12] Simplified Fuzzy Inference [19] Fuzzy Set-Based NFN LinearFuzzy Inference Simplified Fuzzy Inference [19] Fuzzy Relation-Based NFN Linear Fuzzy Inference Advanced Basic Type Architecture Case 2
Simplified Fuzzy Generic Basic Case 2 DL,ME Inference Based Type Architecture Our Model NFN (SONFN) Advanced Modified Type Architecture Case 2 DL,ME Advanced Basic Type Architecture Case 2 Linear Fuzzy Inference Based NFN
1
Generic
Type
,
,
DL
=
0"9f 0.8 i 0.7
\
o.o,\
Premise Part :
Consequence
Part :
PNN
'-~
0.4 0.3 0.2
OA~ 0
0
t
J
1O0 200
i
I
,
300 400
)0
I 1
I
Iteration Layer Figure 19. Learning procedure of the advanced and modified SONFN in Case 1. (Linear fuzzy inference based NFN, 2 --* 4 node inputs, and Type 2 PNN.) 3.2
Medical
Imaging
System
We consider a n m e d i c a l i m a g i n g s y s t e m (MIS [13]) subset of 390 m o d u l e s w r i t t e n in Pascal a n d F O R T R A N for modeling. T h e s e m o d u l e s consist of a p p r o x i m a t e l y 40,000 lines of code. To desiign a n o p t i m a l m o d e l from t h e MIS, we utilize 11 s y s t e m i n p u t variables such as, LOC, CL,
916
S.-K. OH et al.
(A) Type 1 -~ 3; @ (C) Type 2 --~ 3; [] (B)Type3-~ 1; O (D)Type3-~ 2; V 0.2
iiiiiiiiiiiiiiiiiiiiiiiiiii
0.1~
iii ........... i ............
0.11
..............
,~ 0,1, ..~ 0.12
~
o.1 0.08 0.06 0.04
0
1
2
3
Layer (a) Training errors (PI).
[ (A)TypeI: 3; @ (C)'rype2-~ 3; ~l (B)Type3
1; •
(D)Type3-~ 2;
l I
0
0
~
i
0
: ' :
: \
6 1
~
2
~:(9
t6 18) I
3
Layer
(b) Testing errors (E PI). Figure 20. Performance index of the generic and modified SONFN in Case 2. (Linear fuzzy inference-based NFN.)
TCHAR, TCOMN, MCHAR, CDHAR, N, NE, NF, VG, and BW. And as output variable, "NC" is used, See Table i0. Also, we use 4 system inputs [TCOMN, MCHAR, CDHAR, N] among II system inputs. Here 4 system inputs are selected from structure identification by GMDH method. When applying any modeling technique, an assessment of predictive quality is important. Data splitting is a modeling technique that is often applied to test predictive quality. Applying this technique, one randomly partitions the data set to produce two data sets. The first 60% data
Relation-Based Neurofuzzy Networks
917
(A) Type2;@ (C) Type3:[] (B) Type2;O (D) T y p e 3 ; V 0.16
I
I
I
E
0.1~ 0.12
E
0.1
.=
.~ 0.08 0.06
0.04 0.02 0
0 Layer (a) Training errors (PI). 1
(A) Type 2 ; @ (C)Type3 ; I~[ (B) Type 2 ; O (D) Type 3 ; ~'] I
0.18
-
-
i
==1
:~
~. oo
i
........ ~
~
i
-
~
!
i
~
~
i Tt'! ~
;
\.
~.x,,
,o~ i .......
1
\B:o ~ 6 |
I
oo, ........ i ........ i ........ i ........... "i ........ i ........... 0.02
-
-
'
'
L 1
L 2
Layer
(b) Testing errors (E_PI). Figure 21. Performance index of the advanced and modified SONFN in Case 1. (Linear fuzzy inference-based NFN.) set is used for fitting t h e models. T h e r e m a i n i n g 40% d a t a set, t h e t e s t i n g d a t a set, provides for q u a n t i f y i n g t h e predictive q u a l i t y of the fitted models. U s i n g MIS d a t a s e t , t h e regression e q u a t i o n is o b t a i n e d as follows. N C = - 1 . 8 3 7 1 + 0.14097- L O C - 0.14115 • CL - 0.0026487 • T C H A R +0.10312 • T C O M N + 0.0019233- M C H A R + 0.0086053. C D H A R
(28)
- 0 . 0 2 4 5 0 8 • N + 0.16458 • N E - 0.20678 • N F + 0.16353 • V G - 0.95354 • B W T h i s simple m o d e l comes with t h e value of P I = 40.056 a n d E_PI = 36.322 b y (7) for 11 system i n p u t s . W e will b e u s i n g as a reference p o i n t w h e n discussing S O N F N proposed in this paper.
S.-K. OH et aL
918
Table 10. Input and output variables of medical imaging system. length of code with comments
LOC
length of code without comments
CL TCHAR
number of characters
TCOMN
number of comments
MCHAR
number of comments characters
CDHAR
number of code characters program length
N NE
estimated program length
NF
Jensen's program length
VG
McCabe's cyclomatic number
BW
Belady's bandwidth metric
NC
number of changes
3.2.1. R e s u l t s o f N F N m o d e l i n g For t h e N F N s t r u c t u r e s , t h e i d e n t i f i c a t i o n errors of P I a n d E_PI are sh o w n in b el o w T a b l e 11. G A s help o p t i m i z e l e a r n i n g rate, m o m e n t u m coefficient, an d t h e p a r a m e t e r s of t h e m e m b e r s h i p functions. In N F N s t r u c t u r e , t w o m e m b e r s h i p f u n c ti o n s for each i n p u t v a r i a b l e are used. So N F N struct u r e is r e p r e s e n t e d by 16 fuzzy rules for t h e 4 s y s t e m inputs. For t h e l e a r n i n g of N F N , t h e n u m b e r of t r a i n i n g cycles is 500 iterations. Table 11. Performance index of the NFN. Model
Number of System Inputs
PI
E_PI
Regression
11
40.056
36.322
Simplified Fuzzy Inference Based NFN
4
46.084
38.559
Linear Fuzzy Inference Based NFN
4
41.412
34.048
Table 12. Performance index of the SONFN with 4 system inputs.
SONFN
Premise Part; NFN
Consequence Part; PNN
Number of MFs Fuzzy Inference
Number of Inputs & Polynomial Type Layer
2x2x2x2 Simplified 2x2x2x2 Generic Basic Case 1 Type Architecture
Simplified 2x2x2x2 Simplified 2x2x2x2
Simplified
E_PI
3 inputs
1
49.264 27.766
Type 2
2
40.753 17.898
3 inputs
1
50.957 26.599
Type 3
2
4 8 . 3 1 23.265
4 inputs
1
36.602 26.516
Type 2
2
42.776 16.066
4 inputs
1
48.69 25.441
Type 3
2
45.187 18.025
2x2x2x2
3 inputs
1
47.229 27.176
Linear
Type 2
2
43.595 21.375
3inputs
1
48.439 29.423
Type 3
2
44.344 22.186
4inputs
1
40.271 23.677
Linear
Type 2
2
37.373 16.679
2x 2x 2x2
4inputs
1
40.055 23.025
Linear
Type 3
2
35.748 17.807
2× 2 x 2 x Generic Basic Case 1 Type Architecture
PI
2
Linear 2x 2x2x
2
Relation-Based Neurofuzzy Networks
919
3.2.2. R e s u l t s of S O N F N modeling Table 12 summarizes the results of the preferred architectures according to SONFN. Here we select the SONFN in Case 1 with the 3 inputs-Type 2 for the simplified fuzzy inference based NFN and its detailed topology is visualized in Figure 22. Also, for the SONFN with linear fuzzy inference based NFN, the 4 inputs-Type 3 is selected and shown in Figure 23. Table 13 contains a comparative analysis including the previous model. Regression models are constructed by a linear equation. The comparative analysis reveals that the SONFN comes with high accuracy and improved prediction (generalization) capabilities. Table 13. Comparison of identification error with previous modeling methods.)
Number of System Training Inputs (PI)
Model Regression Model
Generalization (E_PI)
11
40.056
36.322
Fuzzy Relation-
Based on Simplified fuzzy inference [19]
4
46.084
38.559
based NFN
Based on linear fuzzy inference
4
41.412
34.048
Simplified Generic Basic Type Architecture Case 1
4
40.753
17.898
Generic Basic Type Architecture Case 1
4
35.748
17.807
Our Model (SONFN)
Linear
TCOMN
MCHAR Y
CDHAR
N
Figure 22. The optimal topology of the generic and basic SONFN in Case 1. (Simplified fuzzy inference based NFN, 3 inputs-Type 2 PNN.)
4. C O N C L U D I N G
REMARKS
In this study, we have introduced a class of SONFN regarded as a modeling vehicle for nonlinear and complex systems, studied its properties, came up with a detailed design procedure and used
920
S.-K. OH et al. im
TCOMN
MCHAR
CDHAR
N
Figure 23. The optimal topology of the generic and basic SONFN in Case 1. (Linear fuzzy inference based NFN, 4 inputs-Type 3 PNN.)
these networks to model a well-known NASA dataset and MIS dataset which are experimental data widely used in software engineering. SONFN is constructed by combining fuzzy relationbased NFN(Simplified or Linear fuzzy inference) with PNN. In this sense, we have constructed a coherent platform in which all components of CI are fully utilized. The model is inherently dynamic--the architecture of the PNN is not fully predetermined (as it usually happens in case of e.g., multiplayer perceptron) and can be generated (adjusted) during learning. A comprehensive design procedure was developed. The series of experiments helped compare the network with other fuzzy models--in all cases the previous models came with higher values of the performance index. REFERENCES 1. W. Pedrycz, Computational Intelligence: A n Introduction, CRC Press, Florida, (1998). 2. J.F. Peters and W. Pedrycz, Computational intelligence, In Encyclopedia of Electrical and Electronic Engineering, Volume 22, (Edited by J.G. Webster), John Wiley ~ Sons, New York, (1999). 3. W. Pedrycz and J.F. Peters (Editers), Computational Intelligence in Software Engineering, World Scientific, Singapore, (1998). 4. H. Takagi and I. Hayashi, NN-driven fuzzy reasoning, Int. J. of Approximate Reasoning 5 (3), 191-212, (1991). 5. S. Horikawa, T. Furuhashi and Y. Uchigawa, On fuzzy modeling using fuzzy neural networks with the back propagation algorithm, I E E E Trans. Neural Networks 3 (5), 801-806, (1992). 6. S.-K. Oh, T.-C. Ahn and W. Pedrycz, A study on the self-organizing polynomial neural networks, Joint 9 th IFSA World Congress, 1690-1695, (2001). 7. S.-K. Oh, D.-W. Kim and B.-J. Park, Transactions on Control, Automation and Systems Engineering, vol. 3, (2001). 8. D.-W. Kim, S.-K. Oh and H.-K. Kim, A Study on the self-organizing fuzzy polynomial neural networks, K I E E International Transactions on Systems and Control 12D (1), 12-16, (2002). 9. S.-K. Oh, D.-W. Kim and W. Pedrycz, Hybrid fuzzy polynomial neural networks, International Journal of Uncertainty Fuzziness and Knowledge-Based Systems 10 (3), 257-280, (2002).
Relation-Based Neurofuzzy Networks
921
10. G.E.P. Box, W.C. Hunter and J.S. Hunter, Statistics for Experimenters, John Wiley K: Sons, (1978). 11. A.G. Ivahnenko, The group method of data handling: A rival of method of stochastic approximation, Soviet Automatic Control 13 (3), 43-55, (1968). 12. M. Shin and A.L. Goel, Empirical data modeling in software engineering using radial basis functions, IEEE Trans on Software Engineering 26 (6), 567-579, (June 2000). 13. M.R. Lyu (Editor), Handbook of Software Reliability Engineering, pp. 510-514, McGraw-Hill, (1995). 14. T. Yamakawa, A new effective learning algorithm for a neo fuzzy neuron model, 5 TM IFSA World Conference, pp. 1017-1020, 1993. 15. D.B. Goldberg, Genetic Algorithms in Search, Optimization ~4 Machine Learning, Addison-Wesley, (1989). 16. C.M. Bishop, Neural Networks for Pattern Recognition, Oxford University, (1995). 17. M. Kearns and D. Ron, Algorithmic Stability and Sanity-Check Bounds for Leave-One-Out Cross-Validation, In Proc. 10th Ann. Conf. Computational Learning Theory, pp. 152-162, (1997). 18. C.F. IKemerer, An empirical validation of software cost estimation models, Comm. ACM 30 (5), 416-429, (May 1987). 19. S.-K. Oh, W. Pedrycz, B.-J. Park and T.-C. Ahn, A hybrid modeling architecture for software engineering, In Proceedings of the 6th IASTED International Conference ASC, pp. 309-314, (July 2002).