A genetic algorithm optimized Morlet wavelet artificial neural network to study the dynamics of nonlinear Troesch’s system

Accepted Manuscript Title: A Genetic Algorithm Optimized Morlet Wavelet Artificial Neural Network to Study the Dynamics of nonlinear Troech’s System Authors: Khalid Majeed, Zaheer Masood, Raza Samar, Muhammad Asif Zahoor Raja PII: DOI: Reference:

S1568-4946(17)30152-7 http://dx.doi.org/doi:10.1016/j.asoc.2017.03.028 ASOC 4113

To appear in:

Applied Soft Computing

Received date: Revised date: Accepted date:

23-1-2016 17-5-2016 22-3-2017

Please cite this article as: Khalid Majeed, Zaheer Masood, Raza Samar, Muhammad Asif Zahoor Raja, A Genetic Algorithm Optimized Morlet Wavelet Artificial Neural Network to Study the Dynamics of nonlinear Troech’s System, Applied Soft Computing Journalhttp://dx.doi.org/10.1016/j.asoc.2017.03.028 This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

A Genetic Algorithm Optimized Morlet Wavelet Artificial Neural Network to Study the Dynamics of nonlinear Troech's System

Khalid Majeed1,a, Zaheer Masood1,b Raza Samar1,c, and Muhammad Asif Zahoor Raja2,d 1

Department of Electrical Engineering, Muhammad Ali Jinnah University, Islamabad, Pakistan

Email: [email protected], [email protected], [email protected], 2

Department of Electrical Engineering, COMSATS Institute of Information Technology, Attock, Pakistan Emails: [email protected] , [email protected], [email protected]

d

Graphical abstract

Highlights     

Novel Design of Morlet Wavelets Neural Networks Models for differential equations Bio-inspired Heuristics integrated with SQP for training of weights of the networks The design scheme is viable to solve Troesch’s problem arising in Plasma Physics The results of the proposed algorithm are in good agreement with Adams methods Accuracy and convergence are validated through results of the statistical Analyses

Abstract In this work, a new stochastic computing technique is developed to study the nonlinear dynamics of Troesch’s problem by designing the mathematical models of Morlet Wavelets Artificial Neural Networks (MW-ANNs) optimized with Genetic Algorithm (GA) integrated with Sequential Quadratic Programming (SQP). The differential equation mathematical

model for MW-ANNs are designed for Troesch’s system by incorporating a windowing kernel based on Morlet Wavelets as an activation function and these networks are constructed to define a fitness function of Troesch’s system in the mean squared sense. The unknown adjustable parameters of MW-ANNs are trained initially by an effective global search using GAs hybridized with SQP for rapid local refinement of the results. The proposed scheme is evaluated to solve the Troesch’s problems for small and large values of the critical parameter in the system. Comparison of the proposed results with standard reference solutions of Adams method shows good agreement. Validation of accuracy and convergence of the proposed scheme is made using statistical analysis based on a sufficiently large number of independent runs, this is done in terms of performance measures of mean absolute deviation and root mean squared error. Keywords: Troesch’s Problem; Wavelet Windowing Kernels, Artificial Neural Networks; Genetic Algorithms; Sequential Quadratic programming; Bio-inspired Computing; 1. Introduction The nonlinear Troesch’s system is an inherently unstable two-point BVP described by an Ordinary Differential Equation (ODE). The Troesch’s problem arises in the study of confinement of a plasma column by radiation pressure and also arises in the theory of gas porous electrodes [1-2]. A comprehensive overview of the nonlinear Troesch’s BVP is presented by Weibel [3]. Due to its practical importance many analytical as well as deterministic numerical methods have been designed for its solution, see [4-11] and reference contained therein. However, most of the techniques applied to solve the problem are based on deterministic solvers with their own intrinsic worth and limitations. Stochastic solvers based on artificial intelligence techniques are relatively less exploited for solving such highly sensitive nonlinear systems. In this paper dynamics of the nonlinear Troesch’s system is investigated through a newly proposed artificial neural network model employing Morlet wavelet kernel function optimized with a hybrid evolutionary computing approach based on Genetic Algorithm (GA) and Sequential Quadratic Programming (SQP). The Troesch’s boundary value problem (BVP) is written as:

d2y   sinh(  y( x)), x  [0, X ] , dx 2 y(0)  0,

(1)

y(1)  1,

where  is a critical parameter of the system. The nonlinear Troesch’s BVP given above become stiff for values of the critical parameter  greater than one. Stochastic solvers based on artificial neural networks (ANNs) have been shown to solve a variety of problems arising in a range of disciplines [12–16]. For instance, solvers based on artificial intelligence techniques have been used for solutions of the nonlinear Van der Pol equations [17], second order BVPs [18], integrodifferential systems [19], singular systems [20-21], Emden–Fowler systems arising in electromagnetic theory [22], systems of secondorder boundary value problems [23], inverse kinematic problems [24], the nonlinear first Painleve system [25-26], nonlinear Navier-Stokes systems [27], pantograph differential equation with delays [28], fluid mechanics problems based on thin film flow of third grade fluids [29], nanotechnology problems based on multi-walled carbon nanotubes [30], Johnson–Segalman fluid flow based on drainage problems [31] MagnetoHydroDynamics 2

(MHD) nonlinear Jeffery-Hamel flow systems [32-33], nonlinear Lane–Emden type equations [34], fuel ignition models arising in combustion theory [35-36], Riccati and Bagley-Torvik fractional order systems [37-38], etc. Recently log-sigmoid function based ANNs trained through Genetic Algorithms (GAs) and Particle Swarm Optimization (PSO) algorithms have been applied for the solution of Troesch’s system [39-41]. An alternate design of an ANN with a wavelet windowing function is a candidate area to explore for improvement in performance. The use of wavelet neural networks (WNN) in different fields based on Morlet windowing kernels has been extensively reported such as for effective shortterm water quality prediction [42], rainfall–runoff modeling [43], wind power forecast [44], enhanced PID controller design [45], reliable modeling of truck engine powertrain components [46], and short term solar irradiance forecasting [47]. The authors are motivated to explore the potential of the Morlet Wavelets windowing kernel in the ANN design to approximate the model given in equation (1). The real strength of these networks is exploited by learning of parameters with bio-inspired computational heuristics based on GA, together with local optimizers such as SQP algorithms. The proposed scheme is evaluated for accuracy and robustness for solving the Troesch’s BVP for small and larger values of the critical parameter  . In this paper, a new ANN based intelligent computing method is developed in which Morlet wavelets windowing kernel is used as the activation function while the optimization task is performed with a hybrid approach based on GA-SQP. Accuracy and convergence of the method are tested for both small and large values of the critical parameter µ. The performance of the schemes is analyzed through analysis of mean absolute deviation (MAE), root mean squared error (RMSE) and their global versions. The computational complexity of the models is examined through analysis of execution time, number of iterations and function counts. This paper is organized as follows. In Section 2, the proposed methodology based on artificial neural network models and the optimization procedure is given. Mathematical expressions for performance indices, including MAE and RMSE are provided in Section 3. The results of detailed numerical experimentations in different cases of the Troesch’s problem are described in Section 4. Comparative analyses through performance indices based on MAE and RMSE values are also given in Section 4 along with an analysis of computational complexity. Section 5 concludes the paper.

2. Methodology The proposed methodology for solving the nonlinear Troesch’s problem consists of three phases. In the first phase, the Morlet Wavelet artificial neural network (MWANN) is designed. In the second phase, the network is used in an unsupervised manner to generate a mean squared error based fitness function. In the third phase, a procedure for the training of weights of the network using Genetic Algorithms (GAs) and Sequential Quadratic Programming (SQP) is developed. The flow diagram of the design scheme is given in Fig. 1. 2.1 Design of Morlet Wavelet Artificial Neural Network (MWANN) Differential equation artificial neural network models are generally constructed using the input, output, and single hidden layer architecture for the solution y( x) and its derivatives (of an ODE), and are mathematically written as:

3

m

yˆ ( x)    i fT ( wi x  i ),

(2)

i 1

and its nth order derivative is given as:

d n yˆ m dn  i fT ( wi x  i ), dx n i 1 dt n

(3)

where m is the number of neurons in the network, fT is the activation function,  = 1 ,  2 , ,  m  ,  =  1 , 2 , , m  , and w =  w1 , w2 , , wm  are unknown adjustable parameters of the network, represented as the weight vector W =  , w,   . A novel design of the differential equation neural network is proposed by exploiting the strength of the Morlet wavelet windowing function. The mathematical representation of the Morlet kernel is given as: fT  x   cos(1.75x )e x

2

/2

(4)

The differential equation neural network as given in equations (2-3) with Morlet function (4) is formulated as: m

 



( 0.5 w x   yˆ ( x)  i cos 1.75  wi x  i  e  i i  i 1

 



2

,

2 d n yˆ m d n ( 0.5 wi x  i    cos 1 .7 5 w x   e   i i i dx n i 1 dx n

(5)

.

(6)

In order to solve Troesch’s problem, the first and second derivatives of the MWANN are also required: 0.5( xwi  i )  sin(1.75( xwi  i )) wi dyˆ m  1.75e    dx i 1  e0.5( xwi  i )2 cos(1.75( xw   )) w ( xw   )  i i i i i  

(7)

 3.0625e 0.5( xwi  i ) cos(1.75( xwi  i )) wi2     2 m  3.5e 0.5( xwi  i ) sin(1.75( xw   )) w2 ( xw   )   d 2 yˆ i i i i i    2 2  0. 5( xw   ) 2 i i dx i 1  cos(1.75( xw   ))( 1.e  w  i i i  0.5( xw   )2 2  2 i i 1.e w ( x w   ) ) i i i  

(8)

2

2

The differential equation network for MWANN models as given in equations (5)-(8) are represented as shown in Fig. 2 to construct a mathematical model of the Troesch’s system. 2.2. Fitness Function The fitness function  of the nonlinear Troesch’s system as given in equation (1) is formulated as the sum of two mean squared errors as: 4

  1   2 ,

(9)

where  1 is the error function for the differential equation and is given as: 2

 1 N  d 2 yˆ m 1   2   sinh(  yˆ m )  , x  [0, 1]  N  1 m 0  dtm  N  1/ h,

yˆ m  yˆ ( xm ), xm  mh

,

where the input grid points N are taken between 0 and 1 as x [ x0  0, x1 , x2 ,

(10)

, xN  1] , h

d yˆ are given in equations (5) and (8) dx 2 2

is the step size, and the networks for yˆ and respectively.

The error function  2 is linked with the boundary conditions and is written as:

2 

1 ( yˆ0 )2  ( yˆ N  1)2  .  2

(11)

The fitness function given in (9) is basically a minimum optimization problem: with trained unknown parameters W of the MWANNs such that the fitness function   0 , then both (1 ,  2 )  0 and eventually, the approximate solution approaches the exact solution, i.e.,

yˆ ( x)  y( x) . 2.3 Optimal Weights search for MWANNs Design parameters of differential equation MWANN models are trained by exploiting the strength of a hybrid computing approach based on Genetic algorithms (GAs) and Sequential Quadratic Programming: GA-SQP. GAs belong to the class of bio-inspired computational heuristic algorithms for global search that are developed on the basis of mathematical modeling of the process of natural selection. The standard procedure of GAs to get a feasible solution is based on three fundamental operations, i.e., selection, crossover, and mutation. The efficiency, accuracy, and robustness of GAs is well established since its introduction by Holland [48], and a number of constrained and unconstrained optimization problems arising in diverse fields of science and engineering have since been solved. Few examples include development of high pressure cyclones for a gas city gate station [49], solving nonlinear systems [50], prediction of minimum miscibility pressure [51], permeability estimation in petroleum reservoir [52], reliable solution of nonlinear equations of single variables [53] close-loop supply chain network design in large-scale networks [54], etc. Hybridization of a local optimizer with GA is done for rapid refinement of the results by taking the best weights of the GA as a starting point of the Sequential Quadratic Programming (SQP) algorithm. The SQP algorithm belongs to the class of local search algorithms and its strength in terms of accuracy, reliability and efficiency is well-established for linear and nonlinear constrained/unconstrained optimization problems [55]. Some recent applications of the SQP algorithm are effective vehicle passing mechanisms on two-lane highways [56], global convergence capability for constrained optimization problems using strength of flexible step acceptance strategy [57], economic load dispatch with stochastic wind power optimization [58], viable optimization of nonlinear complementarity problems [59], and optimization problem of simple processes for the Liquefaction of Natural Gas (LNG) [60]. 5

Keeping in view the potential of both GAs and SQP algorithms, a hybrid GA-SQP algorithm is developed for weight optimization of the MWANN model to solve the Troesch’s problem. Built-in functions available in the Matlab optimization toolbox are used with parameter settings as listed in Table 1. These settings are important, changes in the parameters will affect the optimization results. Workflow of the proposed methodology is depicted in Fig. 2, the essential details of the intermediate steps are listed below: Part 1: Start of Genetic Algorithm Step 1: Initialization: Initial chromosome or individual is set up with randomly generated bounded real values, the number of genes is equal to the number of unknown parameters of the MWANN model. Mathematically the chromosome C is represented as: C  W   , w,    1 ,  2 , ...,  k , w1 , w2 , ..., wk , 1, 2 , ..., k

,

for k neurons. A set of chromosomes or individuals form the population P, given as:  C1   1 , w1 , 1      C  , w2 ,  2  P  2  2 , or    ,  ,      C m    m , wm ,  m  .  1,1 , 1,2 , ..., 1,k , w1,1 , w1,2 , ..., w1, k , 1,1 , 1,2 , ..., 1, k     2,1 ,  2,2 , ...,  2,k , w2,1 , w2,2 , ..., w2,k ,  2,1 ,  2,2 , ...,  2,k   P   , , , , , , ..., , ,     ,  , ...,  , w , w , ..., w ,  ,  , ...,   m,k m ,1 m ,2 2 m,k m ,1 m ,2 m,k   m ,1 m ,2

Here m is the total number of chromosomes in the population P. The parameter settings for GA using the MATLAB ‘gaoptimset’ routine are listed in Table 1. Step 2: Fitness calculation: Fitness for each chromosome C of the population P is determined using equation (9) and its constituent parts given in (10) and (11). Step 3: Termination: Execution of the GA is terminated if any of the following conditions is satisfied:       

Fitness value ε ≤ 10-20. 1000 Generations executed. Function Tolerances (Tolfun) ≤ 10-20. Constraints Tolerance (TolCon) ≤ 10-20. Maximum function evaluations (MaxFunEvals) ≤ 200000. Stall generation limit (StallGenLimit) = 250. Other default values of termination for GA

If any of the above conditions is satisfied, go to part 2 otherwise continue. Step 4: Ranking: Rank each chromosome C on the basis of the minimum value of the fitness ε. The lower the value of the fitness ε, the better is the ranking. Step 5: Reproduction: The population P is updated for each generation using crossover, selection and mutation operators. Built-in functions are invoked: 

Selection: @selectstochunif 6

  

Crossover: @crossoverheuristic Mutations: @mutationadaptfeasible Elite count: 02

Go to step 2 with the newly generated population P. Part 2: Hybridization with Sequential Quadratic Programming (SQP). Refinement in the results of the GA is performed with a local search algorithm based on SQP. A built-in MATLAB routine ‘fmincon’ is used, necessary details are below. Step 1: Initialization: The start point of the SQP algorithm is the final parameter vector of the GA. Other assignments, bounds and declarations of the settings for SQP are done using the ‘optimset’ function, as listed in Table 1. Step 2: Fitness Calculation: The fitness for the weight vector is computed using equation (9) and its constituent parts given in (10-11). Step 3: Termination: The SQP algorithm is stopped if any of the following is attained:  1000 Iterations executed.  Function Tolerances (Tolfun) ≤ 10-25.  Constraints Tolerance (TolCon) ≤ 10-25.  Maximum function evaluations (MaxFunEvals) = 200000  Tolerance on optimization variables (TolX) ≤ 10-15 If terminated, go to part 3 otherwise continue Step 4: Refinements in weights: For each step increment of the SQP, the weight vector is updated. Go to step 2 with the refined weight vector. Part 3: Repository of the results Store the optimized parameters of the GA-SQP for the present run of the hybrid algorithm as listed below:     

The final weight vector Fitness attained, The time consumed Number of generations executed Total number of function calls executed

Part 4: Multiple runs of the GA-SQP algorithm Repeat parts 1 to 3 of the hybrid GA-SQP algorithm for a sufficiently large number of independent runs, each run starts with a different initial random number seed. The dataset is then used to analyze the efficacy of the algorithm using various performance indices. 3. Performance indices In this section, the performance measures are defined which will be used to evaluate the accuracy and convergence of the proposed scheme. The Absolute Error (AE) is defined as the absolute deviation of the proposed solution yˆ ( x) from the reference solution of the Adams method y ( x) , it is given mathematically as:

7

AE  y( xi )  yˆ ( xi ) for i  1, 2,..., N ,

(12)

where N is the number of input grid points. Mean Absolute Error (MAE) is defined as: MAE 

1 N   y( xi )  yˆ ( xi )  , N i 1

(13)

And the Root Mean Squared Error (RMSE) is defined as: N

1 N

RMSE 

  y( x )  yˆ ( x )  i

i 1

2

(14)

i

Global performance measures are also defined: the mean value of MAE (GMAE), global fitness (GFit), and global RMSE (GRMSE). These global operators are defined as follows:

GMAE

Rn

1  Rn

 MAE

1  Rn

1   r 1  N

GRMSE 

r 1 Rn

1 Rn

1  Rn

GFit

1  Rn

N

 i 1

 y ( xi )  yˆ ( xi )   r

,

(15)

Rn

 RMSE r 1 Rn

1 N

 r 1

,

N

  y( x )  yˆ ( x )  i 1

i

(16)

2

i

Rn

 r 1

r

,

where Rn represents the total number of independent runs and  run.

(17)

r

gives the fitness for the rth

4 Numerical Experiments In this section, the results of the proposed scheme based on MWANN optimized with GASQP algorithm are presented for solving the nonlinear Troesch’s system. Two main scenarios of the problem are taken with varying the critical parameter µ so that both the stiff (µ ≥ 1) and non-stiff (µ  1) cases are explored. The exact solution is not available, therefore numerical results determined using the Adam's method are taken as reference in this study. 4.1 Scenario 1: Nonlinear Troesch’s System with µ  [0.1, 2.3] In this scenario, the Troesch’s system (1) is solved for the following values of the critical parameter µ: Case 1: µ = 0.1, Case 2: µ = 1.5 and Case 3: µ = 2.3. Troesch’s problem in these cases is written as:

8

d2y  0.1sinh(0.1y)  0, dx 2

(18)

d2y  1.5sinh(1.5 y )  0, dx 2

(19)

d2y  2.3sinh(2.3 y)  0, dx 2

(20)

The boundary conditions are given as: y(0)  0,

y(1)  1,

(21)

MWANN model for each of these cases is developed by taking 10 neurons, i.e., k = 10, hence we have 30 unknown weights in the network. The number of input grid points is taken as N = 11 and 101, this implies that the span x [0, 1] is divided into 10 and 100 equal intervals with a step size h = 0.1 and h = 0.01, respectively. The fitness function (9) for h = 0.1 is constructed as: 2

 1 1 10  d 2 yˆ     2m  0.1sinh(0.1yˆ m )    yˆ0 2  ( yˆ10  1) 2  11 m0  dxm  2

(22)

For the step size h = 0.01, the fitness function is constructed as: 2

 1 2 1 100  d 2 yˆ m 2   2  0.1sinh(0.1yˆ m )    yˆ0  ( yˆ100  1)   101 m0  dxm  2

(23)

Similarly, the fitness functions can be constructed for the remaining two cases. 4.2 Scenario 2: Nonlinear Troesch’s System with µ  [0.5, 2] Troesch’s problem is solved for three cases based on value of the critical parameter µ as: Case 1: µ = 0.5, Case 2: µ = 1.0 and Case 3: µ = 2.0. The three cases are written as:

d2y  0.5sinh(0.5 y)  0, dx 2

(24)

d2y  sinh( y )  0, dx 2

(25)

d2y  2sinh(2 y)  0, dx 2

(26)

with boundary conditions: y(0)  0,

y(1)  1,

(27)

For h = 0.1, the fitness function for case 1 is written as: 9

2

 1 1 10  d 2 yˆ m     2  0.5sinh(0.5 yˆ m )    yˆ0 2  ( yˆ10  1) 2  11 m0  dxm  2

(28)

For h = 0.01, the fitness function is given as: 2

 1 2 1 100  d 2 yˆ m 2   2  0.5sinh(0.5 yˆ m )    yˆ0  ( yˆ100  1)   101 m0  dxm  2

(29)

Fitness functions for the remaining two cases can be constructed accordingly. The proposed procedure as described in the last section based on GA-SQP is applied for optimizing the fitness function  for each case of both scenarios. Optimized weights for scenario 1 yielding fitness values 1.2722e-12, 5.8636e-10, and 1.5843e-07 for µ=0.1, 1.5, 2.3, respectively for h = 0.1 are given below. For h=0.01 we have fitness values of 6.4804e13, 3.4071e-09, and 3.3006e-07 for cases 1, 2 and 3, respectively for h = 0.01. In scenario 2 with h = 0.1, we achieve fitness values of 8.1369e-12, 4.6008e-10, and 7.7631e08 for µ = 0.5, 1, and 2, respectively. Another optimization yields for h = 0.01 yields fitness values of 8.4059e-11, 4.8493e-11, and 3.36637e-08 for cases 1, 2 and 3 respectively. The optimized weights are shown graphically in Fig. 3. Using these weights the proposed solutions (equation (5)) for the Troesch’s problem for inputs x [0, 1] for case 1 of the scenario 1 with step size h = 0.1 is written as: yˆ c 1  0.2163cos 1.75  0.3392x  0.2784   e0.5 0.3392x 0.2784 

2

 1.2062 cos 1.75  0.1130 x  0.8742   e0.5 0.1130 x 0.8742   ... 2

 1.6707 cos 1.75  0.1539 x  0.7634   e0.5 0.1539 x 0.7634 

(30)

2

For case 1 of the scenario 1 by taking h = 0.01, the proposed solutions are written as: yˆ c 1  0.7925 cos 1.75  0.5021x  0.6327   e0.5 0.5021x  0.6327 

2

 0.5884 cos 1.75  0.2937x  1.2087   e0.5 0.2937x 1.2087   ... 2

 1.2529 cos 1.75  0.2641x  0.6586   e0.5 0.2641x 0.6586 

(31)

2

Similarly the solution for each case of both scenarios are derived and results are provided in Appendix Tables A-1 and A-2 for h = 0.01 and 0.1, respectively, with weights up to 14 decimal places of accuracy in order to avoid the round off error problems. The solutions are determined using equations similar to (30-31) and the results plotted in Fig 4 for both scenarios for inputs x [0, 1] with a step size of 0.05. The values of the reference numerical solution with Adam's method are also plotted in Fig 4 for the same inputs. It is seen from Fig. 4 (a) and (b) that the proposed solutions consistently overlap the reference numerical results. In order to assess the accuracy of the proposed scheme, values of AE are calculated and results are given in Fig 4 (c) and (d) for scenario 1 and in Fig. 4 (e) and (f) for scenario 2. Values of AE are listed in Tables 2 and 3 for each case of both scenarios. It is seen that the values of AE are of the order of 10-08 to 10-09, 10-07 to 10-08 and 10-05 to 10-06 for cases 1, 2 and 3 for both scenarios of the problem. The accuracy of results of the MWANN-GA-SQP decreases with increase in the value of the critical parameter μ. This is understandable since 10

the stiffness of the problem increase with an increase in the value of μ, and vice-versa. There is no appreciable difference in the proposed results due to changes in the value of the step size (0.1 or 0.01), however, the computational time increases significantly in the smaller step size. Accuracy of the designed scheme is further analyzed on the basis of 100 independent runs, and the results of the statistical analysis in terms of Mean and STandard Deviation (STD) values are listed in Tables 4 and 5 (for inputs between 0 and 1 with a step size of 0.05) for each case of both scenarios. It is observed that mean values for cases 1, 2 and 3 are of the order of 10-07, 10-06 and 10-05, respectively. Smaller values of STD are consistently observed in each case which establishes the reliability of the proposed scheme to solve Troesch's problem. Performance indices as given in equations (13-14) are applied to analyze the efficacy of the proposed MWANN-GA-SQP scheme, and the results in terms of fitness ε, MAE, and RMSE against the number of independent runs are plotted in Figs. 5, 6 and 7 respectively, for each case of both scenarios of the Troesch’s system. When the values of fitness achieved are small, corresponding values of MAE and RMSEs are also small, and vice-versa. Better values of fitness, MAE and RMSE are consistently obtained for small values of the critical parameter μ, i.e., for μ = 0.1 or 0.5. Results degrade slightly for μ = 2 or 2.3, but still an accuracy of 4 to 7 decimal places exists in comparison to the reference numerical solution. Additionally, for all three cases of both scenarios, convergent results are consistently obtained. To assess reliability and convergence, a criterion is defined as follows: fitness ε ≤ 10-05, MAE ≤ 10-03 and RMSE ≤ 10-03. Based on this criterion, results of percentage convergent runs for each case of both scenarios of Troesch’s system are tabulated in Table 6. It is seen that 100% convergent results are obtained for each case, however, if the criterion is made stricter, results for case 3 of both scenarios show degradation due to increase in stiffness of the system. Performance analysis of the designed MWANN-GA-SQP methodology is further carried out based on global operators defined in equations (15-17), results for 100 independent runs of the scheme in terms of GFit, GMAe, and GRMSE are given in Table 7. It is seen that GFit, GMAe, and GRMSE values lie in the range 10-06 to 10-09, 10-05 to 10-07 and 10-05 to 10-07 respectively, for both step sizes h = 0.1 and 0.01. For higher values of µ, GFit, GMAE, and GRMSE degrade indicating reduced accuracy. However, in these cases performance of the proposed algorithm with h = 0.01 is slightly better than for h = 0.1. Small values of STD for the global operators are consistently obtained for each case of both scenarios of the problem, this shows the effectiveness of the proposed MWANN-GA-SQP scheme in terms of accuracy and convergence. The computational complexity of the proposed MWANN-GA-SQP model is evaluated by logging the mean and STD values of the execution time, the number of generations/iterations and the total number of function evaluations (Function Counts (FCs)). This is summarized for 100 independent runs of the MWANN-GA-SQP model for each case of both scenarios in Table 8. The average time, number of iterations and function count in case of h = 0.1 is 16 ± 2, 1990 ± 10 and 450000 ± 20000, respectively. For h = 0.01 these respective values are 115 ± 20, 1990 ± 10 and 440000 ± 20000. Value of execution time for h = 0.01 is almost 8 to 10 times greater than for h = 0.1 for all cases. All simulations presented in this study are performed on a Dell Server R920, CPU 4x Intel Xeon 2.5 GHz, with 768 GB RAM, running Matlab 2015a in a Windows 2012R2 environment. Comparative studies of the proposed Morlet wavelet ANN (MWANN) are performed with different type of network models based on traditional activation functions like log-sigmoid and tan-sigmoid, as well as, modern wavelet kernels. In order to have a reliable and effective 11

comparison, a fixed parameter setting of the algorithm SQP is used and 10 numbers of neurons are taken in the hidden layer of each model. Hundred independent runs of the each model based on Mexican Hat ANN (MHANN), log-sigmoid ANN (LSANN), tan-sigmoid ANN (TSANN) is performed to find the solution of Troesch’s system for each case. The results on the basis of fitness and MAE values against a number of runs of each model are presented in Figures 7 and 8, respectively. These graphs are plotted on semi-logarithmic scale in order to observe the small differences. The minimum and mean values of fitness, MAE and RMSE are presented in Figures 9 and 10 for all four ANN models in each case of scenario 1 and 2 of the problem, respectively. It is seen that with the increase of parameter μ decrease in the accuracy of all four ANN models is observed for both scenarios. Few runs of the algorithms based on MWANN, MHANN and LSANN could not give the reasonably accurate results and share of trapping in local minima is further elevated by increasing the values of critical parameter μ in the system. However, the problems of premature convergence can be avoided through the process of hybridization in which global search is performed earlier and later on efficient local search is conducted. It is evidently decipherable from the results of the proposed method GA-SQP presented in Figures 5, 6 and 7, that the hybrid algorithm consistently accurate, while the big oscillation in the results is observed for the optimization mechanism based on a pure SQP algorithm. 5 Conclusions On the basis of the results presented in the last section, the following conclusions are drawn: 

A novel design of a Morlet Wavelet Artificial Neural Network (MWANN) optimized with Genetic Algorithm (GA) integrated with Sequential Quadratic Programming is developed to study the dynamics of the nonlinear Troesch’s system.



The designed scheme is applied to different scenarios by varying the critical parameter μ of the system, as well as the number of input grid points. Results match with a solution from Adam's method with 5 to 9 decimal places of accuracy. Increasing the value of the critical parameter µ of the system has a significant impact on the accuracy of the solution, however, an accuracy of the order of 10-05 to 10-06 is still achieved.



Accuracy and convergence of the MWANN-GA-SQP algorithm are evaluated through results of statistical operators based on mean and standard deviation values that are calculated for 100 independent runs for each variant of the nonlinear Troesch’s system. Small values of the mean and STD of the solutions establish consistency in the results of the algorithm.



Performance indicators based on mean absolute error and root mean squared error and their global versions are seen to consistently achieve a close match with the reference solution. This shows that the proposed algorithm is reliable and robust.



Computational complexity in terms of mean execution time, the number of generations/iterations, and function counts show that the MWANN-GA-SQP algorithm is efficient in the case h = 0.1. For h = 0.01 the computational complexity increases by a factor of 10, however, there is a slight improvement in accuracy for stiff values of the critical parameter μ.



Morlet Wavelet ANN can solve the Troesch’s problem with almost the same accuracy as obtained by its state of art counterparts in neural network methodologies based on log-sigmoid, tan-sigmoid and Mexican hat activation functions.

12

Morlet wavelet ANN is a good alternative to be explored/exploited in future for better results in case of difficult nonlinear systems based on ordinary, partial, fractional, fuzzy, and functional differential equations and their systems. Additionally, the role of modern heuristics computing techniques based on immune or particle swarms could not be denied for the improvement in accuracy and convergence. The few potential alternatives based on Moth flame optimization algorithm, fireworks algorithm, particle swarm optimization (PSO), catfish PSO, fractional PSO, differential search algorithm, differential evolution, and backtracking search algorithms are good choices to be implemented for the improved performance.

13

References [1].

Markin, V.S., Chernenko, A.A., Chizmadehev, Y.A. and Chirkov, Y.G., 1966. Aspects of the theory of gas porous electrodes. Fuel Cells: Their Electrochemical Kinetics, Consultants Bureau, New York, pp.21-33.

[2].

Gidaspow, D. and Baker, B.S., 1973. A model for discharge of storage batteries. Journal of the electrochemical Society, 120(8), pp.1005-1010.

[3]

Weibel, E.S., 1958. Confinement of a plasma column by radiation pressure. The plasma in a magnetic field, pp.60-76.

[4]

Troesch, B.A., 1976. A simple approach to a sensitive two-point boundary value problem. Journal of Computational Physics, 21(3), pp.279-290.

[5].

Chang, S.H., 2010. Numerical solution of Troesch’s problem by simple shooting method. Applied Mathematics and Computation, 216(11), pp.3303-3306.

[6].

Vazquez-Leal, H., Khan, Y., Fernandez-Anaya, G., Herrera-May, A., SarmientoReyes, A., Filobello-Nino, U., Jimenez-Fernandez, V.M. and Pereyra-Diaz, D., 2012. A general solution for troesch's problem. Mathematical Problems in Engineering, 2012.

[7].

Caglar, H., Caglar, N. and Ozer, M., 2014. B-Spline Solution and the Chaotic Dynamics of Troesch's Problem. Acta Physica Polonica A, 125(2), pp.554-560.

[8].

Doha, E.H., Baleanu, D., Bhrawi, A.H. and Hafez, R.M., 2014. A Jacobi collocation method for Troesch’s problem in plasma physics. Proceedings Of The Romanian Academy, Series A.

[9].

Hilal, K., 2014. New numerical scheme for solving Troesch’s Problem. Mathematical Theory and Modeling, 4(2), pp.65-72.

[10].

Kouibia, A., Pasadas, M., Belhaj, Z. and Hananel, A., 2015. The variational spline method for solving Troesch’s problem. Journal of Mathematical Chemistry, 53(3), pp.868-879.

[11].

Temimi, H. and Kurkcu, H., 2014. An accurate asymptotic approximation and precise numerical solution of highly sensitive Troesch’s problem. Applied Mathematics and Computation, 235, pp.253-260.

[12]

Zeng, X.Y., Shu, L., Huang, G.M. and Jiang, J., 2015. Triangular fuzzy series forecasting based on grey model and neural network. Applied Mathematical Modelling in press DOI: 10.1016/j.apm.2015.08.009.

[13]

Yuan, H., Lu, C., Ma, J. and Chen, Z.H., 2015. Neural network-based fault detection method for aileron actuator. Applied Mathematical Modelling. In press. DOI:

[14]

Jwo, D.J., Wen, Z.M. and Lee, Y.C., 2015. Vector tracking loop assisted by the neural network for GPS signal blockage. Applied Mathematical Modelling, 39(19), pp.59495968.

[15]

Jafarian, A., Measoomy, S. and Abbasbandy, S., 2015. Artificial neural networks based modeling for solving Volterra integral equations system. Applied Soft Computing, 27, pp.391-398.

[16]

Rudd, K. and Ferrari, S., 2015. A constrained integration (CINT) approach to solving partial differential equations using artificial neural networks. Neurocomputing, 155, pp.277-285.

14

[17]

Khan, J.A., Raja, M.A.Z., Syam M.I., Tanoli S.A.K., and Awan S.E., 2015. Design and application of nature inspired computing approach for nonlinear stiff oscillatory problems, Neural Computing and Application 26(7), pp 1763-1780.

[18]

Abu-Arqub, O., Abo-Hammour, Z. and Momani, S., 2014. Application of continuous genetic algorithm for nonlinear system of second-order boundary value problems. Appl. Math, 8(1), pp.235-248.

[19]

Al-Smadi, M., Abu Arqub, O. and Momani, S., 2013. A computational method for two-point boundary value problems of fourth-order mixed integrodifferential equations. Mathematical Problems in Engineering, 2013.

[20]

Abo-Hammour, Z., Arqub, O.A., Alsmadi, O., Momani, S. and Alsaedi, A., 2014. An optimization algorithm for solving systems of singular boundary value problems.Appl. Math, 8(6), pp.2809-2821.

[21]

Mall, S. and Chakraverty, S., 2015. Numerical solution of nonlinear singular initial value problems of Emden–Fowler type using Chebyshev Neural Network method. Neurocomputing, 149, pp.975-982.

[22]

Khan, J.A., Raja, M.A.Z., Rashidi, M.M., Syam, M.I. and Wazwaz, A.M., 2015. Nature-inspired computing approach for solving non-linear singular Emden–Fowler problem arising in electromagnetic theory. Connection Science, pp.1-20.

[23]

Arqub, O.A. and Abo-Hammour, Z., 2014. Numerical solution of systems of secondorder boundary value problems using continuous genetic algorithm.Information sciences, 279, pp.396-415.

[24]

Momani, S., Abo-Hammour, Z.S. and Alsmadi, O.M., 2016. Solution of Inverse Kinematics Problem using Genetic Algorithms. Appl. Math, 10(1), pp.1-9.

[25]

Raja, M.A.Z., Khan, J.A., Siddiqui, A.M., Behloul, D., Haroon, T. and Samar, R., 2015. Exactly satisfying initial conditions neural network models for numerical treatment of first Painlevé equation. Applied Soft Computing, 26, pp.244-256.

[26]

Raja, M.A.Z., Khan, J.A., Shah, S.M., Samar, R. and Behloul, D., 2014. Comparison of three unsupervised neural network models for first Painlevé Transcendent. Neural Computing and Applications, 26(5), pp 1055-1071.

[27]

Abo-Hammour, Z.S., Samhouri, A.D. and Mubarak, Y., 2014. Continuous Genetic Algorithm as a Novel Solver for Stokes and Nonlinear Navier Stokes Problems. Mathematical Problems in Engineering, 2014.

[28]

Raja, M.A.Z., 2014. Numerical treatment for boundary value problems of pantograph functional differential equation using computational intelligence algorithms. Applied Soft Computing, 24, pp.806-821.

[29]

Raja, M.A.Z., Khan, J.A. and Haroon, T., 2015. Stochastic Numerical treatment for thin film flow of third grade fluid using unsupervised neural networks. Journal of the Taiwan Institute of Chemical Engineers, 48, pp.26-39.

[30]

Raja, M.A.Z., Farooq, U., Chaudhary, N.I. and Wazwaz, A.M., 2016. Stochastic numerical solver for nanofluidic problems containing multi-walled carbon nanotubes. Applied Soft Computing, 38, pp.561-586.

[31]

Raja, M.A.Z., Shah, F.H., Khan, A.A. and Khan, N.A., 2015. Design of bio-inspired computational intelligence technique for solving steady thin film flow of Johnson– Segalman fluid on vertical cylinder for drainage problems.Journal of the Taiwan Institute of Chemical Engineers. In press. 15

[32]

Raja, M.A.Z., Samar, R., Haroon, T. and Shah, S.M., 2015. Unsupervised neural network model optimized with evolutionary computations for solving variants of nonlinear MHD Jeffery-Hamel problem. Applied Mathematics and Mechanics, 36(12), pp.1611-1638.

[33]

Raja, M.A.Z. and Samar, R., 2014. Numerical treatment for nonlinear MHD Jeffery– Hamel problem using neural networks optimized with interior point algorithm. Neurocomputing, 124, pp.178-193.

[34]

Mall, S. and Chakraverty, S., 2014. Chebyshev Neural Network based model for solving Lane–Emden type equations. Applied Mathematics and Computation, 247, pp.100-114.

[35]

Raja, M.A.Z., 2014. Solution of the one-dimensional Bratu equation arising in the fuel ignition model using ANN optimised with PSO and SQP. Connection Science, 26(3), pp.195-214.

[36]

Raja, M.A.Z., 2014. Numerical treatment for solving one-dimensional Bratu problem using neural networks. Neural Computing and Applications, 24(3-4), pp.549-561.

[37]

Raja, M.A.Z., Manzar, M.A. and Samar, R., 2015. An Efficient computational intelligence approach for solving fractional order Riccati equations using ANN and SQP. Applied Mathematical Modelling, 39(10), pp.3075-3093.

[38]

Raja, M.A.Z., Khan, J.A. and Qureshi, I.M., 2011. Solution of fractional order system of Bagley-Torvik equation using evolutionary computational intelligence. Mathematical Problems in Engineering, 2011.

[39]

Raja, M.A.Z., 2014. Stochastic numerical treatment for solving Troesch’s problem. Information Sciences, 279, pp.860-873.

[40]

Raja, M.A.Z., 2014. Unsupervised neural networks for solving Troesch's problem. Chinese Physics B, 23(1), p.018903.

[41]

Yadav, N., Yadav, A., Kumar, M. and Kim, J.H., 2015. An efficient algorithm based on artificial neural networks and particle swarm optimization for solution of nonlinear Troesch’s problem. Neural Computing and Applications, pp.1-8.

[42]

Xu, L. and Liu, S., 2013. Study of short-term water quality prediction model based on wavelet neural network. Mathematical and Computer Modelling,58(3), pp.807-813.

[43]

Shoaib, M., Shamseldin, A.Y. and Melville, B.W., 2014. Comparative study of different wavelet based neural network models for rainfall–runoff modeling.Journal of Hydrology, 515, pp.47-58.

[44]

Chitsaz, H., Amjady, N. and Zareipour, H., 2015. Wind power forecast using wavelet neural network trained by improved Clonal selection algorithm.Energy Conversion and Management, 89, pp.588-598.

[45]

Zhao, Y., Du, X., Xia, G. and Wu, L., 2015. A novel algorithm for wavelet neural networks with application to enhanced PID controller design.Neurocomputing, 158, pp.257-267.

[46]

Klein, C.E., Bittencourt, M. and dos Santos Coelho, L., 2015. Wavenet using artificial bee colony applied to modeling of truck engine powertrain components. Engineering Applications of Artificial Intelligence, 41, pp.41-55.

[47]

Sharma, V., Yang, D., Walsh, W. and Reindl, T., 2016. Short term solar irradiance forecasting using a mixed wavelet neural network. Renewable Energy, 90, pp.481492. 16

[48]

Goldberg, D.E. and Holland, J.H., 1988. Genetic algorithms and machine learning. Machine learning, 3(2), pp.95-99.

[49]

Fathizadeh, N., Mohebbi, A., Soltaninejad, S. and Iranmanesh, M., 2015. Design and simulation of high pressure cyclones for a gas city gate station using semi-empirical models, genetic algorithm and computational fluid dynamics. Journal of Natural Gas Science and Engineering, 26, pp.313-329.

[50]

Abo-Hammour, Z., Abu Arqub, O., Momani, S., & Shawagfeh, N. (2014). optimization solution of Troesch’s and Bratu’s problems of ordinary type using novel continuous genetic algorithm. Discrete Dynamics in Nature and Society, 2014.

[51]

Kaydani, H., Mohebbi, A., Hajizadeh, A. and Dakhelpour, J., 2014. Developing a Formula Based on a Hybrid Neural Genetic Algorithm for the Prediction of Minimum Miscibility Pressure. Energy Sources, Part A: Recovery, Utilization, and Environmental Effects, 36(6), pp.679-688.

[52]

Mohebbi, A. and Kaydani, H., 2015. Permeability Estimation in Petroleum Reservoir by Meta-heuristics: An Overview. In Artificial Intelligent Approaches in Petroleum Geosciences (pp. 269-285). Springer International Publishing.

[53]

Raja, M.A.Z., Sabir, Z., Mehmood, N., Al-Aidarous, E.S. and Khan, J.A., 2015. Design of stochastic solvers based on genetic algorithms for solving nonlinear equations. Neural Computing and Applications, 26(1), pp.1-23.

[54]

Soleimani, H. and Kannan, G., 2015. A hybrid particle swarm optimization and genetic algorithm for closed-loop supply chain network design in large-scale networks. Applied Mathematical Modelling, 39(14), pp.3990-4012.

[55]

Gill, P.E. and Wong, E., 2012. Sequential quadratic programming methods. In Mixed integer nonlinear programming (pp. 147-224). Springer New York.

[56]

Savsani, P. and Savsani, V., 2015. Passing Vehicle Search (PVS): A novel metaheuristic algorithm. Applied Mathematical Modelling. In-press, DOI: 10.1016/j.apm.2015.10.040.

[57]

Zhu, X. and Pu, D., 2012. Sequential quadratic programming with a flexible step acceptance strategy. Applied Mathematical Modelling, 36(9), pp.3992-4002.

[58]

Morshed, M.J. and Asgharpour, A., 2014. Hybrid imperialist competitive-sequential quadratic programming (HIC-SQP) algorithm for solving economic load dispatch with incorporating stochastic wind power: A comparative study on heuristic optimization techniques. Energy Conversion and Management, 84, pp.30-40.

[59]

Su, K. and Cai, H.P., 2009. A modified SQP-filter method for nonlinear complementarity problem. Applied Mathematical Modelling, 33(6), pp.2890-2896.

[60]

Wahl, P.E. and Løvseth, S.W., 2015. Formulating the optimization problem when using sequential quadratic programming applied to a simple LNG process. Computers & Chemical Engineering, 82, pp.1-12.

17

Optimization Nonlinear Troesch’s BVP Arising in Plasma Physics problem

Optimization Initialization of GAs Random bounds and assignment through ‘gaoptimset’ routine

Fitness Evaluation

Porous media

Initialization (SQP): Start point, Bounds, and assignment of ‘Optimset’ parameters

Fitness Evaluation

Problem Termination Criterion Achieved?

Design of Morlet Wavelets ANN model for the solution and its derivative

No

Reproduction Invoke routines for Crossover, Mutation, Selection, and Elitism

No Iterative Update: Weights are updated as per step increment in SQP

Termination Criterion Achieved?

Yes

Yes

Best Individual of SQP algorithm

Best Individual of GAs Work Flow diagram of SQP

Formulation of fitness function mean square sense Modelling

Storage Store final weights, fitness, function counts for the runs of algorithm

Approximate Solution Through weights of ANNs and compare the results from Adams method

Performance Indices Analysis of results through Fitness, MAE, RMSE

Proposed Results

Figure 1: Schematic workflow diagram of proposed design scheme for Troesch’s Problem

18

Input

Hidden Layer

Output µ f (wx   ).

 1xm

f

 

x  sinh( )

w

+

µ

x

1xm

1

(0,1)

0

wx  

x

f "( wx   ).

 1xm f ( x)  cos(1.75 x )e

x

2

2

d2 f dx 2

+

d 2 yˆ   sinh(  yˆ )  0 dx 2

1

ŷ(x)

d 2 yˆ dx 2

 

Fig. 2 Architecture of the nonlinear Troesch’s problem using differential equation MWANN.

19

Values of weights

Values of weights

Values of weights

(c) Scenario 1: µ = 2.3

Values of weights

Number of neurons

(b) Scenario 1: µ = 1.5

Values of weights

Number of neurons

(a) Scenario 1 for µ = 0.1

Values of weights

Number of neurons

(f) Scenario 1: µ = 2.3

Values of weights

Number of neurons

(e) Scenario 1: µ = 1.5

Values of weights

Number of neurons

(d) Scenario 1 for µ = 0.1

Values of weights

Number of neurons

(i) Scenario 2: µ = 2

Values of weights

Number of neurons

(h) Scenario 2: µ = 1

Values of weights

Number of neurons

(g) Scenario 2 for µ = 0.5

Values of weights

Number of neurons

Number of neurons

Number of neurons

Number of neurons

(j) Scenario 2 for µ = 0.5

(k) Scenario 2: µ = 1

(l) Scenario 2: µ = 2

Figure 3: Optimized weights of MWANN-GA-SQP for solving the Troesch’s problem.

20

Solution “y(x)”

Solution “y(x)”

(b) Proposed solutions for MWANN-GA-SQP for Scenario 2 and h = 0.1

Absolute Error

Input “x”

(a) Proposed solutions for MWANN-GA-SQP for Scenario 1 and h = 0.1

Absolute Error

Input “x”

Input “t” (d) Proposed solutions for MWANN-GA-SQP for Scenario 2 and h = 0.1

Absolute Error

Absolute Error

Input “t” (c) Proposed solutions for MWANN-GA-SQP for Scenario 1 and h = 0.1

Input “t” (e) Proposed solutions for MWANN-GA-SQP for Scenario 1 and h = 0.01

Input “t” (f) Proposed solutions for MWANN-GA-SQP for Scenario 2 and h = 0.01

Figure 4: Comparison of the proposed approximate solutions with the reference numerical results of Adam's Methods for each case (both scenarios) of the Troesch’s problem.

21

Fitness

Fitness

(b) MWANN-GA-SQP for Scenario 2 with step size of h = 0.1

Fitness

Number of Independent runs

(a) MWANN-GA-SQP for Scenario 1 with step size of h = 0.1

Fitness

Number of Independent runs

Number of Independent runs

Number of Independent runs

(c) MWANN-GA-SQP for Scenario 1 with step size of h = 0.01

(d) MWANN-GA-SQP for Scenario 1 with step size of h = 0.01

Figure 5: Plot of fitness against number of independent runs of the proposed algorithm for each case of both scenarios of the Troesch’s problem.

22

MAE

MAE

(b) MWANN-GA-SQP for Scenario 2 with step size of h = 0.1

MAE

Number of Independent runs

(a) MWANN-GA-SQP for Scenario 1 with step size of h = 0.1

MAE

Number of Independent runs

Number of Independent runs

Number of Independent runs

(c) MWANN-GA-SQP for Scenario 1 with step size of h = 0.01

(d) MWANN-GA-SQP for Scenario 1 with step size of h = 0.01

Figure 6: MAE values against number of independent runs of the proposed algorithm for each case of both scenarios of Troesch’s problem.

23

RMSE

RMSE

(b) MWANN-GA-SQP for Scenario 2 with step size of h = 0.1

RMSE

Number of Independent runs

(a) MWANN-GA-SQP for Scenario 1 with step size of h = 0.1

RMSE

Number of Independent runs

Number of Independent runs

Number of Independent runs

(c) MWANN-GA-SQP for Scenario 1 with step size of h = 0.01

(d) MWANN-GA-SQP for Scenario 1 with step size of h = 0.01

Figure 7: RMSE values against number of independent runs of the proposed algorithm for each case of both scenarios of Troesch’s problem.

24

Fitness

Fitness Number of Independent runs

Number of Independent runs

(b) Case 2 of Scenario 1 for h = 0.1

Fitness

Fitness

(a) Case 1 of Scenario 1 for h = 0.1

Number of Independent runs

Number of Independent runs

(d) Case 1 of Scenario 2 for h = 0.1

Fitness

Fitness

(c) Case 3 of Scenario 1 for h = 0.1

Number of Independent runs

Number of Independent runs

(e) Case 2 of Scenario 2 for h = 0.1

(f) Case 3 of Scenario 2 for h = 0.1

Figure 8: Comparison of neural network models optimized with the SQP algorithm on the basis of fitness values for each case of both scenarios of Troesch’s problem.

25

MAE

MAE Number of Independent runs

Number of Independent runs

(b) Case 2 of Scenario 1 for h = 0.1

MAE

MAE

(a) Case 1 of Scenario 1 for h = 0.1

Number of Independent runs

Number of Independent runs

(d) Case 1 of Scenario 2 for h = 0.1

MAE

MAE

(c) Case 3 of Scenario 1 for h = 0.1

Number of Independent runs

Number of Independent runs

(e) Case 2 of Scenario 2 for h = 0.1

(f) Case 3 of Scenario 2 for h = 0.1

Figure 9: Comparison of neural network models optimized with the SQP algorithm on the basis of mean absolute errors for each case of both scenarios of Troesch’s problem.

26

Values

Values

(b) Mean Fitness

Values

Values

(a) Minimum Fitness

(d) Mean MAE

Values

Values

(c) Minimum MAE

(e) Minimum RMSE

(f) Mean RMSE

Figure 10: Comparison of neural network models optimized with the SQP algorithm on the basis of different performance indices for each case of scenario 1 of Troesch’s problem

27

Values

Values

(b) Mean Fitness

Values

Values

(a) Minimum Fitness

(d) Mean MAE

Values

Values

(c) Minimum MAE

(e) Minimum RMSE

(f) Mean RMSE

Figure 11: Comparison of neural network models optimized with the SQP algorithm on the basis of different performance indices for each case of scenario 2 of Troesch’s problem

28

Table 1: Parameters settings used for genetic algorithm (GA) and Sequential Quadratic Programming (SQP) algorithm. Algos Parameters

Settings

Parameters

Settings

GA

Population Creation ‘PopulationSize’ Selection function Initial Population range Crossover function Mutation function ‘EliteCount’ ‘FitnessLimit’

@gacreationuniform 360 Stochastic Uniform [-1.5, 1.5] Heuristics Adaptive feasible 2 10-20

Individual Size ‘Generation’ Function Tolerance ‘TolFun’ ‘StallGenLimit’ Lower bounds Upper bounds Constraint Tolerance ‘TolCon’ Other

30 1000 10-20 250 [-20]1x30 [20]1x30 10-20 Defaults

SQP

Start points Algorithm Iterations Function counts Finite Difference

Final weight of GAs ‘SQP’ 1000 200000 ‘Central’

bounds X-Tolerance ‘TolX’ ‘TolCon’ ‘TolFun’ Other

[-20,20] 10-18 10-25 10-25 defaults

29

Table 2: Absolute error for each case of scenario 1 of Troesch’s Problem. Input “x”

h = 0.1

h = 0.01

μ = 0.1

μ = 1.5

μ = 2.3

μ = 0.1

μ = 1.5

μ = 2.3

0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45 0.50 0.55 0.60 0.65 0.70 0.75 0.80 0.85 0.90 0.95 1.00

2.984E-10 4.948E-09 3.095E-09 3.702E-09 8.402E-09 1.490E-08 2.065E-08 2.377E-08 2.322E-08 1.907E-08 1.220E-08 4.156E-09 3.392E-09 8.876E-09 1.142E-08 1.087E-08 8.097E-09 4.652E-09 2.531E-09 3.164E-09 6.311E-09

1.070E-08 8.733E-08 3.883E-08 1.434E-08 1.062E-09 4.172E-08 2.435E-08 6.305E-08 1.611E-07 1.934E-07 1.401E-07 6.802E-08 6.959E-08 1.772E-07 3.140E-07 3.551E-07 2.739E-07 2.563E-07 4.788E-07 6.253E-07 1.057E-07

1.080E-05 1.135E-05 9.162E-06 5.672E-06 3.085E-06 2.056E-06 1.663E-06 5.743E-07 1.705E-06 4.467E-06 6.522E-06 7.471E-06 8.318E-06 1.058E-05 1.434E-05 1.740E-05 1.810E-05 2.021E-05 3.068E-05 3.808E-05 8.270E-07

6.159E-09 5.178E-09 8.506E-09 1.165E-08 1.413E-08 1.384E-08 1.068E-08 6.246E-09 2.441E-09 6.346E-10 9.113E-10 2.381E-09 3.490E-09 3.033E-09 5.472E-10 3.213E-09 6.878E-09 9.075E-09 9.671E-09 9.894E-09 1.095E-08

1.183E-08 4.053E-08 7.147E-08 1.341E-08 1.466E-07 2.198E-07 1.567E-07 2.365E-09 1.316E-07 1.428E-07 1.464E-08 1.597E-07 2.459E-07 1.678E-07 1.691E-08 1.455E-07 8.999E-08 6.275E-08 7.960E-08 7.014E-08 4.455E-08

8.215E-07 7.249E-07 1.773E-07 4.659E-07 1.551E-06 2.110E-06 1.365E-06 4.509E-08 4.314E-07 4.041E-07 1.458E-06 1.280E-06 1.107E-07 1.096E-06 5.895E-07 1.828E-07 6.328E-07 2.058E-06 1.843E-06 2.104E-06 2.998E-06

30

Table 3: Absolute error for each case of scenario 2 of Troesch’s Problem. Input “x”

h = 0.1

h = 0.01

μ = 0.5

μ=1

μ=2

μ = 0.5

μ=1

μ=2

0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45 0.50 0.55 0.60 0.65 0.70 0.75 0.80 0.85 0.90 0.95 1.00

2.390E-08 1.631E-08 3.981E-09 2.022E-08 4.152E-08 5.229E-08 5.351E-08 5.140E-08 5.403E-08 7.198E-08 8.778E-08 9.887E-08 1.159E-07 1.266E-07 1.303E-07 1.327E-07 1.377E-07 1.523E-07 1.703E-07 1.840E-07 1.863E-07

5.254E-10 7.244E-08 4.677E-08 2.248E-08 6.527E-08 5.235E-08 1.846E-08 8.706E-08 1.133E-07 7.702E-08 8.781E-09 1.038E-07 1.587E-07 1.415E-07 6.450E-08 2.342E-08 5.685E-08 4.070E-09 1.223E-07 1.666E-07 4.064E-08

2.263E-06 3.672E-06 2.896E-06 1.027E-06 1.854E-07 1.041E-07 6.777E-07 1.103E-06 5.684E-07 6.759E-07 1.802E-06 2.128E-06 1.702E-06 1.221E-06 1.300E-06 1.801E-06 2.077E-06 2.231E-06 3.359E-06 4.094E-06 2.318E-06

1.346E-08 1.700E-08 3.473E-08 3.980E-08 3.014E-08 1.657E-08 1.342E-08 2.951E-08 6.469E-08 1.147E-07 1.472E-07 1.510E-07 1.386E-07 1.095E-07 8.081E-08 7.390E-08 9.760E-08 1.472E-07 1.910E-07 1.990E-07 1.821E-07

5.515E-09 1.564E-09 9.267E-09 1.145E-08 4.228E-08 5.257E-08 4.609E-08 1.391E-08 1.978E-08 3.553E-08 2.769E-08 5.901E-09 1.385E-08 2.015E-08 7.882E-09 1.228E-08 2.648E-08 3.280E-08 4.096E-08 5.041E-08 5.940E-08

1.378E-06 1.123E-06 1.136E-06 1.239E-06 1.062E-06 6.190E-07 2.622E-07 2.573E-07 4.931E-07 5.785E-07 2.532E-07 3.132E-07 6.692E-07 5.874E-07 3.986E-07 6.547E-07 1.327E-06 1.711E-06 1.630E-06 2.064E-06 2.534E-06

31

Table 4: Statistical analysis for each case of scenario 1 of Troesch’s Problem. Step size

h = 0.1

h = 0.01

Input “t”

μ = 0.1

μ = 1.5

μ = 2.3

Mean

STD

Mean

STD

Mean

STD

0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45 0.50 0.55 0.60 0.65 0.70 0.75 0.80 0.85 0.90 0.95 1.00 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45 0.50 0.55 0.60 0.65 0.70 0.75 0.80 0.85 0.90 0.95 1.00

3.369E-07 4.065E-07 4.619E-07 4.701E-07 4.712E-07 5.176E-07 5.458E-07 5.268E-07 4.728E-07 3.785E-07 2.931E-07 3.538E-07 4.607E-07 5.512E-07 5.931E-07 5.910E-07 5.817E-07 5.935E-07 5.845E-07 5.165E-07 4.373E-07 1.822E-07 2.153E-07 2.827E-07 3.195E-07 3.306E-07 3.751E-07 4.123E-07 4.141E-07 3.649E-07 2.779E-07 2.409E-07 3.288E-07 4.132E-07 4.623E-07 4.706E-07 4.565E-07 4.693E-07 4.864E-07 4.548E-07 3.830E-07 3.405E-07

1.241E-06 1.211E-06 1.225E-06 1.271E-06 1.302E-06 1.275E-06 1.196E-06 1.083E-06 9.626E-07 9.036E-07 9.521E-07 1.079E-06 1.260E-06 1.448E-06 1.609E-06 1.720E-06 1.766E-06 1.746E-06 1.694E-06 1.656E-06 1.657E-06 4.217E-07 4.062E-07 4.251E-07 5.084E-07 6.076E-07 6.549E-07 6.455E-07 5.830E-07 5.017E-07 4.682E-07 5.237E-07 6.415E-07 7.985E-07 9.424E-07 1.044E-06 1.089E-06 1.062E-06 9.841E-07 8.975E-07 8.405E-07 8.432E-07

1.923E-06 3.042E-06 2.666E-06 1.349E-06 1.544E-06 1.958E-06 1.443E-06 1.090E-06 1.914E-06 1.918E-06 1.112E-06 1.851E-06 3.354E-06 4.031E-06 3.460E-06 2.201E-06 1.556E-06 2.353E-06 4.616E-06 5.840E-06 3.258E-06 2.814E-06 3.162E-06 3.012E-06 2.207E-06 2.119E-06 2.245E-06 1.859E-06 1.339E-06 1.855E-06 1.926E-06 1.409E-06 1.524E-06 2.626E-06 3.286E-06 3.038E-06 2.081E-06 1.355E-06 1.696E-06 3.092E-06 4.245E-06 3.877E-06

1.886E-06 2.400E-06 2.180E-06 1.535E-06 1.598E-06 1.686E-06 1.244E-06 1.103E-06 1.285E-06 1.378E-06 1.319E-06 1.953E-06 2.870E-06 3.055E-06 2.453E-06 1.813E-06 1.755E-06 2.310E-06 3.636E-06 4.375E-06 3.306E-06 4.445E-06 4.232E-06 3.989E-06 3.547E-06 3.141E-06 2.896E-06 2.513E-06 2.260E-06 2.050E-06 2.021E-06 1.889E-06 2.169E-06 2.826E-06 3.120E-06 2.734E-06 2.252E-06 2.226E-06 2.639E-06 3.603E-06 4.293E-06 4.349E-06

2.501E-05 2.677E-05 2.164E-05 1.644E-05 1.508E-05 1.362E-05 1.282E-05 1.378E-05 1.871E-05 2.396E-05 2.617E-05 2.544E-05 2.631E-05 3.299E-05 4.369E-05 5.079E-05 5.093E-05 5.580E-05 8.133E-05 1.019E-04 3.912E-05 3.486E-05 3.339E-05 2.923E-05 2.288E-05 1.917E-05 1.851E-05 1.882E-05 1.832E-05 1.701E-05 1.755E-05 2.079E-05 2.468E-05 2.692E-05 2.793E-05 3.136E-05 3.949E-05 4.913E-05 5.507E-05 6.083E-05 7.564E-05 9.134E-05

4.876E-05 4.457E-05 4.075E-05 3.598E-05 3.075E-05 2.702E-05 2.520E-05 2.451E-05 2.351E-05 2.218E-05 1.974E-05 1.754E-05 1.783E-05 2.145E-05 2.635E-05 2.882E-05 2.881E-05 3.193E-05 4.272E-05 5.327E-05 4.768E-05 5.074E-05 4.542E-05 3.983E-05 3.313E-05 2.673E-05 2.259E-05 2.034E-05 1.812E-05 1.588E-05 1.563E-05 1.824E-05 2.051E-05 2.115E-05 2.261E-05 2.781E-05 3.594E-05 4.278E-05 4.739E-05 5.476E-05 6.829E-05 8.229E-05

32

Table 5: Statistical analysis for each case of scenario 2 of Troesch’s Problem. Step size

h = 0.1

h = 0.01

Input “t”

μ = 0.5

μ=1

μ=2

Mean

STD

Mean

STD

Mean

STD

0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45 0.50 0.55 0.60 0.65 0.70 0.75 0.80 0.85 0.90 0.95 1.00 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45 0.50 0.55 0.60 0.65 0.70 0.75 0.80 0.85 0.90 0.95 1.00

4.681E-07 5.744E-07 6.132E-07 5.389E-07 4.303E-07 4.952E-07 5.948E-07 6.473E-07 6.287E-07 5.451E-07 4.620E-07 4.999E-07 5.935E-07 6.057E-07 5.308E-07 4.118E-07 4.553E-07 6.319E-07 7.421E-07 6.998E-07 5.592E-07 8.410E-07 8.590E-07 9.397E-07 9.472E-07 9.112E-07 9.705E-07 1.073E-06 1.126E-06 1.081E-06 9.482E-07 8.200E-07 8.872E-07 1.023E-06 1.095E-06 1.081E-06 1.050E-06 1.137E-06 1.226E-06 1.214E-06 1.099E-06 1.011E-06

1.061E-06 1.016E-06 9.883E-07 9.460E-07 9.053E-07 8.644E-07 8.782E-07 8.919E-07 8.573E-07 7.985E-07 7.544E-07 7.292E-07 7.141E-07 7.091E-07 6.867E-07 6.890E-07 7.233E-07 7.856E-07 8.319E-07 8.300E-07 8.204E-07 4.307E-06 4.181E-06 4.027E-06 3.940E-06 3.920E-06 3.900E-06 3.839E-06 3.733E-06 3.621E-06 3.572E-06 3.635E-06 3.797E-06 4.032E-06 4.272E-06 4.436E-06 4.466E-06 4.339E-06 4.129E-06 3.945E-06 3.870E-06 3.909E-06

2.564E-06 3.080E-06 3.228E-06 2.874E-06 2.294E-06 1.688E-06 2.066E-06 2.692E-06 2.886E-06 2.586E-06 1.934E-06 1.585E-06 1.922E-06 2.067E-06 1.864E-06 1.592E-06 1.940E-06 2.705E-06 3.466E-06 3.488E-06 2.762E-06 2.179E-06 2.296E-06 2.410E-06 2.308E-06 2.071E-06 1.749E-06 1.617E-06 1.854E-06 2.022E-06 1.911E-06 1.683E-06 1.775E-06 2.012E-06 2.095E-06 2.014E-06 1.983E-06 2.258E-06 2.641E-06 2.941E-06 2.975E-06 2.872E-06

5.306E-06 5.269E-06 5.321E-06 4.959E-06 4.275E-06 4.053E-06 4.313E-06 4.681E-06 4.751E-06 4.402E-06 3.942E-06 3.743E-06 3.895E-06 4.182E-06 4.162E-06 3.923E-06 4.092E-06 4.976E-06 5.724E-06 5.889E-06 5.816E-06 5.198E-06 5.061E-06 5.030E-06 4.858E-06 4.514E-06 4.333E-06 4.466E-06 4.693E-06 4.827E-06 4.817E-06 4.727E-06 4.638E-06 4.691E-06 4.887E-06 5.159E-06 5.569E-06 6.201E-06 6.970E-06 7.534E-06 7.759E-06 8.009E-06

7.707E-06 8.972E-06 7.762E-06 6.223E-06 5.395E-06 4.956E-06 5.591E-06 6.852E-06 8.257E-06 8.677E-06 7.917E-06 7.110E-06 7.850E-06 1.056E-05 1.366E-05 1.468E-05 1.348E-05 1.429E-05 2.127E-05 2.643E-05 1.015E-05 9.845E-06 1.008E-05 8.837E-06 7.029E-06 5.994E-06 5.074E-06 5.543E-06 5.973E-06 6.414E-06 7.924E-06 8.624E-06 7.787E-06 6.976E-06 8.037E-06 1.114E-05 1.461E-05 1.580E-05 1.511E-05 1.669E-05 2.195E-05 2.461E-05

1.088E-05 1.039E-05 9.797E-06 8.381E-06 7.091E-06 6.730E-06 7.283E-06 8.093E-06 7.791E-06 6.602E-06 5.667E-06 5.784E-06 7.052E-06 9.080E-06 1.057E-05 1.067E-05 1.028E-05 1.138E-05 1.524E-05 1.822E-05 1.285E-05 1.381E-05 1.333E-05 1.299E-05 1.123E-05 8.680E-06 7.275E-06 7.589E-06 9.149E-06 1.018E-05 9.552E-06 8.081E-06 7.544E-06 9.375E-06 1.278E-05 1.579E-05 1.699E-05 1.697E-05 1.805E-05 2.196E-05 2.695E-05 3.023E-05

33

Table 6: Convergence analysis through different performance indices. Step size h

Scenario μ number 1

0.1 2

1 0.01 2

0.1 1.5 2.3 0.5 1.0 2.0 0.1 1.5 2.3 0.5 1.0 2.0

Fitness ≤

MAE ≤

RMSE ≤

10-05

10-07

10-08

10-03

10-05

10-08

10-03

10-05

10-08

100 100 100 100 100 100 100 100 100 100 100 100

100 82 4 99 100 18 100 81 26 100 88 73

100 27 0 99 78 2 100 33 5 100 69 3

100 100 79 100 100 97 100 100 85 100 100 99

100 90 7 97 90 38 97 92 0 99 78 18

83 23 0 70 69 0 85 11 0 68 59 0

100 100 67 100 100 97 100 100 77 100 100 99

100 88 5 97 87 25 97 86 0 99 75 4

81 21 0 63 68 0 85 6 0 59 52 0

34

Table 7: Global performance indicators for each case of Troesch’s Problem. Step size h

Scenario μ number 1

0.1

2

1 0.01

2

0.1 1.5 2.3 0.5 1.0 2.0 0.1 1.5 2.3 0.5 1.0 2.0

GFit

GMAE

GRMSE

Values

STD

Values

STD

Values

STD

1.443E-09 2.780E-07 3.067E-06 3.484E-09 1.302E-07 6.191E-07 1.506E-09 2.935E-07 6.264E-06 1.114E-08 6.473E-08 1.924E-06

2.910E-09 2.868E-07 4.178E-06 3.486E-09 2.113E-07 8.504E-07 2.749E-09 3.048E-07 5.948E-06 7.892E-08 1.212E-07 2.657E-06

4.831E-07 2.499E-06 3.439E-05 5.585E-07 2.442E-06 1.037E-05 3.658E-07 2.418E-06 3.490E-05 1.016E-06 2.175E-06 1.067E-05

1.243E-06 1.767E-06 2.623E-05 7.371E-07 4.026E-06 8.153E-06 6.397E-07 2.611E-06 2.899E-05 3.915E-06 5.128E-06 1.236E-05

5.430E-07 2.966E-06 4.368E-05 6.363E-07 2.815E-06 1.241E-05 4.197E-07 2.793E-06 4.266E-05 1.109E-06 2.411E-06 1.270E-05

1.354E-06 2.058E-06 2.972E-05 7.860E-07 4.522E-06 9.009E-06 7.197E-07 2.907E-06 3.640E-05 3.979E-06 5.455E-06 1.443E-05

35

Table 8: Computational Complexity of the proposed MWANN-GA-SQP scheme for each case of Troesch’s Problem. Step size h

Scenario μ number 1

0.1

2

1 0.01

2

0.1 1.5 2.3 0.5 1.0 2.0 0.1 1.5 2.3 0.5 1.0 2.0

Time

Iterations

Function Counts

Mean

STD

Mean

STD

Mean

STD

16.60 16.61 17.09 15.86 16.05 16.83 134.88 126.37 128.52 125.70 130.65 128.96

1.69 2.60 2.74 2.18 2.50 2.79 17.72 18.85 22.06 20.21 24.33 21.02

1972.57 1974.57 2000.00 1990.86 1980.19 1973.84 1997.25 1987.69 1972.70 1972.37 1987.94 1979.58

138.33 106.22 0.00 55.29 103.08 133.78 27.50 80.28 123.56 144.58 84.99 98.97

434586.75 457844.84 477470.46 434040.84 444225.21 468520.69 438277.96 455921.48 463980.32 430264.80 443289.16 463264.91

23126.94 27153.50 12454.45 19487.14 25998.80 24409.33 21665.68 24378.65 23905.96 21042.34 24135.47 23087.02

36

Appendix: Set of weighs up to 14 decimal places of accuracy are given here to reproduce results without round off error problems

Table A-1. Set of optimized weights of MW-ANNs trained through the GA-SQP algorithm for both scenarios of Troesch’s problem for h = 0.01 Scenario 1 h = 0.01 Scenario 2 h = 0.01

µ=0.1

µ=1.5

µ=2.3

µ=0.5

µ=1

µ=2

0.79253536476 8229

0.6632492915 56960

6.6156615137 6284

1.26645249167 083

3.786349480517 36

2.9875426319 7322

α2

0.58845036268 6836

1.8411393633 3902

3.5589955086 3535

0.54866476045 1224

1.960253872366 19

1.1847076740 7662

α3

0.22683143173 2120

0.5687704505 10270

1.6595543183 5041

0.10828438001 6611

0.001119719407 01343

2.0425411045 7975

α4

0.38444985063 5053

2.1368310542 8165

0.6447714103 37136

1.30506788548 431

0.230079277010 012

3.8613509951 8090

0.47182514282 3763

1.7219692147 9920

3.6707011221 0082

1.60824292798 657

1.621882862486 14

0.1345076873 26033

1.31226108971 967

2.4069261876 1234

2.2259527370 4425

1.49411649843 947

2.059107893091 07

1.6671589186 8016

0.03702740007 18122

1.8514883762 7103

1.5982439731 4342

1.91427338493 096

1.138656274535 14

2.6854704064 2823

1.19737167086 006

3.2489487509 5891

1.2151933828 3476

1.27557987438 735

0.140645329484 766

1.4253806493 5783

α9

0.14806956866 7742

1.1605985746 7228

1.4589283345 1563

0.18472349579 5037

2.409763339240 86

1.8707186587 9182

α1

1.25292779511 648

0.6131001547 22528

2.3964332190 7129

0.34668697935 8204

3.840662636057 99

6.8371135931 0993

0.50218777801 2517

1.3985401126 8758

3.3490891322 7372

0.67270923721 9525

0.241575404108 187

1.4461782071 6576

α1

α5

α6

α7

α8

0

w 1

37

w 2

w 3

w 4

w 5

w 6

w 7

w 8

w 9

w 10

β1

β2

β3

β4

β5

0.29378633036 8932

1.2612349247 5002

2.6593139635 8065

0.87532222115 2743

0.320450825978 907

0.4221371574 98307

0.37648967391 1374

0.4866586304 59615

0.8064370100 08524

1.00695318062 302

2.107020395086 80

1.1912563470 1933

0.56544890365 2805

0.5271536325 94950

3.2427648660 5602

0.28438087113 8225

0.876255307739 824

0.4538642084 45204

0.67200827044 0501

0.3220279999 68125

1.6508263406 1538

0.28677811565 5660

0.216953424434 062

2.4710328211 6232

0.18830552526 7424

0.7503404145 11505

0.2104159015 35424

0.13665764316 7918

0.799362691372 617

1.8754350109 2580

0.83800624518 3832

0.2693957714 31782

1.1947458262 7852

0.17393100922 8727

0.458551123973 507

0.1251737980 82636

0.32125238004 9769

2.1859656017 5842

0.4260080560 39339

0.56518723857 4301

1.242687496148 40

0.4018389093 40445

0.21107581969 1103

1.1250425743 2274

0.4721489795 54133

0.15440800021 4853

0.082887996570 0797

1.0755947078 0640

0.26416618738 2393

0.7891079967 69755

0.9185810220 82928

0.76009264214 8236

0.614988102078 141

3.1689738245 2188

0.63273970860 6807

2.5617440969 0227

6.0011509724 7191

0.65049224342 3474

0.654409542110 612

1.6072066499 6243

1.20875071468 886

3.9428182264 3820

4.2144777511 2137

0.91349598635 2955

1.305125965097 70

0.2122065914 46879

0.02412483221 31591

2.1320109784 0907

2.5211670520 5466

0.05975107394 36155

2.358302398766 21

0.7155270368 11235

0.99297178653 9207

1.5322996396 0645

4.3991732701 1409

0.13534174093 7658

0.412218369117 399

1.1744713926 2435

2.61730085863 682

0.9003142397 08204

2.8167907538 6962

1.02498555917 675

0.474745283713 822

2.9741811970 9734

38

β6

0.84219857900 9770

1.5258529334 6161

0.3416356138 37056

0.53800672166 3999

1.718790862606 68

2.6997981408 4622

0.49406130540 5607

0.7519182426 71905

0.8041425949 56787

0.53038096434 7729

0.324106246316 712

0.1589765421 92969

β8

0.86155212495 1632

5.6888737976 1342

3.0935637868 6143

0.87374217454 8123

1.960705868578 41

0.7213605316 45607

β9

1.46384011491 384

1.5798968969 8748

0.2691435307 76647

1.37841712540 477

1.829286338361 09

1.2907167670 7890

0.65862495750 9464

0.2461980659 23970

1.0101492881 5730

0.29383807146 2402

1.872138903185 04

6.7429717296 5823

β7

β1 0

Table A-2 Set of optimized weights of MW-ANNs trained through the GA-SQP algorithm for both scenarios of Troesch’s problem for h = 0.1 Scenario 1 h = 0.1 Scenario 2 h = 0.1

µ=0.1

µ=1.5

µ=2.3

µ=0.5

µ=1

µ=2

α1

0.21633056926 1420

0.5416096099 41722

1.1499870345 0331

0.10415799372 2057

1.89742572020 267

0.34144677542 1823

α2

1.20628000314 886

2.2394179615 0338

12.950351056 5654

0.75223840674 6961

0.76984051768 9791

8.32761333267 172

α3

1.07379047439 892

1.8688021162 0197

3.1931493976 8875

1.11734941933 955

0.70246287205 5100

1.06621783104 878

α4

0.79308385934 4078

5.3496576535 5001

3.0280863411 7571

0.33071283454 4801

1.68145065759 720

5.25840440837 067

α5

1.38821223980 330

0.8062510106 42917

1.8952043852 4526

0.93428986726 2472

1.20094887510 348

0.37856602061 5277

α6

1.74511471472 542

1.6349370040 8374

1.7770400697 6174

0.66821442631 4175

0.98018671945 1858

0.39251331475 2041

α7

1.59514059364 761

2.3543704882 9784

0.4294712623 94808

0.42751746875 7226

6.57661962406 339

4.91638510762 982

39

α8

1.31119420866 495

0.2961720821 27132

0.7615923698 60713

0.16030060309 8804

0.25086322435 1037

4.88684473943 794

α9

0.20176649775 6602

0.7437856663 25031

2.0650905530 6643

1.54900161114 465

0.52808491425 3489

3.09804328960 389

α1

1.67079741554 461

5.2808275644 8602

2.8813868338 9067

1.62584825240 610

1.98536871081 746

5.50325653576 007

0.33928814274 3815

0.9839632122 29332

0.5230428986 88566

0.01890354041 96863

0.66345021904 1742

1.24089529629 155

0.11305231858 7898

0.3107814340 62458

3.0978916464 6108

0.18114732763 2286

0.19225173277 1157

2.00791829013 429

0.21313819155 8175

0.1344779290 15473

1.3324747725 8250

0.14216111373 2367

0.76232472887 4944

0.04862667193 59771

0.02895986418 49868

0.2298674541 46136

0.4113240064 74776

0.81810177214 0586

0.18564620741 3583

2.46254345917 517

0.13626866135 2598

0.2534663515 13625

0.8422602776 65320

0.58120787811 4409

0.33572736531 8442

1.32475985054 441

0.15728384500 3707

0.5169071172 22763

0.4641439626 98238

0.28843244334 2379

0.16421113452 6907

1.73441829191 385

0.13198500050 3225

0.5259651237 43035

1.0713697023 7401

0.37747306456 8869

1.06725676231 166

1.96407381552 131

0.11972985223 0234

1.8804766369 8408

1.6824426197 6663

0.91490681565 2173

0.17613669727 7256

1.15320128712 649

0.36013282581 5756

1.9376061306 0363

0.7088261782 64471

0.31312500092 9095

0.45129751107 3582

0.66697951935 8898

0.15395105780 5725

1.1028931641 7830

2.0461449912 0536

0.16243620149 7146

0.29922938828 5712

0.22907936738 7752

0.27842245703 1564

0.5693248108 59325

0.5032755845 60483

1.32896741187 753

0.95975639795 8051

1.22851285913 393

0

w 1

w 2

w 3

w 4

w 5

w 6

w 7

w 8

w 9

w 10

β1

40

β2

0.87422150082 5589

0.3168888658 53338

6.5471738523 0252

0.84626894316 5604

0.22789483200 2856

4.05503217847 560

β3

0.53582991109 2254

0.6152213653 24499

2.3032289437 3803

0.77577515681 9926

0.00550426121 435235

0.90746350331 5287

β4

1.03029686772 013

1.1525292749 3847

0.8833289370 92219

0.53802307956 9464

0.66381876071 4619

5.18703492690 646

β5

0.79001605743 1554

0.7153464917 02169

2.5433552249 0674

0.36915482947 4946

1.05190721466 550

3.15554359259 211

β6

1.18050958192 058

0.6185763669 33279

1.0269352270 5027

1.20407909159 555

0.49277948351 7104

2.18393452688 477

β7

0.99242744524 7332

0.4357531692 70015

0.5164282809 29061

0.57191134259 6748

4.36608043207 952

4.10357563047 593

β8

0.74524914370 9863

2.8316167656 8086

2.1762534219 9325

1.42381948425 143

0.69732525272 3103

2.32917919279 543

β9

0.79401184290 9965

3.6672031562 3122

1.3725496522 7026

1.22360885858 234

0.11748752388 1932

1.66056951601 982

β1

0.76349869415 9937

2.7232859789 8630

3.9780355543 6338

0.84947796784 8992

0.77746607753 2415

0.75516591297 4250

0

41