Measurement of Fitness Function efficiency using Data Envelopment Analysis

ESWA 9363

No. of Pages 14, Model 5G

17 June 2014 Expert Systems with Applications xxx (2014) xxx–xxx 1

Contents lists available at ScienceDirect

Expert Systems with Applications journal homepage: www.elsevier.com/locate/eswa 5 6

Measurement of Fitness Function efficiency using Data Envelopment Analysis

3 4 7

Q1

David A. Silva a, Gabriela I. Alves a, Paulo S.G. de Mattos Neto b, Tiago A.E. Ferreira a,⇑ a

8 9

b

10 11

Department of Statistics and Informatics, Federal Rural University of Pernambuco, Recife, Pernambuco, Brazil Department of Computing, University of Pernambuco, Garanhuns, Pernambuco, Brazil

a r t i c l e

1 2 3 5 14 15 16 17 18 19 20 21 22 23 24

i n f o

Keywords: Efficiency measure Fitness Function Data Envelopment Analysis Time series forecasting Artificial Neural Networks Evolutionary Strategy Hybrid Intelligent Systems Optimization

a b s t r a c t Over the last years, Evolutionary Algorithms (EAs) have been proposed aiming to find the best configuration of the Artificial Neural Networks (ANN) parameters. Among several parameters of an EA that can influence the quality of the found solution, the choice of the Fitness Function is the most important for its effectiveness and efficiency, given that different Fitness Functions have distinct fitness landscapes. In other words, the Fitness Function guides the evolutionary process of the candidate solutions according with a given criterion of the performance. However, there is not an universal criterion to identify the best performance measure. Thus, what is the Fitness Function more efficient among a set of several possible options? This paper presents a methodology based on Data Envelopment Analysis (DEA) to find the more efficient Fitness Function among candidates. The DEA is used to determine the best combination of statistical measures to build the more efficient Fitness Function for a EA. The case study employed here consists of a hybrid system composed by Evolutionary Strategy and ANN applied to solve the time series forecasting problem. The data analyzed are composed by financial, agribusiness and natural phenomena. The results show that establishment of the Fitness Function is a crucial point in the EA design, being a key factor to obtain the best solution for a limited number of EA’s iteration. Ó 2014 Published by Elsevier Ltd.

26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42

43

1. Introduction

44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59

In literature, Artificial Neural Networks (ANN) has been widely used in different tasks, such as: function approximation (Petkovic´, Q3 C´ojbašic´, & Lukic´, 2013), classification (Fernandes, Cavalcanti, & Ren, 2013), pattern recognition (Ma, Chan, Saha, & Ekanayake, 2013), time series forecasting (Ferreira, Vasconcelos, & Adeodato, 2008; da S. Gomes & Ludermir, 2013), among them. Regardless of the application, the most important step after the choice of the ANN type, is the adjustment of the its parameters, as the network topology, number of layers, number of neurons per layer, activation function, etc. The determination of the optimal values of these parameters generally is a huge task. In this sense, Evolutionary Algorithms (EAs) (Eiben & Smith, 2003) has been used to adjust ANN parameters (Donate, Li, Sánchez, & de Miguel, 2013; Ferreira et al., 2008; Rodrigues, de Mattos Neto, & Ferreira, 2009, Rodrigues et al., 2010). The EAs are heuristic optimization algorithms,

Q2 Q1

⇑ Corresponding author. Tel.: +55 8133206490. E-mail addresses: [email protected] (D.A. Silva), gabbybel@hotmail. com (G.I. Alves), [email protected] (P.S.G. de Mattos Neto), [email protected]. br (T.A.E. Ferreira). URL: http://www.ppgia.ufrpe.br/tiago (T.A.E. Ferreira).

inspired by biological evolution, used commonly to find optimal configuration of a specific system. The combination EA with ANN is very popular in the literature (Belfore & Arkadan, 1997; Bhuiyan, 2009; Ferreira et al., 2008; Gonzalez, Donate, Cortez, Sanchez, & de Miguel, 2012; Grzesiak, Meganck, Sobolewski, & Ufnalski, 2007; Guo, Kang, Liu, Sun, & Mei, 2007; Liao, 2012; Lima, Cannon, & Hsieh, 2012; Tomczak, 2011; Sotiroudis, Goudos, Gotsis, Siakavara, & Sahalos, 2013), where this combination is commonly called of intelligent hybrid systems. In general, for any EA (Eiben & Smith, 2003), trial solutions are represented by individuals of a population, where each one of these individuals has a chance of being selected to generate the next offspring. This procedure is repeated until an optimal solution is found. In each iteration, the recombination and mutation operators form the basis to create new offspring aiming to preserve the diversity in the population. Despite the importance of these operators, the selection and survival of each individual at each generation is guided by Fitness Function. This Fitness Function is used to measure the quality of the individuals. Different Fitness Functions can lead to different solutions, whereas each Fitness Function have its own fitness landscape (Kitts, Edvinsson, & Beding, 2001; Merz, 2004), that exerts strong influence in the effectiveness of the evolutionary search.

http://dx.doi.org/10.1016/j.eswa.2014.06.001 0957-4174/Ó 2014 Published by Elsevier Ltd.

Q1 Please cite this article in press as: Silva, D. A., et al. Measurement of Fitness Function efficiency using Data Envelopment Analysis. Expert Systems with Applications (2014), http://dx.doi.org/10.1016/j.eswa.2014.06.001

60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82

ESWA 9363

No. of Pages 14, Model 5G

17 June 2014 Q1

2

D.A. Silva et al. / Expert Systems with Applications xxx (2014) xxx–xxx

133

For a intelligent hybrid system (here, a EA combined with ANN), the fitness landscape will not depend of the Evolutionary Algorithm used to evolve the ANN. The fitness landscape will depend of the ANN’s structure and the data of the problem studied. However, in general, the performance of the EA will be dependent of the fitness landscape. In literature, some works investigated the Fitness Functions (Chen, Yang, Dong, & Abraham, 2005; Fan, Fox, Pathak, & Wu, 2004; Ferreira et al., 2008; Pai & Hong, 2005; Rodrigues et al., 2009; Rodrigues, Silva, de Mattos Neto, & Ferreira, 2010), as references to understand the performance of the hybrid systems (EA + ANN), where different fitness functions guide the EA to different solution. However, how is possible to determine the Fitness Function most efficient to solve a given problem? Wang et al. (2011) proposed a systematic approach of constructing fitness function, where the multi-objective Fitness Function is built from the combination of the simple fitness functions, but with the Wang’s work is not possible measure the relative efficiency of different fitness functions. In this paper, a methodology based on Data Envelopment Analysis (DEA) (Charnes et al., 1994) is proposed to analyze the Fitness Function efficiency. The DEA is a non-parametric methodology that constructs an efficiency frontier with the best units. Its applications involve many topics as banking (Luo, Bi, & Liang, 2012; Shyu & Chiang, 2012), mining (Touloo, Sohrabi, & Nalchigar, 2009), industry (Sarkis & Cordeiro, 2012), neural networks (Desheng, 2009; Makui & Noushabadi, 2012). The motivation for using this approach in this work is due few studies involving DEA to analyze the efficiency of the Fitness Function, especially applied to time series forecasting problem. The junction of this study is interesting for both researchers investigating the DEA and for research in the area of Evolutionary Algorithms and time series. In this work, the DEA is used to find the Fitness Function that increases the performance of the EA. To check the efficiency of the best fitness functions, graphs comparing the predicted value to the original data are used. Therefore, this article proposes a methodology to guide the project of an expert system based on Evolutionary Algorithms to solve real world problems, in particular to solve the time series forecasting problem. For Evolutionary Algorithms the Fitness Function is a fundamental point to be defined. The Fitness Function will guide the Evolutionary Algorithm to reach a good (maybe, an optimal) solution. Thus, the correct choice of the Fitness Function is fundamental to guarantee the good performance of the Evolutionary Algorithm. In the next Section is introduced the time series forecasting problem that is the case study addressed in this paper. In Section 3 is presented the DEA concepts, the determination of the DEA model and the definition of its variables. In Section 4 is presented the methodology to be used in this article. The experimental results are presented in Section 5 and the conclusions and future works are presented in Section 6.

134

2. Time series forecasting problem

135

139

A time series is a sequence of observations chronologically ordered about a given phenomenon. This series can be composed by discrete or continuous data. In general practical terms, an observation of a phenomenon results in a discrete sampling of data. A discrete time series is represented as a set

140 142

Z t ¼ fzt 2 R j t ¼ 1; 2; 3; . . . ; Ng

83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132

136 137 138

143 144 145 146

ð1Þ

where t is an chronological index, commonly the time, and N the total number of observations. The study of the time series behavior has evolved considerably over the years. The use of more advanced computational resources

combined with statistical techniques resulted in more accurate prediction, leading to lower cost in time to generate the forecasting. However, despite all these features, the ability to perform a good prediction will depend basically on the methodology used and the complexity of the phenomenon studied by the researcher. Thus, reducing the prediction error as much as possible for best results have been faced with the problem studied. The choice of model or technique to forecast, among many factors, depends on the level of precision that is required, desired forecast horizon, type of data used and the cost to produce the forecasts (Abraham & Ledolter, 2009). Alternatives approaches to statistical models (Box & Jenkins, 1994) for time series analysis and forecasting problem have been developed based on Artificial Intelligence techniques, like ANN (Areekul, Senjyu, Toyama, & Yona, 2010; Ferreira et al., 2008; Haykin, 1998; Mandal, Senjyu, Urasaki, Funabashi, & Srivastava, 2007; Yan, 2012) and EA (Amjady & Keynia, 2009; Eiben & Smith, 2003; Hinojosa & Hoese, 2010). Among those techniques, the hybrid systems based on the combination of ANNs with EAs have been used with success reaching relevant results (Donate et al., 2013; da S. Gomes & Ludermir, 2013; Ferreira et al., 2008; Rodrigues et al., 2009; Rodrigues et al., 2010; Stepnicka, Cortez, Donate, & Stepnicka, 2013).

147

3. Data Envelopment Analysis

169

Data Envelopment Analysis (DEA) (Charnes et al., 1994) is a methodology originally used in the area of operational research, where the interest is to compare different and independent units (firms, departments, etc.) in relation to their productive efficiency. In DEA, these units are represented by a variable called DMUs (Decision Make Unit). Each DMU is composed of a number of inputs required for a given amount of products. The main idea is compare these units to obtain the best combination of inputs and outputs aiming to better production efficiency. The DEA methodology was based on the concepts of relative efficiency proposed by Farrell in 1957 (Farrell, 1957). According Farrell, a company is considered efficient when it is able to produce a large quantity of products given a mix of resources. The inefficiency would be obtained when this company failed to get the most of your products from a number of resources. The formal concept about the DEA methodology was introduced by Charnes, Cooper and Rhodes in 1978 (Charnes et al., 1978), where to each DMU the DEA determines the maximum ratio between inputs and outputs, weighted by real factors determined by the model.

170

3.1. DEA models

190

3.1.1. The CCR model The CCR model (Charnes, Cooper, Rhodes) was the first proposed DEA model (Charnes et al., 1978) and assumes constant returns to scale. The solution of this model is given by the linear programming problem (multiplier form) below (Cooper, Seiford, & Tone, 2007),

191

max wk ¼

s X

uj yjk

150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168

171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189

192 193 194 195 196

197

199

subject to

200

201

r X

v i xik ¼ 1

ð3Þ

i¼1 s X

r X

j¼1

i¼1

v i xik  0

ð4Þ

where

Q1 Please cite this article in press as: Silva, D. A., et al. Measurement of Fitness Function efficiency using Data Envelopment Analysis. Expert Systems with Applications (2014), http://dx.doi.org/10.1016/j.eswa.2014.06.001

149

ð2Þ

j¼1

uj yjk 

148

203 204

ESWA 9363

No. of Pages 14, Model 5G

17 June 2014 Q1 205 206 207 208 209 210 211

D.A. Silva et al. / Expert Systems with Applications xxx (2014) xxx–xxx

 i represents the inputs, where r is the maximum number of inputs (i ¼ 1; . . . ; r)  j represents the outputs, where s is the maximum number of outputs (j ¼ 1; . . . ; s)  wk – relative efficiency for kth DMU.  xik ; yjk – amount for input i, output j respectively for kth DMU.  v i ; uj – weights (P 0) for input i, output j respectively.

212 213 214 215 216 217 218 219 220 221

222 224 225

226

The DMU is considered efficient, if the optimal objective value w 1 is equal to 1 in the Section 3.4 and there exists at least one optimal solution v  ; u in Eqs. (3) and (4), with v  > 0; u > 0. Otherwise, the DMU is considered inefficient (Cooper et al., 2007). For inefficient units, the DEA model identifies a set of efficient DMUs, called the set of reference or peer group, taken as benchmarks for the projection of inefficient units onto efficient frontier. The identification of this set of reference is better viewed through the dual model (envelopment form) shown below

min zk

ð5Þ

subject to

ð6Þ

k¼1 n X

kk yjk P yjk

ð7Þ

k¼1

228

kk P 0; 8k

229

where kk 2 R is the intensity of unit k. The optimal objective value z in Eq. (5) is equivalent to the optimal value w in Eq. (2) due to the duality relation of these models (Cooper et al., 2007). While the weights (v  ; u ) are important variables to the multipliers form in Eqs. (3) and (4), the optimal solution (k ; z ) has its relative importance to the form enveloped in (6) and (7). It is possible to obtain, from the values of lambda, the points (xo ; yo ) for which the inefficient units are projected onto the efficient frontier (defined by all efficient units). In this process of projection, inefficient units do not quite fit the surface causing slacks which are related to inputs excesses (s ) or outputs shortfalls (sþ ). These slacks are defined in the equations below

230 231 232 233 234 235 236 237 238 239 240

241

n X kk xik

s ¼ zo xjo 

ð8Þ

k¼1

sþ ¼

n X

kk yjk  yjo

ð9Þ

k¼1

243

s 2 R i ;

244

The ideal goal is to obtain the optimal objective value z ¼ 1 and all   slacks equal to zero (s ¼ 0; sþ ¼ 0) for a unit to be considered efficient, known as Pareto Efficiency (Cooper et al., 2007). Otherwise, the unit is considered inefficient (Cooper et al., 2007; Thanassoulis, 2001). From Eqs. (8) and (9), it is possible to obtain the coordinates of the points that represents the direction for the improvement of ^o ) is obtained by inefficient units. The coordinate points (^xo ; y

245 246 247 248 249 250

251

sþ 2 Rr



^xo ¼ z xo  s ^ o ¼ y o þ sþ y

254

Eqs. (10) and (11) suggest that the efficiency for a given unit can be improved if the values of the inputs are reduced radially by the opti mal objective value (z ) and the excesses of inputs (s ) are eliminated. In the same way, the efficiency can be attained if the  output values are augmented by the output shortfalls (sþ )

256 257 258

259

3.1.2. The BCC model The BCC model (Banker, Charnes, & Cooper, 1984) assumes variable returns of scale (VRS), where the variations of inputs and outputs are not proportional. The efficiency of the kth DMU hk is defined by (multipliers form) (Cooper et al., 2007),

264

max hk ¼

s X

1 The superscript 2001).

uj yjk þ jk

263

265 266 267 268

269

273

r X

v i xik ¼ 1

ð13Þ

i s X

r X

j

i

v i xik þ jk  0

ð14Þ

k2R

275

where the variable j indicates the return of scale. Due to the relation between the CCR model and BCC model, the considerations made in the previous subsection are also emphasized for the BCC model. The dual model (form enveloped) is given by

min bk

ð15Þ

subject to

276 277 278 279 280

281 283 284

285

n X kk xik 6 bk xjk

ð16Þ

k¼1 n X

kk yjk P yjk

ð17Þ

k¼1 n X

kk ¼ 1

ð18Þ

k¼1

kk P 0; 8k

287

where Eq. (18) is a restriction of convexity related by the variable j. The value taken by this restriction indicates whether the firm (DMU) is operating in an area of decreasing (< 1), constant (¼ 1) or increasing (> 1) returns to scale. If the constraint is equal to 1, the BCC model is known as the CCR model. Fig. 1 shows the possible operating regions to the production possibility set. The DMUs P under the efficiency frontier is consid-

288

is used to denote the optimal value of a variable (Thanassoulis, Fig. 1. Efficiency frontier in a DEA model.

Q1 Please cite this article in press as: Silva, D. A., et al. Measurement of Fitness Function efficiency using Data Envelopment Analysis. Expert Systems with Applications (2014), http://dx.doi.org/10.1016/j.eswa.2014.06.001

262

272

ð11Þ



261

271

subject to

v i ; uj P 0;

260

ð12Þ

j¼1

ð10Þ



253

255

(Cooper et al., 2007). Since the inefficient unit is below of the efficiency frontier, the comparison between the targets and evaluated units allows to obtain the percentage of improvement potential for reducing inputs and increasing products for the evaluate unit to become efficient (Coll & Blasco, 2006).

uj yjk 

n X kk xik 6 zk xjk

3

289 290 291 292 293 294

ESWA 9363

No. of Pages 14, Model 5G

17 June 2014 Q1

4

D.A. Silva et al. / Expert Systems with Applications xxx (2014) xxx–xxx

310

ered inefficient. The way that this unit is projected onto efficient frontier determines the orientation of the model. If ‘‘input oriented’’, the DMU is projected onto the efficient frontier, reducing the input and maintaining constant the output (projection of P to B or C). If ‘‘output oriented’’, the input is constant and the output is variable (projection of P to O or F). Under the assumption of variable returns to scale input oriented, the fraction xp =xb (see Fig. 1) is the pure technical efficiency (PTE) of ‘‘P’’. The unit ‘‘E’’ (xe ; ye ) has the largest average productivity within the production possibility set and represents an aggregate technically and scale efficient (know as overall efficiency (OE)) unit for the input/output mix (X; Y) (Boussofiane, Dyson, & Thanassoulis, 1991). The measure of the overall efficiency (OE) of ‘‘P’’ in comparison to unit ‘‘C’’ is the ratio DC=DP. This measure can be decomposed into pure technical efficiency (ratio DB=DP) and scale efficiency (SE) (ratio DC=DB) by the relation

313

OE ¼ ðPTEÞðSEÞ ¼

295 296 297 298 299 300 301 302 303 304 305 306 307 308 309

311

314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332

DB DC DP DB

ð19Þ

This relation indicates that the efficiency of a model with returns constant to scale (overall efficiency) can be decomposed into a model with variable returns to scale (pure technical efficiency) and its scale efficiency. If scale efficiency (SE) is equal to one, the efficient unit following the BCC model has characteristics of the CCR model and operates in the most productive scale size (Cooper et al., 2007). The scale inefficiency occurs when SE < 1 and it is caused by the distance between DC and DB (see Fig. 1). So, when SE < 1 is necessary to verify whether this was due to unit operating under increasing returns to scale or decreasing returns to scale. In order to determine this, one must calculate an additional efficiency measure (DRS) that follows the decreasing return to scale and then determine the following relation (Cooper et al., 2007; Fare, Grosskopf, & Lovell, 1994) 1. If OE ¼ PTE, returns to scale is constant. 2. If OE < PTE and OE ¼ DRS, scale inefficiency is due to increasing returns to scale. 3. If OE < PTE and OE < DRS, scale inefficiency is due to decreasing returns to scale.

333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357

The determination of slacks and benchmarks for the improvement of inefficient units follows the same model presented in Eqs. (8)–(11). An unit will be efficient if it obtains optimal objective    value b ¼ 1 and all slacks equal to zero (s ¼ 0; sþ ¼ 0) (Cooper et al., 2007; Thanassoulis, 2001), following the same path as the CCR model. 3.2. Definition and Selection of DMUs A group of DMUs is considered homogeneous if the analyzed units perform the same objectives under the same market conditions and the inputs and outputs that characterize the performance of all units of the group are the same differing only in intensity or magnitude (Golany & Roll, 1989). Typically, in an Evolutionary Algorithm, the selection process of candidate solution (individual selection) is guided by the Fitness Function. This Fitness Function ranks the individuals in the population of the Evolutionary Algorithm, where the best individual (best solution) will have the bigger value of Fitness Function and the worse individual will have the smaller value of fitness. Therefore, the Fitness Function is a key element in the project of Evolutionary Algorithm, where how the Fitness Function ranks the individual will influence directly in the algorithm performance. Thus, the correct choice of this Fitness Function will influence the performance of evolution to guide individuals to an optimal solution. Therefore, the Fitness Function will be considered in this

paper as a variable of decision in the analysis of efficiencies in DEA, i.e., each Fitness Functions analyzed will be a DMU in the DEA methodology.

358

3.3. Determination of inputs and outputs factors

361

The input and output factors are chosen based on the importance of these variables to the performance of DMUs in the DEA model (Kittelsen, 1993). This step is very important because the results of the model are highly influenced by the input and output choice. The procedures for selection of these factors can be made through the critical judgment of the researcher or by statistical analysis (Kittelsen, 1993). In this work, the time series forecasting problem employ as natural measure of performance the prediction error (Rodrigues et al., 2009). The performance measures are based on forecast error, but there is not consensus in the research area about which error measure provides the best results. Some works in the literature using performance measures based on different types of statistical errors (Armstrong & Collopy, 1992; Rodrigues et al., 2009; Rodrigues et al., 2010) have shown that the prediction accuracy reached by a evolutionary predictive model, when each performance measure is applied, is dependent on the Time Series (and its features). These works also shown that if the performance measure uses more than one statistical error simultaneously then there is a coupling between the statistical errors (Rodrigues et al., 2009; Rodrigues et al., 2010). The forecast error et is formulated as the difference between the current value of the series (Z t ) and the predicted value (Ot )

362

et ¼ ðZ t  Ot Þ

ð20Þ

360

363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384

385 387

From Eq. (20) is possible define several performance measures.

388

3.3.1. MSE (Mean Square Error) The mean square error (MSE) is the performance measure most commonly used in the literature for time series forecasting. Its equation is given by

389

N 1X MSE ¼ ðet Þ2 N t¼1

ð21Þ

390 391 392

393

395

where N indicates the number of points in the time series.

396

3.3.2. MAPE (Mean Absolute Percent Error) The mean absolute percentage error (MAPE) is expressed by the following formula:

397

N  X

  1  et  MAPE ¼  N t¼1 Z t 

ð22Þ

398 399

400

402

where Z t is the current observation of the time series at time t and N the number of points.

403

3.3.3. U of Theil statistics This metric is based on the U-statistic developed by Theil et al. (1966). The U-statistic is an accuracy measure that emphasizes the relevancy base for comparison, where this comparison is commonly done with a naive forecasting method. Here, the MSE error of the predictive model is compared with the MSE error of a random walk model like. This metric is done by the equation:

405

PN 

THEIL ¼

2

t¼1 Z j  Oj 2 PN  t¼1 Z j  Z jþ1

404

406 407 408 409 410 411

412

ð23Þ

If THEIL ¼ 1, the predictive model has a performance equals to a random walk. If THEIL > 1, a performance of the predictive model is worse than a random walk. And if THEIL < 1, the predictive model

Q1 Please cite this article in press as: Silva, D. A., et al. Measurement of Fitness Function efficiency using Data Envelopment Analysis. Expert Systems with Applications (2014), http://dx.doi.org/10.1016/j.eswa.2014.06.001

359

414 415 416 417

ESWA 9363

No. of Pages 14, Model 5G

17 June 2014 Q1 418 419 420 421 422 423 424 425 426

427

429 430 431 432 433 434 435 436 437 438 439

D.A. Silva et al. / Expert Systems with Applications xxx (2014) xxx–xxx

has a performance better than a random walk model like. In the ideal case, this metric will go to zero (THEIL ¼ 0). 3.3.4. ARV (Average Relative Variance) The performance measure Average Relative Variance (ARV) (Nowlan & Hinton, 1992; Shadbolt & Taylor, 2002), like the U-statistic, is also a comparative metric. In the ARV metrics the MSE error is compared with the quadratic deviation of the prediction with respect to the mean of the time series. The ARV performance measure is given by

2 PN  Oj  Z j ARV ¼ Pt¼1  2 N t¼1 Z j  Z

where Z represents the mean of the time series data. If ARV ¼ 1, the model is equal to predict the mean of the time series. If ARV > 1, the predictive model is worse than predict the mean of the time series, and if ARV < 1, the model is better than predict the mean of the set. In the ideal case, this metric will go to zero (ARV ¼ 0). 3.3.5. POCID (Prediction of Change in Direction) The Prediction of Change in Direction (Rodrigues et al., 2009), or POCID for short, is a metric that measure the local trend of the predictive model. The POCID measure is given by the equation:

440

PN

442

POCID ¼ 100

443

where

444

446

ð24Þ

( Dj ¼

t¼1 Dt

ð25Þ

N 





1; if Z j  Z j1 Oj  Oj1 > 0 0;

otherwise

447

For the perfect predictive model, the POCID is equal to 100%.

448

3.4. Definition of the DEA model

449

Before proceeding with the application of the DEA model, it is necessary to decide what kind of model will be used for each time series: the CCR model or the BCC model. One factor used to differentiate them is the return of scale, as described in Section 3.1.2. Simar and Wilson (2002) believes that the wrong choice of this factor may have economic implications because if some DMU does not exhibit constant returns to scale then some unit of production may have obtained larger or smaller weights. Also, if a particular model is assumed to be ‘‘constant returns to scale’’ when in fact it is ‘‘variable returns to scale’’, this mistake can cause loss of statistical efficiency. In an attempt to avoid this mistake, Simar and Wilson (1998, 2000) proposed a bootstrap procedure to test the hypothesis of scale return. Bogetoft and Otto (2010) explain this bootstrap procedure in their book clearly. Consider the set of combinations of inputs and outputs, where the input can produce the output. This combination is called a set of technology T and represented by

450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465

466 468

469 470 471 472 473 474 475

476

478

T ¼ fðx; yÞ j x can produce yg

ð26Þ

where x is the input and y is the output. The interest is to test whether the technology T exhibits constant return to scale (null hypothesis) against the alternative hypothesis of variable returns to scale. The statistic test for this hypothesis is formulated based on the concept of efficiency in scale, where for a given set of K firms observations has the following test statistic S

PK

S¼

k k¼1 ECRS PK k k¼1 EVRS

5

where ECRS and EVRS are respectively the efficiencies of scale of a model with constant returns to scale and variable returns to scale. The null hypothesis is rejected if S < ca , where ca is the critical value (here a ¼ 0:05). Since the distribution of S under the null hypothesis is not known it is impossible to calculate ca directly. Alternatively, the bootstrap method (Simar & Wilson, 1998, 2000) is used to simulate the empirical distribution of S under the null hypothesis and thus obtain the results of the testing of hypotheses (Banker, 1996; Bogetoft & Otto, 2010; Simar & Wilson, 1998; Simar & Wilson, 2002). The results of the hypothesis test for return scales using the bootstrap method is presented in the next section.

479

4. Applied methodology and experimental setup

491

A set of three relevant time series was used to evaluate the relative efficiency of different Fitness Functions proposed in this work. The first time series is a stock market index based on the common stock prices of 500 leading companies publicly traded American Companies, named S&P500 Index (Standard & Poor 500) (S.P.S. Index, 2011). Here, the S&P500 series consists of 369 monthly observations from January 1970 to August 2003. This time series has a inherent random behavior of financial market and a pronounced trend. The second time series analyzed is the sunspot series (der Linden, 2011). This time series has a non linear and quasi-periodic behavior without a trend. The sunspot population quickly rise and more slowly fall on irregular cycle around of 11 years. Finally, the third time series boarded here is the monthly milk production in the United States (N.A.S. Service(Nass)), consisting of 168 points collected monthly between January 1962 and December 1975. This series combine two behaviors: seasonality and trend. Before of the simulations, all series employed were normalized between ½0; 1 and divided into three parts: training set (50% of the data), validation set (25% of the data) and test set (25% of the data). In experiments, a Multi-layer Perceptron (MLP) Artificial Neural Network (ANN) was used with fixed architecture 3–5–1, i.e. 3 nodes in input layer, 5 nodes in hidden layer and 1 neuron in output layer (forecasting horizon of one step ahead). The training of the ANNs is driven by an Evolutionary Algorithm, where here was applied an Evolutionary Strategy (Beyer & Schwefel, 2002; Rechenberg, 1978; Schwefel, 1981) (ES) to evolve the ANN weights to minimize the prediction errors. The ES implemented in this paper uses the Sum Strategy (l þ K) with l ¼ 1 and K ¼ 1, where in each iteration an individual (solution) generates an offspring (another solution) through a mutation following a Normal Distribution and both solutions compete to generate the population of the next generation. Here, an individual, or a solution, is an ANN and the routine to adjust the ANN weights follows the steps bellow:

492

1. The initial population of ANNs is randomly generated using a Normal Distribution in the range [0; 1]; 2. The fitness of the individual (ANN) is calculated using the evaluation measures (Fitness Function); 3. An offspring is generated through a mutation using a Normal Distribution with mean equals to zero and standard deviation equal to r; 4. The offspring is evaluated. If the offspring’s fitness is at least as good as the parent one, it becomes the parent in the next generation. Otherwise, the offspring is disregarded.

528

481 482 483 484 485 486 487 488 489 490

493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527

529 530 531 532 533 534 535 536 537 538

ð27Þ

For all series the same parameters were used. Computationally, the individual is a vector composed by ANN weights plus the

Q1 Please cite this article in press as: Silva, D. A., et al. Measurement of Fitness Function efficiency using Data Envelopment Analysis. Expert Systems with Applications (2014), http://dx.doi.org/10.1016/j.eswa.2014.06.001

480

539 540

ESWA 9363

No. of Pages 14, Model 5G

17 June 2014 Q1

6

D.A. Silva et al. / Expert Systems with Applications xxx (2014) xxx–xxx

Table 1 Fitness Functions.

1

1 1þARV 1 1þMSE 1 1þMAPE 1 1þTHEIL 1 1þMSEþARV 1 1þMSEþMAPE 1 1þMSEþTHEIL 1 1þARVþMAPE 1 1þARVþTHEIL 1 1þMAPEþTHEIL

f1 f2 f3 f4 f5 f6 f7 f8 f9 f10

542 543

544 546

POCID 1þARV POCID 1þMAPE POCID 1þMSE POCID 1þTHEIL POCID 1þMSEþARV POCID 1þMSEþMAPE POCID 1þMSEþTHEIl POCID 1þARVþMAPE POCID 1þARVþTHEIL POCID 1þMAPEþTHEIL

f11 f12 f13 f14 f15 f16 f17 f18 f19 f20

ðjÞ

ðjÞ

ðjÞ

ðjÞ

ðjÞ indiv idualj ¼ w1 ; w2 ; . . . ; ww ; r1 ; r2 . . . ; rðjÞ wtotal total

E

554

r0i ¼ ri es Nð0;1ÞþsNð0;1Þ w0i ¼ wi þ r0i Nð0; 1Þ

549 550

552

0

555 556 557 558 559 560 561 562 563 564 565 566 567 568

where Nð0; 1Þ is a Normal Distribution with mean equals to zero and standard deviation equals to one. The initial value of r is randomly generated by Uniform Distribution in the range ½0; 1. The pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi pffiffiffiffiffiffiffi1 pffiffiffiffiffiffiffiffiffiffiffiffi1 pffiffiffiffiffiffiffiffiffiffiffiffi 2 57 learning rates are given by s ¼ ¼ 2 wtotal pffiffiffiffiffiffiffiffiffi 1 pffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1 e s0 ¼ ð 2wtotal Þ ¼ ð 114Þ following the heuristics rules reported by Bäck (1996) and Eiben and Smith (2003). Here, a boundary rule is also applied to prevent standard deviation r very close to zero, where if ri < 0 then ri ¼ 0 (where 0 ¼ 103 ). The ES will stop if the maximum number of ES iterations is reached (where was used the maximum number of iteration equals to 106 ) or if a fraction of the maximum number of iterations (this fraction is 10%) is reached without an improvement in the fitness. The ES procedure combined with MLP is demonstrated in the Algorithm 1.

Initialize a 0 // Iterations Number Generate PðaÞ // Initial Population offspring Mutation (PðaÞ); Evaluate f (offspring)// f (.) is Fitness Function while stop criterion not satisfied do if f ðoffspringÞ P f ðPðaÞÞ then Pða þ 1Þ ¼ offspring; else Pða þ 1Þ ¼ PðaÞ; end if a ¼ a þ 1; offspring Mutation (PðaÞ); end while 586

588 589 590 591

0

100

200

300

400 350

After the end of the simulations, the Fitness Functions value of the best individuals of the Evolutionary Strategy will be used as a Q4 DMUs in DEA (as seen in Section 3.2). The efficiencies are calculated and the generated predictions are analyzed, correlating the prediction accuracy with the Fitness Function efficiency.

250 200 150 100

ð29Þ ð30Þ

Algorithm 1. ES procedure

587

0.2

300

551

548

0.4

Fig. 2. S&P500 Index.

ð28Þ

where j ¼ 1; 2; . . . ; ðl þ KÞ. Here, the size of the population is l þ K ¼ 2. The mutation mechanism and the criterion of r coevolution (Eiben & Smith, 2003) used were the non-correlated mutation, given by equations:

547

0.6

Monthly Records

mutations steps ri , where i ¼ 1; . . . ; wtotal (wtotal is the number of ANN weights, where here wtotal ¼ 57). Therefore, the individual j is represented by a chromosome given by the vector:

D

0.8

0

Frequency

541

Normalized Index

Fitness Functions

50 0 0

0.2

0.4 0.6 Efficiency

0.8

1

Fig. 3. Histogram for the distribution of efficiencies of the Fitness Functions (S&P500 series).

Table 2 Hypothesis Testing using the bootstrap method to choose the DEA model (a ¼ 0:05). Series

S

ca

S&P500 Sunspot Milk

0.868312 0.936844 0.783449

0.715752 0.955787 0.869256

For the analysis, 20 Fitness Function were built (Table 1), and for each one of them was executed 30 simulations. Since for each simulation there is an elected individual (the best individual) with different characteristics, 20 groups of 30 DMUs were created. Therefore, a total of 600 DMUs per time series were generated. The input and output variables were defined (as cited in Section 3.3) based on measures of forecast error, calculated after the election of the best individuals for each simulation with a Fitness Function. In general, in the DEA, the variables which should be minimized are considered inputs and one that which should be maximized are considered outputs. Here, the variables are:  Inputs: ARV, MAPE, MSE, THEIL  Outputs: POCID

593 594 595 596 597 598 599 600 601 602 603 604 605

4.1. Hypothesis test

606

As described in Section 3.4, the bootstrap method was used in this work to determine the best DEA model for each time series. The results of the statistical test, where the hypothesis of constant returns to scale and the alternative hypothesis of return variables of scale are compared using the bootstrap method (Bogetoft & Otto, 2010). Table 2 shows the results of the statistical test using

607

Q1 Please cite this article in press as: Silva, D. A., et al. Measurement of Fitness Function efficiency using Data Envelopment Analysis. Expert Systems with Applications (2014), http://dx.doi.org/10.1016/j.eswa.2014.06.001

592

608 609 610 611 612

ESWA 9363

No. of Pages 14, Model 5G

17 June 2014 Q1

7

D.A. Silva et al. / Expert Systems with Applications xxx (2014) xxx–xxx Table 3 Metrics for efficient and less efficient Fitness Functions for the S&P500 series and the throughput with respect to best individual. Order DMUs

OE ðz Þ k

Observed Values

f7rep1 MSE

ARV

Slacks MAPE

1 2 3 4 5

f7rep1 f9rep10 f9rep1 f4rep24 f14rep20

1.0000 0.8824 0.7973 0.6782 0.6403

1.0000 1.1739 1.1086 1.0000 1.0652

0.0002 0.0003 0.0004 0.0004 0.0006

0.0169 0.0303 0.0359 0.0381 0.0479

0.0139 0.0185 0.0193 0.0205 0.0231

596 597 598 599 600

f13rep29 f13rep24 f13rep21 f3rep24 f2rep12

0.0114 0.0112 0.0105 0.0099 0.0094

0.8260 0.7826 0.7608 0.6956 0.6739

0.6910 0.6615 0.6913 0.6673 0.6812

53.4892 51.2021 53.5108 51.6543 52.7292

0.9997 0.9694 0.9999 0.9750 0.9893

THEIL 2.1185 3.7019 4.3920 4.6331 6.0052 6600.61 6313.91 6603.29 6370.78 6505.33

POCID

 s arv

 s mape

0.5111 0.6000 0.5666 0.5111 0.5444

0.0000 8.9E-5 0.0001 0.0001 0.0001

0.0000 0.0068 0.0098 0.0089 0.0126

0.0000 0.0000 0.0000 0.0000 0.0000

0.4222 0.4000 0.3888 0.3555 0.3444

0.0077 0.0072 0.0071 0.0064 0.0063

0.6009 0.5618 0.5535 0.5009 0.4882

0.0000 0.0000 0.0000 0.0000 0.0000

621

the bootstrap method to choose the appropriate DEA model for the three times series investigated here, the sunspot series, the S&P500 index series and the milk series. The values of the statistic S and the critical value ca for the three series are shown. The null hypothesis is rejected for the sunspot and milk series, because the value of the statistic S is smaller than the critical value. Thus, the variable returns to scale is adopted for the sunspot series and milk. For the S&P500 series (Table 2, S > ca ) is adopted the model with constant return to scale.

622

5. Experimental results

623

5.1. S&P500 series

624

The S&P500 (Standard & Poor 500) index series presents a regular movement of growth. This growth trend is viewed in Fig. 2. The model chosen for the S&P500 according to hypotheses set out in Section 3.4 follows the assumption of constant returns to scale input oriented. Fig. 3 shows frequency distribution histograms of efficiency to the 600 evaluated units following the CCR model. Note that the highest concentration of efficiencies is located in a region with efficiency below 0.3 (91.83%). This indicates that most of the evaluated units have been unable to achieve the efficient frontier. Only one unit assessed (0.17%) was able to achieve it. The results of the 5 highest and 5 lowest evaluated units for the S&P500 series are shown in Table 3, along with the corresponding overall efficiency (OE), weight peers, observed values, slacks and the values for the improvement of inefficient units (projected values). We must remember that the evaluated unit is considered efficient if it obtains the optimal objective value z ¼ 1 and all slacks are null in the solution of the dual model. Table 3 also shows that only one unit f 7rep12 is considered efficient, following these conditions. The unit f 7rep1 is taken as benchmark for all other units considered inefficient. Still in the Table 3, the 4th column shows the weights attributed to the improvement of inefficient units. Through these weights, the inefficient units are projected on the surface of efficiency. In this process of projection, there are residues (slacks) and through them is built the coordinates for the optimal values of inputs and outputs (projected values). For example, for the unit (f 9rep10) becomes effective is necessary to reduce the inputs proportionally in (10.882435) 11.75% and reduce the slacks corresponding to each input. Thus, the optimal value for the input ‘‘MSE’’ (column projected value) will be 0.000257 [(0.882435⁄0.00392)0.00089] while the observed value was 0.00392 (column observed value).

613 614 615 616 617 618 619 620

625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655

2

Projected Values

 s mse

The notation utilized to represent the Fitness Function as a DMU is fXrepY, where X is the number of the Fitness Function defined in Table 1 and the Y is the simulation number for the respective X Fitness Function (Y is an integer 2 ½1; 30).

 s theil

 sþ pocid

MSE

ARV

MAPE

THEIL

POCID

0.0000 0.7797 1.1532 1.0240 1.5888

0.0000 0.0000 0.0000 0.0000 0.0000

0.0002 0.0002 0.0002 0.0002 0.0002

0.0169 0.0198 0.0187 0.0169 0.0180

0.0139 0.0163 0.0154 0.0139 0.0148

2.1185 2.4870 2.3488 2.1185 2.2567

0.5111 0.6000 0.5666 0.5111 0.5444

74.1286 69.2555 68.2902 61.7642 60.2202

0.0000 0.0000 0.0000 0.0000 0.0000

0.0001 0.0001 0.0001 0.0001 0.0001

0.0139 0.0132 0.0128 0.0117 0.0114

0.0114 0.0108 0.0105 0.0096 0.0093

1.7501 1.6580 1.6119 1.4737 1.4277

0.4222 0.4000 0.3888 0.3555 0.3444

Table 4 Potential improvement for inefficient units in series S&P500. Order

DMUs

Improvement Potential (%) MSE

ARV

MAPE

THEIL

POCID

1 2 3 4 5

f7rep1 f9rep10 f9rep1 f4rep24 f14rep20

0.00 34.42 47.75 55.56 62.37

0.00 34.42 47.75 55.56 62.37

0.00 11.76 20.26 32.17 35.96

0.00 32.82 46.52 54.27 62.42

0.00 0.00 0.00 0.00 0.00

596 597 598 599 600

f13rep29 f13rep24 f13rep21 f3rep24 f2rep12

99.97 99.97 99.98 99.98 99.98

99.97 99.97 99.98 99.98 99.98

98.85 98.88 98.94 99.01 99.05

99.97 99.97 99.98 99.98 99.98

0.00 0.00 0.00 0.00 0.00

Table 5 Weights and percentage contribution input/output for the efficiency (multiplier form)—S&P500 series. Order

DMUs

v mse

v arv

v mape

v theil

upocid

1 2 3 4 5

f7rep1 f9rep10 f9rep1 f4rep24 f14rep24

0.000 0.000 0.000 0.000 0.000

0.000 0.000 0.000 0.000 0.000

71.876(100%) 54.029(100%) 51.693(100%) 48.752(100%) 43.209(100%)

0.000 0.000 0.000 0.000 0.000

1.957(100%) 1.471(100%) 1.407(100%) 1.327(100%) 1.176(100%)

596 597 598 599 600

f14rep20 f13rep29 f13rep21 f13rep24 f2rep12

0.000 0.000 0.000 0.000 0.000

0.000 0.000 0.000 0.000 0.000

1.000(100%) 1.032(100%) 1.000(100%) 1.026(100%) 1.011(100%)

0.000 0.000 0.000 0.000 0.000

0.027(100%) 0.028(100%) 0.027(100%) 0.028(100%) 0.028(100%)

Table 4 shows the potential improvement needed for the inefficient units become efficient. Note that the unit f 1rep7 was the only efficient and does not require improvement in their inputs/outputs. Also note that the values of the output (POCID) have not additional expansion (shortfalls). For the values of inputs ‘‘MSE’’ and ‘‘ARV’’ is given the same improvement conditions, indicating a common characteristic for these two variables. Inefficient units need to improve around 100% in order to achieve the efficiency frontier, indicating that optimum values are very far from ideal. It is also important to note that the input ‘‘MAPE’’ had no slack in its projection (Table 3), but requires improvement in the value of their input. The potential improvement was lower than all other entries indicating that the reduction in the consumption of this variable is more important for obtaining the efficiency of that reduction of other inputs. This fact can be better seen in Table 5, where it is shown the weights that contributed to find the optimal objective value w (2).

Q1 Please cite this article in press as: Silva, D. A., et al. Measurement of Fitness Function efficiency using Data Envelopment Analysis. Expert Systems with Applications (2014), http://dx.doi.org/10.1016/j.eswa.2014.06.001

656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673

ESWA 9363

No. of Pages 14, Model 5G

17 June 2014 8

D.A. Silva et al. / Expert Systems with Applications xxx (2014) xxx–xxx

Fitness Function−f9rep10

Fitness Function−f9rep1

1

1

0.95

0.95

0.95

Normalized Index

Normalized Index

Fitness Function−f7rep1 1

0.9 0.85 0.8 0.75

0.9 0.85 0.8 0.75

0.7 0.65 0

Normalized Index

Q1

0.9 0.85 0.8 0.75

0.7

20

40

60

0.65 0

80

Monthly Records

0.7

20

40

60

0.65 0

80

Monthly Records

20

40

60

80

Monthly Records

Fig. 4. Figure comparing the actual series (solid line) and the prediction generated by the neural network (dashed line) using the DMU more efficient to the S&P500 series.

Fitness Function−f3rep24 1

Fitness Function−f2rep12 1

0.9

0.9

0.8

0.8

0.8

0.7 0.6 0.5 0.4 0.3

Normalized Index

0.9 Normalized Index

Normalized Index

Fitness Function−f13rep21 1

0.7 0.6 0.5 0.4 0.3

0.7 0.6 0.5 0.4 0.3

0.2

0.2

0.2

0.1

0.1

0.1

0 0

50 Monthly Records

100

0 0

50 Monthly Records

100

0 0

50 Monthly Records

100

Fig. 5. Predicted values for S&P500 series to the less efficient DMUs. The solid line is the real data and the dashed line is the generated prediction.

676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699

The CCR model assigned weights only for input ‘‘MAPE’’ showing the importance of this variable. The highest weights were for the most efficient units and smaller weights for less efficient. The contribution of the weight assigned to the product ‘‘output’’ will always be 100% because it is the only variable considered. The overall efficiency (OE) of the unit is obtained by multiplying the weight by the value produced by the output POCID. For example, the unit more efficient (f7rep1) has 100% [(1.957)⁄ (0.511111)]. As defined in Table 1, the evaluated units (DMU) are the functions of fitness of the best individuals of the Evolutionary Strategy. These functions values indicate the performance of the model obtained by correlating the observed data with its prediction. If the model was able to find only one efficient unit projected correctly on the efficiency frontier, then when the original data series are compared with predicted values of this efficient unit is expected to obtain a good fit. Fig. 4 shows the predicted values (compared with the actual series) for the three most efficient DMUs to DEA model adopted (f 7rep1; f 9rep10 and f 9rep1), where the dashed line is the prediction and the solid lines is the real data. Visually is noted that the function f 7 obtained the best fit compared the other two DMUs. Fig. 5 presents the predicted values for the three DMUs less efficiently and visually note that these Fitness Functions can not train a model with sufficient accuracy to adjust the actual series data.

1 Normalized Index

674 675

0.8 0.6 0.4 0.2 0 0

100

200

300

Annual Records Fig. 6. The annual mean sunspot number time series.

5.2. Sunspot series

700

The time series of sunspots shows a quasi-periodic behavior along time as observed in Fig. 6. In the DEA analysis, the model employed for the sunspot series follows the assumption of variable returns to scale input oriented. Fig. 7 shows the efficiency distribution for the 600 units evaluated according to the BCC model. It should be noted that the efficiencies were concentrated on the right side of the distribution representing about 85% of the evaluated units. From the samples analyzed, 1.33% of the units

701

Q1 Please cite this article in press as: Silva, D. A., et al. Measurement of Fitness Function efficiency using Data Envelopment Analysis. Expert Systems with Applications (2014), http://dx.doi.org/10.1016/j.eswa.2014.06.001

702 703 704 705 706 707 708 709

ESWA 9363

No. of Pages 14, Model 5G

17 June 2014 Q1 200

Frequency

150

100

50

0 0

0.2

0.4

0.6

0.8

1

Efficiency Fig. 7. Histogram for the distribution of efficiencies (sunspot series).

710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727

9

D.A. Silva et al. / Expert Systems with Applications xxx (2014) xxx–xxx

showed maximum efficiency. The number of efficiency observations made by the Fitness Function are summarized in Table 6, which is classified according to overall measures of efficiency (OE), purely technical efficiency (PTE) and scale efficiency (SE) obtained. It should be noted that 19 units presented scalar efficiency and 8 units presented pure technical efficiency, however, only 4 units (f 1; f 5; f 15; f 17) showed global efficiency. These 4 units met the criteria of the CCR model, a derivative of the BCC model, and thus were able to reach a maximum scale of productivity. Of the 8 units with pure technical efficiency, 4 needed to improve their global efficiency in order to reach a maximum scale of productivity. In the 30 repetitions, 8 units (f 4; f 7; f 8; f 9; f 10; f 11; f 19; f 20) never attained scalar efficiency which implies that their inefficiency was due to the increasing and decreasing factors of their corresponding scale of production. In Table 7 the results of the 8 highest and 8 lowest evaluated units for the sunspot series, along with their corresponding pure technical efficiencies (PTE), Returns,

Reference Set (Peer), slacks and the values for the improvement of inefficient units (projected values) are shown. Constant scalar returns were seen in only 4 DMUs (f 1rep22; f 5rep5; f 15rep2; f 17rep26) of the 8 most efficient DMUs. The remaining 4 (f 1rep20; f 14rep29; f 19rep19; f 11rep30) units were characterized by decreasing returns. For these 4 units, the input variation produces inversely proportional variation in the final output. In the 6th column of the Table 7 the weights assigned by the BCC model (multiplier form) are shown. Notice that the inputs MSE and MAPE had the highest contributions to the optimal objective value. Under the potential improvement, there is not need to improve the units that are already efficient. In other words, the projection of these units on the frontier efficiency does not produce slacks, thus signifying that the adjustment was adequate. On the other hand, the most inefficient units presented a very high percent of improvement which is indicative of a high need to improve inputs. For example, the unit f 13rep6 needs to reduce its consumption by approximately 694% for the inputs MSE e ARV, 223.4% for the input MAPE and 761% for the input THEIL. These values indicate that the DMUs are more inefficient further away they are from the frontier efficiency. For each of these inefficient units, a reference set (based on the efficiency units) at which the inefficient units would be efficiency. In addition, for f 13rep6, the benchmarks were f 1rep20 and f 15rep2. For each reference, (k ) determines the weight of the contribution for the constructions of projected values. Table 8 shows the average contributions of each benchmark, aggregated by the Fitness Function. Observing the Table 8, the unit f 1rep22 have the largest global averages (72:22%) contribution to the projection of the inefficient units. The weight given to this DMU was always more decisive for all the Fitness Functions groups (f 1; f 2; . . . ; f 20). In second place, the DMU f 15rep2 had a global average contribution of 10:26%, followed by the DMU f 5rep5 (9:36%). The efficiency unit that less contributed was the DMU f 19rep19, with 0:17%.

Table 6 Number of efficient observations by Fitness Functions for the Sunspot Series. DMU

f1

f2

f3

f4

f5

f6

f7

f8

f9

f10

f11

f12

f13

f14

f15

f16

f17

f18

19

f20

SUM

Observations OE PTE DRS SE

30 1 2 2 2

30 0 0 0 2

30 0 0 0 2

30 0 0 0 0

30 1 1 1 1

30 0 0 0 1

30 0 0 0 0

30 0 0 0 0

30 0 0 0 0

30 0 0 0 0

30 0 1 1 0

30 0 0 0 1

30 0 0 0 3

30 0 1 1 1

30 1 1 1 2

30 0 0 0 1

30 1 1 1 2

30 0 0 0 1

30 0 1 1 0

30 0 0 0 0

600 4 8 8 19

Table 7 Metrics for the efficient and less efficient Fitness Functions for the sunspot series and the throughput with respect to best individual. Order

DMU

PE

Return

Peer

1 2 3 4 5 6 7 8

f1rep22 f5rep5 f15rep2 f17rep26 f1rep20 f14rep29 f19rep19 f11rep30

1 1 1 1 1 1 1 1

con. con. con. con. dec. dec. dec. dec.

f1rep22 f5rep5 f15rep2 f17rep26 f1rep20 f14rep29 f19rep19 f11rep30

593 594 595 596 597 598 599 600

f13rep1 f11rep11 f13rep11 f13rep30 f13rep21 f13rep27 f13rep14 f13rep6

0.4349 0.4271 0.3959 0.3956 0.3808 0.3796 0.3673 0.3092

dec. Inc. dec. Inc. con. Inc. Inc. dec.

f1rep20,f15rep2 f1rep22 f1rep20,f15rep2 f1rep22 15rep2 f1rep22, f5rep5, f17rep26 f1rep22,f15rep2 f1rep20,f15rep2

Weight (%)

Potential improvement

v mse

v arv

v mape

v theil

upocid

30.989 30.989 0.000 2.947 0.000 39.309 0.000 41.816

0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000

1.766 1.766 3.308 3.020 2.968 0.894 0.230 0.326

0.000 0.000 0.000 0.000 0.000 0.000 0.713 0.000

0.000 0.000 1.034 1.592 1.442 8.707 8.278 6.217

0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000

0.000 0.000 0.000 0.000 0.000 0.737 0.000 0.000

1.407 1.456 1.252 1.349 1.260 0.676 1.234 1.000

0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000

0.683 0.000 0.608 0.000 0.394 0.238 0.385 0.486

213.17 153.77 280.11 236.20 383.87 163.41 281.54 694.05

213.17 153.77 280.11 236.20 383.87 163.41 281.54 694.05

129.89 134.12 152.58 152.77 162.56 163.41 172.19 223.40

229.00 181.28 299.71 275.76 432.42 183.90 308.14 761.28

0.00 0.00 0.00 7.27 0.00 0.00 0.00 0.00

MSE

ARV

MAPE

THEIL

POCID

Q1 Please cite this article in press as: Silva, D. A., et al. Measurement of Fitness Function efficiency using Data Envelopment Analysis. Expert Systems with Applications (2014), http://dx.doi.org/10.1016/j.eswa.2014.06.001

728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761

ESWA 9363

No. of Pages 14, Model 5G

17 June 2014 Q1

10

D.A. Silva et al. / Expert Systems with Applications xxx (2014) xxx–xxx

Table 8 Average contribution of the weights peer by Fitness Functions for the sunspot series and the Global Average (GA). k

Fitness Function f1

f2

f3

f4

f5

f6

f7

f8

f9

f10

f11

f12

f13

f14

f15

f16

f17

f18

19

f20

GA

f1rep22 66.95 62.44 93.33 58.32 65.19 79.74 67.20 75.91 75.61 93.78 73.74 87.36 48.16 67.49 74.57 66.67 55.21 78.98 69.02 84.71 72.22 f1rep20 5.83 3.67 0.00 2.62 3.82 2.36 2.67 4.10 0.00 0.00 5.45 0.00 11.84 3.00 1.28 5.86 5.63 2.19 5.21 2.08 3.38 f5rep5 14.72 16.95 0.00 34.29 14.12 0.48 21.73 2.43 21.69 4.55 6.92 0.00 1.82 11.64 12.40 0.00 14.31 1.02 4.55 3.62 9.36 f11rep30 0.71 0.53 0.00 0.00 2.20 0.00 0.00 0.00 0.00 0.00 3.46 0.00 0.00 0.26 0.00 0.00 0.63 0.00 1.25 1.22 0.51 f14rep29 1.99 5.12 0.00 0.00 2.13 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.03 4.36 0.00 0.00 4.23 0.00 1.27 0.71 0.99 f15rep2 9.10 8.09 6.67 2.43 5.82 15.43 2.88 14.74 1.45 1.67 4.25 11.25 32.20 9.76 10.87 23.42 10.30 15.42 11.84 7.67 10.26 f17rep26 0.70 3.20 0.00 2.34 6.72 1.99 5.53 2.82 1.25 0.00 6.10 1.39 5.95 3.49 0.87 4.05 9.67 2.39 3.53 0.00 3.10 f19rep19 0.00 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.07 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.33 0.00 0.17

0 0

Normalized Index

Normalized Index

0.4 0.2 0 0

20 40 60 Annual Records

0.8 0.6 0.4 0.2 0 0

20 40 60 Annual Records

0.6 0.4 0.2 0 0

20 40 60 Annual Records

0.8 0.6 0.4 0.2 0 0

20 40 60 Annual Records

0.8 0.6 0.4 0.2 0 0

20 40 60 Annual Records

20 40 60 Annual Records

Fitness Function−f19rep19 1

Fitness Function−f11rep30 1

Fitness Function−f5rep5 1

Fitness Function−f1rep20 1

0.6

0.2 0 0

20 40 60 Annual Records

0.8

0.4

Normalized Index

0.2

0.6

0.8

Normalized Index

0.4

0.8

Normalized Index

0.6

Normalized Index

Normalized Index

Normalized Index

0.8

Fitness Function−f17rep26 1

Fitness Function−f14rep29 1

Fitness Function−f1rep22 1

Fitness Function−f15rep2 1

0.8 0.6 0.4 0.2 0 0

20 40 60 Annual Records

Fig. 8. Figure comparing the actual series (solid line) and the prediction generated by the neural network (dashed line) using the DMU more efficient to the sunspot series.

1 f13rep14

f13rep6

1 Normalized Index

Normalized Index

Sunspot

0.8 0.6 0.4 0.2 0 0

10

20

30 40 50 Annual Records

60

763 764 765 766 767 768 769 770

0.6 0.4 0.2

70

Fig. 9. Sunspot series forecasting with the smallest efficient DMUs. 762

0.8

By observing the units that achieved a technical pure efficiency, and comparing the sunspot series (original data) to the predicted values obtained by the respective units, it is hoped that these two values show a good adjustment. Fig. 8 presents the predicted values for the sunspot series, based on the eight most efficient DMUs (Table 7), where the dashed line is the prediction and the solid lines is the real data. The Fitness Functions with global efficiency, by the DEA analysis, obtained a good fit with the real time series data

0 0

50 100 Monthly Records

150

Fig. 10. The series of monthly milk production.

(Fig. 8). Visually, the units f 11rep30 and f 14rep29, that had higher weights assigned to input ‘‘MSE’’ (see Table 7), shown a good fit to the sunspot series data (Fig. 8). In contrast, Fig. 9 shows the forecast for the sunspot series based on the two least efficient DMUs. It is verified that the Fitness Function f 13 did not reach a good forecast for the sunspot series, where the small efficiency of the Fitness Function implies a poor

Q1 Please cite this article in press as: Silva, D. A., et al. Measurement of Fitness Function efficiency using Data Envelopment Analysis. Expert Systems with Applications (2014), http://dx.doi.org/10.1016/j.eswa.2014.06.001

771 772 773 774 775 776 777

ESWA 9363

No. of Pages 14, Model 5G

17 June 2014 Q1

Frequency

150

100

50

0 0

0.2

0.4 0.6 Efficiency

0.8

1

Fig. 11. Histogram for the distribution of efficiencies (milk series).

779

forecasting capacity of the predictive model. Therefore, the Fitness Function f 13 is not a good choice for the sunspot series.

780

5.3. Milk series

778

781 782 783 784 785 786

11

D.A. Silva et al. / Expert Systems with Applications xxx (2014) xxx–xxx

The time series of monthly milk production in the United States shows a seasonal behavior and a tendency. These two features of the milk time series can be observed in Fig. 10. The model chosen for the milk series follows the assumption of variable returns to scale input oriented, similar to the sunspot series. Fig. 11 shows the efficiency distribution of the 600 evaluated

units following the BCC model. Note that the majority of the analyzed units had pure technical efficiency (PTE) scores lower than 0.6 (approximately 66.51%). Only 5 units (0.83%) were able to reach the frontier of efficiency following the BCC model. In Table 9, the number of evaluated units that were efficient, grouped by the Fitness Function, is shown. In this table, note that only 4 units showed global efficiencies (CCR model), 5 showed pure technical efficiency according to the BCC model and 4 units were efficiency according to the model of decreasing returns to scale. It is important to observe that no unit evaluated by the milk series presented scalar efficiency, which indicates that all of the units are operating above or below the optimal scale. Table 10 shows the results of the 5 highest and 5 lowest evaluated units for the milk series, along with their corresponding technical efficiencies (PTE), Returns, Reference Set (Peer), slacks, and the values for the improvement of inefficient units (projected values). The results presented in Table 10 verify that units demonstrating pure technical efficiency (PE) as well as the most inefficient units, had decreasing returns to scale, working above the optimal scale. The weights were attributed by the dual model (multiplier form), the emphasis was given the input ‘‘MAPE’’, with emphasis on the units f 2rep17 and f 19rep25 that had the largest weights. The bigger weight attributed to the ‘‘POCID’’ output was f 19rep25, however this unit had the lowest weight ‘‘MAPE’’ input, in other words for this unit the highest weight was given to the product. The seventh column of the Table 10 contains the potential improvement of the inefficient units. The efficiency units do not

Table 9 Number of efficient observations by Fitness Functions for the Milk Series. DMU

f1

f2

f3

f4

f5

f6

f7

f8

f9

f10

f11

f12

f13

f14

f15

f16

f17

f18

19

f20

SUM

Observations OE PTE DRS SE

30 1 0 0 0

30 0 1 0 0

30 0 0 0 0

30 0 0 0 0

30 1 0 0 0

30 0 0 0 0

30 0 1 1 0

30 0 0 0 0

30 0 0 0 0

30 0 0 0 0

30 0 0 0 0

30 0 1 1 0

30 0 0 0 0

30 0 0 0 0

30 1 0 0 0

30 0 0 0 0

30 1 0 0 0

30 0 0 0 0

30 0 1 1 0

30 0 1 1 0

600 4 5 4 0

Table 10 Metrics for the efficient and less efficient Fitness Functions for the milk series and the throughput with respect to best individual. Order

DMU

PE

Return

Peer

Weight (%)

v

 mse

v

 arv

Potential Improvement

v

 mape

v

 theil

upocid

MSE

ARV

MAPE

THEIL

POCID

1 2 3 4 5

f2rep17 f7rep26 f12rep23 f19rep25 f20rep22

1 1 1 1 1

dec. dec. dec. dec. dec.

f2rep17 f7rep26 f12rep23 f19rep25 f20rep22

0.000 0.000 0.000 0.000 0.000

0.000 0.000 0.000 0.000 0.000

8.709 6.324 2.102 7.216 4.345

0.000 0.000 0.000 0.000 0.000

0.896 4.823 20.135 0.743 6.101

0.000 0.000 0.000 0.000 0.000

0.000 0.000 0.000 0.000 0.000

0.000 0.000 0.000 0.000 0.000

0.000 0.000 0.000 0.000 0.000

0.000 0.000 0.000 0.000 0.000

596 597 598 599 600

f12rep7 f16rep19 f13rep29 f16rep8 f3rep8

0.1748 0.1649 0.1609 0.1575 0.1489

dec. dec. dec. dec. dec.

f2rep17, f2rep17, f2rep17, f2rep17, f2rep17,

0.000 0.000 0.000 0.000 0.000

0.000 0.000 0.000 0.000 0.000

2.569 3.602 3.591 2.347 4.490

0.000 0.000 0.000 0.000 0.000

0.264 0.371 0.370 0.242 3.424

2771.815 2564.160 2159.784 1975.774 2607.480

2771.815 2564.160 2159.784 1975.774 2607.480

471.948 506.196 521.121 534.815 571.432

2793.932 2565.303 2136.770 1949.200 2589.380

0.000 0.000 0.000 0.000 0.000

f19rep25 f19rep25 f19rep25 f19rep25 f19rep25

Table 11 Average contribution of the weights peer by Fitness Functions to the Milk Series and the Global Average (GA). k

Fitness Function f1

f2

f3

f4

f5

f6

f7

f8

f9

f10

f11

f12

f13

f14

f15

f16

f17

f18

f19

f20

GA

f2rep17 57.52 65.38 58.16 59.73 59.57 67.04 49.07 52.96 58.84 51.77 20.00 7.41 10.00 35.93 23.70 17.04 32.22 29.63 35.50 32.60 41.20 f7rep26 0.00 0.00 0.00 0.00 0.00 0.00 3.33 6.67 0.00 0.00 3.33 15.00 28.33 3.33 0.00 15.00 5.00 11.67 8.33 3.33 5.17 f12rep23 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.33 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.17 f19rep25 42.48 34.62 41.84 40.27 40.43 32.96 47.60 40.37 41.16 48.23 73.33 69.26 53.33 57.41 72.96 62.96 61.11 53.70 54.50 60.74 51.46 f20rep22 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.33 5.00 8.33 3.33 3.33 5.00 1.67 5.00 1.67 3.33 2.00

Q1 Please cite this article in press as: Silva, D. A., et al. Measurement of Fitness Function efficiency using Data Envelopment Analysis. Expert Systems with Applications (2014), http://dx.doi.org/10.1016/j.eswa.2014.06.001

787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813

ESWA 9363

No. of Pages 14, Model 5G

17 June 2014 12

D.A. Silva et al. / Expert Systems with Applications xxx (2014) xxx–xxx

0.8 0.6 0.4 0.2 0

Fitness Function−f7rep26 1 Normalized Index

Normalized Index

Fitness Function−f2rep17 1

20 Monthly Records

40

0.6 0.4 0.2 0

20 Monthly Records

0.6 0.4 0.2 20 Monthly Records

40

0.8 0.6 0.4 0.2 0

20 Monthly Records

40

Fitness Function−f20rep22 1 Normalized Index

Normalized Index

0.8

0.8

0

Fitness Function−f19rep25 1

Fitness Function−f12rep23 1 Normalized Index

Q1

40

0.8 0.6 0.4 0.2 0

20 Monthly Records

40

Normalized Index

Normalized Index

Fitness Function−f12rep7 1 0.8 0.6 0.4 0.2 0 0

20 Monthly Records

Fitness Function−f16rep19 1 0.8 0.6 0.4 0.2 0 0

40

0.8 0.6 0.4 0.2 0 0

20 Monthly Records

40

Fitness Function−f13rep29 1 0.8 0.6 0.4 0.2 0 0

20 Monthly Records

40

Fitness Function−f3rep8 Normalized Index

Normalized Index

Fitness Function−f16rep8 1

20 Monthly Records

Normalized Index

Fig. 12. Figure comparing the actual series (solid line) and the prediction generated by the neural network (dashed line) using the DMU more efficient to the milk series.

40

1 0.8 0.6 0.4 0.2 0 0

20 Monthly Records

40

Fig. 13. Figure comparing the actual series (solid line) and the prediction generated by the neural network (dashed line) using the DMU less efficient to the milk series.

814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829

need to be improved. On the other hand, the most inefficient units need a high reduction of their inputs in order for them to reach the frontier efficiency. For example, f 3rep8 needs to reduce the observed values of ‘‘MSE’’ inputs by almost 2607%, 2607% of ‘‘ARV’’ inputs, 571% of ‘‘MAPE’’ inputs and around 2589% of the ‘‘THEIL’’ inputs. In the fifth column of the Table 10, the reference set is presented for the inefficient units projected unto the frontier efficiency. The efficiency units are self-referencing. The inefficient units have references f 2rep17 e f 19rep25, the same units which demonstrated the biggest weight assigned by the ‘‘MAPE’’ input as previously stated. In Table 11, the average percentage contribution for each aggregated benchmark of the Fitness Function is presented. Note that the units f 2rep17 (41,20%) and f 19rep25 (51,46%) contributed the most to the inefficient units becoming efficient.

The least contributing units were f 7rep26; f 20rep22 and f 12rep23 with average global percentages of 5.17%, 2.00% and 0.17% respectively. Fig. 12 shows the fit of the predictions to the original data. The efficiency units with which were associated low weights for the ‘‘MAPE’’ input (Table 10) were those that needed the fewest adjustments to the original data. The units f 2rep17 and f 19rep25 showed good fits for the upper part of the original series but showed poor fits for the lower periodic regions. Among the efficient DMUs, it is possible to see that f 12rep23 presents the worst forecast, reflecting a lower value of v mape weight (see Table 10). This shows that is possible to sort the efficient units according to the values of v mape , where DMU f 2rep17 on average has the best prediction. The performance of predicted values of the less efficient DMUs is shown in Fig. 13. Neither of the two functions fit well to real values. This fact is in accordance with the result of the efficiencies that were very close to zero (as shown in Table 10).

Q1 Please cite this article in press as: Silva, D. A., et al. Measurement of Fitness Function efficiency using Data Envelopment Analysis. Expert Systems with Applications (2014), http://dx.doi.org/10.1016/j.eswa.2014.06.001

830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845

ESWA 9363

No. of Pages 14, Model 5G

17 June 2014 Q1

D.A. Silva et al. / Expert Systems with Applications xxx (2014) xxx–xxx

873

From the results of experiments in the three time series, we can conclude that DEA was able to identify the best and the worst Fitness Functions. By comparing the prediction with the original series was possible to observe if their Fitness Function, responsible for guiding EA, was really efficient or not efficient. For S&P500 series (exponential trend), the most units evaluated had low scores overall efficiency. Only the unit f 7rep1 was considered efficient. Note that this unit consists by the ‘‘MSE’’ and ‘‘THEIL’’ performance metrics. For sunspot series (quasi-periodic behavior with a constant level), the choice of model with variable returns to scale identified 8 efficient units: f 1rep22, f 5rep5; f 15rep2; f 17rep26; f 1rep20; f 14rep29; f 19rep19; f 11rep30. The units f 14rep29 and f 11rep30 had higher weight assigned to input MSE. For milk series (with trend and a well defined seasonality), the DEA model identified 5 efficient units: f 2rep17; f 7rep26; f 12rep23; f 19rep25; f 20rep22. For this series, the weights of the inputs were assigned only to MAPE input, being the unit f 2rep17 with highest weight. Therefore, these experimental results show that the statistics measures of MSE and MAPE, and the combination of MSE with THEIL are a good choice to compose the Fitness Function for the time series forecasting problem. In more detail, analyzing the functions regarded as more efficient, the experiments results shown that if the time series has a strong trend, then the combination of MSE and THEIL is good to compose the fitness function. However, if the time series has a oscillate behavior, but a constant level, the MSE performance measure is more appropriate. And, if the time series has a seasonality component combined with a trend, then the MAPE performance measure is more indicated to compose the fitness function.

874

6. Conclusion

875

Many intelligent techniques are applied to solve real world problems, where a very studied class of real world problems is the time series forecasting problem. A very popular approach for the time series forecasting problem with expert system is to use an Evolutionary Algorithm to adjust the parameters of a predictive model. In this way, a vital point on the design of the Evolutionary Algorithm is the definition of the fitness function. However, studies for the characterization of the Fitness Function is a branch poorly explored in the literature, where in general just one performance measure, commonly the MSE, is used as the statistical measure to guide the Evolutionary Algorithm. Surely, these fitness functions based on just one performance measure will work, but are these Fitness Function the best choice to guide the Evolutionary Algorithm for search a good solution? In this paper was employed the DEA procedure to point the Fitness Function more efficient used in the Evolutionary Algorithm, with a constant number of iterations, to evolve an ANN as a predictive model. Three Time Series with specific features was used as application: the sunspot time series with a quasi-periodic behavior with a constant level; the S&P500 time series with a exponential trend and random shocks or fluctuations (additive white noise) and the milk time series with trend and a well defined seasonality. After discussing the assumptions and methodology used to set the EA and DEA, the model was idealized using 20 different functions (Table 1), where each one of these function is composed by a combination of up three performance measures. Experimental results showed that regardless of the time series feature, the DEA was able to find the best Fitness Function based on the concept of efficiency. Graphs comparing the original data to the predicted values were used to check the smooth adjustment of Fitness Functions efficient. Some strengths of the proposed method can be highlighted: first, DEA is a non-parametric methodology which does not require, a priori, knowledge of the weights involved in the model.

846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872

876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909

13

The DEA can define which are the DMUs efficient and not efficient just observing the group of all DMUs. Secondly, although of the work to use only one hybrid system (EE þ ANN), it was possible to evaluate five performance measures, where 30 simulations were done to each one of 20 combination of these performance measure, enabling to observe the behavior of each Fitness Function based on behavior of these samples (Figs. 3, 7 and 11). The Fitness Function will create a surface of solution quality (fitness landscape) where the Evolutionary Algorithm will search by the global maximum. In this sense, an efficient Fitness Function should be as smooth as possible and provide a more easily accessible global maximum for any Evolutionary Algorithm. In this way, it is possible expect that an efficient function will be effective regardless of the type of Evolutionary Algorithm applied. However, this statement needs to be confirmed with further experiments using other Evolutionary Algorithms. Therefore, here was expected that there is a correlation between the function and features of the time series, but not with the Evolutionary Algorithm (based on the results of previous work Ferreira et al., 2008; Rodrigues et al., 2009; Rodrigues et al., 2010). This point is the biggest weakness of the proposed methodology, because this point was not tested. In order to remedy this weakness, new experiments are being conducted with other Evolutionary Algorithms such as particle swarm optimization and genetic algorithms. With these new experiments will be possible answer whether there is a dependency of the function with the Evolutionary Algorithm used. In order to evaluate the limitations of the proposed methodology some future works can be listed. Beyond the methodology can be applied to evaluate the efficiency of the Fitness Function for others Evolutionary Algorithms, the methodology can be applied to a new data set, in order to find correlation between the new time series features (not analyzed here, like heteroscedasticity, chaos, spurious dependencies, etc.) and the best configuration of the fitness function. Furthermore, methods for variables selection (Nataraja & Johnson, 2011; Wagner & Shimshak, 2007; Ueda & Hoshiai, 1997) can be combined with the proposed methodology aiming to explore more parameters of the analyzed forecasting model (ANN). Finally, these works also can be used for others problems, like classification, clustering, pattern recognition, and others.

910

References

950

Abraham, B., & Ledolter, J. (2009). Statistical methods for forecasting. Wiley. Amjady, N., & Keynia, F. (2009). Day-ahead price forecasting of electricity markets by mutual information technique and cascaded neuro-evolutionary algorithm. IEEE_J_PWRS, 24(1), 306–318. Areekul, P., Senjyu, T., Toyama, H., & Yona, A. (2010). A hybrid ARIMA and neural network model for short-term price forecasting in deregulated market. IEEE_J_PWRS, 25(1), 524–530. Armstrong, J. S., & Collopy, F. (1992). Error measures for generalizing about forecasting methods: Empirical comparisons. International Journal of Forecasting, 8(1), 69–80. Bäck, T. (1996). Evolutionary algorithms in theory and practice: Evolution strategies, evolutionary programming, genetic algorithms. Oxford, UK: Oxford University Press. Banker, R. D. (1996). Hypothesis tests using data envelopment analysis. Journal of Productivity Analysis, 7(2), 139–159. Banker, R. D., Charnes, A., & Cooper, W. W. (1984). Some models for estimating technical and scale inefficiencies in data envelopment analysis. Management Science, 30(9), 1078–1092. Belfore, L. A., II, & Arkadan, A.-R. (1997). Modeling faulted switched reluctance motors using evolutionary neural networks. IEEE Transactions on Industrial Electronics, 44(2), 226–233. Beyer, H.-G., & Schwefel, H.-P. (2002). Evolution strategies – a comprehensive introduction. Natural Computing, 1(1), 3–52. Bhuiyan, M. (2009). An algorithm for determining neural network architecture using differential evolution. In International conference on business intelligence and financial engineering, 2009. BIFE’09 (pp. 3–7). http://dx.doi.org/10.1109/ BIFE.2009.10. Bogetoft, P., & Otto, L. (2010). Benchmarking with DEA, SFA, and R. Springer. Boussofiane, A., Dyson, R. G., & Thanassoulis, E. (1991). Applied data envelopment analysis. European Journal of Operational Research, 52(1), 1–15.

951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980

Q1 Please cite this article in press as: Silva, D. A., et al. Measurement of Fitness Function efficiency using Data Envelopment Analysis. Expert Systems with Applications (2014), http://dx.doi.org/10.1016/j.eswa.2014.06.001

911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949

Q5

ESWA 9363

No. of Pages 14, Model 5G

17 June 2014 Q1 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061

Q6

14

D.A. Silva et al. / Expert Systems with Applications xxx (2014) xxx–xxx

Box, G. E. P., & Jenkins, G. M. (1994). Time series analysis: Forecasting and control (3rd ed.). Upper Saddle River, NJ, USA: Prentice Hall PTR. Charnes, A. et al. (1978). Measuring the efficiency of decision making units. European Journal of Operational Research, 2(6), 429–444. Charnes, A. et al. (1994). Data envelopment analysis: Theory, methodology, and application. Norwell, MA, USA: Kluwer Academic Publishers. Chen, Y., Yang, B., Dong, J., & Abraham, A. (2005). Time-series forecasting using flexible neural tree model. Information Sciences, 174(3), 219–235. Coll, V., & Blasco, O. M. (2006). Evaluación de la Eficiencia Mediante el Análisis Envolvente de Datos, Juan Carlos Martínez Coll. Cooper, W. W., Seiford, L. M., & Tone, K. (2007). Data envelopment analysis: A comprehensive text with models, applications, references and dea-solver software. Springer. da S. Gomes, G. S., & Ludermir, T. B. (2013). Optimization of the weights and asymmetric activation function family of neural network for time series forecasting. Expert Systems with Applications, 40(16), 6438–6446. der Linden, R. V. (2011). The SIDC team, Online catalogue of the sunspot index. URL . Desheng, W. (2009). Supplier selection: A hybrid model using DEA, decision tree and neural network. Expert Systems with Applications, 36(4), 8503–8508. Donate, J. P., Li, X., Sánchez, G. G., & de Miguel, A. S. (2013). Time series forecasting by evolving artificial neural networks with genetic algorithms, differential evolution and estimation of distribution algorithm. Neural Computing and Applications, 22(1), 11–20. Eiben, A. E., & Smith, J. E. (2003). Introduction to evolutionary computing. SpringerVerlag. Fan, W., Fox, E. A., Pathak, P., & Wu, H. (2004). The effects of fitness functions on genetic programming-based ranking discovery for web search. Journal of the American Society for Information Science and Technology, 55(7), 628–636. Fare, R., Grosskopf, S., & Lovell, C. K. (1994). Production frontiers. Cambridge University Press. Farrell, M. (1957). The measurement of productive efficiency. Journal of the Royal Statistical Society. Series A (General), 120(3), 253–290. Fernandes, B. J., Cavalcanti, G. D., & Ren, T. I. (2013). Autoassociative pyramidal neural network for one class pattern classification with implicit feature extraction. Expert Systems with Applications, 40(18), 7258–7266 . Ferreira, T. A. E., Vasconcelos, G. C., & Adeodato, P. J. L. (2008). A new intelligent system methodology for time series forecasting with artificial neural networks. Neural Processing Letters, 28(2), 113–129. Golany, B., & Roll, Y. (1989). An application procedure for DEA. Omega, 17(3), 237–250. Gonzalez, B., Donate, J., Cortez, P., Sanchez, G., & de Miguel, A. (2012). Parallelization of an evolving artificial neural networks system to forecast time series using openmp and mpi. In 2012 IEEE conference on evolving and adaptive intelligent systems (EAIS) (pp. 186–191). http://dx.doi.org/10.1109/EAIS.2012.6232827. Grzesiak, L., Meganck, V., Sobolewski, J., & Ufnalski, B. (2007). Genetic algorithm for parameters optimization of ann-based speed controller. In EUROCON, 2007. The international conference on #34; Computer as a tool #34 (pp. 1700–1705). http:// dx.doi.org/10.1109/EURCON.2007.4400689. Guo, Y., Kang, L., Liu, F., Sun, H., & Mei, L. (2007). Evolutionary neural networks applied to land-cover classification in zhaoyuan, china. In IEEE symposium on computational intelligence and data mining, 2007. CIDM 2007 (pp. 499–503). http://dx.doi.org/10.1109/CIDM.2007.368916. Haykin, S. (1998). Neural networks: A comprehensive foundation (2nd Edition.). Upper Saddle River, NJ, USA: Prentice Hall PTR. Hinojosa, V., & Hoese, A. (2010). Short-term load forecasting using fuzzy inductive reasoning and evolutionary algorithms. IEEE_J_PWRS, 25(1), 565–574. Kittelsen, S. A. C. (1993). Stepwise DEA. choosing variables for measuring technical efficiency in norwegian electricity distribution, Memorandum 06/1993, Oslo University, Department of Economics. Kitts, B., Edvinsson, L., & Beding, T. (2001). Intellectual capital: From intangible assets to fitness landscapes. Expert Systems with Applications, 20(1), 35–50. Liao, G. -C. (2012). Application a novel evolutionary computation algorithm for load forecasting of air conditioning. In Power and energy engineering conference (APPEEC), 2012 Asia-Pacific (pp. 1–4). http://dx.doi.org/10.1109/APPEEC.2012. 6307573. Lima, A., Cannon, A., & Hsieh, W. (2012). Downscaling temperature and precipitation using support vector regression with evolutionary strategy. In The 2012 international joint conference on neural networks (IJCNN) (pp. 1–8). http://dx.doi.org/10.1109/IJCNN.2012.6252383. Luo, Y., Bi, G., & Liang, L. (2012). Input/output indicator selection for DEA efficiency evaluation: An empirical study of chinese commercial banks. Expert Systems with Applications, 39(1), 1118–1123. Ma, H., Chan, J., Saha, T., & Ekanayake, C. (2013). Pattern recognition techniques and their applications for automatic classification of artificial partial discharge sources. IEEE Transactions on Dielectrics and Electrical Insulation, 20(2), 468–478. http://dx.doi.org/10.1109/TDEI.2013.6508749. Makui, A., & Noushabadi, M. (2012). An empirical study for ranking insurance firms using a hybrid of data envelopment analysis and neural network. Management Science Letters, 2(8), 2923–2928.

Mandal, P., Senjyu, T., Urasaki, N., Funabashi, T., & Srivastava, A. K. (2007). A novel approach to forecast electricity price for PJM using neural network and similar days method. IEEE_J_PWRS, 22(4), 2058–2065. Merz, P. (2004). Advanced fitness landscape analysis and the performance of memetic algorithms. Evolutionary Computation, 12(3), 303–325. N.A.S. Service(Nass). (2011). Milk production. URL http://www.nass.usda.gov/. Nataraja, R. N., & Johnson, A. L. (2011). Guidelines for using variable selection techniques in data envelopment analysis. European Journal of Operational Research, 215(3), 662–669. Nowlan, S. J., & Hinton, G. E. (1992). Simplifying neural networks by soft weightsharing. Neural Computation, 4(4), 473–493. Pai, P.-F., & Hong, W.-C. (2005). Forecasting regional electricity load based on recurrent support vector machines with genetic algorithms. Electric Power Systems Research, 74(3), 417–425. Petkovic´, D., C´ojbašic´, Zˇarko, & Lukic´, S. (2013). Adaptive neuro fuzzy selection of heart rate variability parameters affected by autonomic nervous system. Expert Systems with Applications, 40(11), 4490–4495 . Rechenberg, I. (1978). Evolutionsstrategien. In Simulationsmethoden in der Medizin und Biologie. Medizinische Informatik und Statistik (Vol. 8, pp. 83–114). Springer. Rodrigues, A., de Mattos Neto, P. S. G., & Ferreira, T. A. E. (2009). A prime step in the time series forecasting with hybrid methods: The fitness function choice. In International joint conference on neural networks, 2009. IJCNN 2009 (pp. 2703– 2710). Rodrigues, A., Silva, D. A., de Mattos Neto, P. S. G., & Ferreira, T. A. E. (2010). An experimental study of fitness function and time series forecasting using artificial neural networks. In Proceedings of the 12th annual conference companion on Genetic and evolutionary computation, GECCO’10 (pp. 2015–2018). New York, NY, USA: ACM. Sarkis, J., & Cordeiro, J. J. (2012). Ecological modernization in the electrical utility industry: An application of a bads—goods {DEA} model of ecological and technical efficiency. European Journal of Operational Research, 219(2), 386–395. Schwefel, H. P. (1981). Numerical optimization of computer models. New York, NY, USA: John Wiley & Sons Inc.. Shadbolt, J., & Taylor, J. G. (2002). Neural networks and the financial markets: Predicting, combining, and portfolio optimisation. Springer. Shyu, J., & Chiang, T. (2012). Measuring the true managerial efficiency of bank branches in taiwan: A three-stage DEA analysis. Expert Systems with Applications, 39(13), 11494–11502. Simar, L., & Wilson, P. (1998). Sensitivity analysis of efficiency scores: How to bootstrap in nonparametric frontier models. Management Science, 44(1), 49–61. Simar, L., & Wilson, P. (2000). A general methodology for bootstrapping in nonparametric frontier models. Journal of Applied Statistics, 27(6), 779–802. Simar, L., & Wilson, P. W. (2002). Non-parametric tests of returns to scale. European Journal of Operational Research, 139(1), 115–132. Sotiroudis, S., Goudos, S., Gotsis, K., Siakavara, K., & Sahalos, J. (2013). Application of a composite differential evolution algorithm in optimal neural network design for propagation path-loss prediction in mobile communication systems. Antennas and Wireless Propagation Letters, IEEE, 12, 364–367. http://dx.doi.org/ 10.1109/LAWP.2013.2251994. S.P.S. Index (2011). . Stepnicka, M., Cortez, P., Donate, J. P., & Stepnicka, L. (2013). Forecasting seasonal time series with computational intelligence: On recent methods and the potential of their combinations. Expert Systems with Applications, 40(6), 1981–1992. Thanassoulis, E. (2001). Introduction to the theory and application of data envelopment analysis: A foundation text with integrated software. Norwell, MA, USA: Kluwer Academic Publishers.. Theil, H. et al. (1966). Applied economic forecasting (Vol. 4). Amsterdam: NorthHolland Publishing Company. Tomczak, E. (2011). Application of {ANN} and {EA} for description of metal ions sorption on chitosan foamed structure—equilibrium and dynamics of packed column. Computers & Chemical Engineering, 35(2), 226–235 . Touloo, M., Sohrabi, B., & Nalchigar, S. (2009). A new method for ranking discovered rules from data mining by DEA. Expert Systems with Applications, 36(5), 9105–9112. Ueda, T., & Hoshiai, Y. (1997). Application of principal component analysis for parsimonious summarization of DEA inputs and/or outputs. Journal of the Operations Research Society of Japan, 40(4), 466–478. Wagner, J. M., & Shimshak, D. G. (2007). Stepwise selection of variables in data envelopment analysis: Procedures and managerial perspectives. European Journal of Operational Research, 180(1), 57–67. Wang, P., Zhang, J., Xu, L., Wang, H., Feng, S., & Zhu, H. (2011). How to measure adaptation complexity in evolvable systems – {A} new synthetic approach of constructing fitness functions. Expert Systems with Applications, 38(8), 10414–10419. Yan, W. (2012). Toward automatic time-series forecasting using neural networks. IEEE Transactions on Neural Networks and Learning Systems, 23(7), 1028–1039.

Q1 Please cite this article in press as: Silva, D. A., et al. Measurement of Fitness Function efficiency using Data Envelopment Analysis. Expert Systems with Applications (2014), http://dx.doi.org/10.1016/j.eswa.2014.06.001

1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142