Evolutionary fuzzy k-nearest neighbors algorithm using interval-valued fuzzy sets

Information Sciences 329 (2016) 144–163

Contents lists available at ScienceDirect

Information Sciences journal homepage: www.elsevier.com/locate/ins

Evolutionary fuzzy k-nearest neighbors algorithm using interval-valued fuzzy sets Joaquín Derrac a,∗, Francisco Chiclana b, Salvador García c,d, Francisco Herrera e a

Affectv: Affectv Limited, 77 Oxford Street, London W1D 2ES, United Kingdom Centre for Computational Intelligence (CCI), Faculty of Technology, De Montfort University, Leicester LE1 9BH, United Kingdom c Department of Computer Science, University of Jaén, Jaén 23071, Spain d Department of Information Systems, Faculty of Computing and Information Technology, King Abdulaziz University, Jeddah, Saudi Arabia e Department of Computer Science and Artificial Intelligence, University of Granada, Granada 18071, Spain b

a r t i c l e

i n f o

Article history: Received 26 June 2014 Revised 13 November 2014 Accepted 1 September 2015 Available online 24 September 2015 Keywords: Fuzzy nearest neighbor Interval-valued fuzzy sets Evolutionary algorithms Supervised learning Classification

a b s t r a c t One of the most known and effective methods in supervised classification is the k-nearest neighbors classifier. Several approaches have been proposed to enhance its precision, with the fuzzy k-nearest neighbors (fuzzy-kNN) classifier being among the most successful ones. However, despite its good behavior, fuzzy-kNN lacks of a method for properly defining several mechanisms regarding the representation of the relationship between the instances and the classes of the classification problems. Such a method would be very desirable, since it would potentially lead to an improvement in the precision of the classifier. In this work we present a new approach, evolutionary fuzzy k-nearest neighbors classifier using interval-valued fuzzy sets (EF-kNN-IVFS), incorporating interval-valued fuzzy sets for computing the memberships of training instances in fuzzy-kNN. It is based on the representation of multiple choices of two key parameters of fuzzy-kNN: one is applied in the definition of the membership function, and the other is used in the computation of the voting rule. Besides, evolutionary search techniques are incorporated to the model as a self-optimization procedure for setting up these parameters. An experimental study has been carried out to assess the capabilities of our approach. The study has been validated by using nonparametric statistical tests, and remarks the strong performance of EF-kNN-IVFS compared with several state of the art techniques in fuzzy nearest neighbor classification. © 2015 Elsevier Inc. All rights reserved.

1. Introduction The k-nearest neighbors classifier (kNN) [16] is one of the most popular supervised learning methods. It is a nonparametric method which does not rely on building a model during the training phase, and whose classification rule is based on a given similarity function between the training instances and the test instance to be classified. Since its definition, kNN has become one of most relevant algorithms in data mining [42], and it is an integral part of many applications of machine learning in various domains [35,39]. In nearest neighbor classification, fuzzy sets can be used to model the degree of membership of each instance to the classes of the problem. This approach, known as the fuzzy k-nearest neighbor (fuzzy-kNN) classifier [31], has been shown to be an effective improvement of kNN. ∗

Corresponding author. Tel.: +34 958 240598; fax: +34 958 243317. E-mail addresses: [email protected] (J. Derrac), [email protected] (F. Chiclana), [email protected] (S. García), [email protected] (F. Herrera).

http://dx.doi.org/10.1016/j.ins.2015.09.007 0020-0255/© 2015 Elsevier Inc. All rights reserved.

J. Derrac et al. / Information Sciences 329 (2016) 144–163

145

This fuzzy approach overcomes a drawback of the kNN classifier, in which equal importance is given to every instance in the decision rule, regardless of its typicalness as a class prototype and its distance to the pattern to be classified. Fuzzy memberships enable fuzzy-kNN to achieve higher accuracy rates in most classification problems. This is also the reason why it has been the preferred choice in several applications in medicine [9,12], economy [11], bioinformatics [30], industry [33] and many other fields. The definition of fuzzy memberships is a fundamental issue in fuzzy-kNN. Although they can be set through expert knowledge, or by analyzing local data around each instance (as in [31] or [44]), there may be still a lack of knowledge associated with the assignation of a single value to the membership. This is caused by the necessity of fixing in advance two parameters: kInit in the definition of the initial membership values and m in the computation of the votes of the neighbors. To overcome this difficulty, interval valued fuzzy sets (IVFSs) [4,26], a particular case of type-2 fuzzy sets [5,34], may be used. IVFSs allow membership values to be defined by using a lower and an upper bound. The interval based definition includes not only a greater degree of flexibility than just using a single value, but also enables us to measure the degree of ignorance with the length of the interval [8,20]. Following this approach, IVFSs have been successfully applied in the development of fuzzy systems for classification [36–38]. In the case of nearest neighbor classification, this enables the representation of the uncertainty associated with the true class (or classes) to which every instance belongs, in the context of most standard, supervised classification problems. The optimization capabilities of evolutionary algorithms can also help to overcome this issue. In recent years, they have become a very useful tool in the design of fuzzy learning systems. For example, genetic fuzzy systems [14,15] show how the incorporation of evolutionary algorithms allows to enhance the performance of the learning model through parameter adjustment. Nearest neighbor classifiers’ performance is also prone to be improved by the use of evolutionary algorithms [10,18]. Considering the aforementioned issue, in this paper we propose an evolutionary fuzzy k-nearest neighbors classifier using interval-valued fuzzy sets (EF-kNN-IVFS). On the one hand, it tackles the problem of setting up the parameters via the implementation of interval values to represent both the membership of each training instance to the classes and the votes cast by each neighbor in the decision rule. The introduction of intervals in this approach allows us to consider different values for the kInit and m parameters, obtaining as a result different degrees of membership per each training instance. On the other hand, the implementation of evolutionary in the model would enable us to optimize the selection of both parameters, thus improving the accuracy of the whole classifier algorithm. Specifically, it is proposed to use evolutionary algorithms to develop an automatic method, driven by the CHC evolutionary algorithm [21], for optimizing the procedure to build the intervals in the interval-valued model and, following a wrapper based approach, to adapt the intervals to the specific chosen data set. The methodology developed in [19] for the field of fuzzy nearest neighbor classification is followed to carry out an experimental study to compare the EF-kNN-IVFS and various advanced fuzzy nearest neighbor classifiers. In this study, the classification accuracy is tested over several well-known classification problems. The results are contrasted using nonparametric statistical procedures, validating the conclusions drawn from them. The rest of the paper is organized as follows. Section 2 describes the kNN and fuzzy-kNN classifiers, highlighting the enhancements to the former introduced by the latter. Section 3 presents the EF-kNN-IVFS model, as a natural extension of fuzzy-kNN. Section 4 is devoted to the experimental study and the analysis of its results. Finally, conclusions are drawn in Section 5. 2. kNN and fuzzy-kNN classifiers The kNN and fuzzy-kNN classifiers require to measure the similarity of a new query instance (the new instance to be classified) to the instances stored in the training set. In the next step, a set of k nearest neighbors is found. Every neighbor casts a vote on the class to which the query instance should be assigned. Finally, a class is assigned to the query instance by combining these votes. The above procedure can be formally described as follows: let X be a training set, composed of N instances X = {x0 , x1 , . . . , xN } which belong to C classes. Each instance xi = (xi0 , xi1 , . . . , xiM , xiω ) is characterized by M input attributes and one output attribute ω (ω ∈ C). For a new query instance Q, a nearest neighbor classifier finds its k nearest neighbors in X, using a particular similarity function. Next, the class of Q is predicted as the aggregation of the class attributes ω of the k nearest neighbors. Initially, training instances of kNN are labeled using a hard scheme: The membership U of an instance x to each class of C is given by an array of values in {0, 1}, where Uω (x) = 1 and Uc (x) = 0, c ∈ C, c = ω. In this scheme, each instance belongs completely to one class and does not belong to any of the rest. In the case of fuzzy-kNN [31], the above scheme is extended using a continuous range of membership: memberships are quantified in [0, 1], and obtained using the following membership function



Uc (x) =

0.51 + (nnc /kInit ) ∗ 0.49 if c = ω

(nnc /kInit ) ∗ 0.49

otherwise

(1)

where nnc are the number of instances belonging to class c found among the kInit 1 neighbors of x. This fuzzy scheme causes instances close to the center of the classes to keep the original crisp memberships in {0, 1}, but instances close to the boundaries will spread half of their membership between the neighbors’ classes. 1

kInit is usually set to an integer value between [3, 9].

146

J. Derrac et al. / Information Sciences 329 (2016) 144–163

Once a query instance Q has been presented, its k nearest neighbors are searched in the training set. Although many different similarity functions can be considered for this task, the preferred choice in nearest neighbor classification is to define it via the Euclidean distance, which should suit most classification problems if the training data is normalized in the domain [0, 1]. Throughout the rest of the paper we will follow this methodology: the Euclidean distance is used and the attributes are normalized. Once the k nearest neighbors have been determined, the final output of the classifier is obtained by aggregating the votes cast by its neighbors. In the case of kNN, the votes are obtained by simply adding the memberships of the k neighbors. In the case of fuzzy-kNN, the Euclidean norm and the memberships are weighted to produce a final vote for each class and neighbor using Eq. (2):

V (k j , c) =

Uc (k j ) · 1/(Q − k j )2/(m−1) k 2/(m−1) i=1 1/(Q − ki )

(2)

where kj is the jth nearest neighbor (kj ∈ k) and m, m > 1 is a parameter used to intensify the distances between the query Q and the training data set elements (generally m = 2). The votes of each neighbor are finally added to obtain the final classification, as in the case of kNN. Thus, both classifiers obtain their final output applying the majority simple rule to the classes of the k nearest neighbors. However, in the case of fuzzy-kNN, the use of the soft labeling scheme and the weighted votes allows it to achieve a more precise classification, particularly for instances located next to the decision boundaries, whose crisp classification would otherwise be unclear. 3. EF-kNN-IVFS: evolutionary fuzzy k-nearest neighbors classifier using interval-valued fuzzy sets EF-kNN-IVFS is proposed to tackle the problem of membership assignation through the introduction of IVFS and evolutionary algorithms. As a consequence, the membership values of every instance in the training set are represented as an array of intervals, depicting a more flexible representation of the typicalness of the instances in every class of the problem. Intervals are also considered in the computation of the votes cast by each of the k nearest neighbors in the decision rule. Using this approach we aim at reducing the sensitivity of the original fuzzy-kNN classifier to the kInit and m parameters, removing the necessity of computing explicitly an ad-hoc configuration for each different data set tackled. After the definition of the core model of EF-kNN-IVFS, evolutionary algorithms are introduced for optimizing the representation of both the membership values and the votes. These representations are upgraded through a second redefinition of the way in which the kInit and m parameters are interpreted. In our implementation, we have chosen the CHC evolutionary model [21] to conduct the search. The description of our proposals is organized as follows: •

• • •

First, we present the mechanism developed to perform the computation of interval-valued memberships to the classes (Section 3.1). Second, we describe the voting procedure based on intervals (Section 3.2). Third, we show how the votes can be combined in order to obtain the final classification (Section 3.3). Finally, we detail how the evolutionary search is conducted, describing how the kInit and m parameters are adjusted (Section 3.4).

3.1. Computation of interval-valued memberships to the classes In fuzzy-kNN, the definition of the memberships of the training instances is governed by Eq. (1). It is designed so that the class to which an instance originally belongs obtains more than half (0.51) of the total membership, whereas the rest is shared among the rest of the classes of the problem. By searching for the kInit nearest instances, local information about the relative neighborhood of the instance is considered. Therefore, this set up incorporates both expert knowledge (the ω classes already assigned in the original data) and structural knowledge, thereby obtaining a more accurate representation of the true nature of the instance than by use of the kNN classifier [31]. However, a drawback of this approach is that kInit must be fixed in advance. Some rules of thumb may be considered when aiming for a proper set up, such as not setting it to an extremely low value—with kInit = 1 or kInit = 2 very few neighbors are included, and hence most of the local structural information about the data is lost—or not setting it to a very high value—which would make memberships approximately equal to the global distribution of classes in the training data, and thus discarding again the local information. Beyond this, any fixed value of kInit could potentially be selected. We argue that the use of IVFSs to represent membership of classes could provide an alternative and efficient solution to the above drawback, and therefore make the fixing of a specific value to kInit superfluous. Indeed, the use of interval-values for membership could accommodate the simultaneous use of different values of kInit. That is, Eq. (1) can be parameterized with kInit



Uc (x, kInit ) =

0.51 + (nnc /kInit ) ∗ 0.49 if c = ω

(nnc /kInit ) ∗ 0.49

otherwise

(3)

J. Derrac et al. / Information Sciences 329 (2016) 144–163

147

and then the membership of a training instance x to a class c can be represented as an interval





Uc (x) = Uc−le f t (x), Uc−right (x)

 Uc−le f t (x) = min Uc (x, kInit )   Uc−right (x) = max Uc (x, kInit )

(4)



(5)

with kInit belonging to a particular set of values derived from the application of the considerations mentioned before. Following this scheme, a more flexible and accurate representation of the training instances is obtained: •

•

•

Instances located at the center of their respective classes, surrounded only by instances of the same class, will maintain full membership to it ([1.0, 1.0]) and null membership to the rest of the classes ([0.0, 0.0]). This is equivalent to fuzzy-kNN and kNN. Instances located near the boundaries between classes, surrounded by instances of the same class, but also by some instances of other classes, will have their memberships modified as follows: – The lower value of the membership to ω, Uc−left (x), may be regarded as a measure of how many neighboring instances with a class different to ω there are, and of their relevance. The higher the number of these neighboring instances to the training instance they will be, the closer to 0.51 this lower value will be. – The upper value of the membership to ω, Uc−right (x), is a direct measure of how far away the first neighbor not belonging to ω is. It will be 1.0 if it is not among the first nearest neighbors (in accordance with the set up for kInit chosen), and slightly lower if it is, with the specific value again dependent on the number and position of the neighboring instances not belonging to ω. – The lower value of the membership to the rest of classes will be 0.0, unless one of the first nearest neighbors belongs to that class. The upper value can be regarded as a relative measure of the presence of this class among the neighborhood of the training instance, never greater than 0.49. Instances badly identified (possibly noise), surrounded only by instances of other classes, will get only half membership to ω ([0.51, 0.51]) whereas the membership to the rest of the classes will be a representation of the true nature of the instances.

3.2. Interval-valued voting procedure The votes cast by each neighbor in the computation of the decision rule (Eq. (2)) can also be represented by intervals. In this expression, the parameter m can be used to vary the influence of the neighbors, depending on the specific value chosen. If m = 2, the vote of each neighbor is weighted by the reciprocal of the squared Manhattan distance. As m increases, distances between the different neighbors will be evenly weighted, and thus the relative distances will have less effect on the determination of the votes (with m = 3 the weight becomes the reciprocal of the Euclidean distance). Similarly, if m is decreased, the relative distances will have a greater effect, reducing the contribution of the furthest instances (as m approaches 1). Although the choice recommended in [31] was to simply let m = 2, it is possible to consider this parameter in a more flexible way, by introducing intervals. This allows a range of possible values of m to be considered instead of a single one, resulting in a more general voting mechanism. To represent this, Eq. (2) becomes:

V (k j , c) = Uc (k j ) · D(k j )

(6)

where

D(k j ) = [min (D(k j , ma ), D(k j , mb )), max (D(k j , ma ), D(k j , mb ))]

  1/(Q − k j )2/(m−1)

D(k j , m) = k

i=1

1/(Q − ki )2/(m−1)

(7) (8)

and ma , mb are the minimum and maximum values chosen for the parameter m. Note that since the elements of Eq. (6) are intervals, their product must be computed as follows [27]:

[a1 , a2 ] ∗ [b1 , b2 ] = [min (I j ), max (I j )] I j = {a1 · b1 , a1 · b2 , a2 · b1 , a2 · b2 }.

(9)

3.3. Combination of votes After the votes have been computed, the final classification is obtained as the class with the maximum vote overall. In the case of EF-kNN-IVFS, the votes for every class are computed as

V (c) =

k  j=1

V (k j , c)

(10)

148

J. Derrac et al. / Information Sciences 329 (2016) 144–163

Fig. 1. A valid configuration of EF-kNN-IVFS. The values of kInit chosen are 2, 3, 4 and 8 (from the interval (1, 9)) and the values of ma and mb are 1.48 and 2.07, respectively.

where the addition of two intervals is obtained as follows:

[a1 , a2 ] + [b1 , b2 ] = [a1 + b1 , a2 + b2 ].

(11)

After the votes for every class have been added, the final classification is obtained by applying an order operator to the intervals obtained for each class. Specifically, a lexicographic order with respect to the lower bound [6,7] has been chosen (further details are provided in the experimental study, in Section 4.2). Upon the application of this operator, an order can be established between the classes, choosing the first one as the output of the classifier. 3.4. Evolutionary optimization of the kInit and m parameters As is detailed in Sections 3.1 and 3.2, the performance of the classifier is dependent on the selection of values considered for the kInit, ma and mb parameters. Although proper values may be fixed experimentally, in this subsection we propose the use of evolutionary algorithms for optimizing this selection. For parameters ma and mb , the optimization procedure needs to find two real values within a reasonable range (in accordance with the definition of Eq. (8)). However, the selection of a set of values for kInit may be more sophisticated. Instead of just fixing this selection to an interval (for example, {3, 4, 5, 6, 7, 8, 9}), it is possible to define an optimization procedure for choosing specific, non-correlative values. Fig. 1 shows an example of a valid configuration for EF-kNN-IVFS. It is composed of a binary array of Sn digits, in which a value Sn = 1 shows that n is chosen as a value for kInit, while a value Sn = 0 shows that n is not chosen, and two real values, corresponding to ma and mb . In the example illustrated in Fig. 1, the interval [1.48,2.07] has been chosen for the m parameter. The values 2, 3, 4 and 8 have been chosen for the kInit parameter. Note that this set-up is similar to the configuration {2, 3, 4, 5, 6, 7, 8}, but it will not produce the same memberships, as the values 5–7 are not considered. Thus, this configuration could be useful in such cases where the memberships should be constrained to the four nearest neighbors of each training instance, but then extended by considering an instance at a greater distance (the eighth neighbor). Nevertheless, with this representation, an optimization algorithm may be applied in order to obtain the best possible configuration for the specific data analyzed. In this sense, genetic algorithms are a suitable option given its well-known capabilities for optimization problems. We have chosen the CHC evolutionary algorithm because of its outstanding capabilities for binary optimization [21] and realcoded optimization [22]. It is a robust generational genetic algorithm which involves the combination of an advanced selection strategy with a very highly selective pressure, and several components inducing a strong diversity. The CHC algorithm is configured as follows: •

•

•

•

Representation of solutions. Binary chromosomes of Sn bits (representing the values chosen for kInit) and two real-coded values (representing the ma and mb values). Fig. 1 is, in fact, a valid chromosome of the algorithm. Initialization. All chromosomes are initialized randomly. Real values are initialized into the interval [1, 4] (m cannot take a value of 1 due to the definition of Eq. (8), and should never be set to a very high value). Crossover operator. HUX and BLX-0.5. The HUX operator (the classic crossover operator of CHC, which interchanges half of the non-matching genes of each parent) is used for the binary part of the solutions, whereas the BLX-0.5 operator [22] is used for crossing the real coded values. Fitness function. The core model is used as a wrapper. To evaluate a solution, EF-kNN-IVFS is configured with the parameters represented by the current solution, and its accuracy over the training data (using a leave-one-out validation scheme) is measured. This accuracy is considered to be the fitness of the solution.

The rest of the elements follow the same configuration as in the original definition of CHC. In summary, the application of CHC to optimize the configuration of the core model enables us to obtain a suitable configuration, adapted to the training data available, without the necessity of a specific set-up provided by the practitioner. This increases the adaptability of the whole model, EF-kNN-IVFS, for general supervised learning problems, providing that the training data available is representative of the problem considered. 4. Experimental study An experimental study has been carried out to test the performance of EF-kNN-IVFS. The experiments will involve several well-known classification problems and various state of the art algorithms in fuzzy nearest neighbor classification, chosen according to the revision presented in [19]. Section 4.1 describes the experimental framework in which all the experiments have

J. Derrac et al. / Information Sciences 329 (2016) 144–163

149

Table 1 Data sets considered in the study. Data set

#Ins.

#At.

#Cl.

Data set

#Ins.

#At.

#Cl.

Appendicitis Balance Banana Bands Bupa Cleveland Dermatology Ecoli Glass Haberman Hayes-roth Heart Hepatitis Ionosphere Iris Led7Digit Mammographic Marketing Monk-2 Movement New thyroid Page-blocks

106 625 5300 539 345 297 358 336 214 306 160 270 80 351 150 500 830 6876 432 360 215 5472

7 4 2 19 6 13 34 7 9 3 4 13 19 33 4 7 5 13 6 90 5 10

2 3 2 2 2 5 6 8 7 2 3 2 2 2 3 10 2 9 2 15 3 5

Penbased Phoneme Pima Ring Satimage Segment Sonar Spambase Spectfheart Tae Texture Thyroid Titanic Twonorm Vehicle Vowel Wdbc Wine Winequality-red Winequality-white Wisconsin Yeast

10,992 5404 768 7400 6435 2310 208 4597 267 151 5500 7200 2201 7400 946 990 569 178 1599 4898 683 1484

16 5 8 20 36 19 60 57 44 5 40 21 3 20 18 13 30 13 11 11 9 8

10 2 2 2 7 7 2 2 2 3 11 3 2 2 4 11 2 3 11 11 2 10

been carried out. Section 4.2 provides a description on the study performed for choosing the right order operator for comparing the intervals that represent the output of EF-kNN-IVFS. Section 4.3 shows the results obtained in the main experiments. Finally, Section 4.4 shows a study on the improvement obtained through the incorporation of evolutionary algorithms to the core model of EF-kNN-IVFS. 4.1. Experimental framework The experiments has been conducted over 44 classification data sets, whose partitions are available at the KEEL-Dataset repository2 [1,2]. Table 1 summarizes their characteristics: number of instances (#Ins.), number of attributes (#At.) and number of classes (#Cl.). None of the data sets includes missing values, and no nominal (discrete) attributes have been considered. All attribute values have been normalized at [0, 1] and a 10-folds cross validation procedure has been followed throughout the experiments (the cross validation procedure has been repeated five times and the results have been averaged for stochastic algorithms). Besides EF-kNN-IVFS, we have considered eight comparison algorithms: • • •

•

•

•

•

•

2

The kNN classifier [16]. The fuzzy-kNN classifier [31]. D-SKNN [17] is a nearest neighbor classifier based on the Dempster–Shafer theory, incorporating mechanisms to manage uncertainty and rejection of unclear instances. It estimates expected lower and higher classification costs of the instances according with their k nearest neighbors in the training set, assigning the instances to the class with minimum cost (maximum belief). IF-KNN [32] is a method which incorporates intuitionistic fuzzy sets into the voting rule. Votes are weighted using the concepts of membership and nonmembership to a certain class: If the membership of a voting instance is above a lower threshold and the nonmembership below a higher threshold, the vote is considered as positive. Else, the vote is considered as negative. FENN [43] is the fuzzy version of on Wilson’s editing rule for k-NN, ENN [41]: all instances in the training set are checked and those whose classification by the fuzzy-kNN classifier does not agree with its original class are removed. IT2FKNN [13] is a fuzzy nearest neighbor classifier based on interval type-2 fuzzy sets. They are introduced to represent the memberships of the training instances. In this way, several values for the parameter kInit are considered, thus including flexibility in the memberships’ representation. Type-2 fuzzy sets are built considering all the different memberships computed, and then a type reduction operation is performed to obtain a combined value, representative of all the choices considered initially. The algorithm is similar to fuzzy-kNN in respect to the rest of the phases. GAfuzzy-kNN [29] is the only evolutionary approach available for fuzzy nearest neighbor classification, to the best of our knowledge. It is a wrapper-based fuzzy-kNN classifier, which works by optimizing the m and kInit parameters using a binary genetic algorithm. The fitness function described uses the accuracy of an underlying fuzzy-kNN classifier as the fitness value. PFKNN [3] is a fuzzy prototype reduction method which starts by building a set of prototypes representing the border points of different clusters in the data. Then, it adds to this reference set those instances which could be misclassified. The algorithm also includes a third pruning phase in which non relevant prototypes for the classification phase are discarded. http://www.keel.es/datasets.php.

150

J. Derrac et al. / Information Sciences 329 (2016) 144–163 Table 2 Parameters configuration of the algorithms. Algorithm

Reference

Parameters

kNN Fuzzy-kNN D-SKNN IF-KNN FENN IT2FKNN GAfuzzy-kNN

[16] [31] [17] [32] [43] [13] [29]

PFKNN EF-kNN-IVFS

[3] –

k value: 7 k value: 5, kInit: 3, m: 2 k value: 5, α : 0.95, β : 1.0 k value: 3, mA: 0.6, vA: 0.4, mR: 0.3, vR: 0.7, kInit: 3 k value: 5, k (edition): 5, kInit: 3 k value: 7, kInit: {1,3,5,7,9}, m: 2 k value: 5, kInit: 3, m: 2, population size: 50, Evaluations: 500, crossover probability: 0.8, Mutation probability: 0.01 k value: 9 k value: 9, kInit range: (1,32), m range: [1,4], Sn :32, population size: 50, evaluations: 500

Table 2 shows the parameters configuration selected for each algorithm (we have followed the same configuration as in [19]). The parameter k is chosen for each algorithm in the {3,5,7,9} range,3 selecting the best performing value. That is, every algorithm has been tested considering each different value of k, and the best value found (the one that maximizes the average accuracy) has been chosen. For EF-kNN-IVFS, the set of kInit values allowed and the range in which the maximum and minimum value of m can be established have been set-up according to suggestions of [29], in which GAfuzzy-kNN was presented. The population size and the number of evaluations have been also chosen as in [29], for the sake of a fair comparison. The rest of the parameters have been set up following the recommendations given by the authors of each technique. The similarity measure used in all the experiments is the Euclidean distance. The results obtained in all the experiments will be contrasted through the use of nonparametric statistical tests [24,25]. Specifically, we will use the Wilcoxon signed-ranks test [40] for pairwise comparisons, and the Friedman test [23] (together with the Holm procedure [28] as post-hoc) for performing multiple comparisons. More information about these statistical procedures specifically designed for use in the field of machine learning can be found at the SCI2S thematic public website on Statistical Inference in Computational Intelligence and Data Mining, http://sci2s.ugr.es/sicidm/. 4.2. Selection of the order operator in the voting process The last step of the classification phase of EF-kNN-IVFS requires to provide an ordering on the intervals representing the votes assigned to each class. As in most of cases there is no direct way of providing such order, different ordering mechanisms have been tested, aiming to improve further the classification accuracy. The works [6,7] elaborate on this matter, suggesting different alternatives to provide an ordering between two intervals, [a, b] and [c, d]. Among those alternatives, we have selected the four following procedures: •

•

•

•

Xu. Xu and Yager order. The order is obtained by comparing a + b and c + d. If both sums are equal, then the expressions b − a and d − c are compared to break the tie. Lex-up. Lexicographic order with respect to the upper bound. The order is obtained by comparing b and d. If both values are equal, then a and c are compared to break the tie. Lex-down. Lexicographic order with respect to the lower bound. The order is obtained by comparing a and c. If both values are equal, then b and d are compared to break the tie. K-order. Kα , β mapping based order. The order is obtained by comparing a + α(b − a) and c + α(d − c). If both values are equal, then a + β(b − a) and c + β(d − c) are compared to break the tie. In our implementation, we chosen the values α = 1/3 and β = 2/3

The full results of this study can be found in Appendix A, and explore the accuracy obtained in test phase with each of these operators, considering four different configurations of the k parameter (k ∈ {3, 5, 7, 9}). Table 3 shows a summary of the results obtained over the 44 data sets, including the average accuracy and the number of wins of each algorithm (times in which it has obtained the best result). The best results are highlighted in bold. Given the results obtained, it has been decided that EF-kNN-IVFS will incorporate the lexicographic order of the class intervals with respect to the lower bound as the operator to determine the classification output. 4.3. Analysis of the EF-kNN-IVFS algorithm This study has been performed to appraise the performance of EF-kNN-IVFS in contrast with the comparison algorithms described in Section 4.1. Full results of the study can be found in Appendix A. The tables in the appendix report the average 3 Note that k=1 has been excluded since most of the fuzzy nearest neighbor algorithms would become the 1NN rule. Also, no higher values of k have been chosen because most of the classifiers would degenerate to a majority classifier, discarding the locality capabilities of nearest neighbor algorithms.

J. Derrac et al. / Information Sciences 329 (2016) 144–163

151

Table 3 Comparison of order operators. k=3

k=5

Operator Average accuracy Number of wins

Xu 0.8115 11

Lex-Up 0.8114 13

Lex-Down 0.8151 25

K-Order 0.8123 14

Xu 0.8169 14

Lex-Up 0.8159 9

Lex-Down 0.8191 25

K-Order 0.8165 14

Operator Average accuracy Number of wins

k=7 Xu 0.8172 10

Lex-Up 0.8178 6

Lex-Down 0.8200 22

K-Order 0.8183 14

k=9 Xu 0.8177 11

Lex-Up 0.8183 11

Lex-Down 0.8202 20

K-Order 0.8173 11

Table 4 Summary results of the main comparisons. Algorithm

Accuracy

Friedman rank

Holm p-value

Running time

kNN PFKNN FENN IF-KNN D-SKNN FuzzyKNN GAfuzzy-kNN IT2FKNN EF-kNN-IVFS Friedman p-value

78.15 78.77 79.26 79.72 79.85 81.10 81.30 81.11 82.02

5.5000 5.9205 5.8523 5.6477 5.6477 4.5455 4.6818 4.3068 2.8977 ≤10−10

0.00003 ≤10−5 ≤10−5 0.00002 0.00002 0.00954 0.00674 0.01581 –

3.78 952.21 6.40 6.83 2.86 6.29 1246.47 13.26 1025.81

accuracy obtained by each algorithms over the 44 datasets, the number of wins of each algorithm, and the times over the median (times in which its performance is above the median of all the algorithms of the experiment). A table is included per each setting of the k parameter considered (k ∈ {3, 5, 7, 9}). Also, an additional table is included showing the running time of the best performing setup of each algorithm in every problem. Table 4 shows a summary of the results, considering their best performing setup. These results have been contrasted by using nonparametric statistical procedures. We have chosen the Friedman test, since it provides us with a procedure for simultaneously contrasting the comparisons of EF-kNN-IVFS with all the techniques included in the study. As the table shows, EF-kNN-IVFS achieves the best accuracy result of all the algorithms in the comparison. Regarding the statistical tests, the Friedman p-value is lower than 10−10 , which means that significant differences are found among EF-kNN-IVFS and the rest of methods. The Holm procedure reports a p-value lower than 0.05 for the comparison between EF-kNN-IVFS and IT2FKNN, which means that the differences are significant at a α = 0.05 level. The p-values obtained in the rest of comparisons are lower, depicting that the differences are significant at a α = 0.01 level, at least. Considering these results, we can make the following analysis: •

•

•

•

EF-kNN-IVFS achieves a better average result than all the comparison techniques chosen, including the evolutionary fuzzy nearest neighbor classifier (GAfuzzy-kNN). This good behavior can be also observed by looking at the detailed results (in Appendix A) where EF-kNN-IVFS obtains always the highest number of wins for each of the different settings of the k parameter. Moreover, EF-kNN-IVFS is always the algorithm whose results are more times over the median, which shows the robustness of the method. The statistical analysis performed states that the differences found between EF-kNN-IVFS and the comparison algorithms are always significant, at least at a α = 0.05 level. This confirms our conclusions showing that EF-kNN-IVFS is significantly more accurate than the rest of techniques. EF-kNN-IVFS running time is comparable with other evolutionary or search based techniques, such as GAfuzzy-kNN and PFKNN. Other comparison techniques show a much lower running time, but it comes at a cost of a lower accuracy.

In summary, these results show us that EF-kNN-IVFS is a very competitive fuzzy nearest neighbor classifier. The use of interval values for defining memberships and computing the neighbors’ votes, together with the inclusion of the evolutionary optimization, produces better results than those using the classical kNN classifier, fuzzy-kNN (fuzzy sets), D-SKNN (possibilistic classification), IF-KNN (intuitionistic fuzzy sets), IT2FKNN (type-2 fuzzy sets), FENN and PFKNN (fuzzy prototype reduction techniques). The comparison has been completed introducing an evolutionary fuzzy nearest neighbor classifier, GAfuzzy-kNN, whose results have been also improved. 4.4. Analysis of the evolutionary optimization process Another study has been performed, aimed at analyzing whether the inclusion of evolutionary optimization for parameter optimization is effective for enhancing the performance of EF-kNN-IVFS.

152

J. Derrac et al. / Information Sciences 329 (2016) 144–163 Table 5 Analysis of the evolutionary optimization process.

Average Number of wins

EF-kNN-IVFS (non optimized)

EF-kNN-IVFS (optimized)

81.18 13

82.02 32

Table 6 Results of the Wilcoxon test. Comparison

R+

R−

p-value

EF-kNN-IVFS (optimized) vs. EF-kNN-IVFS (non optimized)

701.5

244.5

0.00509

This analysis have been carried out by comparing the performance of the full EF-kNN-IVFS model with a fixed version, in which the kInit and m parameters are not determined by the evolutionary search. Instead, a fixed configuration is setting up: •

• • •

The kInit parameter is valued in the interval (3, 9), covering the same range of values considered for the k value in the final application of the nearest neighbor rule. Several configurations have been tested experimentally for the m parameter. The best one, [1.5, 2] has been chosen. The k value is again chosen as the best one in the {3,5,7,9} range. In this case, k = 7 has been chosen. The rest of parameters remain the same, including the use of the lexicographic order with respect to the lower bound for making the final decision during the voting process.

Table 5 shows a summary of the accuracy results obtained in this study, including the average accuracy obtained over the 44 datasets and the number of wins of each version (full results can be found in Appendix A). In this table, the full EF-kNNIVFS model is denoted as EF-kNN-IVFS (optimized), whereas the fixed, non-evolutionary version is denoted as EF-kNN-IVFS (non optimized). We have chosen the Wilcoxon test as a pairwise procedure for contrasting this experiment. Table 6 shows the results of the test, including the ranks obtained in each comparison (R+ and R− ) and the p-value associated. The results reported in Tables 5 and 6 show that the optimized version achieves better results, thus justifying the usefulness of the evolutionary method for determining which values of the kInit and m parameters to consider during the construction of the classifier. Although a better result is not achieved in all the datasets, it can be stated that, in general, the evolutionary optimization enables to obtain a better adjusted classifier, both in the sense of average accuracy and in the number of data sets whose results have improved. These results are corroborated by the Wilcoxon test, contrasting these conclusions. 5. Conclusion In this paper we have proposed a new evolutionary interval-valued nearest neighbor classifier, EF-kNN-IVFS. IVFS are chosen as an appropriate tool for representing the instances’ memberships to the different classes of the problem. They also enable our classifier to represent several votes as a single interval, thus giving more flexibility to the decision rule computation, and ultimately, improving the generalization capabilities of the nearest neighbor rule. The evolutionary optimization process has eased the task of properly searching for the best intervals of the kInit and m parameters, and thus further improving its accuracy. This has also been corroborated experimentally and it may be concluded that EF-kNN-IVFS is significantly more accurate than a selection of classical and new approaches in the state of the art of fuzzy nearest neighbor classification. Acknowledgments This work was supported by the Spanish Ministry of Science and Technology (Project TIN2014-57251-P) and by the Andalusian Government (Junta de Andalucía - Regional Projects P10-TIC-06858 and P11-TIC-7765). Appendix A. Full results of the experimental study The full results of the experimental study can be found below. Tables A.7–A.10 show the accuracy results obtained in the test classification phase for the different configurations of the k parameter considered (k ∈ {3, 5, 7, 9}). For each data set, the best results obtained have been highlighted in bold. The table also includes the average accuracy obtained over the 44 datasets, the number of wins of each algorithm (times in which it has obtained the best result) and the times over the median (times in which its performance is above the median of all the algorithms of the experiment). Additionally, Table A.11 shows an analysis on the running time of all the algorithms (measured in seconds), considering their best performing configuration. Table A.12 shows the accuracy results obtained when analyzing the performance of EF-kNN-IVFS without the evolutionary optimization process, highlighting the best results obtained and including the average accuracy obtained over the 44 datasets. Note that, in this table, the full EF-kNN-IVFS model is denoted as EF-kNN-IVFS (optimized), whereas the fixed, non-evolutionary version is denoted as EF-kNN-IVFS (non optimized).

J. Derrac et al. / Information Sciences 329 (2016) 144–163

153

Table A.7 Analysis of the EF-kNN-IVFS algorithm (k = 3). Data sets

EF-kNN-IVFS

kNN

Fuzzy-kNN

D-SKNN

IF-KNN

FENN

IT2FKNN

GAfuzzy-kNN

PFKNN

Appendicitis Balance Banana Bands Bupa Cleveland Dermatology Ecoli Glass Haberman Hayes-roth Heart Hepatitis Ionosphere Iris Led7Digit Mammographic Marketing Monk-2 Movement New thyroid Page-blocks Penbased Phoneme Pima Ring Satimage Segment Sonar Spambase Spectfheart Tae Texture Thyroid Titanic Twonorm Vehicle Vowel Wdbc Wine Winequality-red Winequality-white Wisconsin Yeast Average Number of wins Times over the median

84.18 84.16 89.26 71.85 61.19 56.64 96.04 83.07 73.66 70.61 66.25 79.26 82.51 86.32 94.00 71.60 79.16 30.35 84.33 86.11 96.80 95.98 99.34 90.38 74.36 73.77 90.88 96.80 86.00 91.26 73.45 67.67 98.96 93.99 78.83 96.61 70.21 99.39 96.65 95.49 67.42 67.31 96.82 57.75 81.51 21 35

84.27 83.37 88.64 71.46 60.66 55.14 96.90 80.67 70.11 70.58 25.00 77.41 82.51 94.00 85.18 45.20 80.12 27.97 96.29 78.61 95.37 95.91 99.32 88.49 72.93 71.82 90.94 95.41 83.07 89.23 71.20 41.13 98.75 93.89 62.61 96.49 71.75 97.78 96.48 95.49 52.41 48.92 96.38 53.17 77.57 4 10

87.00 84.97 89.06 70.99 60.27 54.94 96.62 82.77 72.83 68.31 64.38 80.74 82.51 94.00 84.61 69.60 79.40 29.84 78.33 84.72 96.32 95.96 99.34 89.60 73.18 63.07 90.61 96.15 83.55 90.91 73.85 65.62 98.75 93.92 61.66 96.61 70.57 98.38 96.30 95.49 67.54 66.89 96.82 56.74 80.31 3 24

84.27 81.45 88.62 71.46 60.95 54.80 96.90 80.67 71.00 69.94 60.00 77.41 82.51 94.00 85.18 65.20 78.68 30.18 96.29 83.06 95.37 95.91 99.34 88.56 72.93 71.82 90.96 96.02 83.07 89.80 71.20 53.75 98.76 93.88 61.66 96.49 71.39 97.88 96.48 95.49 60.35 59.51 96.38 55.73 79.66 5 10

87.91 83.84 89.40 71.02 59.73 55.58 96.90 82.17 69.99 71.55 50.00 79.26 82.51 94.00 84.90 69.80 80.49 30.11 96.29 80.56 94.44 95.81 99.33 88.45 74.10 64.38 90.68 95.80 82.10 90.17 72.72 51.08 98.62 93.92 73.65 96.49 70.45 97.47 96.31 95.49 59.66 57.35 96.53 56.88 79.72 7 11

87.91 87.84 89.28 69.40 62.04 54.92 96.06 82.17 69.39 69.94 56.25 79.26 82.69 95.33 84.04 70.40 80.38 30.88 79.77 76.67 94.00 95.76 99.31 88.01 73.70 56.24 90.24 95.45 81.64 89.28 75.70 48.38 98.49 93.75 61.66 96.80 69.75 96.36 96.65 95.49 57.78 56.08 97.11 57.21 78.85 10 15

87.00 83.53 89.09 69.41 62.67 55.61 96.90 82.78 73.82 68.30 65.00 79.63 82.51 94.00 84.61 70.00 78.56 30.26 79.47 84.17 95.87 96.03 99.34 89.95 74.23 62.41 90.77 96.15 82.57 90.97 73.45 68.29 98.78 93.96 61.66 96.58 70.69 98.48 96.48 95.49 67.85 67.48 96.67 57.82 80.44 9 27

84.27 84.01 89.19 68.62 60.90 52.22 96.90 82.75 71.85 69.63 63.13 78.52 81.99 94.00 84.90 70.60 78.43 29.20 81.80 84.72 95.41 96.07 99.30 89.62 73.58 66.32 90.41 96.23 85.52 90.91 74.22 66.29 98.89 93.89 78.69 96.41 69.62 99.09 96.65 95.49 67.23 66.40 96.67 56.54 80.62 4 21

82.36 73.91 87.74 67.25 59.99 53.29 74.85 79.77 71.09 67.01 64.38 78.52 80.26 86.67 79.76 53.80 77.72 29.48 72.52 72.78 91.69 95.45 98.85 86.53 71.62 58.81 88.27 81.04 79.24 86.84 68.96 63.04 88.91 92.78 61.61 95.65 64.54 83.54 95.43 93.79 66.79 66.21 95.34 55.80 76.00 0 1

Finally, Tables A.13–A.16 show the accuracy results obtained when analyzing the four possible order operators for the voting process of EF-kNN-IVFS, following the same format as the previous tables.

154

J. Derrac et al. / Information Sciences 329 (2016) 144–163

Table A.8 Analysis of the EF-kNN-IVFS algorithm (k = 5). Data sets

EF-kNN-IVFS

kNN

Fuzzy-kNN

D-SKNN

IF-KNN

FENN

IT2FKNN

GAfuzzy-kNN

PFKNN

Appendicitis Balance Banana Bands Bupa Cleveland Dermatology Ecoli Glass Haberman Hayes-roth Heart Hepatitis Ionosphere Iris Led7Digit Mammographic Marketing Monk-2 Movement New thyroid Page-blocks Penbased Phoneme Pima Ring Satimage Segment Sonar Spambase Spectfheart Tae Texture Thyroid Titanic Twonorm Vehicle Vowel Wdbc Wine Winequality-red Winequality-white Wisconsin Yeast Average Number of wins Times over the median

86.00 87.84 89.34 69.61 65.44 55.60 96.62 82.49 73.38 67.98 66.88 79.63 86.51 87.17 95.33 71.40 80.12 31.11 91.97 85.56 95.87 96.13 99.37 90.41 74.49 73.73 90.80 96.80 83.10 91.86 73.11 66.33 98.96 93.96 78.65 97.05 69.74 99.39 97.18 94.93 68.23 67.97 97.25 58.90 81.91 20 31

85.91 86.24 89.11 68.46 61.31 55.60 96.33 81.27 66.85 66.95 27.50 80.74 86.27 96.00 85.17 41.40 81.69 29.47 94.75 73.06 93.98 95.83 99.23 87.93 73.06 69.22 90.85 95.15 83.10 89.78 71.97 45.79 98.49 93.99 75.10 96.99 71.75 94.44 96.83 96.05 54.29 50.80 96.95 56.74 78.01 7 10

87.91 88.63 89.19 71.31 62.50 55.97 96.33 82.46 72.57 67.34 65.63 80.37 83.42 96.00 84.04 71.60 80.37 30.79 89.69 82.22 93.98 95.91 99.24 89.36 72.93 60.77 90.55 96.36 81.64 91.15 74.23 66.29 98.53 93.97 75.69 97.01 70.81 97.47 96.65 96.01 67.98 67.52 97.25 58.89 81.10 4 26

85.91 86.56 89.11 68.46 61.31 55.60 96.33 80.97 68.76 67.28 50.63 80.74 86.27 96.00 85.17 70.40 81.34 30.55 94.75 79.17 93.98 95.81 99.24 87.99 73.06 69.22 90.77 95.76 83.10 90.34 71.97 51.75 98.53 93.89 75.69 96.99 71.51 95.15 96.83 96.05 59.10 57.35 96.95 57.01 79.85 3 10

87.91 88.00 89.55 68.40 60.78 56.62 96.06 80.70 68.33 69.26 50.63 80.00 86.27 96.00 84.33 71.60 80.86 30.50 94.52 75.28 93.53 95.54 99.22 87.66 73.06 61.64 90.40 95.28 81.17 90.15 72.35 51.04 98.31 93.79 75.69 96.97 70.34 93.74 97.01 96.05 58.60 57.11 96.81 57.69 79.52 2 11

87.82 89.27 89.72 68.01 61.11 56.91 96.35 81.56 69.06 71.24 58.13 78.52 85.02 96.00 84.88 72.40 80.62 31.25 81.82 71.11 92.58 95.69 99.20 88.29 73.31 52.92 89.82 95.24 80.21 89.52 78.28 48.38 98.22 93.74 75.69 96.92 68.69 93.54 96.83 96.60 59.04 58.02 96.66 59.10 79.26 9 12

87.91 87.83 89.51 69.97 64.01 55.28 96.35 82.17 71.56 67.66 63.75 80.37 83.42 96.00 84.04 71.60 80.25 31.15 84.84 82.22 93.98 95.92 99.25 89.64 73.58 60.34 90.50 96.41 82.14 91.23 73.48 66.29 98.49 93.97 75.83 97.01 69.86 97.58 96.65 96.01 68.35 67.82 97.10 59.50 80.93 3 26

86.09 88.00 89.28 69.63 61.68 53.55 96.35 82.48 73.38 68.95 64.38 78.52 83.42 95.33 87.75 71.20 79.53 29.51 92.42 84.72 95.89 96.05 99.32 89.60 74.36 63.39 90.40 96.54 84.55 91.26 75.00 66.96 98.91 93.93 78.51 96.97 70.33 98.79 96.48 95.49 67.10 66.40 96.96 57.89 81.30 3 25

87.91 81.13 89.42 67.55 62.51 56.30 74.86 80.38 71.08 68.61 70.00 80.00 83.42 96.00 80.90 62.80 80.37 30.57 86.01 74.72 92.16 95.83 98.89 88.93 72.92 56.96 89.73 81.21 78.79 88.69 71.60 67.00 88.89 93.67 71.61 96.65 66.08 88.59 96.30 96.08 68.23 68.11 96.37 59.77 78.58 6 11

J. Derrac et al. / Information Sciences 329 (2016) 144–163

155

Table A.9 Analysis of the EF-kNN-IVFS algorithm (k = 7). Data sets

EF-kNN-IVFS

kNN

Fuzzy-kNN

D-SKNN

IF-KNN

FENN

IT2FKNN

GAfuzzy-kNN

PFKNN

Appendicitis Balance Banana Bands Bupa Cleveland Dermatology Ecoli Glass Haberman Hayes-roth Heart Hepatitis Ionosphere Iris Led7Digit Mammographic Marketing Monk-2 Movement New thyroid Page-blocks Penbased Phoneme Pima Ring Satimage Segment Sonar Spambase Spectfheart Tae Texture Thyroid Titanic Twonorm Vehicle Vowel Wdbc Wine Winequality-red Winequality-white Wisconsin Yeast Average Number of wins Times over the median

87.91 88.64 89.60 69.96 65.75 55.59 96.63 83.07 71.86 69.28 66.25 80.00 85.08 86.89 95.33 71.00 79.65 30.70 86.16 85.56 96.32 96.16 99.38 90.49 73.84 73.27 90.85 96.88 84.57 91.65 77.21 67.71 98.95 93.97 78.38 97.16 70.45 99.39 96.83 95.49 69.29 68.34 97.10 59.31 82.00 21 30

87.91 88.48 89.58 69.75 62.53 56.92 96.34 82.45 66.83 69.90 28.75 79.26 89.19 96.00 84.03 43.40 81.71 29.51 89.16 72.50 92.58 95.47 99.13 87.75 72.93 67.46 90.52 94.81 80.21 89.34 77.58 45.08 98.31 93.99 76.24 97.07 72.34 88.69 97.18 96.63 55.29 50.92 97.25 57.49 78.15 7 16

87.91 88.80 89.21 70.22 63.13 56.28 96.06 82.75 72.57 68.97 63.75 79.63 85.08 95.33 84.32 70.80 80.25 30.65 84.11 80.00 93.98 95.87 99.15 89.34 72.80 59.11 90.24 96.19 82.60 90.76 77.95 66.33 98.33 93.90 78.28 97.01 71.40 96.36 97.01 97.19 68.10 67.89 97.10 59.84 81.06 3 23

87.91 89.12 89.58 69.75 62.53 56.59 96.34 81.86 66.83 70.23 40.00 79.26 89.19 96.00 84.03 70.80 80.50 30.81 89.16 75.56 92.58 95.50 99.12 87.80 72.93 67.46 90.47 95.15 80.21 89.89 77.58 48.42 98.27 93.96 78.06 97.07 72.81 90.00 97.18 96.63 58.91 56.57 97.25 58.02 79.50 6 17

87.82 88.48 89.62 68.71 62.26 55.62 95.78 82.77 67.38 71.55 40.63 78.15 89.19 96.00 84.60 71.20 80.73 30.06 87.34 71.67 92.12 95.30 99.11 87.34 72.54 59.92 89.95 95.06 78.79 89.54 76.81 46.42 98.18 93.82 78.06 97.00 70.57 87.88 97.01 96.63 58.29 56.53 97.25 58.02 78.90 3 10

87.91 88.79 90.11 68.04 61.67 58.24 96.06 80.08 66.74 73.86 54.38 79.63 82.36 94.00 84.31 72.60 80.50 31.48 80.17 66.94 91.65 95.30 99.09 87.71 73.70 51.72 89.50 95.02 73.05 89.45 77.17 51.08 98.05 93.75 78.06 97.09 67.39 90.71 96.30 97.19 59.16 58.76 96.66 58.90 78.73 7 10

87.91 88.80 89.51 69.51 65.21 56.97 96.34 82.45 72.11 68.96 63.75 78.52 84.67 95.33 84.32 71.40 79.54 30.81 82.50 81.11 93.98 95.87 99.14 89.65 73.58 58.84 90.12 96.15 82.14 90.93 77.55 67.00 98.35 93.94 78.15 97.04 71.16 97.07 97.18 97.19 68.73 68.38 96.96 59.91 81.11 2 26

87.91 89.28 89.17 69.29 62.86 54.91 95.78 82.76 73.30 67.33 63.75 78.15 84.32 95.33 87.19 71.00 78.33 29.89 84.33 85.00 95.87 96.20 99.32 89.47 73.83 62.59 90.41 96.32 84.55 91.19 78.69 67.67 98.85 93.85 78.74 97.03 71.16 98.79 96.30 96.05 67.48 66.74 97.25 58.29 81.28 7 21

87.91 85.75 89.57 67.14 65.17 57.27 74.83 81.57 70.75 69.61 68.75 79.26 86.09 95.33 81.74 62.20 80.01 30.75 82.97 74.44 92.58 95.69 98.81 89.30 72.67 55.81 89.57 81.43 77.38 89.19 77.18 67.00 88.85 93.78 61.61 96.92 68.08 90.91 97.36 95.49 69.10 68.74 96.51 60.18 78.76 5 10

156

J. Derrac et al. / Information Sciences 329 (2016) 144–163

Table A.10 Analysis of the EF-kNN-IVFS algorithm (k = 9) Data sets

EF-kNN-IVFS

kNN

Fuzzy-kNN

D-SKNN

IF-KNN

FENN

IT2FKNN

GAfuzzy-kNN

PFKNN

Appendicitis Balance Banana Bands Bupa Cleveland Dermatology Ecoli Glass Haberman Hayes-roth Heart Hepatitis Ionosphere Iris Led7Digit Mammographic Marketing Monk-2 Movement New thyroid Page-blocks Penbased Phoneme Pima Ring Satimage Segment Sonar Spambase Spectfheart Tae Texture Thyroid Titanic Twonorm Vehicle Vowel Wdbc Wine Winequality-red Winequality-white Wisconsin Yeast Average Number of wins Times over the median

86.00 89.28 89.83 69.30 65.13 56.28 95.78 83.33 71.34 71.25 66.25 79.26 82.66 85.75 94.67 71.60 79.63 31.03 85.78 85.00 96.32 96.22 99.38 90.34 73.71 73.34 90.88 96.88 85.05 91.84 78.30 71.00 99.00 93.99 78.69 97.19 70.57 99.39 96.65 96.05 69.36 68.54 97.25 59.91 82.02 16 30

87.91 89.44 89.89 68.88 62.44 55.95 95.78 81.26 66.08 71.55 30.00 80.37 83.61 95.33 84.32 45.20 81.23 30.43 86.20 65.00 92.55 95.47 99.04 86.93 73.19 66.20 90.15 94.72 74.95 88.88 76.79 43.75 98.15 94.00 75.38 97.11 71.40 79.29 96.83 96.08 55.60 51.04 96.95 57.62 77.34 6 11

87.91 88.96 89.42 69.96 66.06 56.95 96.06 83.34 71.66 68.30 63.13 80.74 83.42 94.67 84.03 71.40 80.14 30.65 82.76 78.33 93.98 95.80 99.14 89.64 73.45 57.80 90.15 95.84 81.64 90.71 78.69 67.67 98.27 93.89 78.83 97.14 71.16 96.26 96.83 96.08 67.92 68.27 96.95 59.98 81.00 6 29

87.91 89.44 89.89 68.88 62.44 55.95 95.78 80.96 65.60 71.23 35.63 80.37 83.61 95.33 84.32 73.00 80.73 30.98 86.20 70.83 92.55 95.47 99.03 87.08 73.19 66.20 90.16 94.98 74.95 89.56 76.79 51.00 98.09 93.94 78.19 97.11 71.16 83.74 96.83 96.08 58.98 55.74 96.95 59.57 78.78 4 13

86.00 88.63 89.91 68.56 61.90 55.60 95.50 81.54 65.02 72.56 39.38 80.74 84.61 95.33 84.31 72.60 80.72 30.44 86.20 64.72 92.55 95.25 98.99 86.71 72.93 58.35 89.79 94.72 73.55 89.15 77.15 49.67 97.96 93.76 78.19 97.11 70.56 76.06 96.83 96.08 57.72 55.57 97.10 59.04 78.16 4 10

86.91 88.48 90.23 66.49 62.88 56.92 95.22 80.67 65.68 74.84 53.13 79.26 82.18 95.33 84.32 72.60 79.91 31.60 79.45 64.44 91.15 95.34 98.94 87.58 74.23 51.26 89.28 94.72 71.14 89.36 76.44 51.08 98.00 93.75 78.19 97.18 66.21 90.61 96.65 95.49 59.16 58.62 96.37 59.64 78.43 5 7

87.91 88.95 89.62 70.35 66.92 57.64 96.06 83.64 69.76 68.98 63.13 80.00 84.32 94.67 84.03 71.60 79.29 30.90 81.39 78.89 94.44 95.85 99.14 89.77 74.10 57.49 90.01 95.80 80.26 90.89 78.29 68.33 98.20 93.88 78.33 97.09 70.81 96.26 97.18 96.05 68.98 68.07 96.66 60.25 81.00 6 28

86.09 89.60 89.09 69.22 65.57 57.55 95.49 83.92 70.09 70.30 61.88 77.41 83.00 94.00 87.75 71.40 79.06 29.98 83.21 84.72 96.34 96.05 99.33 89.47 73.18 61.70 90.24 96.32 85.05 91.15 76.82 70.25 98.82 93.90 78.69 97.08 70.81 98.69 97.01 96.60 67.35 66.25 96.96 57.82 81.26 5 23

87.00 87.52 89.55 69.67 65.40 57.31 74.27 83.64 69.74 70.59 69.38 80.74 83.59 94.00 81.47 65.20 80.13 30.82 81.87 72.50 92.55 95.56 98.74 89.49 72.79 54.78 89.48 81.08 70.67 90.30 76.81 67.00 88.73 93.72 66.79 96.97 66.19 92.22 97.18 95.49 69.35 68.56 96.51 60.45 78.77 5 11

J. Derrac et al. / Information Sciences 329 (2016) 144–163

157

Table A.11 Running time (best performing versions). Data sets

EF-kNN-IVFS

kNN

Fuzzy-kNN

D-SKNN

IF-KNN

FENN

IT2FKNN

GAfuzzy-kNN

PFKNN

Appendicitis Balance Banana Bands Bupa Cleveland Dermatology Ecoli Glass Haberman Hayes-roth Heart Hepatitis Ionosphere Iris Led7Digit Mammographic Marketing Monk-2 Movement New Thyroid Page-blocks Penbased Phoneme Pima Ring Satimage Segment Sonar Spambase Spectfheart Tae Texture Thyroid Titanic Twonorm Vehicle Vowel Wdbc Wine Winequality-red Winequality-white Wisconsin Yeast Average

1.26 25.87 1548.21 9.83 6.85 7.17 11.92 7.35 3.62 5.30 1.90 5.19 0.84 9.94 1.89 14.75 39.19 3498.21 10.77 19.24 3.14 2117.05 9488.33 1894.40 36.41 4827.28 4142.37 405.53 6.08 2633.29 6.98 2.00 3314.79 4323.87 301.67 4223.25 50.20 71.37 25.51 2.62 194.47 1658.91 29.15 147.88 1025.81

0.11 0.25 3.43 0.42 0.14 0.21 0.33 0.12 0.10 0.13 0.07 0.22 0.06 0.13 0.23 0.19 0.39 12.30 0.27 0.16 0.14 5.95 35.21 4.11 0.45 17.55 16.59 1.09 0.15 10.53 0.14 0.13 13.11 16.95 0.77 17.24 0.32 0.47 0.39 0.08 0.59 4.43 0.39 0.44 3.78

0.07 0.39 4.82 0.47 0.28 0.35 0.31 0.26 0.18 0.24 0.11 0.33 0.06 0.30 0.13 0.43 0.61 17.66 0.33 0.34 0.18 7.48 61.08 5.56 0.58 29.41 30.37 1.76 0.21 19.17 0.25 0.12 23.21 28.09 1.16 30.03 0.56 0.62 0.46 0.14 0.99 6.26 0.52 0.78 6.29

0.05 0.25 3.19 0.49 0.16 0.14 0.25 0.19 0.10 0.17 0.05 0.15 0.06 0.07 0.39 0.32 0.45 8.79 0.21 0.23 0.11 5.90 19.66 4.26 0.62 11.32 13.49 1.53 0.24 10.26 0.23 0.05 9.56 12.16 0.99 11.19 0.61 0.57 0.45 0.06 0.96 4.44 0.52 0.74 2.86

0.07 0.39 6.32 0.53 0.27 0.39 0.32 0.28 0.15 0.24 0.10 0.31 0.06 0.11 0.33 0.46 0.60 21.68 0.35 0.34 0.16 9.49 63.04 10.01 0.55 32.49 30.74 1.82 0.18 18.98 0.23 0.11 23.79 30.38 1.10 31.21 0.61 0.65 0.45 0.14 0.92 8.96 0.50 0.76 6.83

0.09 0.34 4.32 0.46 0.25 0.32 0.30 0.24 0.16 0.23 0.09 0.34 0.05 0.10 0.29 0.40 0.53 15.80 0.30 0.29 0.17 8.93 62.09 6.04 0.49 27.38 32.10 1.59 0.18 21.28 0.25 0.09 25.38 28.58 1.13 32.33 0.49 0.53 0.40 0.13 0.78 5.28 0.58 0.66 6.40

0.13 0.42 15.16 0.81 0.31 0.35 0.51 0.32 0.26 0.28 0.20 0.42 0.12 0.24 0.63 0.51 0.91 47.93 0.38 0.65 0.30 38.03 125.95 19.15 0.80 45.38 61.25 5.71 0.32 43.80 0.34 0.15 52.00 44.38 2.50 45.59 1.09 1.12 0.68 0.16 2.32 19.26 0.72 1.81 13.26

1.23 25.73 1442.70 14.22 9.33 8.82 18.63 9.37 4.38 6.47 2.12 7.40 0.97 2.07 17.35 17.36 43.96 4136.22 19.89 49.45 5.92 2525.24 11,094.88 1751.10 44.18 5302.07 5770.96 537.58 9.33 4021.70 12.19 2.20 4537.03 5263.68 260.61 5360.67 75.83 93.02 42.81 3.61 214.32 1880.17 35.64 162.45 1246.47

0.12 0.72 104.40 0.87 0.68 0.64 0.30 0.45 0.38 0.48 0.26 0.41 0.12 0.14 0.53 0.87 2.69 16,106.10 0.66 1.63 0.23 117.14 731.86 277.56 3.33 16,862.85 2159.46 14.70 0.33 917.56 0.35 0.21 288.65 1062.26 6.30 659.62 7.44 3.79 0.38 0.15 55.54 2464.20 0.39 40.46 952.21

158

J. Derrac et al. / Information Sciences 329 (2016) 144–163 Table A.12 Analysis of the evolutionary optimization process.

Data sets

IVF-kNN (non optimized)

EF-kNN-IVFS (optimized)

Appendicitis Balance Banana Bands Bupa Cleveland Dermatology Ecoli Glass Haberman Hayes-roth Heart Hepatitis Ionosphere Iris Led7Digit Mammographic Marketing Monk-2 Movement New thyroid Page-blocks Penbased Phoneme Pima Ring Satimage Segment Sonar Spambase Spectfheart Tae Texture Thyroid Titanic Twonorm Vehicle Vowel Wdbc Wine Winequality-red Winequality-white Wisconsin Yeast Average Number of wins

87.00 88.95 89.17 69.52 63.23 56.65 96.34 82.48 73.50 68.96 65.00 80.00 84.67 84.32 94.67 71.00 79.88 30.92 82.52 82.50 95.87 96.13 99.20 90.03 73.06 58.86 90.32 96.49 82.62 91.30 76.05 65.71 98.40 94.00 78.06 97.03 72.23 97.78 97.18 96.60 68.85 68.38 96.96 59.50 81.18 13

86.00 89.28 89.83 69.30 65.13 56.28 95.78 83.33 71.34 71.25 66.25 79.26 82.66 85.75 94.67 71.60 79.63 31.03 85.78 85.00 96.32 96.22 99.38 90.34 73.71 73.34 90.88 96.88 85.05 91.84 78.30 71.00 99.00 93.99 78.69 97.19 70.57 99.39 96.65 96.05 69.36 68.54 97.25 59.91 82.02 32

J. Derrac et al. / Information Sciences 329 (2016) 144–163

159

Table A.13 Study on the accuracy of the order operators (k = 3). Data sets

Xu

Lex-Up

Lex-Down

K-Order

Appendicitis Balance Banana Bands Bupa Cleveland Dermatology Ecoli Glass Haberman Hayes-roth Heart Hepatitis Ionosphere Iris Led7Digit Mammographic Marketing Monk-2 Movement New thyroid Page-blocks Penbased Phoneme Pima Ring Satimage Segment Sonar Spambase Spectfheart Tae Texture Thyroid Titanic Twonorm Vehicle Vowel Wdbc Wine Winequality-red Winequality-white Wisconsin Yeast Average Number of wins

0.8427 0.8416 0.8925 0.7123 0.6086 0.5558 0.9633 0.8217 0.7328 0.6963 0.6500 0.7926 0.7983 0.8460 0.9400 0.7080 0.7905 0.3046 0.8362 0.8639 0.9677 0.9627 0.9931 0.9023 0.7410 0.7189 0.9078 0.9654 0.8457 0.9134 0.7308 0.6696 0.9898 0.9403 0.7883 0.9654 0.7021 0.9929 0.9665 0.9549 0.6704 0.6750 0.9667 0.5789 0.8115 11

0.8327 0.8449 0.8930 0.7123 0.6201 0.5490 0.9633 0.8336 0.7230 0.6930 0.6563 0.7852 0.8108 0.8489 0.9400 0.7100 0.7867 0.3005 0.8406 0.8500 0.9723 0.9618 0.9933 0.9004 0.7384 0.7181 0.9094 0.9632 0.8455 0.9139 0.7308 0.6629 0.9902 0.9396 0.7883 0.9654 0.7033 0.9909 0.9665 0.9549 0.6773 0.6727 0.9653 0.5816 0.8114 13

0.8418 0.8416 0.8926 0.7185 0.6119 0.5664 0.9604 0.8307 0.7366 0.7061 0.6625 0.7926 0.8251 0.8632 0.9400 0.7160 0.7916 0.3035 0.8433 0.8611 0.9680 0.9598 0.9934 0.9038 0.7436 0.7377 0.9088 0.9680 0.8600 0.9126 0.7345 0.6767 0.9896 0.9399 0.7883 0.9661 0.7021 0.9939 0.9665 0.9549 0.6742 0.6731 0.9682 0.5775 0.8151 25

0.8427 0.8480 0.8930 0.7071 0.6151 0.5593 0.9633 0.8218 0.7331 0.6898 0.6500 0.7926 0.7983 0.8460 0.9400 0.7100 0.7904 0.3045 0.8384 0.8722 0.9680 0.9627 0.9934 0.9030 0.7475 0.7186 0.9091 0.9658 0.8507 0.9117 0.7308 0.6696 0.9898 0.9401 0.7883 0.9657 0.7045 0.9929 0.9630 0.9549 0.6785 0.6746 0.9667 0.5755 0.8123 14

160

J. Derrac et al. / Information Sciences 329 (2016) 144–163 Table A.14 Study on the accuracy of the order operators (k = 5). Data sets

Xu

Lex-Up

Lex-Down

K-Order

Appendicitis Balance Banana Bands Bupa Cleveland Dermatology Ecoli Glass Haberman Hayes-roth Heart Hepatitis Ionosphere Iris Led7Digit Mammographic Marketing Monk-2 Movement New thyroid Page-blocks Penbased Phoneme Pima Ring Satimage Segment Sonar Spambase Spectfheart Tae Texture Thyroid Titanic Twonorm Vehicle Vowel Wdbc Wine Winequality-red Winequality-white Wisconsin Yeast Average Number of wins

0.8509 0.8768 0.8949 0.7023 0.6459 0.5560 0.9605 0.8246 0.7152 0.6831 0.6375 0.8000 0.8399 0.8603 0.9533 0.7120 0.8012 0.3055 0.9129 0.8611 0.9543 0.9620 0.9933 0.9030 0.7449 0.7147 0.9086 0.9667 0.8360 0.9145 0.7460 0.6896 0.9902 0.9401 0.7865 0.9703 0.6987 0.9939 0.9718 0.9493 0.6817 0.6742 0.9725 0.5869 0.8169 14

0.8509 0.8768 0.8938 0.7071 0.6606 0.5559 0.9633 0.8246 0.7162 0.6862 0.6438 0.7963 0.8233 0.8603 0.9533 0.7120 0.8000 0.3048 0.9175 0.8444 0.9543 0.9622 0.9936 0.9019 0.7449 0.7059 0.9071 0.9671 0.8214 0.9156 0.7460 0.6696 0.9891 0.9403 0.7865 0.9697 0.6998 0.9919 0.9683 0.9549 0.6835 0.6780 0.9710 0.5863 0.8159 9

0.8600 0.8784 0.8934 0.6961 0.6544 0.5560 0.9662 0.8249 0.7338 0.6798 0.6688 0.7963 0.8651 0.8717 0.9533 0.7140 0.8012 0.3111 0.9197 0.8556 0.9587 0.9613 0.9937 0.9041 0.7449 0.7373 0.9080 0.9680 0.8310 0.9186 0.7311 0.6633 0.9896 0.9396 0.7865 0.9705 0.6974 0.9939 0.9718 0.9493 0.6823 0.6797 0.9725 0.5890 0.8191 25

0.8509 0.8784 0.8942 0.6999 0.6491 0.5460 0.9633 0.8277 0.7241 0.6799 0.6500 0.8111 0.8324 0.8517 0.9533 0.7100 0.8011 0.3100 0.9129 0.8472 0.9539 0.9627 0.9931 0.9028 0.7436 0.7161 0.9088 0.9667 0.8360 0.9154 0.7272 0.6829 0.9896 0.9404 0.7865 0.9701 0.7010 0.9939 0.9665 0.9493 0.6823 0.6799 0.9725 0.5896 0.8165 14

J. Derrac et al. / Information Sciences 329 (2016) 144–163

161

Table A.15 Study on the accuracy of the order operators (k = 7). Data sets

Xu

Lex-Up

Lex-Down

K-Order

Appendicitis Balance Banana Bands Bupa Cleveland Dermatology Ecoli Glass Haberman Hayes-roth Heart Hepatitis Ionosphere Iris Led7Digit Mammographic Marketing Monk-2 Movement New thyroid Page-blocks Penbased Phoneme Pima Ring Satimage Segment Sonar Spambase Spectfheart Tae Texture Thyroid Titanic Twonorm Vehicle Vowel Wdbc Wine Winequality-red Winequality-white Wisconsin Yeast Average Number of wins

0.8700 0.8880 0.8964 0.6882 0.6629 0.5529 0.9606 0.8276 0.7148 0.7026 0.6438 0.7963 0.8342 0.8490 0.9533 0.7100 0.7989 0.3083 0.8571 0.8500 0.9587 0.9622 0.9932 0.9039 0.7436 0.7104 0.9077 0.9649 0.8407 0.9143 0.7795 0.6767 0.9902 0.9394 0.7838 0.9711 0.7092 0.9929 0.9665 0.9549 0.6854 0.6807 0.9682 0.5957 0.8172 10

0.8700 0.8864 0.8958 0.6985 0.6543 0.5592 0.9579 0.8244 0.7140 0.6994 0.6438 0.8000 0.8342 0.8602 0.9533 0.7120 0.7963 0.3070 0.8525 0.8500 0.9680 0.9620 0.9934 0.9027 0.7397 0.7147 0.9075 0.9662 0.8455 0.9158 0.7796 0.6767 0.9898 0.9396 0.7838 0.9709 0.7069 0.9919 0.9665 0.9608 0.6886 0.6782 0.9682 0.5950 0.8178 6

0.8791 0.8864 0.8960 0.6996 0.6575 0.5559 0.9663 0.8307 0.7186 0.6928 0.6625 0.8000 0.8508 0.8689 0.9533 0.7100 0.7965 0.3070 0.8616 0.8556 0.9632 0.9616 0.9938 0.9049 0.7384 0.7327 0.9085 0.9688 0.8457 0.9165 0.7721 0.6771 0.9895 0.9397 0.7838 0.9716 0.7045 0.9939 0.9683 0.9549 0.6929 0.6834 0.9710 0.5931 0.8200 22

0.8609 0.8864 0.8957 0.6969 0.6600 0.5662 0.9579 0.8336 0.7152 0.7025 0.6563 0.8037 0.8199 0.8432 0.9533 0.7100 0.7916 0.3109 0.8593 0.8583 0.9587 0.9624 0.9935 0.9030 0.7410 0.7219 0.9096 0.9667 0.8407 0.9134 0.7795 0.6833 0.9898 0.9392 0.7838 0.9716 0.7104 0.9919 0.9683 0.9549 0.6886 0.6852 0.9696 0.5951 0.8183 14

162

J. Derrac et al. / Information Sciences 329 (2016) 144–163 Table A.16 Study on the accuracy of the order operators (k = 9). Data sets

Xu

Lex-Up

Lex-Down

K-Order

Appendicitis Balance Banana Bands Bupa Cleveland Dermatology Ecoli Glass Haberman Hayes-roth Heart Hepatitis Ionosphere Iris Led7Digit Mammographic Marketing Monk-2 Movement New thyroid Page-blocks Penbased Phoneme Pima Ring Satimage Segment Sonar Spambase Spectfheart Tae Texture Thyroid Titanic Twonorm Vehicle Vowel Wdbc Wine Winequality-red Winequality-white Wisconsin Yeast Average Number of wins

0.8518 0.8912 0.8985 0.7034 0.6399 0.5691 0.9578 0.8304 0.7198 0.7158 0.6500 0.8111 0.8199 0.8489 0.9400 0.7180 0.7989 0.3098 0.8345 0.8444 0.9634 0.9620 0.9934 0.9030 0.7357 0.7141 0.9071 0.9658 0.8307 0.9145 0.7754 0.7029 0.9904 0.9400 0.7869 0.9708 0.7081 0.9919 0.9701 0.9660 0.6848 0.6799 0.9681 0.5991 0.8177 11

0.8700 0.8896 0.8979 0.6988 0.6604 0.5658 0.9607 0.8363 0.7051 0.7158 0.6438 0.8111 0.8199 0.8517 0.9400 0.7140 0.7964 0.3060 0.8368 0.8528 0.9680 0.9616 0.9933 0.9023 0.7410 0.7070 0.9083 0.9675 0.8262 0.9158 0.7829 0.7029 0.9898 0.9397 0.7869 0.9703 0.7104 0.9919 0.9683 0.9660 0.6879 0.6793 0.9667 0.5964 0.8183 11

0.8600 0.8928 0.8983 0.6930 0.6513 0.5628 0.9578 0.8333 0.7134 0.7125 0.6625 0.7926 0.8266 0.8575 0.9467 0.7160 0.7963 0.3103 0.8578 0.8500 0.9632 0.9622 0.9938 0.9034 0.7371 0.7334 0.9088 0.9688 0.8505 0.9184 0.7830 0.7100 0.9900 0.9399 0.7869 0.9719 0.7057 0.9939 0.9665 0.9605 0.6936 0.6854 0.9725 0.5991 0.8202 20

0.8518 0.8896 0.8977 0.6930 0.6460 0.5596 0.9579 0.8333 0.7243 0.7158 0.6438 0.7963 0.8199 0.8603 0.9467 0.7200 0.7964 0.3105 0.8348 0.8528 0.9587 0.9618 0.9930 0.9041 0.7344 0.7165 0.9080 0.9675 0.8164 0.9147 0.7755 0.7029 0.9898 0.9394 0.7869 0.9709 0.7140 0.9919 0.9683 0.9549 0.6904 0.6801 0.9710 0.6011 0.8173 11

J. Derrac et al. / Information Sciences 329 (2016) 144–163

163

References [1] J. Alcalá-Fdez, A. Fernández, J. Luengo, J. Derrac, S. García, L. Sánchez, F. Herrera, Keel data-mining software tool: data set repository, integration of algorithms and experimental analysis framework, J. Multiple Valued Logic Soft Comput. 17 (2–3) (2011) 255–287. [2] J. Alcalá-Fdez, L. Sánchez, S. García, M.J. del Jesus, S. Ventura, J.M. Garrell, J. Otero, C. Romero, J. Bacardit, V.M. Rivas, J.C. Fernández, F. Herrera, KEEL: a software tool to assess evolutionary algorithms for data mining problems, Soft Comput. 13 (3) (2008) 307–318. [3] M. Arif, M.U. Akram, F.A. Afsar, F.A.A. Minhas, Pruned fuzzy k-nearest neighbor classifier for beat classification, J. Biomed. Sci. Eng. 3 (2010) 380–389. [4] H. Bustince, Indicator of inclusion grade for interval-valued fuzzy sets. Application to approximate reasoning based on interval-valued fuzzy sets, Int. J. Approx. Reason. 23 (3) (2000) 137–209. [5] H. Bustince, J. Fernández, H. Hagras, F. Herrera, M. Pagola, E. Barrenechea, Interval type-2 fuzzy sets are generalization of interval-valued fuzzy sets: towards a wider view on their relationship, IEEE Trans. Fuzzy Syst. (2015). in press. [6] H. Bustince, J. Fernández, A. Kolesárová, R. Mesiar, Generation of linear orders for intervals by means of aggregation functions, Fuzzy Sets Syst. 220 (1) (2013) 69–77. [7] H. Bustince, M. Galar, B. Bedregal, A. Kolesárová, R. Mesiar, A new approach to interval-valued Choquet integrals and the problem of ordering in intervalvalued fuzzy sets applications, IEEE Trans. Fuzzy Syst. 21 (2013) 1150–1162. [8] H. Bustince, M. Pagola, E. Barrenechea, P. Melo-Pinto, P. Couto, H. Tizhoosh, J. Montero, Ignorance functions. An application to the calculation of the threshold in prostate ultrasound images, Fuzzy Sets Syst. 161 (1) (2010) 20–36. [9] D. Cabello, S. Barro, J. Salceda, R. Ruiz, J. Mira, Fuzzy k-nearest neighbor classifiers for ventricular arrhythmia detection, Int. J. Biomed. Comput. 27 (1991) 77–93. [10] J.R. Cano, F. Herrera, M. Lozano, Using evolutionary algorithms as instance selection for data reduction in KDD: an experimental study, IEEE Trans. Evol. Comput. 7 (6) (2003) 561–575. [11] H.-L. Chen, B. Yang, G. Wang, J. Liu, X. Xu, S. Wang, D. Liu, A novel bankruptcy prediction model based on an adaptive fuzzy k-nearest neighbor method, Knowl. Based Syst. 24 (8) (2011) 1348–1359. [12] M.A. Chikh, M. Saidi, N. Settouti, Diagnosis of diabetes diseases using an artificial immune recognition system2 (AIRS2) with fuzzy k-nearest neighbor, J. Med. Syst. 36 (2012) 2721–2729. [13] F. Chung-Hoon, C. Hwang, An interval type-2 fuzzy k-nearest neighbor, in: Proceedings of the 12th IEEE International Conference on Fuzzy Systems (FUZZIEEE’03), St. Louis, Missouri, USA, May 25–28 (2003), 2003, pp. 802–807. [14] O. Cordón, F. Gomide, F. Herrera, F. Hoffmann, L. Magdalena, Ten years of genetic fuzzy systems: current framework and new trends, Fuzzy Sets Syst. 141 (1) (2004) 5–31. [15] O. Cordón, F. Herrera, F. Hoffmann, L. Magdalena, Genetic fuzzy systems: evolutionary tuning and learning of fuzzy knowledge bases, Advances in Fuzzy Systems—Applications and Theory, World Scientific, 2001. [16] T.M. Cover, P.E. Hart, Nearest neighbor pattern classification, IEEE Trans. Inf. Theory 13 (1) (1967) 21–27. [17] T. Denoeux, A k-nearest neighbor classification rule based on Dempster–Shafer theory, IEEE Trans. Syst. Man Cybern. 25 (5) (1995) 804–813. [18] J. Derrac, C. Cornelis, S. García, F. Herrera, Enhancing evolutionary instance selection algorithms by means of fuzzy rough set based feature selection, Inf. Sci. 186 (2012) 73–92. [19] J. Derrac, S. García, F. Herrera, Fuzzy nearest neighbor algorithms: taxonomy, experimental analysis and prospects, Inf. Sci. 260 (2014) 98–119. [20] D. Dubois, H. Prade, An introduction to bipolar representations of information and preference, Int. J. Intell. Syst. 23 (8) (2008) 866–877. [21] L.J. Eshelman, The CHC adaptative search algorithm: how to have safe search when engaging in nontraditional genetic recombination, in: G.J.E. Rawlins (Ed.), Foundations of Genetic Algorithms, Morgan Kaufmann, San Mateo, California, 1991, pp. 265–283. [22] L.J. Eshelman, J.D. Schaffer, Real-coded genetic algorithms and interval-schemata, in: Proceedings of the Second Workshop on Foundations of Genetic Algorithms. Vail, Colorado, USA, July 26–29, 1992, pp. 187–202. [23] M. Friedman, The use of ranks to avoid the assumption of normality implicit in the analysis of variance, J. Am. Stat. Assoc. 32 (1937) 674–701. [24] S. García, A. Fernández, J. Luengo, F. Herrera, Advanced nonparametric tests for multiple comparisons in the design of experiments in computational intelligence and data mining: experimental analysis of power, Inf. Sci. 180 (2010) 2044–2064. [25] S. García, F. Herrera, An extension on statistical comparisons of classifiers over multiple data sets for all pairwise comparisons, J. Mach. Learn. Res. 9 (2008) 2677–2694. [26] S. Greenfield, F. Chiclana, S. Coupland, R.I. John, The collapsing method of defuzzification for discretised interval type-2 fuzzy sets, Inf. Sci. 179 (13) (2009) 2055–2069. [27] M. Hanss, Applied Fuzzy Arithmetic. An Introduction with Engineering Applications, Springer-Verlag, 2005. [28] S. Holm, A simple sequentially rejective multiple test procedure, Scand. J. Stat. 6 (1979) 65–70. [29] X. Hu, C. Xie, Improving fuzzy k-nn by using genetic algorithm, J. Comput. Inf. Syst. 1 (2) (2005) 203–213. [30] Y. Huang, Y. Li, Prediction of protein subcellular locations using fuzzy k-nn method, Bioinformatics 20 (1) (2004) 21–28. [31] J.M. Keller, M.R. Gray, J.A. Givens, A fuzzy k-nearest neighbor algorithm, IEEE Trans. Syst. Man Cybern. 15 (4) (1985) 580–585. [32] L.I. Kuncheva, An intuitionistic fuzzy k-nearest neighbors rule, Notes Intuitionistic Fuzzy Sets 1 (1995) 56–60. [33] T. Liao, D. Li, Two manufacturing applications of the fuzzy k-nn algorithm, Fuzzy Sets Syst. 92 (1997) 289–303. [34] J.M. Mendel, Uncertain Rule-based Fuzzy Logic Systems: Introduction and New Directions, Prentice-Hall PTR, 2001. [35] A.N. Papadopoulos, Y. Manolopoulos, Nearest Neighbor Search: A Database Perspective, Springer-Verlag Telos, 2004. [36] J.A. Sanz, A. Fernández, H. Bustince, F. Herrera, Improving the performance of fuzzy rule-based classification systems with interval-valued fuzzy sets and genetic amplitude tuning, Inf. Sci. 180 (19) (2010) 3674–3685. [37] J.A. Sanz, A. Fernández, H. Bustince, F. Herrera, A genetic tuning to improve the performance of fuzzy rule-based classification systems with interval-valued fuzzy sets: degree of ignorance and lateral position, Int. J. Approx. Reason. 52 (6) (2011) 751–766. [38] J.A. Sanz, A. Fernández, H. Bustince, F. Herrera, IVTURS: a linguistic fuzzy rule-based classification system based on a new interval-valued fuzzy reasoning method with tuning and rule selection, IEEE Trans. Fuzzy Syst. 21 (3) (2013) 399–411. [39] G. Shakhnarovich, T. Darrell, P. Indyk (Eds.), Nearest-neighbor Methods in Learning and Vision: Theory and Practice, The MIT Press, 2006. [40] F. Wilcoxon, Individual comparisons by ranking methods, Biometrics Bull. 1 (6) (1945) 80–83. [41] D.L. Wilson, Asymptotic properties of nearest neighbor rules using edited data, IEEE Trans. Syst. Man Cybern. 2 (3) (1972) 408–421. [42] X. Wu, V. Kumar (Eds.), Data Mining and Knowledge Discovery, The Top Ten Algorithms in Data Mining, Chapman & Hall/CRC, 2009. [43] M. Yang, C. Cheng, On the edited fuzzy k-nearest neighbor rule, IEEE Trans. Syst. Man Cybern. Part B: Cybern. 28 (3) (1998) 461–466. [44] L.M. Zouhal, T. Denoeux, An evidence-theoretic k-nn rule with parameter optimization, IEEE Trans. Syst. Man Cybern. Part C 28 (2) (1998) 263–271.