A hybrid modified step Whale Optimization Algorithm with Tabu Search for data clustering

Journal Pre-proofs A Hybrid Modified Step Whale Optimization Algorithm with Tabu Search for Data Clustering Amr Mohamed AbdelAziz, Taysir Hassan A. Soliman, Kareem Kamal A.Ghany, Adel Abu El-Magd Sewisy PII: DOI: Reference:

S1319-1578(19)31154-1 https://doi.org/10.1016/j.jksuci.2020.01.015 JKSUCI 742

To appear in:

Journal of King Saud University - Computer and Information Sciences

Received Date: Revised Date: Accepted Date:

7 September 2019 2 January 2020 28 January 2020

Please cite this article as: AbdelAziz, A.M., Soliman, T.H.A., Kamal A.Ghany, K., El-Magd Sewisy, A.A., A Hybrid Modified Step Whale Optimization Algorithm with Tabu Search for Data Clustering, Journal of King Saud University - Computer and Information Sciences (2020), doi: https://doi.org/10.1016/j.jksuci.2020.01.015

This is a PDF file of an article that has undergone enhancements after acceptance, such as the addition of a cover page and metadata, and formatting for readability, but it is not yet the definitive version of record. This version will undergo additional copyediting, typesetting and review before it is published in its final form, but we are providing this version to give early visibility of the article. Please note that, during the production process, errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

© 2020 Production and hosting by Elsevier B.V. on behalf of King Saud University.

A Hybrid Modified Step Whale Optimization Algorithm with Tabu Search for Data Clustering

Abstract Clustering is a popular data analysis tool that groups similar data objects into disjoint clusters. Clustering algorithms have major drawbacks, such as: sensitivity to initialization and getting trapped in local optima. Swarm Intelligence (SI ) have been combined with clustering techniques to enhance their performance. Despite their improvements, some SI methods, such as Particle Swarm Optimization (PSO ) and its rivals, had their drawbacks, such as low convergence rates and generating low quality solutions. This is due to storing a single best solution about solution space, which can lead swarm members to local optima. Whale Optimization Algorithm (WOA) is a recent SI method, which has been proved as a global optimizer over multiple optimization problems. Unlikely, WOA uses the same concept as PSO, it uses the best solution to reposition swarm members. To overcome this problem, WOA needs to store multiple best solutions about solution space. Tabu Search (TS ) is a meta-heuristic method, which uses memory components to explore and exploit search space. In this paper, we propose combining WOA with TS (WOATS ) for data clustering. To assess the efficiency of WOATS, it has been applied in data clustering. WOATS uses an objective function inspired by partitional clustering to maintain the quality of clustering solutions. WOATS was tested over multiple real life datasets, which their sizes vary from small to large-sized datasets. WOATS was able to find centers with high quality in a small number of iterations regardless of the size of the dataset, which clarifies its ability to cover the solution space efficiently. The generated clusters were evaluated according to different outlier criteria: Davies-Bouldin Index and Dunn Index. According to these criteria,

Preprint submitted to Elsevier

January 30, 2020

WOATS presented its superiority over multiple recent original and hybrid SI methods for different datasets. Results obtained by WOATS were better than the results of the other methods with big differences regarding to the different outlier criteria, which ensured the efficiency of WOATS. Keywords: Data Clustering, Computational Intelligence, Data Mining, Swarm Intelligence, Tabu Search, Swarm Intelligence, Whale Optimization Algorithm

1. Introduction Clustering has been an important data analysis tool, used in different fields, such as network intrusion detection [1], and bioinformatics [2]. Clustering methods can be classified into hierarchical clustering and partitional clustering meth5

ods [3]. During past decades, partitional clustering methods experienced certain problems, such as sensitivity to the initial center points, long execution time, and convergence to local optima [4]. To mitigate these drawbacks, researchers paid attention to the combination of Swarm Intelligence (SI ) methods with traditional clustering algorithms [4].

10

SI methods are population-based meta-heuristics, which mimic the intelligent behavior of organisms that live into groups [4]. Many SI methods have been proposed for data clustering, such as Glowworm based clustering [5] and Crow Search Algorithm (CSA) for data clustering [6]. These methods represented clustering solutions as organisms in the swarm. Then, these methods employ

15

intelligent intensification and diversification techniques to cover search space in real time. Although SI methods minimized the execution time of traditional clustering techniques, they had their own limitations. Considering methods, such as Particle Swarm Optimization (PSO ) and its rivals, the limitations of these methods can be summarized into: lack the diversity of swarm members

20

[7] and lack mature memory elements. Lacking diversity of swarm members leads PSO and its rivals to premature convergence [7]. Second, PSO and its rivals store a single best solution within solution space and uses it to re-positions 2

swarm members, which can lead them to get trapped in local optima. These drawbacks led PSO and its rivals to achieve low convergence rates and obtain 25

solutions with low quality [8], which explains the emergence of many new and hybrid SI methods in the last two decades [9]. Tabu Search (TS ) is a single-point meta-heuristic method proposed by Glover [10]. It uses mature memory elements, such as Elite List (EL) and Tabu List (TL) to cover search space economically. Combining TS memory elements with

30

an SI method can enhance the performance of the resulted method. Whale Optimization Algorithm (WOA) is an efficient SI method, proposed by Mirjalili et al. [11]. It mimicked the behavior of the humpback whales to heuristically locate the extreme points of arbitrary functions. It was tested over multiple engineering optimization problems and presented a good performance [11]. The

35

WOA is a simple algorithm as it uses a small number of parameters to control the search process, which makes it a good candidate for new modifications. In this paper, a novel hybrid algorithm that combines WOA [11] with TS [10] (WOATS ) is proposed for data clustering. The attempt to combine these two algorithms is new and has not been investigated before. WOATS will be used

40

to find centers of clusters with the best objective values. WOATS uses an objective function inspired by partitional clustering methods to ensure obtaining centers with high quality. WOATS uses the EL to keep a record of the best candidate solutions. These solutions are used to guide the swarm members to avoid getting trapped into local optima. WOATS improves the diversity of the

45

swarm members by employing the crossover operator inspired by genetic algorithms [12] on swarm members. This increases the ability of WOATS to achieve faster convergence rates. Also, WOATS ensures obtaining solutions with high quality by applying crossover among solutions in the EL and swarm members. These modifications enable WOATS to converge in a smaller number of iter-

50

ations than other similar algorithms. WOATS provides the balance between global and local search by proposing a modified diversification step, which focuses on searching neighbors of swarm members instead of neighbors of the best solution. This removes the need of WOATS for a local search method. 3

To quantify the efficiency of WOATS, it has been tested over real life public 55

datasets, available in the UCI repository [13]. These datasets have different sizes, which vary from small to large size datasets. WOATS showed its superiority over multiple original and hybrid SI methods according to several outliers evaluation criteria, such as purity [14], the Davies-Bouldin index [8], and the silhouette index [8].

60

The rest of the paper is organized as follows. Section 2 presents the previously proposed methods to solve data clustering. Section 3 describes both WOA and TS used in the WOATS algorithm. It gives a glance at how these algorithms work. Section 4 gives a detailed description of the proposed method. It describes the steps and the internal functions of WOATS. Section 5 reports

65

a performance analysis for WOATS. Section 6 summarizes the main points of the paper and introduces the future work.

2. Related Work Partitional clustering algorithms have been the most common methods for data clustering [3]. This returns to their simplicity to be implemented. Par70

titional clustering algorithms have certain drawbacks, which are related to selection of initial center points and getting entrapped in local optima [4]. To overcome these problems, SI methods were proposed to be combined with partitional clustering algorithms [4]. Many original SI methods have been proposed for data clustering [5], [15].

75

Despite the improved speed of SI heuristics over traditional clustering methods, SI heuristics had their own drawbacks. Taking the PSO and its rivals as an example, they need a long time to converge to global optima and generate low quality solutions [8]. These drawbacks, as explained before, are due to the lack of the diversity of swarm members [16] and the lack of mature memory

80

elements. When analyzing the WOA, we can notice that WOA can be considered as another variation of PSO because it uses the best solution to reposition swarm 4

members. This makes WOA shares the drawbacks of PSO as explained before. WOA has another drawback in the diversification technique because of WOA 85

repositions the swarm members in the neighborhood of the prey. This leads the swarm members to miss exploring their neighboring regions, which increases the need for local search methods. This problem was noticed by Abdel-Basset et al. [17] and Mohammed et al. [18]. In the first method, Abdel-Basset et al. [17] proposed combining a local search strategy with WOA for the permutation

90

flow shop scheduling problem. In the second method, Mohammed et al. [18] proposed combining the Bat algorithm to be used in the diversification phase, while the WOA was used in the intensification phase. These methods support our analysis for the drawbacks of WOA. Also, the WOA was proposed for data clustering by Nasiri et al. [15] but they did not modify or add new components

95

to the original algorithm, which ensures the novelty of the proposed method. 3. Background In the following subsections, brief descriptions are given for the partitional clustering problem, TS and WOA algorithms. 3.1. Partitional Clustering Partitional clustering algorithms attempt to decompose the dataset into disjoint clusters [3] based on a distance function and an objective function. The distance function is used to identify the similarity among data objects, while the objective function is used to measure the quality of obtained clusters. A mathematical description of the search problem can be given by assuming that D represents the entire dataset, n is the number of data objects, and d is the number of features, i.e., D = {x1 , x2 , . . . , xn }. Partitional clustering methods try to find the best set of centers C = C 1 , C 2 , . . . , C K that partition the dataset D into K disjoint clusters. Here, the Euclidean distance is used as the distance function. The objective function is used to evaluate the quality of the obtained

5

set of centers, which can be computed as:

𝐹=

100

1 𝑛

𝑛

(1)

∑𝑖 = 1min 𝑑 (𝐶𝑗, 𝑥𝑖) 𝑗

where Cj denotes class j and d(Cj , xi) corresponds to the squared Euclidean distance between the point xi and the center of class Cj . SI methods use the objective function (1) to evaluate the quality of each solution (set of centers). 3.2. Tabu Search TS is a single-point meta-heuristic method, proposed by Glover [10] as a

105

global optimizer. TS starts with a single random solution and uses a local search to cover its vicinity. If a better solution is found, the starting solution will be replaced with the best one. TS uses the EL to provide a thorough coverage of promising regions. Conversely, TL is used to avoid pre-visited regions, which ensures an economic coverage of the search space.

110

The main drawback of TS is that it uses a single point to explore the search space. A single point does not provide an effective exploration for the search space in real time, so TS needs a longer time to converge than SI methods. Enhancing TS performance can be implemented by combining TS with an SI method to ensure faster convergence rates.

115

3.3. Whale Optimization Algorithm WOA is an SI method, proposed by Mirjalili et al. [11] to mimic the social behavior of the humpback whale. It uses two methods for exploration: the first → − diversification method selects a random whale X rand (t) from the current swarm to be the prey and remaining whales reposition themselves in its neighborhood according to: a = 2 – t (2/Max Iteration); A = 2ar − a; 𝐷 = | 𝐶 ∙ 𝑋𝑟𝑎𝑛𝑑 (𝑡) ― 𝑋(𝑡)|, 𝑋(𝑡 + 1) = 𝑋𝑟𝑎𝑛𝑑 (𝑡) ― 𝐴 ∙ 𝐷 , 6

C = 2r,

(2) (3) (4)

where r represents a random number, t is the current iteration number, Max Iteration represents the maximum number of iterations, a is a parameter that has a decreasing value in the range of [0, 2], | | represents the element-wise absolute → − → − value of a vector, D is the distance between current swarm member X (t) and → − the random whale X rand (t), C is a controlling coefficient, and (·) represents an element by element multiplication. The C parameter is used to reposition swarm members in a cubic area surrounding the prey. The second diversifica→ − tion method considers the best solution to be the prey X ∗ (t) and the remaining whales reposition themselves in its neighborhood according to:

𝐷 = | 𝐶 ∙ 𝑋 ∗ (𝑡) ― X(𝑡)|,

(5)

∗

𝑋(t + 1) = 𝑋 (𝑡) ― A ∙ D ,

(6)

→ − where X ∗ (t) is the best solution at iteration t and the variables C, A are given by (2). WOA chooses between the two diversification methods based on the absolute value of A parameter.

→ For the intensification phase, WOA considers the best solution X ∗ (t) to be → − the prey and develops a spiral equation between the current whale X (t) and the prey to mimic the helix-shaped movement of the humpback whales as follows:

𝐷 = | 𝑋 ∗ (𝑡) ― X (t)|,

(7)

𝑋(t + 1) = 𝐷 ∙ 𝑒𝑏𝑙 ∙ cos (2𝜋𝑙) + 𝑋 ∗ (𝑡) ,

(8)

→ − where t is the iteration number, D is the distance between current swarm mem120

ber and the best solution, b is a constant that represents the shape of the logarithmic spiral path and l is a random number in the range of [−1, 1]. WOA chooses between the exploration and exploitation phases randomly according to the value of the p parameter, which has a value in the range [0, 1].  − X (t + 1) =

𝑋 ∗ (𝑡) ― A ∙ D 𝐷 ∙ 𝑒𝑏𝑙 ∙ cos (2𝜋𝑙) + 𝑋 ∗ (𝑡) 7

if p < 0.5 if p ≥ 0.5.

(9)

Figure 1: A depiction of WOATS framework.

p is a random number in range [0, 1]. Alternating the two phases ensures the balance between intensification and diversification techniques, applying each 50% of the time. Overall, the structure and operations of WOA are simple, which 125

facilitates its enhancement.

4. Whale Optimization Algorithm combined with Tabu Search 4.1. WOATS Algorithm and its Components WOATS combines TS with WOA to take advantage of efficient search of 130

the solution space in data clustering. In WOATS, each whale in the swarm represents a complete solution (a set of centers of the clusters). These centers are selected based on the best value of the objective function (1). A block diagram of the main steps of WOATS algorithm is given in Figure 1 for reference. The next subsections present the details of the main components of WOATS, such as

135

intensification by crossover, diversification by crossover, modified diversification, and a complete description of WOATS algorithm. 4.1.1. Intensification by Crossover This step tries to enhance the quality of solutions in the current swarm S by combining them with a randomly selected solution from the EL. To do so, it

140

merges two solutions according to a crossover operator inspired by the genetic 8

Algorithm 1 Intensification by Crossover(S, EL) Select η solutions randomly from the swarm S. for m = 1, . . . , η do Select a random solution X randEL from EL for each center C in α random centers in current whale X m

do

Replace the center C with the opposite center from X randEL . end for end for Return the swarm S.

algorithm [12]. At the end of the function, the procedure returns the updated swarm S (see Algorithm 1). 4.1.2. Modified Diversification Step Despite the efficiency of WOA, the algorithm has a conservative diversification step. The main problem stems from the change of the positions of swarm members, which move their current positions to be closer in the neighborhood of the prey. This limits the ability of the swarm individuals to explore beyond their local neighborhood, ultimately increasing the probability of premature convergence to a suboptimal point. Because of this, a modification to the diversification step is proposed as follows. The new method repositions swarm members in new positions relative to their old positions rather than positioning them in the neighborhood of the prey. As we will see later, this method enhances the search capabilities of WOATS and prevents the need for local search algorithms. The new equations can be formalized as follows: 𝐷 = | 𝐶 ∙ 𝑋𝑟𝑎𝑛𝑑 (𝑡) ― 𝑋(𝑡)|,

(10)

𝐷 = | 𝐶 ∙ 𝑋 ∗ (𝑡) ― 𝑋(𝑡)|,

(11)

𝑋(t + 1) = 𝑋 (𝑡) + (𝐴 / 4) ∙ 𝐷 .

(12)

Depending on the used diversification method, the equations identify how to 145

reposition swarm members. Equation 10 computes the value of D with regard to a random whale selected from the current swarm, while equation 11 computes the value of D with regard to a best solution selected randomly from the EL. In

9

Algorithm 2 Diversification by Crossover(S) Select η solutions randomly from the swarm S. for m = 1, . . . , η do Select a random solution X rand from the current swarm other than current whale X m . for each center C in α random centers in current whale X m do Replace the center C with the opposite center from X rand . end for end for Return the swarm S.

equation 12, the value of the new position is computed with regard to the old → − position of current whale X (t). 150

4.1.3. Diversification by Crossover Besides intensification, crossover is also applied to diversify the candidate solutions. Again, WOATS uses a crossover operator inspired from the genetic algorithms domain. Specifically, WOATS selects η > 1 whales randomly from the current swarm. For each selected whale, another whale is randomly selected

155

to be combined with the first as detailed in (Algorithm 2). After the procedure stops, it returns the updated swarm S. 4.1.4. The WOATS Algorithm The complete WOATS pseudocode is listed in (Algorithm 3). It starts with setting the main parameters and an empty EL. First, a population is randomly

160

chosen from the dataset. Then, the EL is updated with the best solutions found in the initial population. WOATS uses the conditions employed in the original WOA to choose whether to apply intensification or diversification on the swarm. Next, the modified step for diversification is applied instead of the original one, and the solutions stored in the EL are used to guide the swarm members during

165

both the intensification and diversification phases. The Max NonImprove parameter is used to set the maximum number of iterations without improvement that after this number WOATS applies randomly either intensification using crossover (with Algorithm 1) or diversification using crossover (with Algorithm 2). At the end of the algorithm, WOATS returns the best solution stored in

170

EL.

10

Algorithm 3 Pseudocode for WOATS algorithm Initialization. Set number of clusters K, number of particles in swarm n, empty EL, set M ax N onImprove to be maximum number of iterations without improvement, and initialize the whale algorithm parameters. for i = 1, . . . , n do Generate initial centers randomly from the dataset. Compute the objective function of current solution by applying (1)). Update EL. end for Main Loop. for t = 1, . . . , M axIt do for i = 1, . . . , n do Update WOA parameters a, A, C, l, p. if (p < 0.5) then if (|A| < 1) then Select a random whale X ∗(t) from EL. Update position of the current whale X i (t) applying (11), (12). else if (|A| ≥ 1) then Select a random whale X rand (t) from current swarm. Update position of current whale X i (t) applying equation (10), (12). end if else if (p ≥ 0.5) then Select a random whale X ∗(t) from EL. Update position of current whale X i (t) applying equation (8). end if Compute the objective function of current whale X i (t) applying equation (1). Update solutions in EL. end for if number of iterations without improvement ≥ M ax N onImprove then if (randomnumber < 0.5) then Apply intensification procedure Intensification by Crossover(S,EL) (Algorithm 1). else Apply diversification procedure Diversification by Crossover(S) (Algorithm 2). end if end if end for Return best solution X ∗(t) in EL

5. Numerical Experiments WOATS was implemented and tested on a virtual machine in a Linux environment (UBUNTU 16.04 distribution), using datasets stored in a Hadoop framework [19]. The code was developed in Scala [20] because the algorithm 175

will be extended to run over a Hadoop computing cluster to deal with large datasets. For the results reported below, WOATS was tested on a computer with a Core-i3 processor and 16 GB memory. For the numerical experiments, WOATS was tested over multiple real datasets, presented in Table 1. These datasets are publicly available in the UCI r e p o s i t o r y [13] and ELKI [21].

180

Ta- ble 1 reports a summary description of these datasets, regarding dataset name, number of classes, number of attributes, and number of instances in each class. In the next subsections, the main parameters of WOATS and comparisons with several alternative optimization methods are given.

11

Table 1: Characteristics of datasets.

Name Iris Glass Cancer CMC Wine Balance Seed Mouse VaryDensity Ecoli Habermans Survival Magic Electricity CoverType Poker

No. of Classes 3 6 2 3 3 3 3 3 3 8 2 2 2 5 10

No. of Features 4 9 9 9 13 4 7 2 2 8 3 11 9 55 11

Instances in each Class (50, 50, 50) (29,76,70,17,13,9) (444,239) (629,333,511) (59,71,48 ) (288,49,288) (70,70,70) (290,100,100) (50,50,50) (143,77,2,2,35,20,5,52) (225, 81) (12332, 6688) (19237, 26075) (211840, 283301, 35754, 2747, 9493) (513702, 433097, 48828, 21634, 3978, 2050, 1460, 236, 17, 8)

5.1. Parameters Setting 185

A list of the values of the parameters of WOATS is presented in Table 2. These parameters are used to adjust the performance of the algorithm: MaxIt bounds the maximum number of iterations; n denotes the number of whales in each swarm; Max NonImprove stands for the maximum number of iterations without improvement that the algorithm can run before entering either a di-

190

versification or intensification phase; finally, Max EL is the maximum number of solutions that can be stored in the EL list. The parameters’ values were set based on different runs and these values were selected to maintain both the stability of the algorithm and obtaining high quality solutions. Some parameters needed to be as small as possible, such as n and MaxIt to minimize the exe-

195

cution time. Also, the max EL parameter value was set to the minimum value to prevent storing non-useful solutions and to minimize time needed to update the EL. Finally, the Max NonImprove parameter value, which was set based on different experiments to ensure the stability of the algorithm.

12

Table 2: Parameters’ values of WOATS.

Parameter MaxIt n Max NonImprove max EL

Definition Maximum number of iterations Population Size Maximum number of iterations without improvement Maximum number of solutions stored in elite list

Value 300 21 9 5

5.2. The Computational Complexity 200

To provide an approximate evaluation of the computational complexity of WOATS, it is defined as a function of number of fitness evaluations. Fitness evaluation is considered as the most time consuming operation because it computes the distance among the center points and the entire instances of the dataset. Analyzing (Algorithm 3), it can be seen that function evaluations mainly occur

205

in the main loop, which makes WOATS complexity as a function in both number of iterations and number of swarm members. From table 2, the maximum number of iterations is 300 and the number of swarm members is 21. Considering the crossover procedures, intensification and diversification ones, it is obvious that they are not executed in every iteration, they are called when

210

the number of non-improvement iterations is more than Max NonImprove. Even when these procedures are executed, their computational complexity is a function in a random number of swarm members multiplied by a random number of center points. This clarifies the low computational complexity of these procedures, which makes it a good candidate to analyze medium and large sized

215

datasets in real time. 5.3. Performance Analysis of WOATS in Data Clustering To assess the effectiveness of the new modifications, the number of iterations the algorithm needs to converge is used to measure the performance of WOATS. We compare the performance of WOATS with WOA [11] and PSO, implemented

220

according to [22]. Figure 2 presents the convergence curves of WOATS, WOA, and PSO algorithms over multiple real life datasets. The figure shows that WOATS reaches

13

Figure 2: A depiction of convergence curves with WOATS, WOA, and PSO over real life datasets.

the best values of the objective function in the least number of iterations, consistently: for all datasets, WOATS reaches the smallest values of objective function 225

compared with the other methods in less than 50 iterations. This demonstrates the efficiency of the modified diversification technique. For the Glass dataset, WOATS enhances the objective value along all the iterations, which reflects the effectiveness of the new diversification and intensification techniques. Thus, WOATS can converge to global optima in less iterations than its competitors. WOATS was compared with a SI method, Glowworm Swarm Optimization (GSO ), proposed for data Clustering (CGSO ) [5]. CGSO used three different distance functions. It reported its results in [5] according to purity criterion. Purity measures the homogeneity of the data instances in generated clusters [14]. The bigger the purity value the better the quality of the clustering algorithm. Purity is defined by

𝑃𝑢𝑟𝑖𝑡𝑦 =

230

1 𝑛

𝑘 × ∑𝑗 = 1max ( | 𝐿𝑖 ∩ 𝐶𝑗 | ), 𝑖

where n is the number of instances in the dataset, C represents the generated clusters C = {c1 , c2 , . . . , cK }, and L represents the original centers L = {l1 , l2 , . . . , lK }. The purity value lies in the range [0, 1]. Table 3 reports a comparison among WOATS, CGSO-F1, CGSO-F2, CGSO-

14

(13)

Table 3: Performance comparison of WOATS with Glowworm method and different clustering algorithms on real datasets according to Purity criterion.

Dataset Iris Glass Balance Seed Mouse Vary Density

WOATS Avg SD [Best] 0.933 0.022 [0.954] 0.608 0.051 [0.677] 0.729 0.025 [0.774] 0.900 0.006 [0.904] 0.846 0.003 [0.859] 0.966 0.016 [0.993]

CGSO-F1 Avg SD [Best] 0.919 0.090 [0.933] 0.541 0.018 [0.570] 0.726 0.039 [ 0.805 ] 0.900 0.016 [ 0.929 ] 0.837 0.013 [ 0.880 ] 0.956 0.006 [ 0.967 ]

CGSO-F2 Avg SD [Best] 0.903 0.014 [0.927] 0.533 0.036 [0.607] 0.685 0.061 [ 0.810 ] 0.889 0.026 [ 0.924 ] 0.834 0.018 [ 0.876 ] 0.956 0.007 [ 0.967 ]

CGSO-F3 Avg SD [Best] 0.909 0.012 [0.933] 0.529 0.020 [0.575] 0.694 0.074 [ 0.882 ] 0.897 0.018 [ 0.929 ] 0.833 0.018 [ 0.896 ] 0.957 0.006 [ 0.967 ]

F3, and K-M according to purity criterion (13), where values are computed over 235

25 independent runs. Table 3 reports mean, Standard Deviation (SD ), and best purity values found during the 25 runs. This form exhibits a better understanding about the algorithms and their performance. Clearly, Table 3 shows the superiority of WOATS over the other algorithms in the average values for all datasets. Considering the SD values, WOATS achieved the best values for Bal-

240

ance, Seed, and Mouse datasets. For the other datasets, WOATS ’ values were bigger than the best ones with very small values. This shows the capability of WOATS in obtaining similar solutions over different runs without being affected with the randomness in choosing the initial centers. Regarding the best values, WOATS reached the best values for Iris, Glass,

245

and Vary Density datasets. WOATS came the forth for the remaining datasets with very small differences. Reaching the best values for stochastic methods is not evidence on the efficiency of these methods because of their dependency on randomness. That is why the average and the SD metrics of multiple runs are considered as the most accurate evidence when analyzing meta-heuristic meth-

250

ods. Also despite using a simple objective function (1), WOATS ’ performance exceeded that of Glowworm, even when the latter was combined with modified objective functions. To further prove the efficiency of WOATS in analyzing medium and large-sized datasets, WOATS is compared with a MapReduce PSO Clustering Algorithm

255

(MR-CPSO ) proposed to analyze large datasets [23]. Table 4 presents the com-

15

K-M Avg 0.887 0.542 0.659 0.876 0.827 0.953

Table 4: Performance comparison of WOATS with MR-CPSO and K-M over large datasets according to Purity criterion.

Dataset Magic Electricity Poker CoverType

WOATS 0.75 0.68 0.67 0.72

MR-CPSO 0.65 0.58 0.51 0.53

K-M 0.60 0.51 0.11 0.32

parisons among WOATS, MR-CPSO, and K-M according to the purity criterion averaged over 25 runs, reported by Aljarah et al. [23]. As shown in the Table, WOATS still achieves the best results for all datasets, despite the large size of the datasets. This reveals the efficiency of WOATS and the impact of 260

the proposed modifications in providing an effective coverage of solution space regardless of the size of the datasets. Additionally, we compare WOATS with an Improved version of Cuckoo Search (ICS ) with a Modified version of Particle Swarm Optimization (PSO ) (MPSO ) with K-Harmonic Mean (KHM ) (ICMPKHM ) for data clustering, proposed by Bouyera et al. [8]. This algorithm combined multiple SI methods to alleviate the effect of randomization in the well-known Cuckoo Search (CS ) algorithm, and also to ensure a better exploration of search space. For measuring the quality of their method, the authors computed several indices, such as the Davies-Bouldin Index (DBI ), and the silhouette index [8]. The DBI measures the compactness of generated clusters and how well these clusters are separated. It is defined by

𝐷𝐵𝐼 =

1 𝑘

𝑚𝑖 + 𝑚𝑗

𝑘

∑𝑖 = 1max ( 𝑑 (𝑐 , 𝑐 ) ) , 𝑖 𝑗

(14)

𝑖 ≠𝑗

where k denotes the number of clusters, ci is the center of cluster i, mi stands for the average distance from the points in cluster i to center of the cluster ci, and d(ci, cj ) is the Euclidean distance between cluster centers i and j. Hence, the smaller the value of DBI, the better the quality of the solution. In turn, the Silhouette index is another important index for measuring the quality of the obtained clusters. It measures the cohesion and separation among clusters over 16

both point and cluster level. It assesses how similar an object is to its cluster

17

Table 5: Performance comparison of WOATS with ICMPKHM and other algorithms according to DBI and silhouette index criteria.

Dataset Iris Wine Cancer CMC Glass Ecoli

Criteria Silhouette DBI Silhouette DBI Silhouette DBI Silhouette DBI Silhouette DBI Silhouette DBI

WOATS 0.7435 0.3342 0.7391 0.2646 0.6980 0.4688 0.6829 0.3845 0.3963 2.069 0.3171 1.4702

ICMPKHM 0.5546 0.42 0.567 2.92 0.6981 0.68 0.4398 5.86 0.3789 2.65 0.2338 1.63

KHM 0.5509 0.65 0.5479 3.49 0.595 0.83 0.4361 8.14 0.3698 2.75 0.2308 0.95

PSO 0.5366 0.68 0.5473 4.55 0.5608 0.75 0.4386 7.93 0.30981 2.72 0.2281 1.74

ICAKHM 0.5442 0.49 0.553255 3.62 0.5964 0.70 0.4372 7.45 0.3205 2.86 0.2297 1.71

PSOKHM 0.5513 0.54 0.5655 3.07 0.6983 0.65 0.4379 5.92 0.36785 2.68 0.236 1.63

(cohesion) compared to other clusters (separation). The silhouette index can be computed for each point i as: 𝑏𝑖 ― 𝑎 𝑖

𝑆𝑖𝑙(𝑖) = max (𝑎𝑖, 𝑏𝑖) ,

(15)

where ai is the average distance among point i and points in the same cluster, bi represents the average distance among point i and points in other clusters. The total silhouette index is computed as the average for all the clusters. Sil265

houette is a maximization objective, the bigger the silhouette value, the better the clustering solution [8]. Its main drawback is that it is expensive to compute. Table 5 presents a comparison among WOATS, ICMPKHM, KHM, PSO, Imperialist Competitive Algorithm with KHM (ICAKHM ), and PSOKHM based on the quality of generated clusters using both the silhouette and DBI indices.

270

As expected, Table 5 describes that WOATS improves clustering solutions found by other algorithms over all the datasets except Cancer, and Ecoli. Although WOATS does not get the best DBI values for the Ecoli dataset, it achieves the best silhouette index values for these datasets. Also, the silhouette index values of WOATS for the Cancer dataset are very near to the results of better

275

algorithms. To finish the comparisons in this section, WOATS was contrasted with a new 18

Table 6: Performance comparison of WOATS with CSAK and other algorithms according to silhouette index and purity criteria.

Dataset Iris Wine Glass CMC Survival

Criteria

WOATS

CSAK

PSOK Means

Silhouette Purity Silhouette Purity Silhouette Purity Silhouette Purity Silhouette Purity

0.7435 93.3 0.7391 71.12 0.3963 60.83 0.6829 43.33 0.5926 73.53

0.7349 89.33 0.7316 70.22 0.5221 44.39 0.6481 40.12 0.5660 52.29

0.7167 88.67 0.7160 50.56 0.4577 40.65 0.6367 39.71 0.5578 51.96

Genetic K-Means 0.7289 44.67 0.7046 47.75 0.6954 37.85 0.6478 40.12 0.5536 75.82

KMeans++ 0.7251 44.00 0.7319 38.20 0.6172 36.92 0.6231 39.0360 0.5594 52.2876

clustering method based on combining CSA with K-means (K-M) (CSAK ), proposed by Lakshmi et al. [6]). The method used CSA to ensure that K-M can escape from local optima and to remove the sensitivity of K-M to initial 280

centers. Table 6 presents the comparison among WOATS, CSAK, PSOK Means, Genetic K-Means, K-Means++, and K-M, based on the silhouette index and purity. K-Means++ is a classic K-M algorithm with an enhancement in selecting initial centers. It selects initial centers that are distributed over the solution

285

space to overcome the sensitivity problem of K-M to initial centers [24]. Table 6 presents explicitly the superiority of WOATS over the other algorithms for all datasets, except the Glass dataset, regarding the Silhouette index values. CSAK and K-Means++ achieve the second-best results for all the datasets. Considering the purity values, WOATS continues its superiority

290

for all datasets, except the Survival datasets, in which WOATS comes the second. These results present the ability of WOATS to reach clusters that are very similar to original classes, which clarify the correctness of our analysis and the importance of the new modifications added to the original WOA. From the previous comparisons, it is noticeable that the hybridization of

295

WOA with TS has achieved its goals and enhanced the original algorithm. The new modifications allowed WOATS to overcome multiple original and hybrid SI 19

K-Means 0.7098 32.00 0.7097 35.39 0.4993 33.64 0.6192 38.43 0.5240 50.0000

methods, which presents the WOATS as a global optimizer and as an effective data clustering method that can be used for real life applications.

6. Conclusions and Future Work 300

In this paper, we proposed a new hybrid swarm intelligence algorithm based on the combination of Whale Optimization Algorithm (WOA) and Tabu Search (TS ) (WOATS ). This method enhanced the diversification step of WOA by changing the position of swarm members according to their original locations rather than the location of the prey. WOATS used an Elite List (EL) mem-

305

ory element of TS to store the best solutions. WOATS used these solutions to guide swarm members during the search process. This technique prevented WOATS from getting trapped in local optima and ensured a thorough exploration for promising regions. WOATS applied the crossover operator inspired by evolutionary computing methods to ensure the diversity of solutions. These

310

techniques enhanced the performance of the original WOA and ensured a better exploration of the search space. To assess the efficiency of WOATS, it was applied for data clustering. WOATS used an objective function inspired by partitional clustering methods to maintain the quality of swarm members. WOATS was tested over multiple real datasets and proved its superiority over

315

multiple original and hybrid swarm intelligence methods. Results obtained by WOATS were far better than those of other methods, which ensured its efficiency. WOATS was developed using Scala over Linux (Ubuntu) operating systems. WOATS is aimed to be used in clustering massive biological datasets based on Big Data technologies.

320

Acknowledgments The authors thank the efforts of the group involved in the joint project KA107 between Vigo University and Beni-Suef University.

20

References [1] K. A. P. Costa, L. A. M. Pereira, R. Y.M.Nakamura, C. R. Pereira, J. P. Papa, A. X. 325

Falcao, A nature-inspired approach to speed up optimum-path forest clustering and its application to intrusion detection in computer networks, Information Sciences 294 (2014) 95–108. doi:10.1016/j.ins.2014.09.025. [2] G. Ivan, V. Grolmusz, On dimension reduction of clustering results in structural bioinformatics, Biochimica et Biophysica Acta (BBA) - Proteins and Proteomics 1844 (2014)

330

2277–2283. doi:10.1016/j.bbapap.2014.08.015. [3] M. Z. Rodriguez, C. H. Comin, D. Casanova, O. M. Bruno, D. R. Amancio, L. da F. Costa, F. A. Rodrigues, Clustering algorithms: A comparative approach, PLOS ONEdoi:10. 1371/journal.pone.0210236. [4] E.-G. Talbi, Metaheuristics: from Design to Implementation, John Wiley & Sons, 2009.

335

[5] I. Aljarah, S. A. Ludwig, A new clustering approach based on glowworm swarm optimization, in: Congress on Evolutionary Computation (CEC), IEEE, 2013. doi: 10.1109/CEC.2013.6557888. [6] K. Lakshm, N. K. Visalakshi, S. Shanthi, Data clustering using k-means based on crow search algorithm, Sadhana 190. doi:10.1007/s12046-018-0962-3.

340

[7] M. R. Tanweer, S. Suresh, N. Sundararajan, Dynamic mentoring and self-regulation based particle swarm optimization algorithm for solving complex real-world optimization problems, Information Sciences 326 (2016) 1–24. doi:10.1016/j.ins.2015.07.035. [8] A. Bouyera, A. Hatamlou, An efficient hybrid clustering method based on improved cuckoo optimization and modified particle swarm optimization algorithms, Applied Soft

345

Computing 67 (2018) 172–182. doi:10.1016/j.asoc.2018.03.011. [9] X.-S. Yang, Nature-inspired metaheuristic algorithms, Luniver Press, 2008. [10] F. Glover, Future paths for integer programming and links to artificial intelligence, Computers & Operations Research 13 (5) (1986) 533–549. doi:10.1016/0305-0548(86) 90048-1.

350

[11] S. Mirjalili, A. Lewis, The whale optimization algorithm, Advances in Engineering Software 95 (2016) 51–67. doi:10.1016/j.advengsoft.2016.01.008. [12] D. E. Goldberg, Genetic Algorithms in Search, Optimization and Machine Learning, Addison-Wesley, 1989.

21

[13] University of california irvine, ucirvine machine learning repository, http://archive. 355

ics.uci.edu/ml/index.php, last accessed 2019 (2019). [14] Y. Zhao, G. Karypis, Evaluation of hierarchical clustering algorithms for document datasets, in: The Eleventh International Conference on Information and Knowledge Management, IEEE, 2002, pp. 515–524. doi:10.1145/584792.584877. [15] J. Nasiri, F. M. Khiyabani, A whale optimization algorithm (woa) approach for clustering,

360

Cogent Mathematics & Statistics 5 (1). doi:10.1080/25742558.2018.1483565. [16] A. P. Engelbrecht, Particle swarm optimization with crossover: a review and empirical analysis, Artificial Intelligence Review 45 (2016) 131–165. doi:10.1109/TEVC.2014. 2336175. [17] M. Abdel-Basset, G. Manogaran, D. El-Shahat, S. Mirjalili, A hybrid whale optimization

365

algorithm based on local search strategy for the permutation flow shop scheduling problem, Future Generation Computer Systems 85. doi:10.1016/j.future.2018.03.020. [18] H. M. Mohammed, S. U. Umar, T. A. Rashid, A systematic and meta-analysis survey of whale optimization algorithm, Computational Intelligence and Neurosciencedoi:10. 1080/25742558.2018.1483565.

370

[19] Apache software foundation, http://hadoop.apache.org (2019). [20] M. Odersky, P. Altherr, V. Cremet, B. Emir, S. Maneth, S. Micheloud, N. Mihaylov, M. Schinz, E. Stenman, M. Zenger, An overview of the scala programming language, Tech. rep., Ecole Polytechnique Federale de Lausanne (EPFL), Lausanne, Switzerland (2004).

375

[21] E. Schubert, A. Zimek, Elki data mining, https://elki-project.github.io/datasets/, last accessed 2019 (2018). [22] S. Mirjalili, S. M. Mirjalili, A. Lewis, Grey wolf optimizer, Advances in Engineering Software 69 (2014) 46–61. doi:10.1016/j.advengsoft.2013.12.007. [23] I. Aljarah, S. A. Ludwig, Parallel particle swarm optimization clustering algorithm based

380

on mapreduce methodology, in: Fourth World Congress on Nature and Biologically Inspired Computing (NaBIC), IEEE, 2012. doi:10.1109/NaBIC.2012.6402247. [24] B. Bahmani, B. Moseley, A. Vattani, R. Kumar, S. Vassilvitskii, Scalable k-means++, in: Very Large Data Bases (VLDB) Endowment, Vol. 5, 2012, pp. 622–633.

22