- Email: [email protected]
Fuzzy clustering based on K -nearest-neighbours rule
Fuzzy Sets and Systems 120 (2001) 239–247
www.elsevier.com/locate/fss
Fuzzy clustering based on K-nearest-neighbours rule N. Zahid, O. Abouelala, M. Limouri ∗ , A. Essaid Laboratoire Conception et Systemes, Faculte des Sciences, Universite Mohammed V-AGDAL, B.P. 1014 Av. Ibn Battouta, Rabat, Morocco Received April 1998; received in revised form November 1998
Abstract In cluster analysis, several algorithms have been made for partitioning a set of objects into c natural clusters. In general, this problem is formulated as being an objective function optimization approach. However, it is known that the function being minimized is nonconvex and hence it may lead to convergence to many local minima, i.e., to di6erent partitions. Thus, the clustering is repeated with di6erent initializations hoping that some runs will lead to the global minimum. Therefore, the performance of these algorithms depends largely on good choice of these initializations. The most widely used algorithm using this function is called fuzzy c-means algorithm (FCMA). In this paper a new algorithm is proposed to carry out fuzzy clustering without a priori assumptions on initial guesses. This algorithm is based on two-layer clustering strategy. During the :rst layer, the K-nearest-neighbours decision rule is used. Then, to achieve an optimal partition, the second layer involves one iteration of FCMA. The performance of the proposed algorithm and that of FCMA have been tested on six data sets. The c 2001 Elsevier Science results obtained show that the new algorithm possesses a number of advantages over the FCMA. B.V. All rights reserved. Keywords: Cluster analysis; Fuzzy c-means algorithm; K-nearest-neighbours decision rule; Fuzzy sets; Pattern recognition
1. Introduction Clustering algorithms are mainly based on partitioning a set of objects into natural clusters. Numerous mathematical tools, investigated for classi:cation, have been made to detect similarities between objects inside a cluster. The hard clustering is described by a conventional crisp membership function. This function assigns each object to one and only one of the clusters, with a degree of membership equal to one. ∗
Corresponding author. Tel.: +212-7-77-39-98; fax: +212-777-39-98. E-mail address: [email protected] (N. Zahid).
However, the boundaries between the clusters are not often well-de:ned and this description does not reBect the reality. The fuzzy clustering, founded upon fuzzy set theory [17], is meant to deal with not well-de:ned boundaries between clusters. Thus, in fuzzy clustering, the membership function is represented by gradual evolution between zero and one. Therefore, this membership indicates how the object is classi:ed into each cluster. This can be advantageous in the boundary region which may not be precisely de:ned. Many fuzzy clustering algorithms have been developed, but the most widely used is the fuzzy c-means algorithm (FCMA). Conceived by Dunn [7] and generalized by Bezdek [4], this family of algorithms is
c 2001 Elsevier Science B.V. All rights reserved. 0165-0114/01/$ - see front matter PII: S 0 1 6 5 - 0 1 1 4 ( 9 9 ) 0 0 0 7 4 - 3
240
N. Zahid et al. / Fuzzy Sets and Systems 120 (2001) 239–247
based on iterative optimization of a fuzzy objective function. The convergence, proved by Bezdek, of this function produces a local minima or partial optimal points [5]. Nevertheless, several algorithms of this family depend on initial guesses (cluster number, clusters centres,...). These a priori assumptions are necessary but do not guarantee the global minimum. The researchers have developed several algorithms, over the last decade, to solve these problems [1, 2, 8, 11]. The algorithm proposed in this study allow us to carry out fuzzy clustering without assumptions on initial guesses. The heuristic method used in this algorithm combines the favourable features of the FCMA and the K-nearest-neighbours (K-NN) decision rule. By testing the performance of the new algorithm and comparing it with FCMA on six data sets, the results obtained are very encouraging. 2. Hard and fuzzy c-partitions 2.1. Hard c-partition Let X = {x1 ; : : : ; xn } be a :nite set of n unlabelled data vectors in Rp . A hard clustering of X is obtained by organising these vectors into c disjoint clusters called c-partition (c¿2). For xk ∈ X , this partition is de:ned using the following characteristic function: ui : X → {0; 1};
xk → ui (xk ) = uik = 16i6c;
1 0
xk ∈ ith cluster; otherwise;
(1)
16k6n:
of the membership of xk in the ith cluster ui . The function ui is obtained by generalizing the concept of the characteristic function {ui : X → [0; 1]}. The following properties must be true for U to be nondegenerate fuzzy c-partition [6]: uik ∈ [0; 1]; ∀i; k; uik = 1; ∀k; i
0¡
uik ¡n;
(2a) (2b)
∀i:
(2c)
k
The distribution of membership among the c fuzzy clusters (2b) speci:es to what degree the object belongs to each cluster. The knowledge of this distribution is particularly interesting for ambiguous objects situated in “bounding region”. For example, in the case of two clusters, membership close to 0.5 indicates that the object can be member in both clusters. Hence, fuzzy c-partition provide member in much information about the structure in each data than hard c-partition. 3. Fuzzy clustering and cluster validity The fuzzy c-means algorithm (FCMA) is the most popular algorithm used in cluster analysis. It is an objective function optimization approach to solve the following problem [5, 4]: m 2 minimize Jm (U; V ) = uik d (xk ; Vi ) (3) i
k
This crisp function represents the membership degree. They specify objects belonging to di6erent clusters. Therefore, hard clustering algorithms may be insuIcient to deal with incomplete, vague or imprecise information. Thus, an ambiguous object will be assigned de:nitely to one cluster only. Nevertheless, such assignment is not preferable in several applications, especially in pattern recognition and cluster analysis.
with respect to U = [uik ] ∈ R cn , a fuzzy c-partition of n unlabelled data set X = {x1 ; : : : ; xn } ∈ Rpn and to V , a set of c fuzzy cluster centres V = (V1 ; : : : ; Vc ) ∈ Rpc . The parameter m¿1 is used as the fuzziness index. If m = 1, then the algorithm is reduced to the hard c-means algorithm (HCMA) [5, 4]. The convergence to the c-partition of X (U ∗ ) and its representative cluster centres (V ∗ ) has been shown in [5, 4]. The necessary conditions, for the minimizer (U ∗ ; V ∗ ) of Jm (U; V ), are de:ned as
2.2. Fuzzy partition
uik = c
(4a)
A fuzzy c-partition of X is de:ned by a (c × n) matrix U = [uik] ∈ R cn , where uik = ui (xk ) is the degree
n (uik ) m xk Vi = k=1 ; n m k=1 (uik )
(4b)
1 ; 2=(m−1) j=1 (d(xk ; Vi )=d(xk ; Vj ))
N. Zahid et al. / Fuzzy Sets and Systems 120 (2001) 239–247
where d2 (xk ; Vi ) = xk − Vi2 is the distance from xk to the cluster centre Vi . The optimal solution (U ∗ ; V ∗ ) is obtained by Picard iteration through (4a) and (4b). A brief speci:cation of iterating process is given as follows (assuming that d2 (xk ; Vi )¿0; ∀i; k at each iteration of 4a): Algorithm FCMA • Given unlabelled data set X = {x1 ; : : : ; xn } ∈ Rnp Initialization: • Fix c; m; · A and ¿0 • Choose initial cluster centres V0 = (V10 ; : : : ; Vc0 ) ∈ R cp arbitrarily • Set t = 1 FCM1: t • Compute all (c × n) memberships U t = [uik ] with (4a) • Update all c fuzzy cluster centres Vi t using (4b) FCM2: • Compute Et = V t+1 − V t 2 FCM3: • If Et ¡ stop; else t = t + 1 and go to FCM1 end The function Jm (U; V ) is nonconvex and hence may possess many local minima. Thus, the convergence of the algorithm can be trapped in undesirable local minima and may lead to di6erent partitions. To avoid this, one must run the iterative looping from di6erent initializations of [U ] or V and for some values of c; m and . Therefore, the performance of FCMA depends largely on good choice of these parameters. Furthermore, the fuzzy c-partition provided by the output of fuzzy clustering algorithms, attempts to identify the natural structure that is present in the data set analysed. In these partitions, the objects assigned to the same cluster are more similar to each other than are objects belonging to di6erent clusters. However, the diIcult question is how these partitions have identi:ed the structure that is imposed by the used fuzzy clustering algorithm?. This problem is called “Cluster Validity”. Many functions have been developed to solve this problem. These functions assign to the obtained partition a number which measures its e6ectiveness. In [13], there are several fuzzy clustering validity functions such as Classi:cation Entropy (H), Partition CoeIcient (PC), Uniform Data Functional (UDF) and Compactness and Separation Criteria (CS). A comparative examination [13] of these cluster validity shows
241
that they have a similar behaviour to evaluate fuzzy c-partition. In this work the PC has been used, it is de:ned as c n u2 (5) PC(U; c) = i=1 k=1 ik ; n where n is the number of objects ∈ X . If the objects of the data set analysed have been well classi:ed (uik → 1) there is no membership sharing between any pairs of fuzzy clusters. In this case (PC is close to 1) U has a tendency to represent hard c-partition. Conversely, the ambiguous situation is mentioned by: (uik → 1=c, PC is close to 1=c), every object in X is assigned to all c clusters with equal membership value (1=c). The resulting matrix U is the fuzziest partition. Hence, the best clustering is obtained by maximizing PC for c = 2; 3; : : : ; cmax . 4. Description of the proposed algorithm The algorithm described in the previous section always converges to a local optima of Jm starting from an initial guess of cluster centres. In practice the location of these cluster centres is not necessarily known a priori. Consequently, di6erent choices of initial cluster centres may lead to di6erent local optima. The simple idea of this work consists of developing a clustering algorithm based on two-layer clustering strategy. During the :rst layer an unsupervised tracking of initial cluster centres is made by using the Knearest-neighbours (K-NN) decision rule [12]. This rule, considered as pre-processing stage, is known to provide a useful nonparametric procedure for pattern recognition problems. It assigns to an unknown object y the cluster that is represented by a majority among its K-nearest-neighbours (K is the number of the closest objects of y). The hard K-NN algorithm is described as follows [12]: Algorithm K -NN Initialization: • Let X = {x1 ; : : : ; xn } ∈ Rnp be a set of n labelled data set • Input y, of unknown classi:cation • Fix K, number of the closest neighbours of y A1: Creation of the K -NN set (E) of y • Set i = 1
242
N. Zahid et al. / Fuzzy Sets and Systems 120 (2001) 239–247
Do until (K-NN objects to y are found) • Calculate d(y; xi ) the distance from y to xi If (i6K) then • Add xi in the set E of K-NN Else If (xi is the closer to y than any previous nearest neighbour) then • Delete the farthest in the set E, include xi in the set E EndIf increment i End Do Until A2: Classi%cation of y • Determine the majority cluster represented in the set E • Classify y in the majority cluster. end To locate the primary cluster centres of X , the :rst layer of the proposed algorithm can be stated as follows: Let X = {x1 ; : : : ; xn } be a :nite set of n unlabelled objects in Rp and let c be a given number of clusters. The :rst layer consists of partitioning the entire space X into c cells. In the same cell the objects are similar, in the Euclidean sense, to each other. These regrouping are realized by using the :rst part (A1) of the K-NN algorithm. In other words, the search of each cell is made by the following sequence: For 16i6c, each cell is represented by a set of K-nearest-neighbours formulated as Ei : (yi ; K-NN of yi ; Gi )
(6)
which means that for each object yi , de:ned below, the set Ei corresponding to its K closest neighbours is found. The mean centre Gi of Ei is de:ned as xk ∈Ei xk Gi = : (7) K +1 De9nition of yi : If i = 1; y1 is submitted to the following conditions: • y1 ∈ E1 • y1 is the farthest object from the global mean (G0 ) of all objects xk ∈ X de:ned as xk ∈X xk G0 = ; (8) n where n is the number of objects ∈ X
If 26i6c, the object yi which allows to create Ei is de:ned by • yi ∈ E i , • yi ∈= E ; 166i − 1, • yi = K-NN of y , • yi is the farthest object from Gi−1 . Furthermore, in practice the number K of the closest neighbours of yi may vary between 1 and (n − 1). On the other hand, the number c of the sets Ei (16i6c) is inBuenced by K and the conditions imposed on yi . In consequence, when K → (n − 1) ⇒ c → 1, and if K → 1 ⇒ c → n=2. This allow us to express K as a function of c in the following manner: n K = Integer −1 : (9) c We note that the number of clusters c may be speci:ed by the user or may be estimated by using the partition coeIcient (PC). In some particular cases, the unclassi:ed objects x (x ∈= (E1 ; E2 ; : : : ; Ec )) are assigned to the nearest centre (G1 ; G2 ; : : : ; Gc ) and then all of these centres are updated. Thus, at the end of this step all objects are assigned to the obtained sets (E1 ; E2 ; : : : ; Ec ). In other words, this procedure allow us to partition the data set X into c homogeneous groups (considered as c clusters), to de:ne the primary clusters centres and to come close to the best partition. To fuzzify the obtained partition, the second layer involves one iteration of FCMA. The Eqs. (4a) and (4b) are used, respectively, to compute the fuzzy membership degrees and the new fuzzy centres. The following is the statement of the algorithm: Algorithm KNN-1FCMA Initial setting: • Given unlabelled data set X = {x1 ; : : : ; xn } ∈ Rnp • Fix c, choose m¿1 and x = xT x • Set K = Integer(n=c − 1); number of K closest objects • Set I = {1; 2; : : : ; n}; indexes of the objects xk ∈ X Step 1: Creation of c sets Ei of K-NN (16i6c) • Compute G0 ∈ Rp of X using (8) • Set i = 1 Do until (i6c) • Search, in I, for the index of the farthest object yi from Gi−1 • Set j = 1
N. Zahid et al. / Fuzzy Sets and Systems 120 (2001) 239–247
Do until (K-NN objects to yi are found) • Calculate d(yi ; xj ) If (j6K) then • Add xj in the set Ei of K-NN Else If (xj is the closer to yi than any previous nearest neighbour) then • Delete the farthest in the set Ei , include xj in the set Ei EndIf End Do Until • Include yi in the set Ei • Calculate Gi ∈ Rp , centre of Ei using (7) • Delete yi index and the K-NN indexes of yi from I End Do Until If I is not empty then • For each remaining object x determine minimum distance to any centre Gi of Ei (1 6 i 6 c) • Classify x to the nearest centre and update all c centres Gi End If Step 2: FCMA one iteration ∗ • Compute U ∗ = [uik ]; (c × n) fuzzy c-partition matrix according to 1 ∗ uik = c 2=(m−1) (d(x ; G )=d(x k i k ; Gj )) j=1 ∗ ∗ • Calculate fuzzy centres V = (V1 ; V2∗ ; : : : ; Vc∗ ) ∈ R cp using (4b) end
5. Numerical examples and results analysis The fuzzy algorithm proposed has been coded in C=Unix, in order to compare its e6ectiveness with that of FCMA, six sets of data are considered. The :rst data set S1 , taken from [9], consists of two touching clusters (Fig. 1). There are 20 objects ∈ R2 in each cluster. The following three clusters (S2 ; S3 ; S4 ), shown in Figs. 2– 4, are arti:cially generated by using a random number function. They consist of 60, 80 and 120 objects in R2 distributed, respectively, on three, four and six clusters. Each cluster contains 20 objects and the overlapping rate varies progressively from S1 (touching clusters) to S4 (superimposed clusters).
243
Fig. 1. Data set S1 .
Fig. 2. Data set S2 .
The :fth data set is the IRIS data of Anderson [3]. It consists of three clusters (subspecies) of irises (Setosa: cluster 1, Virginica: cluster 2 and Versicolor: cluster 3). Each cluster contains 50 objects (plants) in R4 and the four components of an object are the measurements, in centimeter, of sepal length, sepal width, petal length and petal width. The cluster 1 is wellseparated cluster from clusters 2 and 3 which are very overlapping clusters. Therefore, this particular data set o6ers a fuzzy environment which is usually used to test fuzzy clustering algorithms [8, 12]. The last data
244
N. Zahid et al. / Fuzzy Sets and Systems 120 (2001) 239–247 Table 1 Results of clustering Data sets c
S1 S2 S3 S4 IRIS23 IRIS a Average
2 3 4 6 2 3
Misclassi:cation rate
Number of iterations
FCMAa
KNN–1FCMA
FCMAa
0 1 2 13 14 16
0 0 0 13 13 17
11 8 10 19 10 12
number.
Fig. 3. Data set S3 .
stability to the weighting exponent m and quality of clustering. The misclassi:cation rate is computed by using the maximum membership conversion of the obtained fuzzy c-partition to hard one: 1 if uik ¿usk ; 16s6c; s = i; (10) uik; max = 0 otherwise:
Fig. 4. Data set S4 .
Note that in the :rst analysis, the number of clusters is assumed to be known (it is :xed at the right value), and the FCMA is applied over 20 runs with arbitrary initial choices of V0 and the stopping criterion is :xed at 0.001. Table 1 shows, for m = 2, that both algorithms have practically the same misclassi:cation rate. However, for similar c-partitions the FCMA convergence requires between 8 and 19 iterations, whereas in the proposed algorithm one iteration of FCMA in the second layer is suIcient. The stability of both algorithms to the weighting exponent m, in the case of the IRIS data set, is shown in Fig. 5. The result obtained indicates that the KNN– 1FCMA is more stable than FCMA which is quite dependent on the weighting exponent. In fact, for di6erent values of m we obtain di6erent partitions. Similar results have been obtained for the other data sets. Furthermore, the quality of clustering is indicated by how closely the objects are associated to the cluster centres. The measure of this quality can be done by analysing the membership degree of objects. For S1 and S2 Figs. 6 and 7 illustrate the evolution of c × n values of the fuzzy membership degrees calculated by the proposed algorithm. We can see that most objects have a high grade of belongingness to their original clusters. Comparable quality of clustering is obtained for the other data sets.
set, called IRIS23, is a subset of the IRIS data set. It includes clusters 2 and 3. By using this data set we can analyse the behaviour of the algorithm on two nonseparable clusters. We note that various clustering and classi:er algorithms have used the IRIS data set to test their performance. Typical error rates for classi:er (supervised) design are 0 – 5 mistakes and for clustering (unsupervised) design are 10 –17 mistakes [14,15]. The performance of the algorithms are reported in term of number of iterations, misclassi:cation rate,
N. Zahid et al. / Fuzzy Sets and Systems 120 (2001) 239–247
Fig. 5. Misclassi:cation rate as function of m (IRIS, c = 3).
Fig. 6. Fuzzy membership evolution for S1 ; c = 2.
245
246
N. Zahid et al. / Fuzzy Sets and Systems 120 (2001) 239–247
Fig. 7. Fuzzy membership evolution for S2 ; c = 3.
Up to now both algorithms must be run for each desired number of clusters. Consequently, the algorithms can partition a data set into any speci:ed number of clusters even if the data contain no cluster structure [10]. To solve this problem one can use a cluster-validity criterion. In this work we have use the partition coeIcient (PC) described in Section 3. This coeIcient represents an indicator of a good quality of clustering. Therefore, √ by varying the number of clusters (c = 2; : : : ; cmax = n; [13]) and by using the PC we can study the e6ect of the number K on the obtained fuzzy c-partitions. Consequently, the user is not required to :x the number of clusters c and the best clustering is detected by :nding the highest value of PC. For both algorithms we indicate, in Table 2, the maximum value of partition coeIcient (PCmax ) which corresponds, in principle, to the right number of clusters (cr ). As can be seen, in the case of FCMA PC considers two clusters as a natural structure for all data sets. On the other hand, the proposed algorithm identify correctly, at PCmax (more greater than 1=c), the right number of clusters in each case except for IRIS and S4 data sets. This result is due to the very overlapping clusters (Virginica, Versicolor) and to the six superimposed clusters of S4 . It is known that in partitioning
Table 2 Optimal values of c chosen by PCmax FCMAa
Data sets
S1 S2 S3 S4 IRIS IRIS23 a m = 2;
KNN–1FCMA
cr
PCmax
cr
PCmax
2 2 2 2 2 2
0.80 0.77 0.73 0.69 0.89 0.75
2 3 4 2 2 2
0.80 0.75 0.71 0.66 0.88 0.74
= 0:001.
algorithms the search of the right number of clusters, present in the data set, is still an open problem [10, 16]. On the other hand, in hierarchical clustering methods the user is not required to specify the number of clusters. However, the dendrogram resulting from these methods is generally too complex and some heuristic is necessary to determine which level of the hierarchy best partitions the data set [10]. 6. Conclusion The use of the well-known fuzzy c-means algorithm requires determination of several parameters such as
N. Zahid et al. / Fuzzy Sets and Systems 120 (2001) 239–247
the number of clusters, the stopping criterion and the exponent weight. In addition, an initialization procedure must be made to obtain the :rst partition in the iterative process. This initialization is often done by arbitrary initial choices of cluster centres. However, di6erent initializations may lead to di6erent local minimum of the fuzzy objective function. Thus, the clustering is repeated with di6erent initializations hoping that some runs will lead to the global minimum. This problem has been treated by some researchers [1, 2, 8, 10]. In this paper another approach is proposed. It is based on two-layer clustering strategy. During the :rst layer an unsupervised tracking of initial cluster centres is made by using the Knearest-neighbours decision rule. Then, to fuzzify the obtained partition, the second layer involves one iteration of fuzzy c-means algorithm. This algorithm has been implemented, analysed and tested on six data sets. The results of experimental works show that the proposed algorithm can be used to carry out fuzzy clustering without a priori assumptions on the cluster centres, the weighting exponent, the stopping criterion and the number of clusters. In addition, it has been shown that the proposed procedure have also an important impact on the quality of clustering, it performs better with overlapping fuzzy clusters than the fuzzy c-means algorithm. These results encourage further research such as the variability in cluster shapes, the variations in cluster densities and the variability in the number of objects in each clusters. References [1] K.S. Al Sultan, S.Z. Selim, Global algorithm for fuzzy clustering problem, Pattern Recognition 26 (9) (1993) 1357–1361.
247
[2] K.S. Al Sultan, A tabu search approach to the clustering problem, Pattern Recognition 28 (9) (1995) 1443 –1451. [3] E. Anderson, The irises of the Gaspe Peninsula, Bull. Amer. Iris Soc. 59 (1935) 2– 5. [4] J.C. Bezdek, Fuzzy mathematics in pattern classi:cation, Ph.D. Dissertation, Appl. Math. Cornell Univ., Ithaca, NY, 1973. [5] J.C. Bezdek, A convergence theorem for the fuzzy ISODATA clustering algorithms, IEEE Trans. Pattern Anal. Machine Intell. PAMI-2 (1) (1980) 1– 8. [6] J.C. Bezdek, J.D. Haris, Fuzzy partitions and relations; an axiomatic basis for clustering, Fuzzy Sets and Systems 1 (1978) 111–127. [7] J.C. Dunn, A fuzzy relative of the ISODATA process and its use in detecting compact well-separated clusters, J. Cybernet. 3 (3) (1974) 32– 57. [8] J. Gath, A.B. Geva, Unsupervised optimal fuzzy clustering, IEEE Trans. Pattern Anal. Machine Intell. 11 (7) (1989) 773 –781. [9] T. Huntsberger, P. Ajjimarangsee, Parallel self-organizing feature maps for unsupervised pattern recognition, Int. J. General Systems 16 (4) (1990) 357–372. [10] A.K. Jain, J.V. Moreau, Bootstrap technique in cluster analysis, Pattern Recognition 20 (5) (1987) 547– 568. [11] M.S. Kamel, S.Z. Selim, A thresholded fuzzy c-means algorithm for semi-fuzzy clustering, Pattern Recognition 24 (9) (1991) 825 – 833. [12] J.M. Keller, M.R. Gray, J.A. Givens, Jr., A fuzzy K-nearest neighbors algorithm, IEEE Trans. Systems Man, Cybernet. SMC-15 (4) (1985) 580 – 585. [13] N.R. Pal, J.C. Bezdek, On cluster validity for the fuzzy c-means model, IEEE Trans. Fuzzy Systems 3 (3) (1995) 370 –379. [14] N.R. Pal, J.C. Bezdek, E.C.-K. Tsao, Generalized clustering networks and Kohonen’s self-organizing scheme, IEEE Trans. Neural Networks 4 (4) (1993) 549 – 557. [15] E.C-K. Tsao, J.C. Bezdek, N.R. Pal, Fuzzy Kohonen clustering networks, Pattern Recognition 27 (5) (1994) 757–764. [16] X.L. Xie, G. Beni, A validity measure for fuzzy clustering, IEEE Trans. Pattern Anal. Machine Intell. PAMI-13 (1991) 841– 847. [17] L.A. Zadeh, Fuzzy sets, Inform. Control 8 (1965) 338 –353.
www.elsevier.com/locate/fss
Fuzzy clustering based on K-nearest-neighbours rule N. Zahid, O. Abouelala, M. Limouri ∗ , A. Essaid Laboratoire Conception et Systemes, Faculte des Sciences, Universite Mohammed V-AGDAL, B.P. 1014 Av. Ibn Battouta, Rabat, Morocco Received April 1998; received in revised form November 1998
Abstract In cluster analysis, several algorithms have been made for partitioning a set of objects into c natural clusters. In general, this problem is formulated as being an objective function optimization approach. However, it is known that the function being minimized is nonconvex and hence it may lead to convergence to many local minima, i.e., to di6erent partitions. Thus, the clustering is repeated with di6erent initializations hoping that some runs will lead to the global minimum. Therefore, the performance of these algorithms depends largely on good choice of these initializations. The most widely used algorithm using this function is called fuzzy c-means algorithm (FCMA). In this paper a new algorithm is proposed to carry out fuzzy clustering without a priori assumptions on initial guesses. This algorithm is based on two-layer clustering strategy. During the :rst layer, the K-nearest-neighbours decision rule is used. Then, to achieve an optimal partition, the second layer involves one iteration of FCMA. The performance of the proposed algorithm and that of FCMA have been tested on six data sets. The c 2001 Elsevier Science results obtained show that the new algorithm possesses a number of advantages over the FCMA. B.V. All rights reserved. Keywords: Cluster analysis; Fuzzy c-means algorithm; K-nearest-neighbours decision rule; Fuzzy sets; Pattern recognition
1. Introduction Clustering algorithms are mainly based on partitioning a set of objects into natural clusters. Numerous mathematical tools, investigated for classi:cation, have been made to detect similarities between objects inside a cluster. The hard clustering is described by a conventional crisp membership function. This function assigns each object to one and only one of the clusters, with a degree of membership equal to one. ∗
Corresponding author. Tel.: +212-7-77-39-98; fax: +212-777-39-98. E-mail address: [email protected] (N. Zahid).
However, the boundaries between the clusters are not often well-de:ned and this description does not reBect the reality. The fuzzy clustering, founded upon fuzzy set theory [17], is meant to deal with not well-de:ned boundaries between clusters. Thus, in fuzzy clustering, the membership function is represented by gradual evolution between zero and one. Therefore, this membership indicates how the object is classi:ed into each cluster. This can be advantageous in the boundary region which may not be precisely de:ned. Many fuzzy clustering algorithms have been developed, but the most widely used is the fuzzy c-means algorithm (FCMA). Conceived by Dunn [7] and generalized by Bezdek [4], this family of algorithms is
c 2001 Elsevier Science B.V. All rights reserved. 0165-0114/01/$ - see front matter PII: S 0 1 6 5 - 0 1 1 4 ( 9 9 ) 0 0 0 7 4 - 3
240
N. Zahid et al. / Fuzzy Sets and Systems 120 (2001) 239–247
based on iterative optimization of a fuzzy objective function. The convergence, proved by Bezdek, of this function produces a local minima or partial optimal points [5]. Nevertheless, several algorithms of this family depend on initial guesses (cluster number, clusters centres,...). These a priori assumptions are necessary but do not guarantee the global minimum. The researchers have developed several algorithms, over the last decade, to solve these problems [1, 2, 8, 11]. The algorithm proposed in this study allow us to carry out fuzzy clustering without assumptions on initial guesses. The heuristic method used in this algorithm combines the favourable features of the FCMA and the K-nearest-neighbours (K-NN) decision rule. By testing the performance of the new algorithm and comparing it with FCMA on six data sets, the results obtained are very encouraging. 2. Hard and fuzzy c-partitions 2.1. Hard c-partition Let X = {x1 ; : : : ; xn } be a :nite set of n unlabelled data vectors in Rp . A hard clustering of X is obtained by organising these vectors into c disjoint clusters called c-partition (c¿2). For xk ∈ X , this partition is de:ned using the following characteristic function: ui : X → {0; 1};
xk → ui (xk ) = uik = 16i6c;
1 0
xk ∈ ith cluster; otherwise;
(1)
16k6n:
of the membership of xk in the ith cluster ui . The function ui is obtained by generalizing the concept of the characteristic function {ui : X → [0; 1]}. The following properties must be true for U to be nondegenerate fuzzy c-partition [6]: uik ∈ [0; 1]; ∀i; k; uik = 1; ∀k; i
0¡
uik ¡n;
(2a) (2b)
∀i:
(2c)
k
The distribution of membership among the c fuzzy clusters (2b) speci:es to what degree the object belongs to each cluster. The knowledge of this distribution is particularly interesting for ambiguous objects situated in “bounding region”. For example, in the case of two clusters, membership close to 0.5 indicates that the object can be member in both clusters. Hence, fuzzy c-partition provide member in much information about the structure in each data than hard c-partition. 3. Fuzzy clustering and cluster validity The fuzzy c-means algorithm (FCMA) is the most popular algorithm used in cluster analysis. It is an objective function optimization approach to solve the following problem [5, 4]: m 2 minimize Jm (U; V ) = uik d (xk ; Vi ) (3) i
k
This crisp function represents the membership degree. They specify objects belonging to di6erent clusters. Therefore, hard clustering algorithms may be insuIcient to deal with incomplete, vague or imprecise information. Thus, an ambiguous object will be assigned de:nitely to one cluster only. Nevertheless, such assignment is not preferable in several applications, especially in pattern recognition and cluster analysis.
with respect to U = [uik ] ∈ R cn , a fuzzy c-partition of n unlabelled data set X = {x1 ; : : : ; xn } ∈ Rpn and to V , a set of c fuzzy cluster centres V = (V1 ; : : : ; Vc ) ∈ Rpc . The parameter m¿1 is used as the fuzziness index. If m = 1, then the algorithm is reduced to the hard c-means algorithm (HCMA) [5, 4]. The convergence to the c-partition of X (U ∗ ) and its representative cluster centres (V ∗ ) has been shown in [5, 4]. The necessary conditions, for the minimizer (U ∗ ; V ∗ ) of Jm (U; V ), are de:ned as
2.2. Fuzzy partition
uik = c
(4a)
A fuzzy c-partition of X is de:ned by a (c × n) matrix U = [uik] ∈ R cn , where uik = ui (xk ) is the degree
n (uik ) m xk Vi = k=1 ; n m k=1 (uik )
(4b)
1 ; 2=(m−1) j=1 (d(xk ; Vi )=d(xk ; Vj ))
N. Zahid et al. / Fuzzy Sets and Systems 120 (2001) 239–247
where d2 (xk ; Vi ) = xk − Vi2 is the distance from xk to the cluster centre Vi . The optimal solution (U ∗ ; V ∗ ) is obtained by Picard iteration through (4a) and (4b). A brief speci:cation of iterating process is given as follows (assuming that d2 (xk ; Vi )¿0; ∀i; k at each iteration of 4a): Algorithm FCMA • Given unlabelled data set X = {x1 ; : : : ; xn } ∈ Rnp Initialization: • Fix c; m; · A and ¿0 • Choose initial cluster centres V0 = (V10 ; : : : ; Vc0 ) ∈ R cp arbitrarily • Set t = 1 FCM1: t • Compute all (c × n) memberships U t = [uik ] with (4a) • Update all c fuzzy cluster centres Vi t using (4b) FCM2: • Compute Et = V t+1 − V t 2 FCM3: • If Et ¡ stop; else t = t + 1 and go to FCM1 end The function Jm (U; V ) is nonconvex and hence may possess many local minima. Thus, the convergence of the algorithm can be trapped in undesirable local minima and may lead to di6erent partitions. To avoid this, one must run the iterative looping from di6erent initializations of [U ] or V and for some values of c; m and . Therefore, the performance of FCMA depends largely on good choice of these parameters. Furthermore, the fuzzy c-partition provided by the output of fuzzy clustering algorithms, attempts to identify the natural structure that is present in the data set analysed. In these partitions, the objects assigned to the same cluster are more similar to each other than are objects belonging to di6erent clusters. However, the diIcult question is how these partitions have identi:ed the structure that is imposed by the used fuzzy clustering algorithm?. This problem is called “Cluster Validity”. Many functions have been developed to solve this problem. These functions assign to the obtained partition a number which measures its e6ectiveness. In [13], there are several fuzzy clustering validity functions such as Classi:cation Entropy (H), Partition CoeIcient (PC), Uniform Data Functional (UDF) and Compactness and Separation Criteria (CS). A comparative examination [13] of these cluster validity shows
241
that they have a similar behaviour to evaluate fuzzy c-partition. In this work the PC has been used, it is de:ned as c n u2 (5) PC(U; c) = i=1 k=1 ik ; n where n is the number of objects ∈ X . If the objects of the data set analysed have been well classi:ed (uik → 1) there is no membership sharing between any pairs of fuzzy clusters. In this case (PC is close to 1) U has a tendency to represent hard c-partition. Conversely, the ambiguous situation is mentioned by: (uik → 1=c, PC is close to 1=c), every object in X is assigned to all c clusters with equal membership value (1=c). The resulting matrix U is the fuzziest partition. Hence, the best clustering is obtained by maximizing PC for c = 2; 3; : : : ; cmax . 4. Description of the proposed algorithm The algorithm described in the previous section always converges to a local optima of Jm starting from an initial guess of cluster centres. In practice the location of these cluster centres is not necessarily known a priori. Consequently, di6erent choices of initial cluster centres may lead to di6erent local optima. The simple idea of this work consists of developing a clustering algorithm based on two-layer clustering strategy. During the :rst layer an unsupervised tracking of initial cluster centres is made by using the Knearest-neighbours (K-NN) decision rule [12]. This rule, considered as pre-processing stage, is known to provide a useful nonparametric procedure for pattern recognition problems. It assigns to an unknown object y the cluster that is represented by a majority among its K-nearest-neighbours (K is the number of the closest objects of y). The hard K-NN algorithm is described as follows [12]: Algorithm K -NN Initialization: • Let X = {x1 ; : : : ; xn } ∈ Rnp be a set of n labelled data set • Input y, of unknown classi:cation • Fix K, number of the closest neighbours of y A1: Creation of the K -NN set (E) of y • Set i = 1
242
N. Zahid et al. / Fuzzy Sets and Systems 120 (2001) 239–247
Do until (K-NN objects to y are found) • Calculate d(y; xi ) the distance from y to xi If (i6K) then • Add xi in the set E of K-NN Else If (xi is the closer to y than any previous nearest neighbour) then • Delete the farthest in the set E, include xi in the set E EndIf increment i End Do Until A2: Classi%cation of y • Determine the majority cluster represented in the set E • Classify y in the majority cluster. end To locate the primary cluster centres of X , the :rst layer of the proposed algorithm can be stated as follows: Let X = {x1 ; : : : ; xn } be a :nite set of n unlabelled objects in Rp and let c be a given number of clusters. The :rst layer consists of partitioning the entire space X into c cells. In the same cell the objects are similar, in the Euclidean sense, to each other. These regrouping are realized by using the :rst part (A1) of the K-NN algorithm. In other words, the search of each cell is made by the following sequence: For 16i6c, each cell is represented by a set of K-nearest-neighbours formulated as Ei : (yi ; K-NN of yi ; Gi )
(6)
which means that for each object yi , de:ned below, the set Ei corresponding to its K closest neighbours is found. The mean centre Gi of Ei is de:ned as xk ∈Ei xk Gi = : (7) K +1 De9nition of yi : If i = 1; y1 is submitted to the following conditions: • y1 ∈ E1 • y1 is the farthest object from the global mean (G0 ) of all objects xk ∈ X de:ned as xk ∈X xk G0 = ; (8) n where n is the number of objects ∈ X
If 26i6c, the object yi which allows to create Ei is de:ned by • yi ∈ E i , • yi ∈= E ; 166i − 1, • yi = K-NN of y , • yi is the farthest object from Gi−1 . Furthermore, in practice the number K of the closest neighbours of yi may vary between 1 and (n − 1). On the other hand, the number c of the sets Ei (16i6c) is inBuenced by K and the conditions imposed on yi . In consequence, when K → (n − 1) ⇒ c → 1, and if K → 1 ⇒ c → n=2. This allow us to express K as a function of c in the following manner: n K = Integer −1 : (9) c We note that the number of clusters c may be speci:ed by the user or may be estimated by using the partition coeIcient (PC). In some particular cases, the unclassi:ed objects x (x ∈= (E1 ; E2 ; : : : ; Ec )) are assigned to the nearest centre (G1 ; G2 ; : : : ; Gc ) and then all of these centres are updated. Thus, at the end of this step all objects are assigned to the obtained sets (E1 ; E2 ; : : : ; Ec ). In other words, this procedure allow us to partition the data set X into c homogeneous groups (considered as c clusters), to de:ne the primary clusters centres and to come close to the best partition. To fuzzify the obtained partition, the second layer involves one iteration of FCMA. The Eqs. (4a) and (4b) are used, respectively, to compute the fuzzy membership degrees and the new fuzzy centres. The following is the statement of the algorithm: Algorithm KNN-1FCMA Initial setting: • Given unlabelled data set X = {x1 ; : : : ; xn } ∈ Rnp • Fix c, choose m¿1 and x = xT x • Set K = Integer(n=c − 1); number of K closest objects • Set I = {1; 2; : : : ; n}; indexes of the objects xk ∈ X Step 1: Creation of c sets Ei of K-NN (16i6c) • Compute G0 ∈ Rp of X using (8) • Set i = 1 Do until (i6c) • Search, in I, for the index of the farthest object yi from Gi−1 • Set j = 1
N. Zahid et al. / Fuzzy Sets and Systems 120 (2001) 239–247
Do until (K-NN objects to yi are found) • Calculate d(yi ; xj ) If (j6K) then • Add xj in the set Ei of K-NN Else If (xj is the closer to yi than any previous nearest neighbour) then • Delete the farthest in the set Ei , include xj in the set Ei EndIf End Do Until • Include yi in the set Ei • Calculate Gi ∈ Rp , centre of Ei using (7) • Delete yi index and the K-NN indexes of yi from I End Do Until If I is not empty then • For each remaining object x determine minimum distance to any centre Gi of Ei (1 6 i 6 c) • Classify x to the nearest centre and update all c centres Gi End If Step 2: FCMA one iteration ∗ • Compute U ∗ = [uik ]; (c × n) fuzzy c-partition matrix according to 1 ∗ uik = c 2=(m−1) (d(x ; G )=d(x k i k ; Gj )) j=1 ∗ ∗ • Calculate fuzzy centres V = (V1 ; V2∗ ; : : : ; Vc∗ ) ∈ R cp using (4b) end
5. Numerical examples and results analysis The fuzzy algorithm proposed has been coded in C=Unix, in order to compare its e6ectiveness with that of FCMA, six sets of data are considered. The :rst data set S1 , taken from [9], consists of two touching clusters (Fig. 1). There are 20 objects ∈ R2 in each cluster. The following three clusters (S2 ; S3 ; S4 ), shown in Figs. 2– 4, are arti:cially generated by using a random number function. They consist of 60, 80 and 120 objects in R2 distributed, respectively, on three, four and six clusters. Each cluster contains 20 objects and the overlapping rate varies progressively from S1 (touching clusters) to S4 (superimposed clusters).
243
Fig. 1. Data set S1 .
Fig. 2. Data set S2 .
The :fth data set is the IRIS data of Anderson [3]. It consists of three clusters (subspecies) of irises (Setosa: cluster 1, Virginica: cluster 2 and Versicolor: cluster 3). Each cluster contains 50 objects (plants) in R4 and the four components of an object are the measurements, in centimeter, of sepal length, sepal width, petal length and petal width. The cluster 1 is wellseparated cluster from clusters 2 and 3 which are very overlapping clusters. Therefore, this particular data set o6ers a fuzzy environment which is usually used to test fuzzy clustering algorithms [8, 12]. The last data
244
N. Zahid et al. / Fuzzy Sets and Systems 120 (2001) 239–247 Table 1 Results of clustering Data sets c
S1 S2 S3 S4 IRIS23 IRIS a Average
2 3 4 6 2 3
Misclassi:cation rate
Number of iterations
FCMAa
KNN–1FCMA
FCMAa
0 1 2 13 14 16
0 0 0 13 13 17
11 8 10 19 10 12
number.
Fig. 3. Data set S3 .
stability to the weighting exponent m and quality of clustering. The misclassi:cation rate is computed by using the maximum membership conversion of the obtained fuzzy c-partition to hard one: 1 if uik ¿usk ; 16s6c; s = i; (10) uik; max = 0 otherwise:
Fig. 4. Data set S4 .
Note that in the :rst analysis, the number of clusters is assumed to be known (it is :xed at the right value), and the FCMA is applied over 20 runs with arbitrary initial choices of V0 and the stopping criterion is :xed at 0.001. Table 1 shows, for m = 2, that both algorithms have practically the same misclassi:cation rate. However, for similar c-partitions the FCMA convergence requires between 8 and 19 iterations, whereas in the proposed algorithm one iteration of FCMA in the second layer is suIcient. The stability of both algorithms to the weighting exponent m, in the case of the IRIS data set, is shown in Fig. 5. The result obtained indicates that the KNN– 1FCMA is more stable than FCMA which is quite dependent on the weighting exponent. In fact, for di6erent values of m we obtain di6erent partitions. Similar results have been obtained for the other data sets. Furthermore, the quality of clustering is indicated by how closely the objects are associated to the cluster centres. The measure of this quality can be done by analysing the membership degree of objects. For S1 and S2 Figs. 6 and 7 illustrate the evolution of c × n values of the fuzzy membership degrees calculated by the proposed algorithm. We can see that most objects have a high grade of belongingness to their original clusters. Comparable quality of clustering is obtained for the other data sets.
set, called IRIS23, is a subset of the IRIS data set. It includes clusters 2 and 3. By using this data set we can analyse the behaviour of the algorithm on two nonseparable clusters. We note that various clustering and classi:er algorithms have used the IRIS data set to test their performance. Typical error rates for classi:er (supervised) design are 0 – 5 mistakes and for clustering (unsupervised) design are 10 –17 mistakes [14,15]. The performance of the algorithms are reported in term of number of iterations, misclassi:cation rate,
N. Zahid et al. / Fuzzy Sets and Systems 120 (2001) 239–247
Fig. 5. Misclassi:cation rate as function of m (IRIS, c = 3).
Fig. 6. Fuzzy membership evolution for S1 ; c = 2.
245
246
N. Zahid et al. / Fuzzy Sets and Systems 120 (2001) 239–247
Fig. 7. Fuzzy membership evolution for S2 ; c = 3.
Up to now both algorithms must be run for each desired number of clusters. Consequently, the algorithms can partition a data set into any speci:ed number of clusters even if the data contain no cluster structure [10]. To solve this problem one can use a cluster-validity criterion. In this work we have use the partition coeIcient (PC) described in Section 3. This coeIcient represents an indicator of a good quality of clustering. Therefore, √ by varying the number of clusters (c = 2; : : : ; cmax = n; [13]) and by using the PC we can study the e6ect of the number K on the obtained fuzzy c-partitions. Consequently, the user is not required to :x the number of clusters c and the best clustering is detected by :nding the highest value of PC. For both algorithms we indicate, in Table 2, the maximum value of partition coeIcient (PCmax ) which corresponds, in principle, to the right number of clusters (cr ). As can be seen, in the case of FCMA PC considers two clusters as a natural structure for all data sets. On the other hand, the proposed algorithm identify correctly, at PCmax (more greater than 1=c), the right number of clusters in each case except for IRIS and S4 data sets. This result is due to the very overlapping clusters (Virginica, Versicolor) and to the six superimposed clusters of S4 . It is known that in partitioning
Table 2 Optimal values of c chosen by PCmax FCMAa
Data sets
S1 S2 S3 S4 IRIS IRIS23 a m = 2;
KNN–1FCMA
cr
PCmax
cr
PCmax
2 2 2 2 2 2
0.80 0.77 0.73 0.69 0.89 0.75
2 3 4 2 2 2
0.80 0.75 0.71 0.66 0.88 0.74
= 0:001.
algorithms the search of the right number of clusters, present in the data set, is still an open problem [10, 16]. On the other hand, in hierarchical clustering methods the user is not required to specify the number of clusters. However, the dendrogram resulting from these methods is generally too complex and some heuristic is necessary to determine which level of the hierarchy best partitions the data set [10]. 6. Conclusion The use of the well-known fuzzy c-means algorithm requires determination of several parameters such as
N. Zahid et al. / Fuzzy Sets and Systems 120 (2001) 239–247
the number of clusters, the stopping criterion and the exponent weight. In addition, an initialization procedure must be made to obtain the :rst partition in the iterative process. This initialization is often done by arbitrary initial choices of cluster centres. However, di6erent initializations may lead to di6erent local minimum of the fuzzy objective function. Thus, the clustering is repeated with di6erent initializations hoping that some runs will lead to the global minimum. This problem has been treated by some researchers [1, 2, 8, 10]. In this paper another approach is proposed. It is based on two-layer clustering strategy. During the :rst layer an unsupervised tracking of initial cluster centres is made by using the Knearest-neighbours decision rule. Then, to fuzzify the obtained partition, the second layer involves one iteration of fuzzy c-means algorithm. This algorithm has been implemented, analysed and tested on six data sets. The results of experimental works show that the proposed algorithm can be used to carry out fuzzy clustering without a priori assumptions on the cluster centres, the weighting exponent, the stopping criterion and the number of clusters. In addition, it has been shown that the proposed procedure have also an important impact on the quality of clustering, it performs better with overlapping fuzzy clusters than the fuzzy c-means algorithm. These results encourage further research such as the variability in cluster shapes, the variations in cluster densities and the variability in the number of objects in each clusters. References [1] K.S. Al Sultan, S.Z. Selim, Global algorithm for fuzzy clustering problem, Pattern Recognition 26 (9) (1993) 1357–1361.
247
[2] K.S. Al Sultan, A tabu search approach to the clustering problem, Pattern Recognition 28 (9) (1995) 1443 –1451. [3] E. Anderson, The irises of the Gaspe Peninsula, Bull. Amer. Iris Soc. 59 (1935) 2– 5. [4] J.C. Bezdek, Fuzzy mathematics in pattern classi:cation, Ph.D. Dissertation, Appl. Math. Cornell Univ., Ithaca, NY, 1973. [5] J.C. Bezdek, A convergence theorem for the fuzzy ISODATA clustering algorithms, IEEE Trans. Pattern Anal. Machine Intell. PAMI-2 (1) (1980) 1– 8. [6] J.C. Bezdek, J.D. Haris, Fuzzy partitions and relations; an axiomatic basis for clustering, Fuzzy Sets and Systems 1 (1978) 111–127. [7] J.C. Dunn, A fuzzy relative of the ISODATA process and its use in detecting compact well-separated clusters, J. Cybernet. 3 (3) (1974) 32– 57. [8] J. Gath, A.B. Geva, Unsupervised optimal fuzzy clustering, IEEE Trans. Pattern Anal. Machine Intell. 11 (7) (1989) 773 –781. [9] T. Huntsberger, P. Ajjimarangsee, Parallel self-organizing feature maps for unsupervised pattern recognition, Int. J. General Systems 16 (4) (1990) 357–372. [10] A.K. Jain, J.V. Moreau, Bootstrap technique in cluster analysis, Pattern Recognition 20 (5) (1987) 547– 568. [11] M.S. Kamel, S.Z. Selim, A thresholded fuzzy c-means algorithm for semi-fuzzy clustering, Pattern Recognition 24 (9) (1991) 825 – 833. [12] J.M. Keller, M.R. Gray, J.A. Givens, Jr., A fuzzy K-nearest neighbors algorithm, IEEE Trans. Systems Man, Cybernet. SMC-15 (4) (1985) 580 – 585. [13] N.R. Pal, J.C. Bezdek, On cluster validity for the fuzzy c-means model, IEEE Trans. Fuzzy Systems 3 (3) (1995) 370 –379. [14] N.R. Pal, J.C. Bezdek, E.C.-K. Tsao, Generalized clustering networks and Kohonen’s self-organizing scheme, IEEE Trans. Neural Networks 4 (4) (1993) 549 – 557. [15] E.C-K. Tsao, J.C. Bezdek, N.R. Pal, Fuzzy Kohonen clustering networks, Pattern Recognition 27 (5) (1994) 757–764. [16] X.L. Xie, G. Beni, A validity measure for fuzzy clustering, IEEE Trans. Pattern Anal. Machine Intell. PAMI-13 (1991) 841– 847. [17] L.A. Zadeh, Fuzzy sets, Inform. Control 8 (1965) 338 –353.