- Email: [email protected]
A novel fuzzy entropy-constrained competitive learning algorithm for image coding
Neurocomputing 37 (2001) 197}208
A novel fuzzy entropy-constrained competitive learning algorithm for image coding Wen-Jyi Hwang*, Faa-Jeng Lin, Shi-Chiang Liao, Jeng-Hsin Huang Department of Electrical Engineering, Chung Yuan Christian University, Chungli, 32023, Taiwan Accepted 17 November 2000
Abstract A novel variable-rate vector quantizer (VQ) design algorithm using both fuzzy and competitive learning technique is presented. The algorithm enjoys better rate-distortion performance than that of other existing fuzzy clustering and competitive learning algorithms. In addition, the learning algorithm is less sensitive to the selection of initial reproduction vectors. Therefore, the algorithm can be an e!ective alternative to the existing variable-rate VQ algorithms for signal compression. 2001 Elsevier Science B.V. All rights reserved. Keywords: Image coding; Vector quantization; Competitive learning; Fuzzy clustering
1. Introduction Entropy-constrained vector quantization (ECVQ) [1] algorithm is a well-known method for variable-rate VQ design [2]. The technique employs batch training process where components of VQ are constructed iteratively one at a time. In addition, during the course of the iterative construction procedure, ECVQ algorithm utilizes crisp clustering technique where each training vector is assigned to only one cluster for training set partitioning. The algorithm have the following disadvantages. First, the performance of iterative design procedure is sensitive to the selection of initial reproduction vectors. Therefore, a bad local optimum might be achieved if an improper set of initial codewords is selected. Moreover, since many training vectors might have features belonging to more than one clusters, it is not ideal to assign each training vector to only one cluster during set partitioning process.
* Corresponding author. Tel.: 886-3-456-3171-4808; fax: 886-3-456-3160. E-mail address: [email protected] (W.-J. Hwang). 0925-2312/01/$ - see front matter 2001 Elsevier Science B.V. All rights reserved. PII: S 0 9 2 5 - 2 3 1 2 ( 0 0 ) 0 0 3 5 1 - 9
198
W.-J. Hwang et al. / Neurocomputing 37 (2001) 197}208
To alleviate these drawbacks, a number of competitive learning (CL) algorithms [5,7] and fuzzy clustering techniques [4,6] have been proposed for VQ design. As compared with the traditional iterative algorithms, the CL-based algorithms are shown to have more robust rate-distortion performance because the training process of these algorithms are performed in continuous mode. The fuzzy clustering algorithms can also improve the performance of the VQ design since better codevectors might be found by fuzzifying the boundary of clusters. Although using CL or fuzzy clustering algorithm along can improve the design of VQs, it is expected that the fusion of these two techniques can improve the performance of VQs further because the resulting VQs can have the advantages of both algorithms. The objective of this letter is to present a novel variable-rate VQ design algorithm, termed fuzzy entropy-constrained competitive learning (FECCL) algorithm, which performs VQ training in both continuous and fuzzy mode. To achieve continuous training, similar to the CL algorithms, the weight vectors of the algorithm are updated for each input training vector. However, the continuous training process is not based on the winner-take-all scheme adopted by usual CL algorithms, which results in crisp training. In the FECCL algorithm, for any input training vector, its membership grade in each cluster is "rst computed. All the weight vectors are then updated according to these membership grades. In order to control the average rate of the VQ after training process, when computing the membership functions, the length of channel codeword representing each weight vector will be taken into account. Simulation results show that the FECCL algorithm has more robust and superior ratedistortion performance than other existing VQ design algorithms where only crisp or batch training are used.
2. Preliminaries Recall that the encoder W of a variable-rate VQ [1,2] can be decomposed into two parts. The "rst part is the function w which maps a source vector x into a source index i. The second part, , maps the source index i into a channel codeword c . The G decoder Z can be similarly decomposed into two parts: \ and z. The mapping \ is the inverse of , and maps c back to source index i. The mapping z outputs a G reproduction vector z(i) for the source index i. The pair (W, Z) is referred to as a variable-rate VQ because a source vector is represented by a single channel codeword, and the channel codewords for di!erent source vectors are usually not of the same length. Given a source vector x, let (w(x)) be the length of the channel codeword representing x. In addition, let d (u,) be the mean-squared distance between vectors L u and . For a given set of training vectors X"x , j"1,2, t, where t is the number H of training vectors in the training set, the average rate and average distortion of the VQ are then given as R"1/ntR (w(x )) and D"1/ntR d (x , z(w(x ))), respecH H H L H H tively. Let D (R) be the function indicates the lowest possible average distortion that L can be achieved by a VQ with blocklength n under the rate constraint R. Although the
W.-J. Hwang et al. / Neurocomputing 37 (2001) 197}208
199
Lagragian methods cannot be used for "nding D(R) [1], they can be used for "nding the convex hull of D(R) be minimizing the following functional: R C(w, z, )" d (x , z(w(x )))#(w(x )). L H H H H
(1)
Here, has an interpretation as the slope of a line supporting the convex hull. We now discuss the optimal conditions for w, z and that minimizes the functional C(w, z, ). Fix and z, the mapping w that minimizes (1) is the one that satis"es w(x )"arg min J(x , z(i)), H H XGX,
(2)
where N is the number of reproduction vectors, and J(x , z(i))"d (x , z(i))#(i) H L H
(3)
can be viewed as the modi"ed distance measure between x and z(i). H Next, for "xed w and , the mapping z that minimizes (1) is given as z(i)"centroid of x : w(x )"i. H H
(4)
Finally, the optimal channel encoder for "xed w and z is the entropy encoder designed based on the probability p(i) of each reproduction vector z(i). That is, the optimal satis"es (i)"log
1 . p(i)
(5)
There are a number of entropy coding techniques for the implementation of mapping , such as the Hu!man codes [3] and Arithmetic codes [8] which can yield an average rate closed to the entropy of the indices. The ECVQ algorithm uses (2), (4), (5) iteratively for the minimization of C(w, z, ) given in (1). The scheme is sensitive to the initial parameters since reproduction vectors are constructed in batch mode. If the initial and z are improperly chosen, then the ECVQ algorithm can easily converge to a bad local minimum. In addition, the clustering operation is a crisp clustering process. Better codewords might be found if fuzzy clustering process is employed. The objective of ECCL algorithm [5] is also to minimize C(w, z, ). However, unlike the ECVQ algorithm, the ECCL performs the continuous training process for the codebook construction. Given , the ECCL algorithm "nds the optimal w and z satisfying Eqs. (2) and (4), respectively, using the following winner-take-all
200
W.-J. Hwang et al. / Neurocomputing 37 (2001) 197}208
competition scheme [5]: z(iH)Qz(iH)#(iH)(x !z(iH)), H
(6)
z(i)Qz(i), iOiH,
(7)
where x is an input training vector in the training set X, (i) is the learning rate of the H ith weight vector, and iH is the index of the winning weight vector based on the following competition rule: iH"arg min J(x , z(k)). H XIX,
(8)
Since the ECCL algorithm performs weight vector updating for each input training vector, the algorithm has more robust performance as compared with ECVQ algorithm. The FECVQ algorithm [4] performs fuzzy clustering during the course of iterative training process. In order to incorporate the membership functions of training vectors into the clustering process, the objective of FECVQ is to minimize the following objective function: R , C(, z, )" ( )K(J(x , z(i))), G H H H G
(9)
where m'1 indicates the degree of fuzziness, and , i"1,2, N is the set of G H membership values associated with training vector x , satisfying , "1 and H G G H 50. The algorithm employs an iterative training procedure that optimizes , z, G H and one at a time. In particular, for "xed z and , the optimal membership functions minimizing C(, z, ) are given as [4] (1/m(x , z(i))) K\ H " . G H , (1/J(x , z(l))) K\ J H
(10)
Since performing fuzzy clustering for codebook construction, FECVQ can "nd better reconstruction vectors. However, FECVQ also performs training in batch mode, and therefore is sensitive to initial parameters.
3. FECCL algorithm The FECCL algorithm has the advantages of both FECVQ and ECCL algorithms. In the algorithm, similar to ECCL technique, the training process is performed in continuous mode. However, unlike the ECCL algorithm, where the weight vector updating is based on winner-take-all scheme, the FECCL algorithm updates weight vectors according to the membership functions, which measure the degree of closeness between the input training vector to each weight vector. During the course of FECCL
W.-J. Hwang et al. / Neurocomputing 37 (2001) 197}208
201
Fig. 1. Design procedure of FECCL algorithm.
design, the reproduction vectors z(i), i"1,2, N, are "rst updated in fuzzy and continuous mode based on a "xed channel encoder . The source encoder w is then constructed based on the updated z. After that, the channel encoder is designed according to the new w. The same process is repeated until the convergence of the algorithm. The design procedure of FECCL algorithm is summarized in Fig. 1. We now discuss each step of the algorithm in more detail. For any input training vector x 3X, the FECCL algorithm updates weight vectors z(i), i"1,2, N, accordH ing to the following rule: z(i)Qz(i)#(i)( )K(z(i)!x ), (11) G H H where (i) is the learning rate of z(i). At the "rst K iterations of FECVQ algorithm, where K'0 can be prespeci"ed before the design, the membership functions , i"1,2, N, are computed according to (10), and therefore all the weight vectors G H are updated for each input training vector x . Consequently, FECVQ algorithm is H insensitive to the selection of initial weight vectors. After K iterations, since the VQ encoding process is a crisp selection process, in order to "nd codewords well-suited for crisp encoding, the FECCL algorithm employs a novel fuzzy-to-crisp transition
202
W.-J. Hwang et al. / Neurocomputing 37 (2001) 197}208
process for codebook construction. In the process, as the number of iterations increases, the number for weight vectors to be updated for each training vector is gradually reduced until winner-take-all updating mode is reached. To describe the fuzzy-to-crisp transition process in more detail, we "rst denote I (k) as the set of weight vectors to be updated when x is used as the input trainH H ing vector at kth iteration. Therefore, given any training vector x , the set H I (k)"z(i), i"1,2, N for k"1,2, K. Since all weight vectors in the VQ are H included in the set I (k) for updating, similar to FECVQ, the computation of H membership function is based on (10). For k'K, we gradually reduce the number of weight vectors in I (k). In the procedure for removing weight vectors at iteration H k'K, we "rst compute JM (k!1) de"ned as 1 JM (k!1)" J(x , z(i)), H N(I (k!1)) z I H G Z H I\
(12)
where N( I (k!1)) is the number of weight vectors in set I (k!1). Based on H H I (k!1) and JM (k!1), the set I (k) is constructed according to H H I (k)"z(i)3I (k!1): J(x , z(i))4JM (k!1). H H H
(13)
Given x , for the weight vectors which are outside I (k), their corresponding memberH H ship functions are set to zero, and the weight vectors are not updated. On the other hand, for the weight vectors z(i) which are inside I (k), the corresponding membership H functions are then given as (1/J(x , z(i))) K\ H " . G H z I (1/J(x , z(l))) K\ J Z H I H
(14)
These weight vectors are then updated according to (11) with given G H in (14). Note that when k increases, the number of weight vectors in I (k) decreases. H Therefore, in the FECCL algorithm, the iteration number k might achieve a point where, for each of the training vector x 3X, its I (k) contain only one element. In this H H case, the corresponding weight vector updating process is identical to the winnertake-all scheme adopted by the ECCL algorithm. At each iteration of FECCL, after all the training vectors x have been used for H weight vector updating, the source encoder w is then constructed by the following rule: w(x)"arg min J(x, z(i)), XGX,
(15)
where x is any input vector for encoding, and z(i), i"1,2, N, are the reproduction vectors obtained from weight vector updating.
W.-J. Hwang et al. / Neurocomputing 37 (2001) 197}208
203
After w has been determined, we then partition the training set X into N clusters C ,2, C , where C "x : w(x )"i. The probability p(i) that the reproduction , G H H vector z(i) will be selected by the source encoder can then be approximated as p(i)"N(C )/t, where N(C ) is the number of training vectors in C . The optimal G G G channel encoder is then designed according to (5). We note that the selection of value e!ects the resulting rate of the VQ design. However, similar to other variable-rate VQ methods, it is di$cult to "nd the relationship between average rate of a VQ and value analytically. Hence, the value achieving a given target rate R might have to be found on the trial-and-error basis. Finally, as the number of iterations k increases, the FECCL algorithm gradually reduce to ECCL algorithm due to the fuzzy-to-crisp transition process. Since ECCL is guaranteed to converge [5], the weight vectors of the FECCL algorithm also converges for su$cient number of iterations.
4. Simulation results This section presents some simulation results of the FECCL algorithm. The ECVQ, ECCL, and SCS (soft competitive scheme) [9] algorithms are also implemented for comparison purpose. Note that, although the SCS algorithm is primarily used for the "xed-rate VQ design, the algorithm can easily be adapted for constructing a variablerate VQ by replacing the squared distance measure in the algorithm with the modi"ed distance measure shown in Eq. (3) for the training of reproduction vectors z. An iterative scheme similar to that shown in Fig. 1 is then used for the design of of source encoder w and channel encoder . The training images for the VQ designs are two 512;512 images `Peppera and `Lenaa. The dimension of training vectors and weight vectors is 4;4. Fig. 2 elaborates the dependence of the ECVQ, ECCL, FECCL and SCS algorithms on the selection of initial reproduction vectors. Several methods are possible to obtain an initial set of reproduction vectors for initialing these VQ design algorithms. An obvious choice is a random initial set. By constructing these VQ algorithms on di!erent random initial sets independently, we can study the in#uence of the initial reproduction vectors on the behavior of these algorithms in a statistical manner. Figs. 2(a)}(d) show the distribution of C(w, z, ) of the variable-rate VQs constructed by the ECVQ, ECCL, FECCL and SCS algorithms, respectively. The distribution of each algorithm is obtained by independently applying that algorithm 200 times using randomly chosen initial reproduction vectors. In fact, the value of every elements of each reproduction vector is 100.0#r, and r is taken randomly from the interval [!25,25]. The value and the number of codewords N for all the algorithms are the same, and are given by "25 and N"256, respectively. The degree of membership of FECCL is set to be m"1.1. From Fig. 2, we observe that the distribution of FECCL algorithm has a better concentration. To further illustrate this fact, the mean and variance of the distribution of each algorithm are listed in Table 1. Based on the table, it can be seen that the mean
204
W.-J. Hwang et al. / Neurocomputing 37 (2001) 197}208
Fig. 2. Distribution of C(w, z,) of ECVQ, ECCL, FECCL and SCS algorithms. In the histogram, the results of 200 independent runs are displayed for each algorithm (a) histogram for ECVQ, (b) histogram for ECCL, (c) histogram for FECCL, and (d) histogram for SCS.
and variance of the distribution of FECCL algorithm are lower than those of other algorithms. Therefore, as compared with other methods, the FECCL algorithm has superior and more robust performance.
W.-J. Hwang et al. / Neurocomputing 37 (2001) 197}208
205
Table 1 The mean and variance of the distribution of C(w, z, ) of the ECVQ, ECCL, FECCL and SCS algorithms shown in Fig. 2
Mean Variance
ECVQ
ECCL
FECCL
SCS
92.43 5.83
87.74 1.49
78.33 0.83
90.58 1.28
Fig. 3. PSNRS of FECCL algorithm designed subject to di!erent number of codewords N.
To further illustrate the features of the FECCL algorithm, Fig. 3 shows the ratedistortion performance of FECCL algorithm for di!erent number of reproduction vectors N. The PSNR values shown in the "gure are de"ned as 10 log 255/ (MSE of the reconstructed images). It can be observed from Fig. 3 that, for a given average rate, the PSNR value improves as N becomes larger. The selection of value in the FECCL algorithm a!ects the resulting average rate of FECCL algorithm. Fig. 4 shows the relationship between and the average rate of FECCL algorithm with N"128. Based on the "gure, we observe that larger values result in lower average rates after VQ design. Since the parameters N and can be speci"ed independently before the VQ design, from Figs. 3 and 4 we conclude that the FECCL algorithm can achieve di!erent rate-distortion performances by adopting various combinations of and N in accordance with the application requirements.
206
W.-J. Hwang et al. / Neurocomputing 37 (2001) 197}208
Fig. 4. Rate- relationship of FECCL algorithm.
5. Conclusion This paper presents a novel variable-rate VQ design algorithm using fuzzy competitive learning technique. A novel fuzzy-to-crisp transition process is also employed so that the resulting reproduction vectors are well-suited for crisp encoding process. The algorithm has the advantages of both the ECCL and FECVQ algorithms that its rate-distortion performance is more robust and superior than that of other existing variable-rate VQ design algorithms. The algorithm therefore can be an e!ective tool for the applications where high rate-distortion performance is desired. Acknowledgements The authors would like to thank the anonymous reviewers for their valuable comments and suggestions. References [1] P.A. Chou, T. Lookabaugh, R.M. Gray, Entropy-constrained vector quantization, IEEE Trans. Acoust. Speech Signal Process. 37 (1989) 31}42.
W.-J. Hwang et al. / Neurocomputing 37 (2001) 197}208
207
[2] A. Gersho, R.M. Gray, Vector Quantization and Signal Compression, Kluwer, Norwood, MA, 1992. [3] R.W. Hamming, Coding and Information Theory, 2nd Edition, Prentice-Hall, Englewood Cli!s, NJ, 1986. [4] W.J. Hwang, S.L. Hong, A fuzzy entropy-constrained vector quantizer design algorithm and its applications to image coding, IEICE Trans. Fund. Electron. Commun. Comput. Sci. E82-A (1999) 1109}1116. [5] W.J. Hwang, B.Y. Ye, S.C. Liao, A novel entropy-constrained competitive learning algorithm for vector quantization, Neurocomputing 25 (1999) 133}147. [6] N.B. Karayiannis, P.I. Pai, Fuzzy vector quantization algorithms and their application in image compression, IEEE Trans. Image Process. 4 (1995) 1193}1201. [7] T. Kohonen, Self-Organization and Associative Memory, 3rd Edition, Springer, Germany, 1989. [8] I.H. Witten, R.M. Neal, J.G. Cleary, Arithmetic coding for data compression, Commun. ACM 30 (1987) 520}539. [9] E. Yair, K. Zeger, A. Gersho, Competitive learning for vector quantizer design, IEEE Trans. Signal Process. 40 (1992) 294}309.
Wen-Jyi Hwang received his diploma in electronics engineering from National Taipei Institute of Technology, Taiwan, in 1987, and M.S.E.C.E. and Ph.D. degrees from the University of Massachusetts at Amherst in 1990 and 1993, respectively. In 1993, he joined the faculty at Chung Yuan Christian University, where he is currently a Professor of Electrical Engineering. His research interests are in image coding and digital signal processing.
Faa-Jeng Lin received the B.S. and M.S. degrees in electrical engineering from the National Cheng Kung University, Tainan, Taiwan, and the Ph.D. degree in electrical engineering from the National Tsing Hua University, Hsin Chu, Taiwan, in 1983, 1985, and 1993, respectively. During 1985}1989 he was with the Chung Shan Institute of Science and Technology, Tao Yuan, Taiwan, as a team leader of automatic test equipment and microcomputer system design division. He is currently a Professor in the Department of Electrical Engineering, Chung Yuan Christian University, Chung Li, Taiwan. His research interests include motor servo drives, computer-based control systems, control theory applications, and power electronics.
Shi-Chiang Liao obtained his B.S. degree in electronics engineering from Fu Jen Catholic University and M.S. degree in electrical engineering from Chung Yuan Christian University in 1997 and 1999, respectively. His research interests include signal compression, image processing and neural networks.
208
W.-J. Hwang et al. / Neurocomputing 37 (2001) 197}208 Jeng-Hsin Huang was born in Yingko Town, Taipei Hsien, Taiwan, Republic of China, on January 21, 1978. He received the B.S. degrees in electrical engineering from the Chung Yuan Christian University in 2000. Since 2000 he has been working toward the M.S degree in the Department of Electrical Engineering at Chung Yuan Christian University. His research interests include the digital signal processing, computer architecture and programming.
A novel fuzzy entropy-constrained competitive learning algorithm for image coding Wen-Jyi Hwang*, Faa-Jeng Lin, Shi-Chiang Liao, Jeng-Hsin Huang Department of Electrical Engineering, Chung Yuan Christian University, Chungli, 32023, Taiwan Accepted 17 November 2000
Abstract A novel variable-rate vector quantizer (VQ) design algorithm using both fuzzy and competitive learning technique is presented. The algorithm enjoys better rate-distortion performance than that of other existing fuzzy clustering and competitive learning algorithms. In addition, the learning algorithm is less sensitive to the selection of initial reproduction vectors. Therefore, the algorithm can be an e!ective alternative to the existing variable-rate VQ algorithms for signal compression. 2001 Elsevier Science B.V. All rights reserved. Keywords: Image coding; Vector quantization; Competitive learning; Fuzzy clustering
1. Introduction Entropy-constrained vector quantization (ECVQ) [1] algorithm is a well-known method for variable-rate VQ design [2]. The technique employs batch training process where components of VQ are constructed iteratively one at a time. In addition, during the course of the iterative construction procedure, ECVQ algorithm utilizes crisp clustering technique where each training vector is assigned to only one cluster for training set partitioning. The algorithm have the following disadvantages. First, the performance of iterative design procedure is sensitive to the selection of initial reproduction vectors. Therefore, a bad local optimum might be achieved if an improper set of initial codewords is selected. Moreover, since many training vectors might have features belonging to more than one clusters, it is not ideal to assign each training vector to only one cluster during set partitioning process.
* Corresponding author. Tel.: 886-3-456-3171-4808; fax: 886-3-456-3160. E-mail address: [email protected] (W.-J. Hwang). 0925-2312/01/$ - see front matter 2001 Elsevier Science B.V. All rights reserved. PII: S 0 9 2 5 - 2 3 1 2 ( 0 0 ) 0 0 3 5 1 - 9
198
W.-J. Hwang et al. / Neurocomputing 37 (2001) 197}208
To alleviate these drawbacks, a number of competitive learning (CL) algorithms [5,7] and fuzzy clustering techniques [4,6] have been proposed for VQ design. As compared with the traditional iterative algorithms, the CL-based algorithms are shown to have more robust rate-distortion performance because the training process of these algorithms are performed in continuous mode. The fuzzy clustering algorithms can also improve the performance of the VQ design since better codevectors might be found by fuzzifying the boundary of clusters. Although using CL or fuzzy clustering algorithm along can improve the design of VQs, it is expected that the fusion of these two techniques can improve the performance of VQs further because the resulting VQs can have the advantages of both algorithms. The objective of this letter is to present a novel variable-rate VQ design algorithm, termed fuzzy entropy-constrained competitive learning (FECCL) algorithm, which performs VQ training in both continuous and fuzzy mode. To achieve continuous training, similar to the CL algorithms, the weight vectors of the algorithm are updated for each input training vector. However, the continuous training process is not based on the winner-take-all scheme adopted by usual CL algorithms, which results in crisp training. In the FECCL algorithm, for any input training vector, its membership grade in each cluster is "rst computed. All the weight vectors are then updated according to these membership grades. In order to control the average rate of the VQ after training process, when computing the membership functions, the length of channel codeword representing each weight vector will be taken into account. Simulation results show that the FECCL algorithm has more robust and superior ratedistortion performance than other existing VQ design algorithms where only crisp or batch training are used.
2. Preliminaries Recall that the encoder W of a variable-rate VQ [1,2] can be decomposed into two parts. The "rst part is the function w which maps a source vector x into a source index i. The second part, , maps the source index i into a channel codeword c . The G decoder Z can be similarly decomposed into two parts: \ and z. The mapping \ is the inverse of , and maps c back to source index i. The mapping z outputs a G reproduction vector z(i) for the source index i. The pair (W, Z) is referred to as a variable-rate VQ because a source vector is represented by a single channel codeword, and the channel codewords for di!erent source vectors are usually not of the same length. Given a source vector x, let (w(x)) be the length of the channel codeword representing x. In addition, let d (u,) be the mean-squared distance between vectors L u and . For a given set of training vectors X"x , j"1,2, t, where t is the number H of training vectors in the training set, the average rate and average distortion of the VQ are then given as R"1/ntR (w(x )) and D"1/ntR d (x , z(w(x ))), respecH H H L H H tively. Let D (R) be the function indicates the lowest possible average distortion that L can be achieved by a VQ with blocklength n under the rate constraint R. Although the
W.-J. Hwang et al. / Neurocomputing 37 (2001) 197}208
199
Lagragian methods cannot be used for "nding D(R) [1], they can be used for "nding the convex hull of D(R) be minimizing the following functional: R C(w, z, )" d (x , z(w(x )))#(w(x )). L H H H H
(1)
Here, has an interpretation as the slope of a line supporting the convex hull. We now discuss the optimal conditions for w, z and that minimizes the functional C(w, z, ). Fix and z, the mapping w that minimizes (1) is the one that satis"es w(x )"arg min J(x , z(i)), H H XGX,
(2)
where N is the number of reproduction vectors, and J(x , z(i))"d (x , z(i))#(i) H L H
(3)
can be viewed as the modi"ed distance measure between x and z(i). H Next, for "xed w and , the mapping z that minimizes (1) is given as z(i)"centroid of x : w(x )"i. H H
(4)
Finally, the optimal channel encoder for "xed w and z is the entropy encoder designed based on the probability p(i) of each reproduction vector z(i). That is, the optimal satis"es (i)"log
1 . p(i)
(5)
There are a number of entropy coding techniques for the implementation of mapping , such as the Hu!man codes [3] and Arithmetic codes [8] which can yield an average rate closed to the entropy of the indices. The ECVQ algorithm uses (2), (4), (5) iteratively for the minimization of C(w, z, ) given in (1). The scheme is sensitive to the initial parameters since reproduction vectors are constructed in batch mode. If the initial and z are improperly chosen, then the ECVQ algorithm can easily converge to a bad local minimum. In addition, the clustering operation is a crisp clustering process. Better codewords might be found if fuzzy clustering process is employed. The objective of ECCL algorithm [5] is also to minimize C(w, z, ). However, unlike the ECVQ algorithm, the ECCL performs the continuous training process for the codebook construction. Given , the ECCL algorithm "nds the optimal w and z satisfying Eqs. (2) and (4), respectively, using the following winner-take-all
200
W.-J. Hwang et al. / Neurocomputing 37 (2001) 197}208
competition scheme [5]: z(iH)Qz(iH)#(iH)(x !z(iH)), H
(6)
z(i)Qz(i), iOiH,
(7)
where x is an input training vector in the training set X, (i) is the learning rate of the H ith weight vector, and iH is the index of the winning weight vector based on the following competition rule: iH"arg min J(x , z(k)). H XIX,
(8)
Since the ECCL algorithm performs weight vector updating for each input training vector, the algorithm has more robust performance as compared with ECVQ algorithm. The FECVQ algorithm [4] performs fuzzy clustering during the course of iterative training process. In order to incorporate the membership functions of training vectors into the clustering process, the objective of FECVQ is to minimize the following objective function: R , C(, z, )" ( )K(J(x , z(i))), G H H H G
(9)
where m'1 indicates the degree of fuzziness, and , i"1,2, N is the set of G H membership values associated with training vector x , satisfying , "1 and H G G H 50. The algorithm employs an iterative training procedure that optimizes , z, G H and one at a time. In particular, for "xed z and , the optimal membership functions minimizing C(, z, ) are given as [4] (1/m(x , z(i))) K\ H " . G H , (1/J(x , z(l))) K\ J H
(10)
Since performing fuzzy clustering for codebook construction, FECVQ can "nd better reconstruction vectors. However, FECVQ also performs training in batch mode, and therefore is sensitive to initial parameters.
3. FECCL algorithm The FECCL algorithm has the advantages of both FECVQ and ECCL algorithms. In the algorithm, similar to ECCL technique, the training process is performed in continuous mode. However, unlike the ECCL algorithm, where the weight vector updating is based on winner-take-all scheme, the FECCL algorithm updates weight vectors according to the membership functions, which measure the degree of closeness between the input training vector to each weight vector. During the course of FECCL
W.-J. Hwang et al. / Neurocomputing 37 (2001) 197}208
201
Fig. 1. Design procedure of FECCL algorithm.
design, the reproduction vectors z(i), i"1,2, N, are "rst updated in fuzzy and continuous mode based on a "xed channel encoder . The source encoder w is then constructed based on the updated z. After that, the channel encoder is designed according to the new w. The same process is repeated until the convergence of the algorithm. The design procedure of FECCL algorithm is summarized in Fig. 1. We now discuss each step of the algorithm in more detail. For any input training vector x 3X, the FECCL algorithm updates weight vectors z(i), i"1,2, N, accordH ing to the following rule: z(i)Qz(i)#(i)( )K(z(i)!x ), (11) G H H where (i) is the learning rate of z(i). At the "rst K iterations of FECVQ algorithm, where K'0 can be prespeci"ed before the design, the membership functions , i"1,2, N, are computed according to (10), and therefore all the weight vectors G H are updated for each input training vector x . Consequently, FECVQ algorithm is H insensitive to the selection of initial weight vectors. After K iterations, since the VQ encoding process is a crisp selection process, in order to "nd codewords well-suited for crisp encoding, the FECCL algorithm employs a novel fuzzy-to-crisp transition
202
W.-J. Hwang et al. / Neurocomputing 37 (2001) 197}208
process for codebook construction. In the process, as the number of iterations increases, the number for weight vectors to be updated for each training vector is gradually reduced until winner-take-all updating mode is reached. To describe the fuzzy-to-crisp transition process in more detail, we "rst denote I (k) as the set of weight vectors to be updated when x is used as the input trainH H ing vector at kth iteration. Therefore, given any training vector x , the set H I (k)"z(i), i"1,2, N for k"1,2, K. Since all weight vectors in the VQ are H included in the set I (k) for updating, similar to FECVQ, the computation of H membership function is based on (10). For k'K, we gradually reduce the number of weight vectors in I (k). In the procedure for removing weight vectors at iteration H k'K, we "rst compute JM (k!1) de"ned as 1 JM (k!1)" J(x , z(i)), H N(I (k!1)) z I H G Z H I\
(12)
where N( I (k!1)) is the number of weight vectors in set I (k!1). Based on H H I (k!1) and JM (k!1), the set I (k) is constructed according to H H I (k)"z(i)3I (k!1): J(x , z(i))4JM (k!1). H H H
(13)
Given x , for the weight vectors which are outside I (k), their corresponding memberH H ship functions are set to zero, and the weight vectors are not updated. On the other hand, for the weight vectors z(i) which are inside I (k), the corresponding membership H functions are then given as (1/J(x , z(i))) K\ H " . G H z I (1/J(x , z(l))) K\ J Z H I H
(14)
These weight vectors are then updated according to (11) with given G H in (14). Note that when k increases, the number of weight vectors in I (k) decreases. H Therefore, in the FECCL algorithm, the iteration number k might achieve a point where, for each of the training vector x 3X, its I (k) contain only one element. In this H H case, the corresponding weight vector updating process is identical to the winnertake-all scheme adopted by the ECCL algorithm. At each iteration of FECCL, after all the training vectors x have been used for H weight vector updating, the source encoder w is then constructed by the following rule: w(x)"arg min J(x, z(i)), XGX,
(15)
where x is any input vector for encoding, and z(i), i"1,2, N, are the reproduction vectors obtained from weight vector updating.
W.-J. Hwang et al. / Neurocomputing 37 (2001) 197}208
203
After w has been determined, we then partition the training set X into N clusters C ,2, C , where C "x : w(x )"i. The probability p(i) that the reproduction , G H H vector z(i) will be selected by the source encoder can then be approximated as p(i)"N(C )/t, where N(C ) is the number of training vectors in C . The optimal G G G channel encoder is then designed according to (5). We note that the selection of value e!ects the resulting rate of the VQ design. However, similar to other variable-rate VQ methods, it is di$cult to "nd the relationship between average rate of a VQ and value analytically. Hence, the value achieving a given target rate R might have to be found on the trial-and-error basis. Finally, as the number of iterations k increases, the FECCL algorithm gradually reduce to ECCL algorithm due to the fuzzy-to-crisp transition process. Since ECCL is guaranteed to converge [5], the weight vectors of the FECCL algorithm also converges for su$cient number of iterations.
4. Simulation results This section presents some simulation results of the FECCL algorithm. The ECVQ, ECCL, and SCS (soft competitive scheme) [9] algorithms are also implemented for comparison purpose. Note that, although the SCS algorithm is primarily used for the "xed-rate VQ design, the algorithm can easily be adapted for constructing a variablerate VQ by replacing the squared distance measure in the algorithm with the modi"ed distance measure shown in Eq. (3) for the training of reproduction vectors z. An iterative scheme similar to that shown in Fig. 1 is then used for the design of of source encoder w and channel encoder . The training images for the VQ designs are two 512;512 images `Peppera and `Lenaa. The dimension of training vectors and weight vectors is 4;4. Fig. 2 elaborates the dependence of the ECVQ, ECCL, FECCL and SCS algorithms on the selection of initial reproduction vectors. Several methods are possible to obtain an initial set of reproduction vectors for initialing these VQ design algorithms. An obvious choice is a random initial set. By constructing these VQ algorithms on di!erent random initial sets independently, we can study the in#uence of the initial reproduction vectors on the behavior of these algorithms in a statistical manner. Figs. 2(a)}(d) show the distribution of C(w, z, ) of the variable-rate VQs constructed by the ECVQ, ECCL, FECCL and SCS algorithms, respectively. The distribution of each algorithm is obtained by independently applying that algorithm 200 times using randomly chosen initial reproduction vectors. In fact, the value of every elements of each reproduction vector is 100.0#r, and r is taken randomly from the interval [!25,25]. The value and the number of codewords N for all the algorithms are the same, and are given by "25 and N"256, respectively. The degree of membership of FECCL is set to be m"1.1. From Fig. 2, we observe that the distribution of FECCL algorithm has a better concentration. To further illustrate this fact, the mean and variance of the distribution of each algorithm are listed in Table 1. Based on the table, it can be seen that the mean
204
W.-J. Hwang et al. / Neurocomputing 37 (2001) 197}208
Fig. 2. Distribution of C(w, z,) of ECVQ, ECCL, FECCL and SCS algorithms. In the histogram, the results of 200 independent runs are displayed for each algorithm (a) histogram for ECVQ, (b) histogram for ECCL, (c) histogram for FECCL, and (d) histogram for SCS.
and variance of the distribution of FECCL algorithm are lower than those of other algorithms. Therefore, as compared with other methods, the FECCL algorithm has superior and more robust performance.
W.-J. Hwang et al. / Neurocomputing 37 (2001) 197}208
205
Table 1 The mean and variance of the distribution of C(w, z, ) of the ECVQ, ECCL, FECCL and SCS algorithms shown in Fig. 2
Mean Variance
ECVQ
ECCL
FECCL
SCS
92.43 5.83
87.74 1.49
78.33 0.83
90.58 1.28
Fig. 3. PSNRS of FECCL algorithm designed subject to di!erent number of codewords N.
To further illustrate the features of the FECCL algorithm, Fig. 3 shows the ratedistortion performance of FECCL algorithm for di!erent number of reproduction vectors N. The PSNR values shown in the "gure are de"ned as 10 log 255/ (MSE of the reconstructed images). It can be observed from Fig. 3 that, for a given average rate, the PSNR value improves as N becomes larger. The selection of value in the FECCL algorithm a!ects the resulting average rate of FECCL algorithm. Fig. 4 shows the relationship between and the average rate of FECCL algorithm with N"128. Based on the "gure, we observe that larger values result in lower average rates after VQ design. Since the parameters N and can be speci"ed independently before the VQ design, from Figs. 3 and 4 we conclude that the FECCL algorithm can achieve di!erent rate-distortion performances by adopting various combinations of and N in accordance with the application requirements.
206
W.-J. Hwang et al. / Neurocomputing 37 (2001) 197}208
Fig. 4. Rate- relationship of FECCL algorithm.
5. Conclusion This paper presents a novel variable-rate VQ design algorithm using fuzzy competitive learning technique. A novel fuzzy-to-crisp transition process is also employed so that the resulting reproduction vectors are well-suited for crisp encoding process. The algorithm has the advantages of both the ECCL and FECVQ algorithms that its rate-distortion performance is more robust and superior than that of other existing variable-rate VQ design algorithms. The algorithm therefore can be an e!ective tool for the applications where high rate-distortion performance is desired. Acknowledgements The authors would like to thank the anonymous reviewers for their valuable comments and suggestions. References [1] P.A. Chou, T. Lookabaugh, R.M. Gray, Entropy-constrained vector quantization, IEEE Trans. Acoust. Speech Signal Process. 37 (1989) 31}42.
W.-J. Hwang et al. / Neurocomputing 37 (2001) 197}208
207
[2] A. Gersho, R.M. Gray, Vector Quantization and Signal Compression, Kluwer, Norwood, MA, 1992. [3] R.W. Hamming, Coding and Information Theory, 2nd Edition, Prentice-Hall, Englewood Cli!s, NJ, 1986. [4] W.J. Hwang, S.L. Hong, A fuzzy entropy-constrained vector quantizer design algorithm and its applications to image coding, IEICE Trans. Fund. Electron. Commun. Comput. Sci. E82-A (1999) 1109}1116. [5] W.J. Hwang, B.Y. Ye, S.C. Liao, A novel entropy-constrained competitive learning algorithm for vector quantization, Neurocomputing 25 (1999) 133}147. [6] N.B. Karayiannis, P.I. Pai, Fuzzy vector quantization algorithms and their application in image compression, IEEE Trans. Image Process. 4 (1995) 1193}1201. [7] T. Kohonen, Self-Organization and Associative Memory, 3rd Edition, Springer, Germany, 1989. [8] I.H. Witten, R.M. Neal, J.G. Cleary, Arithmetic coding for data compression, Commun. ACM 30 (1987) 520}539. [9] E. Yair, K. Zeger, A. Gersho, Competitive learning for vector quantizer design, IEEE Trans. Signal Process. 40 (1992) 294}309.
Wen-Jyi Hwang received his diploma in electronics engineering from National Taipei Institute of Technology, Taiwan, in 1987, and M.S.E.C.E. and Ph.D. degrees from the University of Massachusetts at Amherst in 1990 and 1993, respectively. In 1993, he joined the faculty at Chung Yuan Christian University, where he is currently a Professor of Electrical Engineering. His research interests are in image coding and digital signal processing.
Faa-Jeng Lin received the B.S. and M.S. degrees in electrical engineering from the National Cheng Kung University, Tainan, Taiwan, and the Ph.D. degree in electrical engineering from the National Tsing Hua University, Hsin Chu, Taiwan, in 1983, 1985, and 1993, respectively. During 1985}1989 he was with the Chung Shan Institute of Science and Technology, Tao Yuan, Taiwan, as a team leader of automatic test equipment and microcomputer system design division. He is currently a Professor in the Department of Electrical Engineering, Chung Yuan Christian University, Chung Li, Taiwan. His research interests include motor servo drives, computer-based control systems, control theory applications, and power electronics.
Shi-Chiang Liao obtained his B.S. degree in electronics engineering from Fu Jen Catholic University and M.S. degree in electrical engineering from Chung Yuan Christian University in 1997 and 1999, respectively. His research interests include signal compression, image processing and neural networks.
208
W.-J. Hwang et al. / Neurocomputing 37 (2001) 197}208 Jeng-Hsin Huang was born in Yingko Town, Taipei Hsien, Taiwan, Republic of China, on January 21, 1978. He received the B.S. degrees in electrical engineering from the Chung Yuan Christian University in 2000. Since 2000 he has been working toward the M.S degree in the Department of Electrical Engineering at Chung Yuan Christian University. His research interests include the digital signal processing, computer architecture and programming.