- Email: [email protected]
Reply to the comments
Signal Processing 20 (1990) 93-94 Elsevier
REPLY
TO THE
93
COMMENTS
T.K. S A R K A R and X. Y A N G Department of Electrical Engineering, Syracuse University, Syracuse, N Y 13244-1240, U.S.A. Abstract. The author is wrong, in assuming that the algorithms in [5] are the same algorithms that were published earlier in
[2]. The algorithm in [2] is the conjugate gradient a la "Fletcher Reeves" whereas the algorithm in [5] is a modified version of the algorithm proposed by Towsend and Johnson [6]. To an interested reader, there are many papers in the mathematical literature which elaborate on the different versions of the conjugate gradient algorithm for nonlinear optimization problems. In [6], the difference between the Fletcher algorithm [2] and the algorithm used in [5l is outlined.
1. Introduction The assumption that the algorithm of [5] is iden= tical to that in [2] is incorrect. The difference is outlined in [6]. For a researcher, who wants to find more about conjugate gradient methods, there are several papers available that outline the differences.
2. Operator or matrix? The normal operator A * A behaves as A only if the operator is bounded. Therefore, A X = Y is equivalent to A * A X = A * Y only for certain special classes of operators, primarily for b o u n d e d operators. This is a well-known result in operator theory and that is why we did not mention it in the paper. But on hindsight, it looks like we should have mentioned it.
4. Application of the conjugate gradient algorithm The point made by the author is that the conjugate gradient method should have been restarted and this would have provided faster convergence. In mathematical literature, the restart of the conjugate gradient algorithm is mentioned. It is also mentioned that the restart is preferred after N iterations. Since in our problem of interest the
convergence is attained in less than N iterations, restart is only deemed inappropriate but there is no scientific basis to restart the iteration after less than N iterations. Hence, in our paper, restart of the algorithm was not mentioned.
5. Computation of more than one eigenvalue In principle, the method mentioned in [5] could be used to find eigenvalues other than the largest or the smallest one. The author claims that 'contamination' of the eigenvector occurs and hence purification is necessary. It turns out that this is a problem depending on how the eigenvalues and eigenvectors are extracted. If one finds the smallest eigenvalue and the eigenvector associated with it and then seeks the next largest eigenvalue by restarting the iteration with an initial guess which is orthogonal to the eigenvector corresponding to the 'smallest eigenvalue', contamination of the eigenvalue occurs as suggested by the author. However, one can do the computation, a little more intelligently, by shifting the smallest eigenvalue and then computing the smallest eigenvalue of [A+otAminpvT]. In this, contamination is not a problem as one is seeking the smallest eigenvalue of the modified matrix [ A + a A m i . vvT]. Here v is the eigenvector corresponding to hm~., and a is a scale factor.
94
Comments
6. Conclusions
The suggestion made by the author that one should always perform a 'partial singular value decomposition' in order to find the eigenvalues of a Hermitian matrix is absolutely ridiculous. This is because in computing the singular values, one is squaring the condition number of the matrix and
Signal Processing
hence the rate of convergence becomes quite slow. For a Hermitian matrix, the paper provides a fast, stable and accurate solution procedure for computing iteratively the smallest/largest eigenvalues and the corresponding eigenvector. For a non Hermitian matrix, the singular value decomposition is a viable way, but that was not the subject matter of our paper.
REPLY
TO THE
93
COMMENTS
T.K. S A R K A R and X. Y A N G Department of Electrical Engineering, Syracuse University, Syracuse, N Y 13244-1240, U.S.A. Abstract. The author is wrong, in assuming that the algorithms in [5] are the same algorithms that were published earlier in
[2]. The algorithm in [2] is the conjugate gradient a la "Fletcher Reeves" whereas the algorithm in [5] is a modified version of the algorithm proposed by Towsend and Johnson [6]. To an interested reader, there are many papers in the mathematical literature which elaborate on the different versions of the conjugate gradient algorithm for nonlinear optimization problems. In [6], the difference between the Fletcher algorithm [2] and the algorithm used in [5l is outlined.
1. Introduction The assumption that the algorithm of [5] is iden= tical to that in [2] is incorrect. The difference is outlined in [6]. For a researcher, who wants to find more about conjugate gradient methods, there are several papers available that outline the differences.
2. Operator or matrix? The normal operator A * A behaves as A only if the operator is bounded. Therefore, A X = Y is equivalent to A * A X = A * Y only for certain special classes of operators, primarily for b o u n d e d operators. This is a well-known result in operator theory and that is why we did not mention it in the paper. But on hindsight, it looks like we should have mentioned it.
4. Application of the conjugate gradient algorithm The point made by the author is that the conjugate gradient method should have been restarted and this would have provided faster convergence. In mathematical literature, the restart of the conjugate gradient algorithm is mentioned. It is also mentioned that the restart is preferred after N iterations. Since in our problem of interest the
convergence is attained in less than N iterations, restart is only deemed inappropriate but there is no scientific basis to restart the iteration after less than N iterations. Hence, in our paper, restart of the algorithm was not mentioned.
5. Computation of more than one eigenvalue In principle, the method mentioned in [5] could be used to find eigenvalues other than the largest or the smallest one. The author claims that 'contamination' of the eigenvector occurs and hence purification is necessary. It turns out that this is a problem depending on how the eigenvalues and eigenvectors are extracted. If one finds the smallest eigenvalue and the eigenvector associated with it and then seeks the next largest eigenvalue by restarting the iteration with an initial guess which is orthogonal to the eigenvector corresponding to the 'smallest eigenvalue', contamination of the eigenvalue occurs as suggested by the author. However, one can do the computation, a little more intelligently, by shifting the smallest eigenvalue and then computing the smallest eigenvalue of [A+otAminpvT]. In this, contamination is not a problem as one is seeking the smallest eigenvalue of the modified matrix [ A + a A m i . vvT]. Here v is the eigenvector corresponding to hm~., and a is a scale factor.
94
Comments
6. Conclusions
The suggestion made by the author that one should always perform a 'partial singular value decomposition' in order to find the eigenvalues of a Hermitian matrix is absolutely ridiculous. This is because in computing the singular values, one is squaring the condition number of the matrix and
Signal Processing
hence the rate of convergence becomes quite slow. For a Hermitian matrix, the paper provides a fast, stable and accurate solution procedure for computing iteratively the smallest/largest eigenvalues and the corresponding eigenvector. For a non Hermitian matrix, the singular value decomposition is a viable way, but that was not the subject matter of our paper.