- Email: [email protected]
Special issue on support vector machines
Neurocomputing 55 (2003) 1 – 3 www.elsevier.com/locate/neucom
Editorial
Special issue on support vector machines Support vector machines (SVMs) are currently a very active research area within machine learning. Motivated by statistical learning theory, SVMs have been successfully applied to numerous tasks, among others in data mining, computer vision, and bioinformatics. SVMs are examples of a broader category of learning approaches which utilize the concept of kernel substitution, which makes the task of learning more tractable by exploiting an implicit mapping into a high-dimensional space. SVMs have many appealing properties for machine learning. For example, the classic SVM learning task involves convex quadratic programming, a problem that does not su,er from the ‘local minima’ problem and whose solution may easily be found by using one of the many specially e/cient algorithms developed for it in the optimization theory. Furthermore, recently developed model selection strategies can be applied, so that few, if any, learning parameters need to be set by the operator. Above all, they have been found to work very well in practice. This special issue on SVMs includes papers from a broad range of topics. We have grouped them under the following two categories: methodology and applications.
1. Methodology There are several papers that consider issues relating to SVM design. Next to the basic theory and the training algorithm, model (feature) selection and hyperparameter tuning are important aspects. As SVM is a kernel method, the issue of choosing kernels for special situations has recently drawn a lot of attention. The standard SVM returns only class labels; therefore, the probabilistic treatment of SVMs needs further investigation. Moreover, in addition to some earlier comparisons between SVMs and other methods, more serious benchmarks are essential. Elaborated techniques and key applications are presented in “Advanced support vector machines and kernel methods” by Sanchez. These include topics of numerical optimization, working set selection, improved generalization, model selection, and parameter tuning as well as classi7cation, regression, text categorization, computer vision, and bioinformatics. In “Feature vector selection and projection using kernels”, Baudat and Anouar discuss the use of kernel methods to extract relevant data points. Data are then projected onto the subspace of the selected points where traditional algorithms can be applied. This reduces the number of data points used in practical situations. c 2003 Published by Elsevier B.V. 0925-2312/03/$ - see front matter doi:10.1016/S0925-2312(03)00428-4
2
Editorial / Neurocomputing 55 (2003) 1 – 3
In “Finite Newton method for Lagrangian support vector machine classi7cation”, Fung and Mangasarian propose an e,ective procedure for solving large linear SVMs. Earlier, Lagrangian SVMs were solved by a procedure with more iterations. Using a Newton method where the cost of each iteration is higher, the total number of iterations is largely reduced. SVM was originally designed for two-class classi7cation. There are several approaches to extend it for multi-class data sets. In “K-SVCR: a support vector machine for multi-class classi7cation”, Angulo, Parra, and Catala discuss a method which improves the pairwise multiclass strategy. In “A geometric approach to support vector regression”, Bi and Bennett give a novel ‘nearest point’ formulation for SVM regression. This work opens the way for the development of new algorithms for designing SVM regressors. Good and easy-to-evaluate estimates of generalization error are necessary for fast tuning of SVM hyperparameters. In “Hyperparameter design criteria for support vector classi7ers”, Anguita, Ridella, Rivieccio and Zunino empirically explore the usefulness of a new estimate called the ‘maximal discrepancy criterion’. In “On problem-oriented kernel re7ning”, Parrado-Hernandez, Arenas-Garcia, Mora-Jimenez, and Navia-Vazques present a method of training support vector machines using a reduced set of basis vectors. This approach represents a means to learn the kernel function from data. Probabilistic adaptation and application of Bayesian design tools to SVM design has been gaining popularity recently. In “SVM regression through variational methods and its online implementation”, Gao, Gunn, Harris and Brown propose a variational approach that provides an interesting alternative to Bayesian designs. In “The support vector machine under test”, Meyer, Leisch, and Hornik benchmark SVM performance against other classi7cation and regression methods. Such benchmarks are essential if relatively new tools such as SVMs have to be accepted by practitioners of machine learning. Burges and Crisp address in “Uniqueness theorems for kernel methods” the question when a kernel algorithm has a unique solution. Necessary and su/cient conditions are given for the solutions as well as illustrating examples and an o,set determination method. In “Model selection for support vector machine classi7cation”, Gold and Sollich discuss the SVM model selection problem. A new probabilistic framework for SVM classi7cation is reviewed 7rst. Numerical experiments on 7ve benchmark data sets using four model criteria are carried out. 2. Applications SVMs have been successfully applied to many applications such as data mining, bioinformatics, text categorization, machine vision, etc. In this special issue, several new applications using SVMs are presented. In “Support vector regression as a signal discriminator in high energy physics”, Naumann and Whiteson demonstrate an application of SVR in high-energy physics, especially for data acquisition and triggering. According to the authors, SVR can
Editorial / Neurocomputing 55 (2003) 1 – 3
3
perform as well as current algorithmic tools which draw attention and spur their use in the 7eld. The paper includes a performance comparison of SVMs with another arti7cial neural network method used for the same application. There are so far very few attempts to realize hardware implementations of SVMs. In “Neural network learning for VLSI implementations of support vector machines: a survey”, Anguita and Boni review several existing approaches. They also discuss a special purpose adaptive hardware for solving a telecommunication application. It can serve as a highly valuable reference in this research domain. Time series prediction is an important application of SVMs. In “Volatility forecasting from multiscale and high-dimensional market data”, Gavrishchaka and Ganguli show that “SVMs can e,ectively work with high-dimensional inputs to account for volatility long-memory and multiscale e,ects”. It demonstrates that SVMs can be alternative to the GARCH process. In “Financial Time Series Forecasting Using Support Vector Machines”, Kim applies SVMs to 7nancial time series forecasting and shows advantages over Backpropagation networks (BPN) and case-based reasoning (CBR). In “A comparison of PCA, KPCA and ICA for feature extraction in support vector machine”, Cao, Chua, Chong, and Gu compare three existing dimensionality reduction techniques for time series prediction using support vector regression. The results of a number of experiments are presented. Drug-related problems are interesting and there have been relatively few works in the literature on using new classi7cation tools on such problems. In “Support vector machine models in drug design: Applications to drug transport processes and QSAR using simplex optimizations and variable selection”, Norinder shares his experiences on the usefulness of SVMs to one such problem. We wish to thank all colleagues whose contributions make this special issue reality. Enjoy the issue! Colin Campbell Department of Engineering Mathematics; Bristol University; Bristol BS8 1TR, UK Chih-Jen Lin Department of Computer Science and Information Engineering; National Taiwan University; Taipei; Taiwan; 106 S. Sathiva Keerthi Department of Mechanical Engineering; National University of Singapore; 10 Kent Ridge Crescent; Singapore; 119260 V. David Sanchez A Neurocomputing - Editor in Chief ACIS Corporation; P.O. Box 1424; La Canada; CA 91012; USA E-mail address: [email protected] The Guest Editorial Team
Editorial
Special issue on support vector machines Support vector machines (SVMs) are currently a very active research area within machine learning. Motivated by statistical learning theory, SVMs have been successfully applied to numerous tasks, among others in data mining, computer vision, and bioinformatics. SVMs are examples of a broader category of learning approaches which utilize the concept of kernel substitution, which makes the task of learning more tractable by exploiting an implicit mapping into a high-dimensional space. SVMs have many appealing properties for machine learning. For example, the classic SVM learning task involves convex quadratic programming, a problem that does not su,er from the ‘local minima’ problem and whose solution may easily be found by using one of the many specially e/cient algorithms developed for it in the optimization theory. Furthermore, recently developed model selection strategies can be applied, so that few, if any, learning parameters need to be set by the operator. Above all, they have been found to work very well in practice. This special issue on SVMs includes papers from a broad range of topics. We have grouped them under the following two categories: methodology and applications.
1. Methodology There are several papers that consider issues relating to SVM design. Next to the basic theory and the training algorithm, model (feature) selection and hyperparameter tuning are important aspects. As SVM is a kernel method, the issue of choosing kernels for special situations has recently drawn a lot of attention. The standard SVM returns only class labels; therefore, the probabilistic treatment of SVMs needs further investigation. Moreover, in addition to some earlier comparisons between SVMs and other methods, more serious benchmarks are essential. Elaborated techniques and key applications are presented in “Advanced support vector machines and kernel methods” by Sanchez. These include topics of numerical optimization, working set selection, improved generalization, model selection, and parameter tuning as well as classi7cation, regression, text categorization, computer vision, and bioinformatics. In “Feature vector selection and projection using kernels”, Baudat and Anouar discuss the use of kernel methods to extract relevant data points. Data are then projected onto the subspace of the selected points where traditional algorithms can be applied. This reduces the number of data points used in practical situations. c 2003 Published by Elsevier B.V. 0925-2312/03/$ - see front matter doi:10.1016/S0925-2312(03)00428-4
2
Editorial / Neurocomputing 55 (2003) 1 – 3
In “Finite Newton method for Lagrangian support vector machine classi7cation”, Fung and Mangasarian propose an e,ective procedure for solving large linear SVMs. Earlier, Lagrangian SVMs were solved by a procedure with more iterations. Using a Newton method where the cost of each iteration is higher, the total number of iterations is largely reduced. SVM was originally designed for two-class classi7cation. There are several approaches to extend it for multi-class data sets. In “K-SVCR: a support vector machine for multi-class classi7cation”, Angulo, Parra, and Catala discuss a method which improves the pairwise multiclass strategy. In “A geometric approach to support vector regression”, Bi and Bennett give a novel ‘nearest point’ formulation for SVM regression. This work opens the way for the development of new algorithms for designing SVM regressors. Good and easy-to-evaluate estimates of generalization error are necessary for fast tuning of SVM hyperparameters. In “Hyperparameter design criteria for support vector classi7ers”, Anguita, Ridella, Rivieccio and Zunino empirically explore the usefulness of a new estimate called the ‘maximal discrepancy criterion’. In “On problem-oriented kernel re7ning”, Parrado-Hernandez, Arenas-Garcia, Mora-Jimenez, and Navia-Vazques present a method of training support vector machines using a reduced set of basis vectors. This approach represents a means to learn the kernel function from data. Probabilistic adaptation and application of Bayesian design tools to SVM design has been gaining popularity recently. In “SVM regression through variational methods and its online implementation”, Gao, Gunn, Harris and Brown propose a variational approach that provides an interesting alternative to Bayesian designs. In “The support vector machine under test”, Meyer, Leisch, and Hornik benchmark SVM performance against other classi7cation and regression methods. Such benchmarks are essential if relatively new tools such as SVMs have to be accepted by practitioners of machine learning. Burges and Crisp address in “Uniqueness theorems for kernel methods” the question when a kernel algorithm has a unique solution. Necessary and su/cient conditions are given for the solutions as well as illustrating examples and an o,set determination method. In “Model selection for support vector machine classi7cation”, Gold and Sollich discuss the SVM model selection problem. A new probabilistic framework for SVM classi7cation is reviewed 7rst. Numerical experiments on 7ve benchmark data sets using four model criteria are carried out. 2. Applications SVMs have been successfully applied to many applications such as data mining, bioinformatics, text categorization, machine vision, etc. In this special issue, several new applications using SVMs are presented. In “Support vector regression as a signal discriminator in high energy physics”, Naumann and Whiteson demonstrate an application of SVR in high-energy physics, especially for data acquisition and triggering. According to the authors, SVR can
Editorial / Neurocomputing 55 (2003) 1 – 3
3
perform as well as current algorithmic tools which draw attention and spur their use in the 7eld. The paper includes a performance comparison of SVMs with another arti7cial neural network method used for the same application. There are so far very few attempts to realize hardware implementations of SVMs. In “Neural network learning for VLSI implementations of support vector machines: a survey”, Anguita and Boni review several existing approaches. They also discuss a special purpose adaptive hardware for solving a telecommunication application. It can serve as a highly valuable reference in this research domain. Time series prediction is an important application of SVMs. In “Volatility forecasting from multiscale and high-dimensional market data”, Gavrishchaka and Ganguli show that “SVMs can e,ectively work with high-dimensional inputs to account for volatility long-memory and multiscale e,ects”. It demonstrates that SVMs can be alternative to the GARCH process. In “Financial Time Series Forecasting Using Support Vector Machines”, Kim applies SVMs to 7nancial time series forecasting and shows advantages over Backpropagation networks (BPN) and case-based reasoning (CBR). In “A comparison of PCA, KPCA and ICA for feature extraction in support vector machine”, Cao, Chua, Chong, and Gu compare three existing dimensionality reduction techniques for time series prediction using support vector regression. The results of a number of experiments are presented. Drug-related problems are interesting and there have been relatively few works in the literature on using new classi7cation tools on such problems. In “Support vector machine models in drug design: Applications to drug transport processes and QSAR using simplex optimizations and variable selection”, Norinder shares his experiences on the usefulness of SVMs to one such problem. We wish to thank all colleagues whose contributions make this special issue reality. Enjoy the issue! Colin Campbell Department of Engineering Mathematics; Bristol University; Bristol BS8 1TR, UK Chih-Jen Lin Department of Computer Science and Information Engineering; National Taiwan University; Taipei; Taiwan; 106 S. Sathiva Keerthi Department of Mechanical Engineering; National University of Singapore; 10 Kent Ridge Crescent; Singapore; 119260 V. David Sanchez A Neurocomputing - Editor in Chief ACIS Corporation; P.O. Box 1424; La Canada; CA 91012; USA E-mail address: [email protected] The Guest Editorial Team