K. Duan, S.S. Keerthi and A.N. Poo
Technical Report CD-01-11
Control Division
Dept. of Mechanical Engineering
National University of Singapore
Singapore-119260
Using the feedback received from several researchers, we revised the technical report on Sept. 12, 2001. The pdf file given below contains the revised version.
Choosing optimal hyperparameter values for support vector machines is an important step in SVM design. This is usually done by minimizing either an estimate of generalization error or some other related performance measure. In this paper, we empirically study the usefulness of several simple performance measures that are inexpensive to compute (in the sense that they do not require expensive matrix operations involving the kernel matrix). The results point out which of these measures are adequate functionals for tuning SVM hyperparameters. For SVMs with L1 soft margin formulation, none of the simple measures yields a performance uniformly as good as k-fold cross validation; Joachims' Xi-Alpha bound and Wahba et al's GACV come next and perform reasonably well. For SVMs with L2 soft margin formulation, the radius margin bound gives a very good prediction of optimal hyperparameter values.
pdf file containing Technical Report (revised on Sept. 12, 2001).
pdf file containing figures not included in the Report (revised on Sept. 12, 2001).
Each of the datasets are described by 4 files: two of them give the training and test input vectors and the other two give the target values for the training and test points. See our paper for details concerning each dataset.
The first 4 datasets are taken from the benchmark repository of Gunnar Raetsch and correspond to the first of the 100 realizations that he has given.
The Tree dataset was taken from the data page of Image Processing Lab, University of Texas at Arlington. The file, COMF18.TRA given there was randomly partitioned to form the training and test data. Also, the multiclass problem given there was converted to the binary classification problem of differentiating tree patterns from non-tree patterns.
Now here are the tarred+gzipped files for the 5 datasets:
If you have questions or comments about the technical report please send email to Sathiya Keerthi We would also be interested in knowing about the comparitive performance of the measures that we have considered, on other data sets that we haven't tried.