The doubly regularized support vector machine

The standard L ₂-norm support vector machine (SVM) is a widely used tool for classification problems. The L ₁-norm SVM is a variant of the standard Lanorm SVM, that constrains the Li-norm of the fitted coefficients. Due to the nature of the L ₁-norm, the L ₁-norm SVM has the property of automatically selecting variables, not shared by the standard L ₂-norm SVM. It has been argued that the L ₁-norm SVM may have some advantage over the L ₂-norm SVM, especially with high dimensional problems and when there are redundant noise variables. On the other hand, the L ₁-norm SVM has two drawbacks: (1) when there are several highly correlated variables, the L ₁-norm SVM tends to pick only a few of them, and remove the rest; (2) the number of selected variables is upper bounded by the size of the training data. A typical example where these occur is in gene microarray analysis. In this paper, we propose a doubly regularized support vector machine (DrSVM). The DrSVM uses the elastic-net penalty, a mixture of the L ₂-norm and the L ₁-norm penalties. By doing so, the DrSVM performs automatic variable selection in a way similar to the L ₁-norm SVM. In addition, the DrSVM encourages highly correlated variables to be selected (or removed) together. We illustrate how the DrSVM can be particularly useful when the number of variables is much larger than the size of the training data (p ≫ n). We also develop efficient algorithms to compute the whole solution paths of the DrSVM.