Abstract
The support vector machine (SVM) is a powerful and widely used classification algorithm. This paper uses the Karush-Kuhn-Tucker conditions to provide rigorous mathematical proof for new insights into the behavior of SVM. These insights provide unexpected relationships between SVM and two other linear classifiers: the mean difference and the maximal data piling direction. For example, we show that in many cases SVM can be viewed as a cropped version of these classifiers. By carefully exploring these connections we show how SVM tuning behavior is affected by data characteristics including: balanced vs. unbalanced classes, low vs. high dimension, separable vs. non-separable data. These results provide further insights into tuning SVM via cross-validation by explaining observed pathological behavior and motivating improved cross-validation methodology.
Funding Statement
This research was supported in part by the National Science Foundation under Grant No. 1633074.
Citation
Iain Carmichael. J. S. Marron. "Geometric insights into support vector machine behavior using the KKT conditions." Electron. J. Statist. 15 (2) 6311 - 6343, 2021. https://doi.org/10.1214/21-EJS1902
Information