## The Annals of Statistics

### Assessing robustness of classification using an angular breakdown point

#### Abstract

Robustness is a desirable property for many statistical techniques. As an important measure of robustness, the breakdown point has been widely used for regression problems and many other settings. Despite the existing development, we observe that the standard breakdown point criterion is not directly applicable for many classification problems. In this paper, we propose a new breakdown point criterion, namely angular breakdown point, to better quantify the robustness of different classification methods. Using this new breakdown point criterion, we study the robustness of binary large margin classification techniques, although the idea is applicable to general classification methods. Both bounded and unbounded loss functions with linear and kernel learning are considered. These studies provide useful insights on the robustness of different classification methods. Numerical results further confirm our theoretical findings.

#### Article information

Source
Ann. Statist., Volume 46, Number 6B (2018), 3362-3389.

Dates
Revised: November 2017
First available in Project Euclid: 11 September 2018

https://projecteuclid.org/euclid.aos/1536631277

Digital Object Identifier
doi:10.1214/17-AOS1661

Mathematical Reviews number (MathSciNet)
MR3852655

Zentralblatt MATH identifier
1408.62121

Subjects
Secondary: 62G35: Robustness

#### Citation

Zhao, Junlong; Yu, Guan; Liu, Yufeng. Assessing robustness of classification using an angular breakdown point. Ann. Statist. 46 (2018), no. 6B, 3362--3389. doi:10.1214/17-AOS1661. https://projecteuclid.org/euclid.aos/1536631277

#### References

• Biggio, B., Nelson, B. and Laskov, P. (2012). Poisoning attacks against support vector machines. In Proceedings of the 29th International Conference on Machine Learning (J. Langford and J. Pineau, eds.) 1807–1814. ACM, New York.
• Christmann, A. and Steinwart, I. (2004). On robustness properties of convex risk minimization methods for pattern recognition. J. Mach. Learn. Res. 5 1007–1034.
• Donoho, D. and Huber, P. J. (1983). The notion of breakdown point. In A Festschrift for Erich L. Lehmann 157–184. Wadsworth, Belmont, CA.
• Freund, Y. and Schapire, R. E. (1997). A decision-theoretic generalization of on-line learning and an application to boosting. J. Comput. System Sci. 55 119–139.
• Genton, M. G. and Lucas, A. (2003). Comprehensive definitions of breakdown points for independent and dependent observations. J. R. Stat. Soc. Ser. B. Stat. Methodol. 65 81–94.
• Hable, R. and Christmann, A. (2011). On qualitative robustness of support vector machines. J. Multivariate Anal. 102 993–1007.
• Hampel, F. R. (1971). A general qualitative definition of robustness. Ann. Math. Stat. 42 1887–1896.
• Hampel, F. R. (1974). The influence curve and its role in robust estimation. J. Amer. Statist. Assoc. 69 383–393.
• Hastie, T., Tibshirani, R. and Friedman, J. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd ed. Springer, New York.
• Horst, R. and Thoai, N. V. (1999). DC programming: Overview. J. Optim. Theory Appl. 103 1–43.
• Hubert, M., Rousseeuw, P. J. and Van Aelst, S. (2008). High-breakdown robust multivariate methods. Statist. Sci. 92–119.
• Kanamori, T., Fujiwara, S. and Takeda, A. (2014). Breakdown point of robust support vector machine. Preprint. Available at arXiv:1409.0934.
• Kimeldorf, G. S. and Wahba, G. (1970). A correspondence between Bayesian estimation on stochastic processes and smoothing by splines. Ann. Math. Stat. 41 495–502.
• Krause, N. and Singer, Y. (2004). Leveraging the margin more carefully. In Proceedings of the Twenty-First International Conference on Machine Learning 63. ACM, New York.
• Lin, X., Wahba, G., Xiang, D., Gao, F., Klein, R. and Klein, B. (2000). Smoothing spline ANOVA models for large data sets with Bernoulli observations and the randomized GACV. Ann. Statist. 28 1570–1600.
• Liu, Y. and Shen, X. (2006). Multicategory $\psi$-learning. J. Amer. Statist. Assoc. 101 500–509.
• Mason, L., Baxter, J., Bartlett, P. and Frean, M. (2000). Boosting algorithms as gradient descent. In In Advances in Neural Information Processing Systems 12 512–518. MIT Press, Cambridge.
• Ronchetti, E. (1997). Robust inference by influence functions. J. Statist. Plann. Inference 57 59–72.
• Sakata, S. and White, H. (1995). An alternative definition of finite-sample breakdown point with applications to regression model estimators. J. Amer. Statist. Assoc. 90 1099–1106.
• Schölkopf, B. and Smola, A. J. (2002). Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. MIT Press, Cambridge, MA.
• Scholkopf, B., Smola, A. J., Williamson, R. C. and Bartlett, P. L. (2000). New support vector algorithms. Neural Comput. 12 1207–1245.
• Shen, X., Tseng, G. C., Zhang, X. and Wong, W. H. (2003). On $\psi$-learning. J. Amer. Statist. Assoc. 98 724–734.
• Stromberg, A. J. and Ruppert, D. (1992). Breakdown in nonlinear regression. J. Amer. Statist. Assoc. 87 991–997.
• Vapnik, V. N. (1998). Statistical Learning Theory. Wiley, New York.
• Wahba, G. (1990). Spline Models for Observational Data 59. SIAM, Philadelphia, PA.
• Wu, Y. and Liu, Y. (2007). Robust truncated hinge loss support vector machines. J. Amer. Statist. Assoc. 102 974–983.
• Xu, L., Crammer, K. and Schuurmans, D. (2006). Robust support vector machine training via convex outlier ablation. In American Association for Artificial Intelligence 6 536–542.
• Yu, Y., Yang, M., Xu, L., White, M. and Schuurmans, D. (2010). Relaxed clipping: A global training method for robust regression and classification. In Advances in Neural Information Processing Systems 2532–2540.
• Zhao, J., Yu, G. and Liu, Y. (2018). Supplement to “Assessing robustness of classification using an angular breakdown point.” DOI:10.1214/17-AOS1661SUPP.

#### Supplemental materials

• Supplement to “Assessing robustness of classification using an angular breakdown point”. The supplementary material contains the remaining proof of the theoretical results.