The Annals of Statistics

Assessing robustness of classification using an angular breakdown point

Junlong Zhao, Guan Yu, and Yufeng Liu

Full-text: Access denied (no subscription detected)

We're sorry, but we are unable to provide you with the full text of this article because we are not able to identify you as a subscriber. If you have a personal subscription to this journal, then please login. If you are already logged in, then you may need to update your profile to register your subscription. Read more about accessing full-text

Abstract

Robustness is a desirable property for many statistical techniques. As an important measure of robustness, the breakdown point has been widely used for regression problems and many other settings. Despite the existing development, we observe that the standard breakdown point criterion is not directly applicable for many classification problems. In this paper, we propose a new breakdown point criterion, namely angular breakdown point, to better quantify the robustness of different classification methods. Using this new breakdown point criterion, we study the robustness of binary large margin classification techniques, although the idea is applicable to general classification methods. Both bounded and unbounded loss functions with linear and kernel learning are considered. These studies provide useful insights on the robustness of different classification methods. Numerical results further confirm our theoretical findings.

Article information

Source
Ann. Statist., Volume 46, Number 6B (2018), 3362-3389.

Dates
Received: April 2016
Revised: November 2017
First available in Project Euclid: 11 September 2018

Permanent link to this document
https://projecteuclid.org/euclid.aos/1536631277

Digital Object Identifier
doi:10.1214/17-AOS1661

Mathematical Reviews number (MathSciNet)
MR3852655

Zentralblatt MATH identifier
1408.62121

Subjects
Primary: 62H30: Classification and discrimination; cluster analysis [See also 68T10, 91C20]
Secondary: 62G35: Robustness

Keywords
Breakdown point classification loss function reproducing kernel Hilbert spaces robustness

Citation

Zhao, Junlong; Yu, Guan; Liu, Yufeng. Assessing robustness of classification using an angular breakdown point. Ann. Statist. 46 (2018), no. 6B, 3362--3389. doi:10.1214/17-AOS1661. https://projecteuclid.org/euclid.aos/1536631277


Export citation

References

  • Biggio, B., Nelson, B. and Laskov, P. (2012). Poisoning attacks against support vector machines. In Proceedings of the 29th International Conference on Machine Learning (J. Langford and J. Pineau, eds.) 1807–1814. ACM, New York.
  • Christmann, A. and Steinwart, I. (2004). On robustness properties of convex risk minimization methods for pattern recognition. J. Mach. Learn. Res. 5 1007–1034.
  • Donoho, D. and Huber, P. J. (1983). The notion of breakdown point. In A Festschrift for Erich L. Lehmann 157–184. Wadsworth, Belmont, CA.
  • Freund, Y. and Schapire, R. E. (1997). A decision-theoretic generalization of on-line learning and an application to boosting. J. Comput. System Sci. 55 119–139.
  • Genton, M. G. and Lucas, A. (2003). Comprehensive definitions of breakdown points for independent and dependent observations. J. R. Stat. Soc. Ser. B. Stat. Methodol. 65 81–94.
  • Hable, R. and Christmann, A. (2011). On qualitative robustness of support vector machines. J. Multivariate Anal. 102 993–1007.
  • Hampel, F. R. (1971). A general qualitative definition of robustness. Ann. Math. Stat. 42 1887–1896.
  • Hampel, F. R. (1974). The influence curve and its role in robust estimation. J. Amer. Statist. Assoc. 69 383–393.
  • Hastie, T., Tibshirani, R. and Friedman, J. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd ed. Springer, New York.
  • Horst, R. and Thoai, N. V. (1999). DC programming: Overview. J. Optim. Theory Appl. 103 1–43.
  • Hubert, M., Rousseeuw, P. J. and Van Aelst, S. (2008). High-breakdown robust multivariate methods. Statist. Sci. 92–119.
  • Kanamori, T., Fujiwara, S. and Takeda, A. (2014). Breakdown point of robust support vector machine. Preprint. Available at arXiv:1409.0934.
  • Kimeldorf, G. S. and Wahba, G. (1970). A correspondence between Bayesian estimation on stochastic processes and smoothing by splines. Ann. Math. Stat. 41 495–502.
  • Krause, N. and Singer, Y. (2004). Leveraging the margin more carefully. In Proceedings of the Twenty-First International Conference on Machine Learning 63. ACM, New York.
  • Lin, X., Wahba, G., Xiang, D., Gao, F., Klein, R. and Klein, B. (2000). Smoothing spline ANOVA models for large data sets with Bernoulli observations and the randomized GACV. Ann. Statist. 28 1570–1600.
  • Liu, Y. and Shen, X. (2006). Multicategory $\psi $-learning. J. Amer. Statist. Assoc. 101 500–509.
  • Mason, L., Baxter, J., Bartlett, P. and Frean, M. (2000). Boosting algorithms as gradient descent. In In Advances in Neural Information Processing Systems 12 512–518. MIT Press, Cambridge.
  • Ronchetti, E. (1997). Robust inference by influence functions. J. Statist. Plann. Inference 57 59–72.
  • Sakata, S. and White, H. (1995). An alternative definition of finite-sample breakdown point with applications to regression model estimators. J. Amer. Statist. Assoc. 90 1099–1106.
  • Schölkopf, B. and Smola, A. J. (2002). Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. MIT Press, Cambridge, MA.
  • Scholkopf, B., Smola, A. J., Williamson, R. C. and Bartlett, P. L. (2000). New support vector algorithms. Neural Comput. 12 1207–1245.
  • Shen, X., Tseng, G. C., Zhang, X. and Wong, W. H. (2003). On $\psi $-learning. J. Amer. Statist. Assoc. 98 724–734.
  • Stromberg, A. J. and Ruppert, D. (1992). Breakdown in nonlinear regression. J. Amer. Statist. Assoc. 87 991–997.
  • Vapnik, V. N. (1998). Statistical Learning Theory. Wiley, New York.
  • Wahba, G. (1990). Spline Models for Observational Data 59. SIAM, Philadelphia, PA.
  • Wu, Y. and Liu, Y. (2007). Robust truncated hinge loss support vector machines. J. Amer. Statist. Assoc. 102 974–983.
  • Xu, L., Crammer, K. and Schuurmans, D. (2006). Robust support vector machine training via convex outlier ablation. In American Association for Artificial Intelligence 6 536–542.
  • Yu, Y., Yang, M., Xu, L., White, M. and Schuurmans, D. (2010). Relaxed clipping: A global training method for robust regression and classification. In Advances in Neural Information Processing Systems 2532–2540.
  • Zhao, J., Yu, G. and Liu, Y. (2018). Supplement to “Assessing robustness of classification using an angular breakdown point.” DOI:10.1214/17-AOS1661SUPP.

Supplemental materials

  • Supplement to “Assessing robustness of classification using an angular breakdown point”. The supplementary material contains the remaining proof of the theoretical results.