Bernoulli

  • Bernoulli
  • Volume 17, Number 4 (2011), 1368-1385.

Support vector machines with a reject option

Marten Wegkamp and Ming Yuan

Full-text: Open access

Enhanced PDF (253 KB)

Abstract

This paper studies $ℓ_1$ regularization with high-dimensional features for support vector machines with a built-in reject option (meaning that the decision of classifying an observation can be withheld at a cost lower than that of misclassification). The procedure can be conveniently implemented as a linear program and computed using standard software. We prove that the minimizer of the penalized population risk favors sparse solutions and show that the behavior of the empirical risk minimizer mimics that of the population risk minimizer. We also introduce a notion of classification complexity and prove that our minimizers adapt to the unknown complexity. Using a novel oracle inequality for the excess risk, we identify situations where fast rates of convergence occur.

Article information

Source
Bernoulli, Volume 17, Number 4 (2011), 1368-1385.

Dates
First available in Project Euclid: 4 November 2011

Permanent link to this document
https://projecteuclid.org/euclid.bj/1320417508

Digital Object Identifier
doi:10.3150/10-BEJ320

Mathematical Reviews number (MathSciNet)
MR2854776

Zentralblatt MATH identifier
1243.68256

Keywords
adaptive prediction classification with a reject option lasso oracle inequalities sparsity support vector machines statistical learning

Citation

Wegkamp, Marten; Yuan, Ming. Support vector machines with a reject option. Bernoulli 17 (2011), no. 4, 1368--1385. doi:10.3150/10-BEJ320. https://projecteuclid.org/euclid.bj/1320417508


Export citation

References

  • [1] Bartlett, P.L. and Wegkamp, M.H. (2008). Classification with a reject option using a hinge loss. J. Mach. Learn. Res. 9 1823–1840.
    Mathematical Reviews (MathSciNet): MR2438825
    Zentralblatt MATH: 1225.62080
  • [2] Bickel, P.J., Ritov, Y. and Tsybakov, A.B. (2009). Simultaneous analysis of Lasso and Dantzig selector. Ann. Statist. 37 1705–1732.
    Mathematical Reviews (MathSciNet): MR2533469
    Zentralblatt MATH: 1173.62022
    Digital Object Identifier: doi:10.1214/08-AOS620
    Project Euclid: euclid.aos/1245332830
  • [3] Devroye, L. and Lugosi, G. (2000). Combinatorial Methods in Density Estimation. New York: Springer.
    Mathematical Reviews (MathSciNet): MR1843146
    Zentralblatt MATH: 0964.62025
  • [4] Hastie, T., Tibshirani, R. and Friedman, J. (2001). The Elements of Statistical Learning. New York: Springer.
    Mathematical Reviews (MathSciNet): MR1851606
    Zentralblatt MATH: 0973.62007
  • [5] Herbei, R. and Wegkamp, M.H. (2006). Classification with reject option. Canad. J. Statist. 34 709–721.
    Mathematical Reviews (MathSciNet): MR2347054
    Digital Object Identifier: doi:10.1002/cjs.5550340410
    Zentralblatt MATH: 1151.62302
  • [6] Koltchinskii, V. (2009). Sparsity in penalized empirical risk minimization. Ann. Inst. H. Poincaré Probab. Statist. 45 7–57.
    Mathematical Reviews (MathSciNet): MR2500227
    Zentralblatt MATH: 1168.62044
    Digital Object Identifier: doi:10.1214/07-AIHP146
    Project Euclid: euclid.aihp/1234469970
  • [7] Ledoux, M. and Talagrand, M. (1991). Probability in Banach Spaces. New York: Springer.
    Mathematical Reviews (MathSciNet): MR1102015
    Zentralblatt MATH: 0748.60004
  • [8] Tarigan, B. and van de Geer, S.A. (2006). Classifiers of support vector machine type with ℓ1 complexity regularization. Bernoulli 12 1045–1076.
    Mathematical Reviews (MathSciNet): MR2274857
    Digital Object Identifier: doi:10.3150/bj/1165269150
    Project Euclid: euclid.bj/1165269150
    Zentralblatt MATH: 1118.62067
  • [9] Tsybakov, A.B. (2004). Optimal aggregation of classifiers in statistical learning. Ann. Statist. 32 135–166.
    Mathematical Reviews (MathSciNet): MR2051002
    Zentralblatt MATH: 1105.62353
    Digital Object Identifier: doi:10.1214/aos/1079120131
    Project Euclid: euclid.aos/1079120131
  • [10] van de Geer, S.A. (2000). Empirical Processes in M-estimation. Cambridge: Cambridge Univ. Press.
    Mathematical Reviews (MathSciNet): MR1739079
  • [11] Wegkamp, M.H. (2007). Lasso type classifiers with a reject option. Electron. J. Statist. 1 155–168.
    Mathematical Reviews (MathSciNet): MR2312148
    Zentralblatt MATH: 05274611
    Digital Object Identifier: doi:10.1214/07-EJS058
    Project Euclid: euclid.ejs/1179332302
  • [12] Yuan, M. and Wegkamp, M.H. (2010). Classification methods with reject option based on convex risk minimization. J. Mach. Learn. Res. 11 111–130.
    Mathematical Reviews (MathSciNet): MR2591623
    Zentralblatt MATH: 1242.62066

Bernoulli Society for Mathematical Statistics and Probability

Bernoulli

New content alerts

Email RSS ToC RSS Article
  • You have access to this content.
  • You have partial access to this content.
  • You do not have access to this content.
Turn Off MathJax

What is MathJax?

More like this

+ See more