The Annals of Statistics
- Ann. Statist.
- Volume 34, Number 6 (2006), 2593-2656.
Local Rademacher complexities and oracle inequalities in risk minimization
Full-text: Open access
Abstract
Let ℱ be a class of measurable functions f:S↦[0, 1] defined on a probability space (S, $\mathcal{A}$, P). Given a sample (X1, …, Xn) of i.i.d. random variables taking values in S with common distribution P, let Pn denote the empirical measure based on (X1, …, Xn). We study an empirical risk minimization problem Pnf→min , f∈ℱ. Given a solution f̂n of this problem, the goal is to obtain very general upper bounds on its excess risk $$\mathcal{E}_{P}(\hat{f}_{n}):=P\hat{f}_{n}-\inf_{f\in \mathcal{F}}Pf,$$ expressed in terms of relevant geometric parameters of the class ℱ. Using concentration inequalities and other empirical processes tools, we obtain both distribution-dependent and data-dependent upper bounds on the excess risk that are of asymptotically correct order in many examples. The bounds involve localized sup-norms of empirical and Rademacher processes indexed by functions from the class. We use these bounds to develop model selection techniques in abstract risk minimization problems that can be applied to more specialized frameworks of regression and classification.
Article information
Source
Ann. Statist., Volume 34, Number 6 (2006), 2593-2656.
Dates
First available in Project Euclid: 23 May 2007
Permanent link to this document
https://projecteuclid.org/euclid.aos/1179935055
Digital Object Identifier
doi:10.1214/009053606000001019
Mathematical Reviews number (MathSciNet)
MR2329442
Zentralblatt MATH identifier
1118.62065
Subjects
Primary: 62H30: Classification and discrimination; cluster analysis [See also 68T10, 91C20] 60B99: None of the above, but in this section 68Q32: Computational learning theory [See also 68T05]
Secondary: 62G08: Nonparametric regression 68T05: Learning and adaptive systems [See also 68Q32, 91E40] 68T10: Pattern recognition, speech recognition {For cluster analysis, see 62H30}
Keywords
Rademacher complexities empirical risk minimization oracle inequalities model selection concentration inequalities classification
Citation
Koltchinskii, Vladimir. Local Rademacher complexities and oracle inequalities in risk minimization. Ann. Statist. 34 (2006), no. 6, 2593--2656. doi:10.1214/009053606000001019. https://projecteuclid.org/euclid.aos/1179935055
References
- Anthony, M. and Bartlett, P. (1999). Neural Network Learning: Theoretical Foundations. Cambridge Univ. Press.
- Baraud, Y. (2002). Model selection for regression on a random design. ESAIM Probab. Statist. 6 127–146. Mathematical Reviews (MathSciNet): MR1918295
Digital Object Identifier: doi:10.1051/ps:2002007
Zentralblatt MATH: 1059.62038 - Barron, A., Birgé, L. and Massart, P. (1999). Risk bounds for model selection via penalization. Probab. Theory Related Fields 113 301–413. Mathematical Reviews (MathSciNet): MR1679028
Digital Object Identifier: doi:10.1007/s004400050210
Zentralblatt MATH: 0946.62036 - Bartlett, P., Boucheron, S. and Lugosi, G. (2002). Model selection and error estimation. Machine Learning 48 85–113.
- Bartlett, P., Bousquet, O. and Mendelson, S. (2005). Local Rademacher complexities. Ann. Statist. 33 1497–1537. Mathematical Reviews (MathSciNet): MR2166554
Digital Object Identifier: doi:10.1214/009053605000000282
Project Euclid: euclid.aos/1123250221
Zentralblatt MATH: 1083.62034 - Bartlett, P., Jordan, M. and McAuliffe, J. (2006). Convexity, classification and risk bounds. J. Amer. Statist. Assoc. 101 138–156. Mathematical Reviews (MathSciNet): MR2268032
Digital Object Identifier: doi:10.1198/016214505000000907
Zentralblatt MATH: 1118.62330 - Bartlett, P. and Mendelson, S. (2006). Empirical minimization. Probab. Theory Related Fields 135 311–334. Mathematical Reviews (MathSciNet): MR2240689
Digital Object Identifier: doi:10.1007/s00440-005-0462-3
Zentralblatt MATH: 1142.62348 - Birgé, L. and Massart, P. (1997). From model selection to adaptive estimation. In Festschrift for Lucien Le Cam. Research Papers in Probability and Statistics (D. Pollard, E. Torgersen and G. Yang, eds.) 55–87. Springer, New York.
- Blanchard, G., Bousquet, O. and Massart, P. (2003). Statistical performance of support vector machines. Preprint.Mathematical Reviews (MathSciNet): MR2396805
Digital Object Identifier: doi:10.1214/009053607000000839
Project Euclid: euclid.aos/1205420509
Zentralblatt MATH: 1133.62044 - Blanchard, G., Lugosi, G. and Vayatis, N. (2003). On the rate of convergence of regularized boosting classifiers. J. Mach. Learn. Res. 4 861–894. Mathematical Reviews (MathSciNet): MR2076000
Digital Object Identifier: doi:10.1162/1532443041424319
Zentralblatt MATH: 1083.68109 - Boucheron, S., Bousquet, O., Lugosi, G. and Massart, P. (2005). Moment inequalities for functions of independent random variables. Ann. Probab. 33 514–560. Mathematical Reviews (MathSciNet): MR2123200
Digital Object Identifier: doi:10.1214/009117904000000856
Project Euclid: euclid.aop/1109868590
Zentralblatt MATH: 1074.60018 - Boucheron, S., Lugosi, G. and Massart, P. (2000). A sharp concentration inequality with applications. Random Structures Algorithms 16 277–292.
- Bousquet, O. (2002). A Bennett concentration inequality and its application to suprema of empirical processes. C. R. Acad. Sci. Paris Ser. I Math. 334 495–500. Mathematical Reviews (MathSciNet): MR1890640
Digital Object Identifier: doi:10.1016/S1631-073X(02)02292-6
Zentralblatt MATH: 1001.60021 - Bousquet, O., Koltchinskii, V. and Panchenko, D. (2002). Some local measures of complexity of convex hulls and generalization bounds. In Computational Learning Theory. Lecture Notes in Comput. Sci. 2375 59–73. Springer, Berlin. Mathematical Reviews (MathSciNet): MR2040405
Zentralblatt MATH: 1050.68055
Digital Object Identifier: doi:10.1007/3-540-45435-7_5 - Devroye, L., Györfi, G. and Lugosi, G. (1996). A Probabilistic Theory of Pattern Recognition. Springer, New York.
- Dudley, R. M. (1999). Uniform Central Limit Theorems. Cambridge Univ. Press.
- Einmahl, U. and Mason, D. (2000). An empirical processes approach to the uniform consistency of kernel type function estimators. J. Theoret. Probab. 13 1–37. Mathematical Reviews (MathSciNet): MR1744994
Digital Object Identifier: doi:10.1023/A:1007769924157
Zentralblatt MATH: 0995.62042 - Giné, E. and Guillou, A. (2001). On consistency of kernel density estimators for randomly censored data: Rates holding uniformly over adaptive intervals. Ann. Inst. H. Poincaré Probab. Statist. 37 503–522. Mathematical Reviews (MathSciNet): MR1876841
Digital Object Identifier: doi:10.1016/S0246-0203(01)01081-0
Zentralblatt MATH: 0974.62030 - Giné, E. and Koltchinskii, V. (2006). Concentration inequalities and asymptotic results for ratio type empirical processes. Ann. Probab. 34 1143–1216. Mathematical Reviews (MathSciNet): MR2243881
Digital Object Identifier: doi:10.1214/009117906000000070
Project Euclid: euclid.aop/1151418495
Zentralblatt MATH: 1152.60021 - Giné, E., Koltchinskii, V. and Wellner, J. (2003). Ratio limit theorems for empirical processes. In Stochastic Inequalities and Applications (E. Giné, C. Houdré and D. Nualart, eds.) 249–278. Birkhäuser, Basel.
- Giné, E. and Zinn, J. (1984). Some limit theorems for empirical processes (with discussion). Ann. Probab. 12 929–998. Mathematical Reviews (MathSciNet): MR0757767
Digital Object Identifier: doi:10.1214/aop/1176993138
Project Euclid: euclid.aop/1176993138 - Györfi, L., Kohler, M., Krzyzak, A. and Walk, H. (2002). A Distribution-Free Theory of Nonparametric Regression. Springer, New York. Mathematical Reviews (MathSciNet): MR1920390
- Johnstone, I. M. (1998). Oracle inequalities and nonparametric function estimation. In Proc. International Congress of Mathematicians 3 267–278.
- Klein, T. (2002). Une inégalité de concentration à gauche pour les processus empiriques. C. R. Acad. Sci. Paris Ser. I Math. 334 501–504. Mathematical Reviews (MathSciNet): MR1890641
Digital Object Identifier: doi:10.1016/S1631-073X(02)02303-8
Zentralblatt MATH: 1003.60024 - Kohler, M. (2000). Inequalities for uniform deviations of averages from expectations with applications to nonparametric regression. J. Statist. Plann. Inference 89 1–23. Mathematical Reviews (MathSciNet): MR1794410
Digital Object Identifier: doi:10.1016/S0378-3758(99)00215-3
Zentralblatt MATH: 0982.62035 - Koltchinskii, V. (2001). Rademacher penalties and structural risk minimization. IEEE Trans. Inform. Theory 47 1902–1914. Mathematical Reviews (MathSciNet): MR1842526
Digital Object Identifier: doi:10.1109/18.930926
Zentralblatt MATH: 1008.62614 - Koltchinskii, V. and Panchenko, D. (2000). Rademacher processes and bounding the risk of function learning. In High Dimensional Probability II (E. Giné, D. Mason and J. Wellner, eds.) 443–457. Birkhäuser, Boston.
- Koltchinskii, V. and Panchenko, D. (2002). Empirical margin distributions and bounding the generalization error of combined classifiers. Ann. Statist. 30 1–50. Mathematical Reviews (MathSciNet): MR1892654
Project Euclid: euclid.aos/1015362183
Zentralblatt MATH: 1012.62004 - Koltchinskii, V. and Panchenko, D. (2005). Complexities of convex combinations and bounding the generalization error in classification. Ann. Statist. 33 1455–1496. Mathematical Reviews (MathSciNet): MR2166553
Digital Object Identifier: doi:10.1214/009053605000000228
Project Euclid: euclid.aos/1123250220
Zentralblatt MATH: 1080.62045 - Koltchinskii, V., Panchenko, D. and Lozano, F. (2003). Bounding the generalization error of convex combinations of classifiers: balancing the dimensionality and the margins. Ann. Appl. Probab. 13 213–252. Mathematical Reviews (MathSciNet): MR1951998
Digital Object Identifier: doi:10.1214/aoap/1042765667
Project Euclid: euclid.aoap/1042765667
Zentralblatt MATH: 1073.62535 - Ledoux, M. and Talagrand, M. (1991). Probability in Banach Spaces. Isoperimetry and Processes. Springer, New York.
- Lee, W. S., Bartlett, P. and Williamson, R. C. (1996). Efficient agnostic learning of neural networks with bounded fan-in. IEEE Trans. Inform. Theory 42 2118–2132. Mathematical Reviews (MathSciNet): MR1447518
Digital Object Identifier: doi:10.1109/18.485715
Zentralblatt MATH: 0874.68253 - Lugosi, G. and Vayatis, N. (2004). On the Bayes-risk consistency of regularized boosting methods. Ann. Statist. 32 30–55. Mathematical Reviews (MathSciNet): MR2051000
Project Euclid: euclid.aos/1079120129
Zentralblatt MATH: 1105.62319 - Lugosi, G. and Wegkamp, M. (2004). Complexity regularization via localized random penalties. Ann. Statist. 32 1679–1697. Mathematical Reviews (MathSciNet): MR2089138
Digital Object Identifier: doi:10.1214/009053604000000463
Project Euclid: euclid.aos/1091626183
Zentralblatt MATH: 1045.62060 - Mammen, E. and Tsybakov, A. (1999). Smooth discrimination analysis. Ann. Statist. 27 1808–1829. Mathematical Reviews (MathSciNet): MR1765618
Digital Object Identifier: doi:10.1214/aos/1017939240
Project Euclid: euclid.aos/1017939240
Zentralblatt MATH: 0961.62058 - Massart, P. (2000). Some applications of concentration inequalities to statistics. Ann. Fac. Sci. Toulouse Math. (6) 9 245–303.
- Mendelson, S. (2002). Improving the sample complexity using global data. IEEE Trans. Inform. Theory 48 1977–1991. Mathematical Reviews (MathSciNet): MR1930004
Digital Object Identifier: doi:10.1109/TIT.2002.1013137
Zentralblatt MATH: 1061.68128 - Mendelson, S. (2002). Geometric parameters of kernel machines. In Computational Learning Theory. Lecture Notes in Comput. Sci. 2375 29–43. Springer, Berlin. Mathematical Reviews (MathSciNet): MR2040403
Zentralblatt MATH: 1050.68070
Digital Object Identifier: doi:10.1007/3-540-45435-7_3 - Shen, X. and Wong, W.H. (1994). Convergence rate of sieve estimates. Ann. Statist. 22 580–615. Mathematical Reviews (MathSciNet): MR1292531
Digital Object Identifier: doi:10.1214/aos/1176325486
Project Euclid: euclid.aos/1176325486
Zentralblatt MATH: 0805.62008 - Steinwart, I. (2005). Consistency of support vector machines and other regularized kernel machines. IEEE Trans. Inform. Theory 51 128–142. Mathematical Reviews (MathSciNet): MR2234577
Digital Object Identifier: doi:10.1109/TIT.2004.839514
Zentralblatt MATH: 1304.62090 - Talagrand, M. (1994). Sharper bounds for Gaussian and empirical processes. Ann. Probab. 22 28–76. Mathematical Reviews (MathSciNet): MR1258865
Digital Object Identifier: doi:10.1214/aop/1176988847
Project Euclid: euclid.aop/1176988847
Zentralblatt MATH: 0798.60051 - Talagrand, M. (1996). A new look at independence. Ann. Probab. 24 1–34. Mathematical Reviews (MathSciNet): MR1387624
Digital Object Identifier: doi:10.1214/aop/1065725175
Project Euclid: euclid.aop/1042644705
Zentralblatt MATH: 0858.60019 - Talagrand, M. (1996). New concentration inequalities in product spaces. Invent. Math. 126 505–563. Mathematical Reviews (MathSciNet): MR1419006
Digital Object Identifier: doi:10.1007/s002220050108
Zentralblatt MATH: 0893.60001 - Tsybakov, A. (2004). Optimal aggregation of classifiers in statistical learning. Ann. Statist. 32 135–166. Mathematical Reviews (MathSciNet): MR2051002
Digital Object Identifier: doi:10.1214/aos/1079120131
Project Euclid: euclid.aos/1079120131
Zentralblatt MATH: 1105.62353 - Tsybakov, A. and van de Geer, S. (2005). Square root penalty: Adaptation to the margin in classification and in edge estimation. Ann. Statist. 33 1203–1224. Mathematical Reviews (MathSciNet): MR2195633
Digital Object Identifier: doi:10.1214/009053604000001066
Project Euclid: euclid.aos/1120224100
Zentralblatt MATH: 1080.62047 - van de Geer, S. (2000). Empirical Processes in M-Estimation. Cambridge Univ. Press.Zentralblatt MATH: 1179.62073
- van der Vaart, A. W. and Wellner, J. A. (1996). Weak Convergence and Empirical Processes. With Applications to Statistics. Springer, New York.
- Vapnik, V. (1998). Statistical Learning Theory. Wiley, New York.
- Vapnik, V. and Chervonenkis, A. (1974). Theory of Pattern Recognition. Nauka, Moscow (in Russian).
- Zhang, T. (2004). Statistical behavior and consistency of classification methods based on convex risk minimization (with discussion). Ann. Statist. 32 56–117, 128–134. Mathematical Reviews (MathSciNet): MR2051001
Digital Object Identifier: doi:10.1214/aos/1079120130
Project Euclid: euclid.aos/1079120130
Zentralblatt MATH: 1105.62323
- You have access to this content.
- You have partial access to this content.
- You do not have access to this content.
More like this
- Concentration inequalities and asymptotic results for ratio type empirical processes
Giné, Evarist and Koltchinskii, Vladimir, The Annals of Probability, 2006 - Sparse recovery in convex hulls via entropy penalization
Koltchinskii, Vladimir, The Annals of Statistics, 2009 - On minimax density estimation on \mathbb{R}}
Juditsky, Anatoli and Lambert-Lacroix, Sophie, Bernoulli, 2004
- Concentration inequalities and asymptotic results for ratio type empirical processes
Giné, Evarist and Koltchinskii, Vladimir, The Annals of Probability, 2006 - Sparse recovery in convex hulls via entropy penalization
Koltchinskii, Vladimir, The Annals of Statistics, 2009 - On minimax density estimation on \mathbb{R}}
Juditsky, Anatoli and Lambert-Lacroix, Sophie, Bernoulli, 2004 - Inverse statistical learning
Loustau, Sébastien, Electronic Journal of Statistics, 2013 - Rademacher complexity for Markov chains: Applications to kernel smoothing and Metropolis–Hastings
Bertail, Patrice and Portier, François, Bernoulli, 2019 - Sparsity in penalized empirical risk minimization
Koltchinskii, Vladimir, Annales de l'Institut Henri Poincaré, Probabilités et Statistiques, 2009 - Bounds on Walsh coefficients by dyadic difference and a new
Koksma-Hlawka type inequality for Quasi-Monte Carlo integration
Yoshiki, Takehito, Hiroshima Mathematical Journal, 2017 - Empirical risk minimization in inverse problems
Klemelä, Jussi and Mammen, Enno, The Annals of Statistics, 2010 - Empirical entropy, minimax regret and minimax risk
Rakhlin, Alexander, Sridharan, Karthik, and Tsybakov, Alexandre B., Bernoulli, 2017 - Asymptotic bounds for the distribution of the sum of dependent random variables
Wang, Ruodu, Journal of Applied Probability, 2014