The Annals of Statistics

Sparsity in multiple kernel learning

Vladimir Koltchinskii and Ming Yuan

Full-text: Open access

Enhanced PDF (303 KB)

Abstract

The problem of multiple kernel learning based on penalized empirical risk minimization is discussed. The complexity penalty is determined jointly by the empirical L2 norms and the reproducing kernel Hilbert space (RKHS) norms induced by the kernels with a data-driven choice of regularization parameters. The main focus is on the case when the total number of kernels is large, but only a relatively small number of them is needed to represent the target function, so that the problem is sparse. The goal is to establish oracle inequalities for the excess risk of the resulting prediction rule showing that the method is adaptive both to the unknown design distribution and to the sparsity of the problem.

Article information

Source
Ann. Statist., Volume 38, Number 6 (2010), 3660-3695.

Dates
First available in Project Euclid: 30 November 2010

Permanent link to this document
https://projecteuclid.org/euclid.aos/1291126969

Digital Object Identifier
doi:10.1214/10-AOS825

Mathematical Reviews number (MathSciNet)
MR2766864

Zentralblatt MATH identifier
1204.62086

Subjects
Primary: 62G08: Nonparametric regression 62F12: Asymptotic properties of estimators
Secondary: 62J07: Ridge regression; shrinkage estimators

Keywords
High dimensionality multiple kernel learning oracle inequality reproducing kernel Hilbert spaces restricted isometry sparsity

Citation

Koltchinskii, Vladimir; Yuan, Ming. Sparsity in multiple kernel learning. Ann. Statist. 38 (2010), no. 6, 3660--3695. doi:10.1214/10-AOS825. https://projecteuclid.org/euclid.aos/1291126969


Export citation

References

  • Aronszajn, N. (1950). Theory of reproducing kernels. Trans. Amer. Math. Soc. 68 337–404.
    Zentralblatt MATH: 0037.20701
    Digital Object Identifier: doi:10.1090/S0002-9947-1950-0051437-7
  • Bach, F. (2008). Consistency of the group lasso and multiple kernel learning. J. Mach. Learn. Res. 9 1179–1225.
    Zentralblatt MATH: 1225.68147
  • Bickel, P., Ritov, Y. and Tsybakov, A. (2009). Simultaneous analysis of lasso and Dantzig selector. Ann. Statist. 37 1705–1732.
    Zentralblatt MATH: 1173.62022
    Digital Object Identifier: doi:10.1214/08-AOS620
    Project Euclid: euclid.aos/1245332830
  • Bousquet, O. and Herrmann, D. (2003). On the complexity of learning the kernel matrix. In Advances in Neural Information Processing Systems 15 415–422. MIT Press, Cambridge.
  • Blanchard, G., Bousquet, O. and Massart, P. (2008). Statistical performance of support vector machines. Ann. Statist. 36 489–531.
    Zentralblatt MATH: 1133.62044
    Digital Object Identifier: doi:10.1214/009053607000000839
    Project Euclid: euclid.aos/1205420509
  • Bousquet, O. (2002). A Bennett concentration inequality and its applications to suprema of empirical processes. C. R. Math. Acad. Sci. Paris 334 495–500.
    Zentralblatt MATH: 1001.60021
    Digital Object Identifier: doi:10.1016/S1631-073X(02)02292-6
  • Candes, E. and Tao, T. (2007). The Dantzig selector: Statistical estimation when p is much larger than n. Ann. Statist. 35 2313–2351.
    Zentralblatt MATH: 1139.62019
    Digital Object Identifier: doi:10.1214/009053606000001523
    Project Euclid: euclid.aos/1201012958
  • Crammer, K., Keshet, J. and Singer, Y. (2003). Kernel design using boosting. In Advances in Neural Information Processing Systems 15 553–560. MIT Press, Cambridge.
  • Koltchinskii, V. (2008). Oracle inequalities in empirical risk minimization and sparse recovery problems. Ecole d’Eté de Probabilités de Saint-Flour, Lecture Notes. Preprint.
  • Koltchinskii, V. (2009a). Sparsity in penalized empirical risk minimization. Ann. Inst. H. Poincaré Probab. Statist. 45 7–57.
    Zentralblatt MATH: 1168.62044
    Digital Object Identifier: doi:10.1214/07-AIHP146
    Project Euclid: euclid.aihp/1234469970
  • Koltchinskii, V. (2009b). The Dantzig selector and sparsity oracle inequalities. Bernoulli 15 799–828.
    Zentralblatt MATH: 05815956
    Digital Object Identifier: doi:10.3150/09-BEJ187
    Project Euclid: euclid.bj/1251463282
  • Koltchinskii, V. (2009c). Sparse recovery in convex hulls via entropy penalization. Ann. Statist. 37 1332–1359.
    Zentralblatt MATH: 1269.62039
    Digital Object Identifier: doi:10.1214/08-AOS621
    Project Euclid: euclid.aos/1239369024
  • Koltchinskii, V. and Yuan, M. (2008). Sparse recovery in large ensembles of kernel machines. In Proceedings of 19th Annual Conference on Learning Theory 229–238. Omnipress, Madison, WI.
  • Lanckriet, G., Cristianini, N., Bartlett, P., Ghaoui, L. and Jordan, M. (2004). Learning the kernel matrix with semidefinite programming. J. Mach. Learn. Res. 5 27–72.
    Zentralblatt MATH: 1222.68241
  • Ledoux, M. and Talagrand, M. (1991). Probability in Banach Spaces. Springer, New York.
    Zentralblatt MATH: 0748.60004
  • Lin, Y. and Zhang, H. (2006). Component selection and smoothing in multivariate nonparametric regression. Ann. Statist. 34 2272–2297.
    Zentralblatt MATH: 1106.62041
    Digital Object Identifier: doi:10.1214/009053606000000722
    Project Euclid: euclid.aos/1169571797
  • Meier, L., van de Geer, S. and Bühlmann, P. (2009). High-dimensional additive modeling. Ann. Statist. 37 3779–3821.
    Zentralblatt MATH: 1360.62186
    Digital Object Identifier: doi:10.1214/09-AOS692
    Project Euclid: euclid.aos/1256303527
  • Mendelson, S. (2002). Geometric parameters of kernel machines. In COLT 2002. Lecture Notes in Artificial Intelligence 2375 29–43. Springer, Berlin.
    Mathematical Reviews (MathSciNet): MR2040403
    Zentralblatt MATH: 1050.68070
  • Micchelli, C. and Pontil, M. (2005). Learning the kernel function via regularization. J. Mach. Learn. Res. 6 1099–1125.
    Zentralblatt MATH: 1222.68265
  • Raskutti, G., Wainwright, M. and Yu, B. (2009). Lower bounds on minimax rates for nonparametric regression with additive sparsity and smoothness. In Advances in Neural Information Processing Systems (NIPS 22) 1563–1570. Curran Associates, Red Hook, NY.
  • Ravikumar, P., Liu, H., Lafferty, J. and Wasserman, L. (2008). SpAM: Sparse additive models. In Advances in Neural Information Processing Systems (NIPS 20) 1201–1208. Curran Associates, Red Hook, NY.
  • Srebro, N. and Ben-David, S. (2006). Learning bounds for support vector machines with learned kernels. In Learning Theory. Lecture Notes in Comput. Sci. 4005 169–183. Springer, Berlin.
    Zentralblatt MATH: 1143.68561
  • Talagrand, M. (1996). New concentration inequalities for product measures. Invent. Math. 126 505–563.
    Zentralblatt MATH: 0893.60001
    Digital Object Identifier: doi:10.1007/s002220050108
  • Tsybakov, A. B. (2009). Introduction to Nonparametric Estimation. Springer, New York.
    Zentralblatt MATH: 1176.62032
  • van der Vaart, A. W. and Wellner, J. A. (1996). Weak Convergence and Empirical Processes. With Applications to Statistics. Springer, New York.
    Zentralblatt MATH: 0862.60002

The Institute of Mathematical Statistics

The Annals of Statistics

New content alerts

Email RSS ToC RSS Article
  • You have access to this content.
  • You have partial access to this content.
  • You do not have access to this content.
Turn Off MathJax

What is MathJax?

More like this

+ See more