The Annals of Statistics

SCAD-penalized regression in high-dimensional partially linear models

Huiliang Xie and Jian Huang

Full-text: Open access

Abstract

We consider the problem of simultaneous variable selection and estimation in partially linear models with a divergent number of covariates in the linear part, under the assumption that the vector of regression coefficients is sparse. We apply the SCAD penalty to achieve sparsity in the linear part and use polynomial splines to estimate the nonparametric component. Under reasonable conditions, it is shown that consistency in terms of variable selection and estimation can be achieved simultaneously for the linear and nonparametric components. Furthermore, the SCAD-penalized estimators of the nonzero coefficients are shown to have the asymptotic oracle property, in the sense that it is asymptotically normal with the same means and covariances that they would have if the zero coefficients were known in advance. The finite sample behavior of the SCAD-penalized estimators is evaluated with simulation and illustrated with a data set.

Article information

Source
Ann. Statist., Volume 37, Number 2 (2009), 673-696.

Dates
First available in Project Euclid: 10 March 2009

Permanent link to this document
https://projecteuclid.org/euclid.aos/1236693146

Digital Object Identifier
doi:10.1214/07-AOS580

Mathematical Reviews number (MathSciNet)
MR2502647

Zentralblatt MATH identifier
1162.62037

Subjects
Primary: 62J05: Linear regression 62G08: Nonparametric regression
Secondary: 62E20: Asymptotic distribution theory

Keywords
Asymptotic normality high-dimensional data oracle property penalized estimation semiparametric models variable selection

Citation

Xie, Huiliang; Huang, Jian. SCAD-penalized regression in high-dimensional partially linear models. Ann. Statist. 37 (2009), no. 2, 673--696. doi:10.1214/07-AOS580. https://projecteuclid.org/euclid.aos/1236693146


Export citation

References

  • Berndt, E. R. (1991). The Practice of Econometrics: Classical and Contemporary. Addison-Wesley, Reading, MA.
  • Chen, H. (1988). Convergence rates for parametric components in a partly linear model. Ann. Statist. 16 136–146.
  • Engle, R. F., Granger, C. W., Rice, J. and Weiss, A. (1986). Semiparametric estimates of the relationship between weather and electricity sales. J. Amer. Statist. Assoc. 81 310–320.
  • Fan, J. and Li, R. (2001). Variable selection via nonconcave penalized likelihood and its oracle properties. J. Amer. Statist. Assoc. 96 1348–1360.
  • Fan, J. and Li, R. (2004). New estimation and model selection procedures for semiparametric modeling in longitudinal data analysis. J. Amer. Statist. Assoc. 99 710–723.
  • Fan, J. and Peng, H. (2004). Nonconcave penalized likelihood with a diverging number of parameters. Ann. Statist. 32 928–961.
  • Friedman, J. H. (1991). Multivariate adaptive regression splines. Ann. Statist. 19 1–68.
  • Heckman, N. E. (1986). Spline smoothing in a partly linear model. J. R. Stat. Soc. Ser. B Stat. Methodol. 48 244–248.
  • Huang, J. (1999). Efficient estimation of the partially linear additive Cox model. Ann. Statist. 27 1536–1563.
  • Huang, J., Horowitz, J. L. and Ma, S. G. (2008). Asymptotic properties of bridge estimators in sparse high-dimensional regression models. Ann. Statist. 36 587–613.
  • Hunter, D. R. and Li, R. (2005). Variable selection using MM algorithms. Ann. Statist. 33 1617–1642.
  • Kim, Y., Kim, J. and Kim, Y. (2006). Blockwise sparse regression. Statistica Sinica 16 375–390.
  • Knight, K. and Fu, W. (2000). Asymptotics for lasso-type estimators. Ann. Statist. 28 1356–1378.
  • Meinshausen, N. and Buhlmann, P. (2006). High dimensional graphs and variable selection with the Lasso. Ann. Statist. 34 1436–1462.
  • Robinson, P. M. (1988). Root-n-consistent semiparametric regression. Econometrica 56 931–954.
  • Schumaker, L. (1981). Spline Functions: Basic Theory. Wiley, New York.
  • Speckman, P. (1988). Kernel smoothing in partial lineal models. J. R. Stat. Soc. Ser. B Stat. Methodol. 50 413–436.
  • Stone, C. J. (1980). Optimal rates of convergence for nonparametric estimators. Ann. Statist. 8 1348–1360.
  • Stone, C. J. (1982). Optimal global rates of convergence for nonparametric regression. Ann. Statist. 10 1040–1053.
  • Stone, C. J. (1985). Additive regression and other nonparametric models. Ann. Statist. 13 689–705.
  • Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. J. R. Stat. Soc. Ser. B Stat. Methodol. 58 267–288.
  • van de Geer, S. (2008). High-dimensional generalized linear models and the Lasso. Ann. Statist. 36 614–645.
  • Wahba, G. (1984). Partial spline models for the semiparametric estimation of functions of several variables. In Statistical Analyses for Time Series, Japan-US Joint Seminar 319–329. Institute of Statistical Mathematics, Tokyo.
  • Wahba, G. (1990). Spline Models for Observational Data. SIAM, Philadelphia, PA.
  • Wang, S. and Jia, Z. (1993). Inequalities in Matrix Theory. Anhui Education Press, Hefei, China.
  • Zhang, C.-H. and Huang, J. (2008). The sparsity and bias of the Lasso selection in high-dimensional linear regression. Ann. Statist. 36 1567–1594.
  • Zhao, P. and Yu, B. (2006). On model selection consistency of LASSO. J. Mach. Learn. Res. 7 2541–2563.