The Annals of Statistics

SCAD-penalized regression in high-dimensional partially linear models

Huiliang Xie and Jian Huang

Full-text: Open access


We consider the problem of simultaneous variable selection and estimation in partially linear models with a divergent number of covariates in the linear part, under the assumption that the vector of regression coefficients is sparse. We apply the SCAD penalty to achieve sparsity in the linear part and use polynomial splines to estimate the nonparametric component. Under reasonable conditions, it is shown that consistency in terms of variable selection and estimation can be achieved simultaneously for the linear and nonparametric components. Furthermore, the SCAD-penalized estimators of the nonzero coefficients are shown to have the asymptotic oracle property, in the sense that it is asymptotically normal with the same means and covariances that they would have if the zero coefficients were known in advance. The finite sample behavior of the SCAD-penalized estimators is evaluated with simulation and illustrated with a data set.

Article information

Ann. Statist., Volume 37, Number 2 (2009), 673-696.

First available in Project Euclid: 10 March 2009

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 62J05: Linear regression 62G08: Nonparametric regression
Secondary: 62E20: Asymptotic distribution theory

Asymptotic normality high-dimensional data oracle property penalized estimation semiparametric models variable selection


Xie, Huiliang; Huang, Jian. SCAD-penalized regression in high-dimensional partially linear models. Ann. Statist. 37 (2009), no. 2, 673--696. doi:10.1214/07-AOS580.

Export citation


  • Berndt, E. R. (1991). The Practice of Econometrics: Classical and Contemporary. Addison-Wesley, Reading, MA.
  • Chen, H. (1988). Convergence rates for parametric components in a partly linear model. Ann. Statist. 16 136–146.
  • Engle, R. F., Granger, C. W., Rice, J. and Weiss, A. (1986). Semiparametric estimates of the relationship between weather and electricity sales. J. Amer. Statist. Assoc. 81 310–320.
  • Fan, J. and Li, R. (2001). Variable selection via nonconcave penalized likelihood and its oracle properties. J. Amer. Statist. Assoc. 96 1348–1360.
  • Fan, J. and Li, R. (2004). New estimation and model selection procedures for semiparametric modeling in longitudinal data analysis. J. Amer. Statist. Assoc. 99 710–723.
  • Fan, J. and Peng, H. (2004). Nonconcave penalized likelihood with a diverging number of parameters. Ann. Statist. 32 928–961.
  • Friedman, J. H. (1991). Multivariate adaptive regression splines. Ann. Statist. 19 1–68.
  • Heckman, N. E. (1986). Spline smoothing in a partly linear model. J. R. Stat. Soc. Ser. B Stat. Methodol. 48 244–248.
  • Huang, J. (1999). Efficient estimation of the partially linear additive Cox model. Ann. Statist. 27 1536–1563.
  • Huang, J., Horowitz, J. L. and Ma, S. G. (2008). Asymptotic properties of bridge estimators in sparse high-dimensional regression models. Ann. Statist. 36 587–613.
  • Hunter, D. R. and Li, R. (2005). Variable selection using MM algorithms. Ann. Statist. 33 1617–1642.
  • Kim, Y., Kim, J. and Kim, Y. (2006). Blockwise sparse regression. Statistica Sinica 16 375–390.
  • Knight, K. and Fu, W. (2000). Asymptotics for lasso-type estimators. Ann. Statist. 28 1356–1378.
  • Meinshausen, N. and Buhlmann, P. (2006). High dimensional graphs and variable selection with the Lasso. Ann. Statist. 34 1436–1462.
  • Robinson, P. M. (1988). Root-n-consistent semiparametric regression. Econometrica 56 931–954.
  • Schumaker, L. (1981). Spline Functions: Basic Theory. Wiley, New York.
  • Speckman, P. (1988). Kernel smoothing in partial lineal models. J. R. Stat. Soc. Ser. B Stat. Methodol. 50 413–436.
  • Stone, C. J. (1980). Optimal rates of convergence for nonparametric estimators. Ann. Statist. 8 1348–1360.
  • Stone, C. J. (1982). Optimal global rates of convergence for nonparametric regression. Ann. Statist. 10 1040–1053.
  • Stone, C. J. (1985). Additive regression and other nonparametric models. Ann. Statist. 13 689–705.
  • Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. J. R. Stat. Soc. Ser. B Stat. Methodol. 58 267–288.
  • van de Geer, S. (2008). High-dimensional generalized linear models and the Lasso. Ann. Statist. 36 614–645.
  • Wahba, G. (1984). Partial spline models for the semiparametric estimation of functions of several variables. In Statistical Analyses for Time Series, Japan-US Joint Seminar 319–329. Institute of Statistical Mathematics, Tokyo.
  • Wahba, G. (1990). Spline Models for Observational Data. SIAM, Philadelphia, PA.
  • Wang, S. and Jia, Z. (1993). Inequalities in Matrix Theory. Anhui Education Press, Hefei, China.
  • Zhang, C.-H. and Huang, J. (2008). The sparsity and bias of the Lasso selection in high-dimensional linear regression. Ann. Statist. 36 1567–1594.
  • Zhao, P. and Yu, B. (2006). On model selection consistency of LASSO. J. Mach. Learn. Res. 7 2541–2563.