The Annals of Statistics

A general theory of hypothesis tests and confidence regions for sparse high dimensional models

Abstract

We consider the problem of uncertainty assessment for low dimensional components in high dimensional models. Specifically, we propose a novel decorrelated score function to handle the impact of high dimensional nuisance parameters. We consider both hypothesis tests and confidence regions for generic penalized M-estimators. Unlike most existing inferential methods which are tailored for individual models, our method provides a general framework for high dimensional inference and is applicable to a wide variety of applications. In particular, we apply this general framework to study five illustrative examples: linear regression, logistic regression, Poisson regression, Gaussian graphical model and additive hazards model. For hypothesis testing, we develop general theorems to characterize the limiting distributions of the decorrelated score test statistic under both null hypothesis and local alternatives. These results provide asymptotic guarantees on the type I errors and local powers. For confidence region construction, we show that the decorrelated score function can be used to construct point estimators that are asymptotically normal and semiparametrically efficient. We further generalize this framework to handle the settings of misspecified models. Thorough numerical results are provided to back up the developed theory.

Article information

Source
Ann. Statist., Volume 45, Number 1 (2017), 158-195.

Dates
Revised: January 2016
First available in Project Euclid: 21 February 2017

https://projecteuclid.org/euclid.aos/1487667620

Digital Object Identifier
doi:10.1214/16-AOS1448

Mathematical Reviews number (MathSciNet)
MR3611489

Zentralblatt MATH identifier
1364.62128

Subjects
Primary: 62E20: Asymptotic distribution theory 62F03: Hypothesis testing
Secondary: 62F25: Tolerance and confidence regions

Citation

Ning, Yang; Liu, Han. A general theory of hypothesis tests and confidence regions for sparse high dimensional models. Ann. Statist. 45 (2017), no. 1, 158--195. doi:10.1214/16-AOS1448. https://projecteuclid.org/euclid.aos/1487667620

References

• [1] Bach, F. (2010). Self-concordant analysis for logistic regression. Electron. J. Stat. 4 384–414.
• [2] Belloni, A., Chernozhukov, V. and Wang, L. (2011). Square-root Lasso: Pivotal recovery of sparse signals via conic programming. Biometrika 98 791–806.
• [3] Belloni, A., Chernozhukov, V. and Wei, Y. (2013). Honest confidence regions for logistic regression with a large number of controls. Preprint. Available at arXiv:1304.3969.
• [4] Bickel, P. J. (1975). One-step Huber estimates in the linear model. J. Amer. Statist. Assoc. 70 428–434.
• [5] Bickel, P. J., Ritov, Y. and Tsybakov, A. B. (2009). Simultaneous analysis of Lasso and Dantzig selector. Ann. Statist. 37 1705–1732.
• [6] Candes, E. and Tao, T. (2007). The Dantzig selector: Statistical estimation when $p$ is much larger than $n$. Ann. Statist. 35 2313–2351.
• [7] Cox, D. R. and Hinkley, D. V. (1979). Theoretical Statistics. CRC Press, Boca Raton, FL.
• [8] Fan, J. and Li, R. (2001). Variable selection via nonconcave penalized likelihood and its oracle properties. J. Amer. Statist. Assoc. 96 1348–1360.
• [9] Fang, E. X., Ning, Y. and Liu, H. (2014). Testing and confidence intervals for high dimensional proportional hazards model. Preprint. Available at arXiv:1412.5158.
• [10] Godambe, V. P. and Kale, B. K. (1991). Estimating functions: An overview. In Estimating Functions. Oxford Statist. Sci. Ser. 7 3–20. Oxford Univ. Press, New York.
• [11] Godfrey, L. G. (1991). Misspecification Tests in Econometrics: The Lagrange Multiplier Principle and Other Approaches. Econometric Society Monographs 16. Cambridge Univ. Press, Cambridge. Reprint of the 1988 original.
• [12] Jankova, J. and van de Geer, S. (2013). Confidence intervals for high-dimensional inverse covariance estimation. Preprint. Available at arXiv:1403.6752.
• [13] Javanmard, A. and Montanari, A. (2014). Confidence intervals and hypothesis testing for high-dimensional regression. J. Mach. Learn. Res. 15 2869–2909.
• [14] Javanmard, A. and Montanari, A. (2014). Hypothesis testing in high-dimensional regression under the Gaussian random design model: Asymptotic theory. IEEE Trans. Inform. Theory 60 6522–6554.
• [15] Knight, K. and Fu, W. (2000). Asymptotics for Lasso-type estimators. Ann. Statist. 28 1356–1378.
• [16] Lee, J. D., Sun, D. L., Sun, Y. and Taylor, J. E. (2013). Exact inference after model selection via the Lasso. Preprint. Available at arXiv:1311.6238.
• [17] Lin, D. Y. and Ying, Z. (1994). Semiparametric analysis of the additive risk model. Biometrika 81 61–71.
• [18] Lin, W. and Lv, J. (2013). High-dimensional sparse additive hazards regression. J. Amer. Statist. Assoc. 108 247–264.
• [19] Lindsay, B. (1982). Conditional score functions: Some optimality results. Biometrika 69 503–512.
• [20] Liu, W. (2013). Gaussian graphical model estimation with false discovery rate control. Ann. Statist. 41 2948–2978.
• [21] Liu, W. and Luo, X. (2012). High-dimensional sparse precision matrix estimation via sparse column inverse operator. Preprint. Available at arXiv:1203.3896.
• [22] Lockhart, R., Taylor, J., Tibshirani, R. J. and Tibshirani, R. (2014). A significance test for the Lasso. Ann. Statist. 42 413–468.
• [23] Martinussen, T. and Scheike, T. H. (2009). Covariate selection for the semiparametric additive risk model. Scand. J. Stat. 36 602–619.
• [24] Meinshausen, N. and Bühlmann, P. (2006). High-dimensional graphs and variable selection with the Lasso. Ann. Statist. 34 1436–1462.
• [25] Meinshausen, N. and Bühlmann, P. (2010). Stability selection. J. R. Stat. Soc. Ser. B. Stat. Methodol. 72 417–473.
• [26] Meinshausen, N., Meier, L. and Bühlmann, P. (2009). $p$-values for high-dimensional regression. J. Amer. Statist. Assoc. 104 1671–1681.
• [27] Ning, Y. and Liu, H. (2016). Supplement to “A general theory of hypothesis tests and confidence regions for sparse high dimensional models.” DOI:10.1214/16-AOS1448SUPP.
• [28] Portnoy, S. (1984). Asymptotic behavior of $M$-estimators of $p$ regression parameters when $p^{2}/n$ is large. I. Consistency. Ann. Statist. 12 1298–1309.
• [29] Portnoy, S. (1985). Asymptotic behavior of $M$ estimators of $p$ regression parameters when $p^{2}/n$ is large. II. Normal approximation. Ann. Statist. 13 1403–1417.
• [30] Ren, Z., Sun, T., Zhang, C.-H. and Zhou, H. H. (2015). Asymptotic normality and optimalities in estimation of large Gaussian graphical models. Ann. Statist. 43 991–1026.
• [31] Small, C. G. and McLeish, D. L. (2011). Hilbert Space Methods in Probability and Statistical Inference 920. John Wiley & Sons, New York.
• [32] Sun, T. and Zhang, C.-H. (2012). Scaled sparse linear regression. Biometrika 99 879–898.
• [33] Tibshirani, R. (1996). Regression shrinkage and selection via the Lasso. J. R. Stat. Soc. Ser. B. Stat. Methodol. 58 267–288.
• [34] van de Geer, S., Bühlmann, P., Ritov, Y. and Dezeure, R. (2014). On asymptotically optimal confidence regions and tests for high-dimensional models. Ann. Statist. 42 1166–1202.
• [35] Van der Vaart, A. W. (2000). Asymptotic Statistics 3. Cambridge Univ. Press, Cambridge.
• [36] Voorman, A., Shojaie, A. and Witten, D. (2014). Inference in high dimensions with the penalized score test. Preprint. Available at arXiv:1401.2678.
• [37] Wang, L., Kim, Y. and Li, R. (2013). Calibrating nonconvex penalized regression in ultra-high dimension. Ann. Statist. 41 2505–2536.
• [38] Wang, Z., Liu, H. and Zhang, T. (2014). Optimal computational and statistical rates of convergence for sparse nonconvex learning problems. Ann. Statist. 42 2164–2201.
• [39] Wasserman, L. and Roeder, K. (2009). High-dimensional variable selection. Ann. Statist. 37 2178–2201.
• [40] White, H. (1982). Maximum likelihood estimation of misspecified models. Econometrica 50 1–25.
• [41] Zhang, C.-H. (2010). Nearly unbiased variable selection under minimax concave penalty. Ann. Statist. 38 894–942.
• [42] Zhang, C.-H. and Zhang, S. S. (2014). Confidence intervals for low dimensional parameters in high dimensional linear models. J. R. Stat. Soc. Ser. B. Stat. Methodol. 76 217–242.
• [43] Zhao, P. and Yu, B. (2006). On model selection consistency of Lasso. J. Mach. Learn. Res. 7 2541–2563.
• [44] Zhong, P.-S., Hu, T. and Li, J. (2015). Tests for coefficients in high-dimensional additive hazard models. Scand. J. Stat. 42 649–664.

Supplemental materials

• Supplement to “A general theory of hypothesis tests and confidence regions for sparse high dimensional models”. The supplementary materials contain additional technical details, simulation results and proofs.