Open Access
February 2017 A general theory of hypothesis tests and confidence regions for sparse high dimensional models
Yang Ning, Han Liu
Ann. Statist. 45(1): 158-195 (February 2017). DOI: 10.1214/16-AOS1448


We consider the problem of uncertainty assessment for low dimensional components in high dimensional models. Specifically, we propose a novel decorrelated score function to handle the impact of high dimensional nuisance parameters. We consider both hypothesis tests and confidence regions for generic penalized M-estimators. Unlike most existing inferential methods which are tailored for individual models, our method provides a general framework for high dimensional inference and is applicable to a wide variety of applications. In particular, we apply this general framework to study five illustrative examples: linear regression, logistic regression, Poisson regression, Gaussian graphical model and additive hazards model. For hypothesis testing, we develop general theorems to characterize the limiting distributions of the decorrelated score test statistic under both null hypothesis and local alternatives. These results provide asymptotic guarantees on the type I errors and local powers. For confidence region construction, we show that the decorrelated score function can be used to construct point estimators that are asymptotically normal and semiparametrically efficient. We further generalize this framework to handle the settings of misspecified models. Thorough numerical results are provided to back up the developed theory.


Download Citation

Yang Ning. Han Liu. "A general theory of hypothesis tests and confidence regions for sparse high dimensional models." Ann. Statist. 45 (1) 158 - 195, February 2017.


Received: 1 September 2015; Revised: 1 January 2016; Published: February 2017
First available in Project Euclid: 21 February 2017

zbMATH: 1364.62128
MathSciNet: MR3611489
Digital Object Identifier: 10.1214/16-AOS1448

Primary: 62E20 , 62F03
Secondary: 62F25

Keywords: Confidence interval , High dimensional inference , hypothesis test , model misspecification , nuisance parameter , score function , Sparsity

Rights: Copyright © 2017 Institute of Mathematical Statistics

Vol.45 • No. 1 • February 2017
Back to Top