Necessary and sufficient conditions for variable selection consistency of the LASSO in high dimensions

Abstract

This paper investigates conditions for variable selection consistency of the LASSO in high dimensional regression models and gives necessary and sufficient conditions for the same, potentially allowing the model dimension p to grow arbitrarily fast as a function of the sample size n. These conditions require both upper and lower bounds on the growth rate of the penalty parameter. It turns out that a variant of the irrepresentable Condition (IRC) of (J. Mach. Learn. Res. 7 (2006) 2541–2563), herein called the lower irrepresentable Condition (or LIRC), is determined by the lower bound considerations while the upper bound considerations lead to a new condition, called the upper irrepresentable Condition (or UIRC) in this paper. It is shown that the LIRC together with the UIRC is necessary and sufficient for the variable selection consistency of the LASSO, thereby settling a conjecture of (J. Mach. Learn. Res. 7 (2006) 2541–2563). Further, it is shown that under some mild regularity conditions, the penalty parameter must necessarily tend to infinity at a certain minimal rate to ensure variable selection consistency of the LASSO and that the corresponding LASSO estimators of the nonzero regression parameters cannot be $\sqrt{\mathit{n}}$-consistent (even for individual parameters). Thus, under fairly general conditions, the LASSO with a single choice of the penalty parameter cannot achieve both variable selection consistency and $\sqrt{\mathit{n}}$-consistency simultaneously.