The Annals of Applied Statistics

A simple forward selection procedure based on false discovery rate control

Yoav Benjamini and Yulia Gavrilov

Full-text: Open access

Enhanced PDF (252 KB)

Abstract

We propose the use of a new false discovery rate (FDR) controlling procedure as a model selection penalized method, and compare its performance to that of other penalized methods over a wide range of realistic settings: nonorthogonal design matrices, moderate and large pool of explanatory variables, and both sparse and nonsparse models, in the sense that they may include a small and large fraction of the potential variables (and even all). The comparison is done by a comprehensive simulation study, using a quantitative framework for performance comparisons in the form of empirical minimaxity relative to a “random oracle”: the oracle model selection performance on data dependent forward selected family of potential models. We show that FDR based procedures have good performance, and in particular the newly proposed method, emerges as having empirical minimax performance. Interestingly, using FDR level of 0.05 is a global best.

Article information

Source
Ann. Appl. Stat., Volume 3, Number 1 (2009), 179-198.

Dates
First available in Project Euclid: 16 April 2009

Permanent link to this document
https://projecteuclid.org/euclid.aoas/1239888367

Digital Object Identifier
doi:10.1214/08-AOAS194

Mathematical Reviews number (MathSciNet)
MR2668704

Zentralblatt MATH identifier
1160.62068

Keywords
Linear regression multiple testing random oracle

Citation

Benjamini, Yoav; Gavrilov, Yulia. A simple forward selection procedure based on false discovery rate control. Ann. Appl. Stat. 3 (2009), no. 1, 179--198. doi:10.1214/08-AOAS194. https://projecteuclid.org/euclid.aoas/1239888367


Export citation

References

  • Abramovich, F. and Benjamini, Y. (1995). Thresholding of wavelet coefficients as multiple hypotheses testing procedure. In Wavelets and Statistics. Lecture Notes in Statistics 103 5–14. Springer, Berlin.
  • Abramovich, F., Benjamini, Y., Donoho, D. and Johnstone, I. (2006). Adapting to unknown sparsity by controlling the false discovery rate. Ann. Statist. 34 584–653.
    Mathematical Reviews (MathSciNet): MR2281879
    Zentralblatt MATH: 1092.62005
    Digital Object Identifier: doi:10.1214/009053606000000074
    Project Euclid: euclid.aos/1151418235
  • Akaike, H. (1973). Information theory and an extension of the maximum likelihood principle. In Second International Symposium on Information Theory (B. N. Petrov and F. Czaki, eds.) 267–281. Akademia Kiadó, Budapest.
    Mathematical Reviews (MathSciNet): MR483125
    Zentralblatt MATH: 0283.62006
  • Andrews, D. F., Bickel, P. J., Hampel, F. R., Huber, P. J., Rogers, W. H. and Tukey, J. W. (1972). Robust Estimates of Location: Survey and Advances. Princeton Univ. Press, Princeton, NJ.
    Mathematical Reviews (MathSciNet): MR331595
    Zentralblatt MATH: 0254.62001
  • Benjamini, Y. and Gavrilov, Y. (2009). Supplement to “A simple forward selection procedure based on false discovery rate control.” DOI: 10.1214/08-AOAS194SUPP.
  • Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. J. Roy. Statist. Soc. Ser. B 57 289–300.
    Mathematical Reviews (MathSciNet): MR1325392
  • Benjamini, Y. and Hochberg, Y. (2000). On the adaptive control of the false discovery fate in multiple testing with independent statistics. J. Educ. Behav. Stat. 25 60–83.
  • Benjamini, Y., Krieger, A. and Yekutieli, D. (2006). Adaptive linear step-up procedures that control the false discovery rate. Biometrica 93 491–507.
    Mathematical Reviews (MathSciNet): MR2261438
    Zentralblatt MATH: 1108.62069
    Digital Object Identifier: doi:10.1093/biomet/93.3.491
  • Bickel, P., Ritov, Y. and Tsybakov, A. (2008a). Simultaneous analysis of Lasso and Dantzig selector. Ann. Statist. To appear. Available at http://www.proba.jussieu.fr/pageperso/tsybakov/.
    Mathematical Reviews (MathSciNet): MR2533469
    Zentralblatt MATH: 1173.62022
    Digital Object Identifier: doi:10.1214/08-AOS620
    Project Euclid: euclid.aos/1245332830
  • Bickel, P., Ritov, Y. and Tsybakov, A. (2008b). Hierarchical selection of variables in sparse high-dimensional regression. Working paper. Available at http://pluto.huji.ac.il/~yaacov/BRT_SM_PostSubmitted.pdf.
  • Birgé, L. and Massart, P. (2001). A generalized Cp criterion for Gaussian model. Technical report, Lab. De Probabilitiés, Univ. Paris VI. Available at http://www.proba.jussieu.fr/mathdoc/preprints/index.html#2001.
  • Candes, E. and Tao, T. (2007). The Dantzig selector: Statistical estimation when p is much larger than n. Ann. Statist. 35 2313–2351.
    Mathematical Reviews (MathSciNet): MR2382644
    Zentralblatt MATH: 1139.62019
    Digital Object Identifier: doi:10.1214/009053606000001523
    Project Euclid: euclid.aos/1201012958
  • Cochran, W. G. (1977). Sampling Techniques, 3rd ed. Wiley, New York.
    Mathematical Reviews (MathSciNet): MR474575
  • Donoho, D. L. and Johnstone, I. M. (1994). Ideal spatial adaptation by wavelet shrinkage. Biometrika 81 425–455.
    Mathematical Reviews (MathSciNet): MR1311089
    Zentralblatt MATH: 0815.62019
    Digital Object Identifier: doi:10.1093/biomet/81.3.425
  • Draper, N. R. and Smith, H. (1998). Applied Regression Analyses, 3rd ed. Wiley, New York.
    Mathematical Reviews (MathSciNet): MR1614335
  • Efron, B., Hastie, T., Johnstone, I. and Tibshirani, R. (2004). Least angle regression. Ann. Statist. 32 407–499.
    Mathematical Reviews (MathSciNet): MR2060166
    Zentralblatt MATH: 1091.62054
    Digital Object Identifier: doi:10.1214/009053604000000067
    Project Euclid: euclid.aos/1083178935
  • Finner, H., Dickhaus, T. and Roters, M. (2009). On the false discovery rate and asymptotically optimal rejection curve. Ann. Statist. To appear.
    Mathematical Reviews (MathSciNet): MR2502644
    Zentralblatt MATH: 1162.62068
    Digital Object Identifier: doi:10.1214/07-AOS569
    Project Euclid: euclid.aos/1236693143
  • Foster, D. and Stine, R. (2004). Variable selection in data mining: Building a predictive model for bankruptcy. J. Amer. Statist. Assoc. 99 303–313.
    Mathematical Reviews (MathSciNet): MR2109312
    Zentralblatt MATH: 1117.62335
    Digital Object Identifier: doi:10.1198/016214504000000287
  • Gavrilov, Y., Benjamini, Y. and Sarkar, S. (2009). An adaptive step-down procedure with proven FDR control under independence. Ann. Statist. 37 619–629.
    Mathematical Reviews (MathSciNet): MR2502645
    Zentralblatt MATH: 1162.62069
    Digital Object Identifier: doi:10.1214/07-AOS586
    Project Euclid: euclid.aos/1236693144
  • Genovese, C. and Wasserman, L. (2002). Operating characteristics and extensions of the false discovery rate procedure. J. Roy. Statist. Soc. Ser. B 64 499–517.
    Mathematical Reviews (MathSciNet): MR1924303
    Zentralblatt MATH: 1090.62072
    Digital Object Identifier: doi:10.1111/1467-9868.00347
  • George, E. I. and Foster, D. P. (2000). Calibration and empirical Bayes variable selection. Biometrika 87 731–747.
    Mathematical Reviews (MathSciNet): MR1813972
    Zentralblatt MATH: 1029.62008
    Digital Object Identifier: doi:10.1093/biomet/87.4.731
  • Johnstone, I. and Silverman, B. W. (2005). Empirical Bayes selection of wavelet thresholds. Ann. Statist. 33 1700–1752.
    Mathematical Reviews (MathSciNet): MR2166560
    Zentralblatt MATH: 1078.62005
    Digital Object Identifier: doi:10.1214/009053605000000345
    Project Euclid: euclid.aos/1123250227
  • Mallows, C. L. (1973). Some comments on Cp. Technometrics 12 661–675.
  • Sarkar, S. K. (2002). Some results on false discovery rate in stepwise multiple testing procedures. Ann. Statist. 30 239–257.
    Mathematical Reviews (MathSciNet): MR1892663
    Zentralblatt MATH: 1101.62349
    Digital Object Identifier: doi:10.1214/aos/1015362192
    Project Euclid: euclid.aos/1015362192
  • Shen, X. and Ye, J. (2002). Adaptive model selection. J. Amer. Statist. Assoc. 97 210–221.
    Mathematical Reviews (MathSciNet): MR1947281
    Zentralblatt MATH: 1073.62509
    Digital Object Identifier: doi:10.1198/016214502753479356
  • Storey, J. D. (2002). A direct approach to false discovery rates. J. Roy. Statist. Soc. Ser. B 64 479–98.
    Mathematical Reviews (MathSciNet): MR1924302
    Zentralblatt MATH: 1090.62073
    Digital Object Identifier: doi:10.1111/1467-9868.00346
  • Storey, J. D., Taylor, J. E. and Siegmund, D. (2004). Strong control, conservative point estimation and simultaneous conservative consistency of false discovery rates: A unified approach. J. Roy. Statist. Soc. Ser. B 66 187–205.
    Mathematical Reviews (MathSciNet): MR2035766
    Zentralblatt MATH: 1061.62110
    Digital Object Identifier: doi:10.1111/j.1467-9868.2004.00439.x
  • Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. J. Roy. Statist. Soc. Ser. B 58 267–288.
    Mathematical Reviews (MathSciNet): MR1379242
  • Tibshirani, R. and Knight, K. (1999). The covariance inflation criterion for adaptive model selection. J. Roy. Statist. Soc. Ser. B 61 529–546.
    Mathematical Reviews (MathSciNet): MR1707859
    Zentralblatt MATH: 0924.62031
    Digital Object Identifier: doi:10.1111/1467-9868.00191
  • Wu, Y., Boos, D. D. and Stefanski, L. A. (2007). Controlling variable selection by the addition of pseudovariables. J. Amer. Statist. Assoc. 102 235–243.
    Mathematical Reviews (MathSciNet): MR2345541
    Zentralblatt MATH: 05191564
    Digital Object Identifier: doi:10.1198/016214506000000843
  • Yuan, M., Ekici, A., Lu, Z. and Monteiro, R. (2007). Dimension reduction and coefficient estimation in multivariate linear regression. J. Roy. Statist. Soc. Ser. B 69 329–346.
    Mathematical Reviews (MathSciNet): MR2323756
    Digital Object Identifier: doi:10.1111/j.1467-9868.2007.00591.x

Supplemental materials

The Institute of Mathematical Statistics

The Annals of Applied Statistics

New content alerts

Email RSS ToC RSS Article
  • You have access to this content.
  • You have partial access to this content.
  • You do not have access to this content.
Turn Off MathJax

What is MathJax?

More like this

+ See more