The Annals of Statistics

Online rules for control of false discovery rate and false discovery exceedance

Adel Javanmard and Andrea Montanari

Full-text: Access denied (no subscription detected)

We're sorry, but we are unable to provide you with the full text of this article because we are not able to identify you as a subscriber. If you have a personal subscription to this journal, then please login. If you are already logged in, then you may need to update your profile to register your subscription. Read more about accessing full-text


Multiple hypothesis testing is a core problem in statistical inference and arises in almost every scientific field. Given a set of null hypotheses $\mathcal{H}(n)=(H_{1},\ldots,H_{n})$, Benjamini and Hochberg [J. R. Stat. Soc. Ser. B. Stat. Methodol. 57 (1995) 289–300] introduced the false discovery rate ($\mathrm{FDR}$), which is the expected proportion of false positives among rejected null hypotheses, and proposed a testing procedure that controls $\mathrm{FDR}$ below a pre-assigned significance level. Nowadays $\mathrm{FDR}$ is the criterion of choice for large-scale multiple hypothesis testing.

In this paper we consider the problem of controlling $\mathrm{FDR}$ in an online manner. Concretely, we consider an ordered—possibly infinite—sequence of null hypotheses $\mathcal{H}=(H_{1},H_{2},H_{3},\ldots)$ where, at each step $i$, the statistician must decide whether to reject hypothesis $H_{i}$ having access only to the previous decisions. This model was introduced by Foster and Stine [J. R. Stat. Soc. Ser. B. Stat. Methodol. 70 (2008) 429–444].

We study a class of generalized alpha investing procedures, first introduced by Aharoni and Rosset [J. R. Stat. Soc. Ser. B. Stat. Methodol. 76 (2014) 771–794]. We prove that any rule in this class controls online $\mathrm{FDR}$, provided $p$-values corresponding to true nulls are independent from the other $p$-values. Earlier work only established $\mathrm{mFDR}$ control. Next, we obtain conditions under which generalized alpha investing controls $\mathrm{FDR}$ in the presence of general $p$-values dependencies. We also develop a modified set of procedures that allow to control the false discovery exceedance (the tail of the proportion of false discoveries). Finally, we evaluate the performance of online procedures on both synthetic and real data, comparing them with offline approaches, such as adaptive Benjamini–Hochberg.

Article information

Ann. Statist., Volume 46, Number 2 (2018), 526-554.

Received: March 2016
Revised: January 2017
First available in Project Euclid: 3 April 2018

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 62F03: Hypothesis testing 62F05: Asymptotic properties of tests
Secondary: 62L99: None of the above, but in this section

Hypothesis testing false discovery rate (FDR) false discovery exceedance (FDX) online decision making


Javanmard, Adel; Montanari, Andrea. Online rules for control of false discovery rate and false discovery exceedance. Ann. Statist. 46 (2018), no. 2, 526--554. doi:10.1214/17-AOS1559.

Export citation


  • [1] Aharoni, E. and Rosset, S. (2014). Generalized $\alpha$-investing: Definitions, optimality results and application to public databases. J. R. Stat. Soc. Ser. B. Stat. Methodol. 76 771–794.
  • [2] Barber, R. F. and Candès, E. J. (2015). Controlling the false discovery rate via knockoffs. Ann. Statist. 43 2055–2085.
  • [3] Barber, R. F. and Candes, E. J. (2016). A knockoff filter for high-dimensional selective inference. Available at arXiv:1602.03574.
  • [4] Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. J. R. Stat. Soc. Ser. B. Stat. Methodol. 57 289–300.
  • [5] Benjamini, Y. and Yekutieli, D. (2001). The control of the false discovery rate in multiple testing under dependency. Ann. Statist. 29 1165–1188.
  • [6] Blanchard, G. and Roquain, É. (2009). Adaptive false discovery rate control under independence and dependence. J. Mach. Learn. Res. 10 2837–2871.
  • [7] Bogdan, M., van den Berg, E., Sabatti, C., Su, W. and Candès, E. J. (2015). SLOPE—Adaptive variable selection via convex optimization. Ann. Appl. Stat. 9 1103–1140.
  • [8] Donoho, D. L. and Johnstone, I. M. (1994). Minimax risk over $l_{p}$-balls for $l_{q}$-error. Probab. Theory Related Fields 99 277–303.
  • [9] Dwork, C., Feldman, V., Hardt, M., Pitassi, T., Reingold, O. and Roth, A. (2015). Preserving statistical validity in adaptive data analysis [extended abstract]. In STOC’15—Proceedings of the 2015 ACM Symposium on Theory of Computing 117–126. ACM, New York.
  • [10] Fithian, W., Sun, D. and Taylor, J. (2014). Optimal inference after model selection. Available at arXiv:1410.2597.
  • [11] Foster, D. P. and Stine, R. A. (2008). $\alpha$-investing: A procedure for sequential control of expected false discoveries. J. R. Stat. Soc. Ser. B. Stat. Methodol. 70 429–444.
  • [12] Genovese, C. R., Lazar, N. A. and Nichols, T. (2002). Thresholding of statistical maps in functional neuroimaging using the false discovery rate. Neuroimage 15 870–878.
  • [13] Genovese, C. R., Roeder, K. and Wasserman, L. (2006). False discovery control with $p$-value weighting. Biometrika 93 509–524.
  • [14] Genovese, C. R. and Wasserman, L. (2006). Exceedance control of the false discovery proportion. J. Amer. Statist. Assoc. 101 1408–1417.
  • [15] G’Sell, M. G., Wager, S., Chouldechova, A. and Tibshirani, R. (2016). Sequential selection procedures and false discovery rate control. J. R. Stat. Soc. Ser. B. Stat. Methodol. 78 423–444.
  • [16] Ioannidis, J. P. (2005). Contradicted and initially stronger effects in highly cited clinical research. Jornal of the American Medical Association 294 218–228.
  • [17] Ioannidis, J. P. A. (2005). Why most published research findings are false. Chance 18 40–47.
  • [18] Javanmard, A. and Montanari, A. (2013). Nearly optimal sample size in hypothesis testing for high-dimensional regression. In 51st Annual Allerton Conference 1427–1434, Monticello, IL.
  • [19] Javanmard, A. and Montanari, A. (2014). Confidence intervals and hypothesis testing for high-dimensional regression. J. Mach. Learn. Res. 15 2869–2909.
  • [20] Javanmard, A. and Montanari, A. (2014). Hypothesis testing in high-dimensional regression under the Gaussian random design model: Asymptotic theory. IEEE Trans. Inform. Theory 60 6522–6554.
  • [21] Javanmard, A. and Montanari, A. (2015). On online control of false discovery rate. Available at arXiv:1502.06197.
  • [22] Javanmard, A. and Montanari, A. (2018). Supplement to “Online rules for control of false discovery rate and false discovery exceedance.” DOI:10.1214/17-AOS1559SUPP.
  • [23] Jin, J. (2008). Proportion of non-zero normal means: Universal oracle equivalences and uniformly consistent estimators. J. R. Stat. Soc. Ser. B. Stat. Methodol. 70 461–493.
  • [24] Jin, J. and Cai, T. T. (2007). Estimating the null and the proportional of nonnull effects in large-scale multiple comparisons. J. Amer. Statist. Assoc. 102 495–506.
  • [25] Johnstone, I. M. (1994). On minimax estimation of a sparse normal mean vector. Ann. Statist. 22 271–289.
  • [26] Lehmann, E. L. and Romano, J. P. (2012). Generalizations of the Familywise Error Rate. Springer, Berlin.
  • [27] Li, A. and Barber, R. F. (2016). Accumulation tests for FDR control in ordered hypothesis testing. J. Amer. Statist. Assoc. 112 1–38.
  • [28] Lin, D., Foster, D. P. and Ungar, L. H. (2011). VIF regression: A fast regression algorithm for large data. J. Amer. Statist. Assoc. 106 232–247.
  • [29] Lockhart, R., Taylor, J., Tibshirani, R. J. and Tibshirani, R. (2014). A significance test for the lasso. Ann. Statist. 42 413–468.
  • [30] Meinshausen, N. and Rice, J. (2006). Estimating the proportion of false null hypotheses among a large number of independently tested hypotheses. Ann. Statist. 34 373–393.
  • [31] Owen, A. B. (2005). Variance of the number of false discoveries. J. R. Stat. Soc. Ser. B. Stat. Methodol. 67 411–426.
  • [32] Pekelis, L., Walsh, D. and Johari, R. (2015). The new stats engine. Available at
  • [33] Prinz, F., Schlange, T. and Asadullah, K. (2011). Believe it or not: How much can we rely on published data on potential drug targets? Nature Reviews Drug Discovery 10 712–712.
  • [34] Reiner, A., Yekutieli, D. and Benjamini, Y. (2003). Identifying differentially expressed genes using false discovery rate controlling procedures. Bioinformatics 19 368–375.
  • [35] Rosset, S., Aharoni, E. and Neuvirth, H. (2014). Novel statistical tools for management of public databases facilitate community-wide replicability and control of false discovery. Genetic Epidemiology 38 477–481.
  • [36] Storey, J. D. (2002). A direct approach to false discovery rates. J. R. Stat. Soc. Ser. B. Stat. Methodol. 64 479–498.
  • [37] Storey, J. D. and Tibshirani, R. (2003). SAM thresholding and false discovery rates for detecting differential gene expression in DNA microarrays. In The Analysis of Gene Expression Data 272–290. Springer, New York.
  • [38] Tibshirani, R. J., Taylor, J., Lockhart, R. and Tibshirani, R. (2016). Exact post-selection inference for sequential regression procedures. J. Amer. Statist. Assoc. 111 600–620.
  • [39] van de Geer, S., Bühlmann, P., Ritov, Y. and Dezeure, R. (2014). On asymptotically optimal confidence regions and tests for high-dimensional models. Ann. Statist. 42 1166–1202.
  • [40] van der Laan, M. J., Dudoit, S. and Pollard, K. S. (2004). Augmentation procedures for control of the generalized family-wise error rate and tail probabilities for the proportion of false positives. Stat. Appl. Genet. Mol. Biol. 3 Art. 15, 27.
  • [41] Zhang, C.-H. and Zhang, S. S. (2014). Confidence intervals for low dimensional parameters in high dimensional linear models. J. R. Stat. Soc. Ser. B. Stat. Methodol. 76 217–242.

Supplemental materials

  • Online rules for control of false discovery rate and false discovery exceedance. Due to space constraints, proof of theorems and some of the technical details as well as additional numerical studies are provided in the Supplementary Material [22].