The Annals of Statistics

Estimating the proportion of false null hypotheses among a large number of independently tested hypotheses

Nicolai Meinshausen and John Rice

Full-text: Open access


We consider the problem of estimating the number of false null hypotheses among a very large number of independently tested hypotheses, focusing on the situation in which the proportion of false null hypotheses is very small. We propose a family of methods for establishing lower 100(1−α)% confidence bounds for this proportion, based on the empirical distribution of the p-values of the tests. Methods in this family are then compared in terms of ability to consistently estimate the proportion by letting α→0 as the number of hypothesis tests increases and the proportion decreases. This work is motivated by a signal detection problem that occurs in astronomy.

Article information

Ann. Statist., Volume 34, Number 1 (2006), 373-393.

First available in Project Euclid: 2 May 2006

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 62H15: Hypothesis testing
Secondary: 62J15: Paired and multiple comparisons 62P35: Applications to physics

Hypothesis testing multiple comparisons sparsity


Meinshausen, Nicolai; Rice, John. Estimating the proportion of false null hypotheses among a large number of independently tested hypotheses. Ann. Statist. 34 (2006), no. 1, 373--393. doi:10.1214/009053605000000741.

Export citation


  • Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. J. Roy. Statist. Soc. Ser. B 57 289–300.
  • Benjamini, Y. and Hochberg, Y. (2000). On the adaptive control of the false discovery rate in multiple hypothesis testing with independent statistics. J. Educational and Behavioral Statistics 25 60–83.
  • Chen, W., Zhang, Z., King, S., Alcock, C., Byun, Y., Cook, K., Dave, R., Giammarco, J., Lee, T., Lehner, M., Liang, C., Lissauer, J., Marshall, S., de Pater, I., Porrata, R., Rice, J., Wang, A., Wang, S. and Wen, C. (2003). Fast CCD photometry in the Taiwanese–American Occultation Survey. Baltic Astronomy 12 568–573.
  • Csörgő, M. and Horvath, L. (1993). Weighted Approximations in Probability and Statistics. Wiley, Chichester.
  • Daniels, H. (1945). The statistical theory of the strength of bundles of threads. I. Proc. Roy. Soc. London Ser. A 183 405–435.
  • Donoho, D. and Jin, J. (2004). Higher criticism for detecting sparse heterogeneous mixtures. Ann. Statist. 32 962–994.
  • Genovese, C. and Wasserman, L. (2004). A stochastic process approach to false discovery control. Ann. Statist. 32 1035–1061.
  • Liang, C.-L., Rice, J., de Pater, I., Alcock, C., Axelrod, T., Wang, A. and Marshall, S. (2004). Statistical methods for detecting stellar occultations by Kuiper Belt objects: The Taiwanese–American Occultation Survey. Statist. Sci. 19 265–274.
  • Meinshausen, N. and Bühlmann, P. (2005). Lower bounds for the number of false null hypotheses for multiple testing of associations under general dependence structures. Biometrika 92 893–907.
  • Nettleton, D. and Hwang, J. (2003). Estimating the number of false null hypotheses when conducting many tests. Technical report, Dept. Statistics, Iowa State Univ.
  • Schweder, T. and Spjøtvoll, E. (1982). Plots of $p$-values to evaluate many tests simultaneously. Biometrika 69 493–502.
  • Shorack, G. and Wellner, J. (1986). Empirical Processes with Applications to Statistics. Wiley, New York.
  • Storey, J. D. (2002). A direct approach to false discovery rates. J. R. Stat. Soc. Ser. B Stat. Methodol. 64 479–498.