Open Access
February 2009 Robustness of multiple testing procedures against dependence
Sandy Clarke, Peter Hall
Ann. Statist. 37(1): 332-358 (February 2009). DOI: 10.1214/07-AOS557


An important aspect of multiple hypothesis testing is controlling the significance level, or the level of Type I error. When the test statistics are not independent it can be particularly challenging to deal with this problem, without resorting to very conservative procedures. In this paper we show that, in the context of contemporary multiple testing problems, where the number of tests is often very large, the difficulties caused by dependence are less serious than in classical cases. This is particularly true when the null distributions of test statistics are relatively light-tailed, for example, when they can be based on Normal or Student’s t approximations. There, if the test statistics can fairly be viewed as being generated by a linear process, an analysis founded on the incorrect assumption of independence is asymptotically correct as the number of hypotheses diverges. In particular, the point process representing the null distribution of the indices at which statistically significant test results occur is approximately Poisson, just as in the case of independence. The Poisson process also has the same mean as in the independence case, and of course exhibits no clustering of false discoveries. However, this result can fail if the null distributions are particularly heavy-tailed. There clusters of statistically significant results can occur, even when the null hypothesis is correct. We give an intuitive explanation for these disparate properties in light- and heavy-tailed cases, and provide rigorous theory underpinning the intuition.


Download Citation

Sandy Clarke. Peter Hall. "Robustness of multiple testing procedures against dependence." Ann. Statist. 37 (1) 332 - 358, February 2009.


Published: February 2009
First available in Project Euclid: 16 January 2009

zbMATH: 1155.62031
MathSciNet: MR2488354
Digital Object Identifier: 10.1214/07-AOS557

Primary: 62G10
Secondary: 62G35

Keywords: False-discovery rate , family-wise error rate , linear process , moving average , multiplicity , significance level , simultaneous hypothesis testing , time-series

Rights: Copyright © 2009 Institute of Mathematical Statistics

Vol.37 • No. 1 • February 2009
Back to Top