The Annals of Statistics

Multiple testing via FDRL for large-scale imaging data

Chunming Zhang, Jianqing Fan, and Tao Yu

Full-text: Open access


The multiple testing procedure plays an important role in detecting the presence of spatial signals for large-scale imaging data. Typically, the spatial signals are sparse but clustered. This paper provides empirical evidence that for a range of commonly used control levels, the conventional FDR procedure can lack the ability to detect statistical significance, even if the p-values under the true null hypotheses are independent and uniformly distributed; more generally, ignoring the neighboring information of spatially structured data will tend to diminish the detection effectiveness of the FDR procedure. This paper first introduces a scalar quantity to characterize the extent to which the “lack of identification phenomenon” (LIP) of the FDR procedure occurs. Second, we propose a new multiple comparison procedure, called FDRL, to accommodate the spatial information of neighboring p-values, via a local aggregation of p-values. Theoretical properties of the FDRL procedure are investigated under weak dependence of p-values. It is shown that the FDRL procedure alleviates the LIP of the FDR procedure, thus substantially facilitating the selection of more stringent control levels. Simulation evaluations indicate that the FDRL procedure improves the detection sensitivity of the FDR procedure with little loss in detection specificity. The computational simplicity and detection effectiveness of the FDRL procedure are illustrated through a real brain fMRI dataset.

Article information

Ann. Statist., Volume 39, Number 1 (2011), 613-642.

First available in Project Euclid: 15 February 2011

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Primary: 62H35: Image analysis 62G10: Hypothesis testing
Secondary: 62P10: Applications to biology and medical sciences 62E20: Asymptotic distribution theory

Brain fMRI false discovery rate median filtering p-value sensitivity specificity


Zhang, Chunming; Fan, Jianqing; Yu, Tao. Multiple testing via FDR L for large-scale imaging data. Ann. Statist. 39 (2011), no. 1, 613--642. doi:10.1214/10-AOS848.

Export citation


  • Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. J. Roy. Statist. Soc. Ser. B 57 289–300.
  • Benjamini, Y. and Heller, R. (2007). False discovery rates for spatial signals. J. Amer. Statist. Assoc. 102 1272–1281.
  • Benjamini, Y. and Yekutieli, D. (2001). The control of the false discovery rate in multiple testing under dependency. Ann. Statist. 29 1165–1188.
  • Casella, G. and Berger, R. L. (1990). Statistical Inference. Wadsworth and Brooks/Cole Advanced Books and Software, Pacific Grove, CA.
  • Chu, C. K., Glad, I., Godtliebsen, F. and Marron, J. S. (1998). Edge preserving smoothers for image processing (with discussion). J. Amer. Statist. Assoc. 93 526–556.
  • Cox, R. W. (1996). AFNI: Software for analysis and visualization of functional magnetic resonance neuroimages. Comput. Biomed. Res. 29 162–173.
  • Dudoit, S., Shaffer, J. P. and Boldrick, J. C. (2003). Multiple hypothesis testing in microarray experiments. Statist. Sci. 18 71–103.
  • Efron, B. (2004). Large-scale simultaneous hypothesis testing: The choice of a null hypothesis. J. Amer. Statist. Assoc. 99 96–104.
  • Fan, J., Hall, P. and Yao, Q. (2007). To how many simultaneous hypothesis tests can normal, Student’s t or bootstrap calibration be applied? J. Amer. Statist. Assoc. 102 1282–1288.
  • Genovese, C. and Wasserman, L. (2002). Operating characteristics and extensions of the false discovery rate procedure. J. R. Stat. Soc. Ser. B Stat. Methodol. 64 499–517.
  • Genovese, C. R. and Wasserman, L. (2004). A stochastic process approach to false discovery control. Ann. Statist. 32 1035–1061.
  • Genovese, C. R., Roeder, K. and Wasserman, L. (2006). False discovery control with p-value weighting. Biometrika 93 509–524.
  • Le Bihan, D., Mangin, J. F., Poupon, C., Clark, C. A., Pappata, S., Molko, N. and Chabriat, H. (2001). Diffusion tensor imaging: Concepts and applications. Journal of Magnetic Resonance Imaging 13 534–546.
  • Leek, J. T. and Storey, J. D. (2008). A general framework for multiple testing dependence. Proc. Natl. Acad. Sci. USA 105 18718–18723.
  • Lehmann, E. L. and Romano, J. P. (2005). Generalizations of the familywise error rate. Ann. Statist. 33 1138–1154.
  • Lehmann, E. L., Romano, J. P. and Shaffer, J. P. (2005). On optimality of stepdown and stepup multiple test procedures. Ann. Statist. 33 1084–1108.
  • Nichols, T. and Hayasaka, S. (2003). Controlling the familywise error rate in functional neuroimaging: A comparative review. Stat. Methods Med. Res. 12 419–446.
  • Owen, A. B. (2005). Variance of the number of false discoveries. J. R. Stat. Soc. Ser. B Stat. Methodol. 67 411–426.
  • Roweis, S. and Saul, L. (2000). Nonlinear dimensionality reduction by locally linear embedding. Science 290 2323–2326.
  • Sarkar, S. K. (2006). False discovery and false nondiscovery rates in single-step multiple testing procedures. Ann. Statist. 34 394–415.
  • Smith, S., Jenkinson, M., Woolrich, M., Beckmann, C. F., Behrens, T. E. J., Johansen-Berg, H., Bannister, P. R., De Luca, M., Drobnjak, I. Flitney, D. E., Niazy, R. K., Saunders, J., Vickers, J., Zhang, Y., De Stefano, N., Brady, J. M. and Matthews, P. M. (2004). Advances in functional and structural MR image analysis and implementation as FSL. NeuroImage 23 208–219.
  • Storey, J. D. (2002). A direct approach to false discovery rates. J. R. Stat. Soc. Ser. B Stat. Methodol. 64 479–498.
  • Storey, J. D., Taylor, J. E. and Siegmund, D. (2004). Strong control, conservative point estimation and simultaneous conservative consistency of false discovery rates: A unified approach. J. R. Stat. Soc. Ser. B Stat. Methodol. 66 187–205.
  • van der Vaart, A. W. (1998). Asymptotic Statistics. Cambridge Univ. Press, Cambridge.
  • Woolrich, M. W., Ripley, B. D., Brady, M. and Smith, S. M. (2001). Temporal autocorrelation in univariate linear modelling of FMRI data. NeuroImage 14 1370–1386.
  • Worsley, K. J., Liao, C. H., Aston, J., Petre, V., Duncan, G., Morales, F. and Evans, A. C. (2002). A general statistical analysis for fMRI data. NeuroImage 15 1–15.
  • Wu, W. B. (2008). On false discovery control under dependence. Ann. Statist. 36 364–380.
  • Zhang, C. M. and Yu, T. (2008). Semiparametric detection of significant activation for brain fMRI. Ann. Statist. 36 1693–1725.
  • Zhang, C. M., Fan, J. and Yu, T. (2010). Supplement to “Multiple testing via FDRL for large scale imaging data.” DOI: 10.1214/10-AOS848SUPP.

Supplemental materials

  • Supplementary material: Proofs and figures. Section 1 gives detailed proofs of Theorems 4.1–4.3, Section 2 gives the figure in Section 5.2, and Section 3 gives the figure in Section 5.3.