June 2022 Large-scale inference with block structure
Jiyao Kou, Guenther Walther
Author Affiliations +
Ann. Statist. 50(3): 1541-1572 (June 2022). DOI: 10.1214/21-AOS2162


The detection of weak and rare effects in large amounts of data arises in a number of modern data analysis problems. Known results show that in this situation the potential of statistical inference is severely limited by the large-scale multiple testing that is inherent in these problems. Here, we show that fundamentally more powerful statistical inference is possible when there is some structure in the signal that can be exploited, for example, if the signal is clustered in many small blocks, as is the case in some relevant applications. We derive the detection boundary in such a situation where we allow both the number of blocks and the block length to grow polynomially with sample size. We derive these results both for the univariate and the multivariate settings as well as for the problem of detecting clusters in a network. These results recover as special cases the sparse signal detection problem (Ann. Statist. 32 (2004) 962–994) where there is no structure in the signal, as well as the scan problem (Statist. Sinica 23 (2013) 409–428) where the signal comprises a single interval. We develop methodology that allows optimal adaptive detection in the general setting, thus exploiting the structure if it is present without incurring a relevant penalty in the case where there is no structure. The advantage of this methodology can be considerable, as in the case of no structure the means need to increase at the rate logn to ensure detection, while the presence of structure allows detection even if the means decrease at a polynomial rate.

Funding Statement

The authors were supported by NSF Grants DMS-1220311 and DMS-1501767.


Download Citation

Jiyao Kou. Guenther Walther. "Large-scale inference with block structure." Ann. Statist. 50 (3) 1541 - 1572, June 2022. https://doi.org/10.1214/21-AOS2162


Received: 1 June 2019; Revised: 1 December 2021; Published: June 2022
First available in Project Euclid: 16 June 2022

MathSciNet: MR4441131
zbMATH: 07547941
Digital Object Identifier: 10.1214/21-AOS2162

Primary: 62G10 , 62G32
Secondary: 62H15

Keywords: block structure , heterogeneous mixture detection , sparse signal detection , structured Berk–Jones statistic , structured higher criticism , structured ϕ-divergence , tail bound for higher criticism statistic and Berk–Jones statistic , tail bound for supremum of binomial log likelihood ratio process , tail bound for supremum of standardized Brownian bridge

Rights: Copyright © 2022 Institute of Mathematical Statistics


This article is only available to subscribers.
It is not available for individual sale.

Vol.50 • No. 3 • June 2022
Back to Top