Open Access
November 2023 Online Multiple Hypothesis Testing
David S. Robertson, James M. S. Wason, Aaditya Ramdas
Author Affiliations +
Statist. Sci. 38(4): 557-575 (November 2023). DOI: 10.1214/23-STS901


Modern data analysis frequently involves large-scale hypothesis testing, which naturally gives rise to the problem of maintaining control of a suitable type I error rate, such as the false discovery rate (FDR). In many biomedical and technological applications, an additional complexity is that hypotheses are tested in an online manner, one-by-one over time. However, traditional procedures that control the FDR, such as the Benjamini–Hochberg procedure, assume that all p-values are available to be tested at a single time point. To address these challenges, a new field of methodology has developed over the past 15 years showing how to control error rates for online multiple hypothesis testing. In this framework, hypotheses arrive in a stream, and at each time point the analyst decides whether to reject the current hypothesis based both on the evidence against it, and on the previous rejection decisions. In this paper, we present a comprehensive exposition of the literature on online error rate control, with a review of key theory as well as a focus on applied examples. We also provide simulation results comparing different online testing algorithms and an up-to-date overview of the many methodological extensions that have been proposed.

Funding Statement

DS Robertson was supported by the Biometrika Trust, the UK Medical Research Council (MC_UU_0002/14) and the NIHR Cambridge Biomedical Research Centre (BRC1215-20014). The views expressed in this publication are those of the authors and not necessarily those of the NHS, the National Institute for Health Research or the Department of Health and Social Care (DHCS). For the purpose of open access, the author has applied a Creative Commons Attribution (CC BY) licence to any Author Accepted Manuscript version arising.
JMS Wason was supported by a NIHR Research Professorship (NIHR301614).
A Ramdas was supported by NSF DMS CAREER award 1916320.


Download Citation

David S. Robertson. James M. S. Wason. Aaditya Ramdas. "Online Multiple Hypothesis Testing." Statist. Sci. 38 (4) 557 - 575, November 2023.


Published: November 2023
First available in Project Euclid: 6 November 2023

Digital Object Identifier: 10.1214/23-STS901

Keywords: A/B testing , data repositories , platform trials , type I error rate

Rights: This research was funded, in whole or in part, by [UK Medical Research Council, MC_UU_00002/14]. A CC BY 4.0 license is applied to this article arising from this submission, in accordance with the grant's open access conditions

Vol.38 • No. 4 • November 2023
Back to Top