Open Access
June 2020 Post hoc confidence bounds on false positives using reference families
Gilles Blanchard, Pierre Neuvial, Etienne Roquain
Ann. Statist. 48(3): 1281-1303 (June 2020). DOI: 10.1214/19-AOS1847

Abstract

We follow a post hoc, “user-agnostic” approach to false discovery control in a large-scale multiple testing framework, as introduced by Genovese and Wasserman [J. Amer. Statist. Assoc. 101 (2006) 1408–1417], Goeman and Solari [Statist. Sci. 26 (2011) 584–597]: the statistical guarantee on the number of correct rejections must hold for any set of candidate items, possibly selected by the user after having seen the data. To this end, we introduce a novel point of view based on a family of reference rejection sets and a suitable criterion, namely the joint familywise error rate over that family (JER for short). First, we establish how to derive post hoc bounds from a given JER control and analyze some general properties of this approach. We then develop procedures for controlling the JER in the case where reference regions are $p$-value level sets. These procedures adapt to dependencies and to the unknown quantity of signal (via a step-down principle). We also show interesting connections to confidence envelopes of Meinshausen [Scand. J. Stat. 33 (2006) 227–237]; Genovese and Wasserman [J. Amer. Statist. Assoc. 101 (2006) 1408–1417], the closed testing based approach of Goeman and Solari [Statist. Sci. 26 (2011) 584–597] and to the higher criticism of Donoho and Jin [Ann. Statist. 32 (2004) 962–994]. Our theoretical statements are supported by numerical experiments.

Citation

Download Citation

Gilles Blanchard. Pierre Neuvial. Etienne Roquain. "Post hoc confidence bounds on false positives using reference families." Ann. Statist. 48 (3) 1281 - 1303, June 2020. https://doi.org/10.1214/19-AOS1847

Information

Received: 1 March 2017; Revised: 1 February 2019; Published: June 2020
First available in Project Euclid: 17 July 2020

zbMATH: 07241591
MathSciNet: MR4124323
Digital Object Identifier: 10.1214/19-AOS1847

Subjects:
Primary: 62G10
Secondary: 62H15

Keywords: Dependence , family-wise error rate , higher criticism , multiple testing , Post hoc inference , Simes inequality , step-down algorithm

Rights: Copyright © 2020 Institute of Mathematical Statistics

Vol.48 • No. 3 • June 2020
Back to Top