Open Access
2021 Practical valid inferences for the two-sample binomial problem
Michael P. Fay, Sally A. Hunsberger
Author Affiliations +
Statist. Surv. 15: 72-110 (2021). DOI: 10.1214/21-SS131


Our interest is whether two binomial parameters differ, which parameter is larger, and by how much. This apparently simple problem was addressed by Fisher in the 1930’s, and has been the subject of many review papers since then. Yet there continues to be new work on this issue and no consensus solution. Previous reviews have focused primarily on testing and the properties of validity and power, or primarily on confidence intervals, their coverage, and expected length. Here we evaluate both. For example, we consider whether a p-value and its matching confidence interval are compatible, meaning that the p-value rejects at level α if and only if the 1α confidence interval excludes all null parameter values. For focus, we only examine non-asymptotic inferences, so that most of the p-values and confidence intervals are valid (i.e., exact) by construction. Within this focus, we review different methods emphasizing many of the properties and interpretational aspects we desire from applied frequentist inference: validity, accuracy, good power, equivariance, compatibility, coherence, and parameterization and direction of effect. We show that no one method can meet all the desirable properties and give recommendations based on which properties are given more importance.


The authors thank Erica Brittain and anonymous reviewers for comments that improved the paper.


Download Citation

Michael P. Fay. Sally A. Hunsberger. "Practical valid inferences for the two-sample binomial problem." Statist. Surv. 15 72 - 110, 2021.


Received: 1 June 2020; Published: 2021
First available in Project Euclid: 16 April 2021

arXiv: 1904.05416
Digital Object Identifier: 10.1214/21-SS131

Primary: 62F03
Secondary: 62F25

Keywords: 2 by 2 table , Barnard’s test , Fisher’s exact test , unconditional exact test

Vol.15 • 2021
Back to Top