We propose well-calibrated null preference priors for use with one-sided hypothesis tests, such that resulting Bayesian and frequentist inferences agree. Null preference priors mean that they have nearly of their prior belief in the null hypothesis, and well-calibrated priors mean that the resulting posterior beliefs in the alternative hypothesis are not overconfident. This formulation expands the class of problems giving Bayes-frequentist agreement to include problems involving discrete distributions such as binomial and negative binomial one- and two-sample exact (i.e., valid) tests. When applicable, these priors give posterior belief in the null hypothesis that is a valid p-value, and the null preference prior emphasizes that large p-values may simply represent insufficient data to overturn prior belief. This formulation gives a Bayesian interpretation of some common frequentist tests, as well as more intuitively explaining lesser known and less straightforward confidence intervals for two-sample tests.
The authors thank Dean Follmann, Jon Fintzi, Sander Greenland, as well as the Editor and three referees for helpful comments on this paper.
Ram Tiwari primarily worked on this when he was Director for the Division of Biostatistics, CDRH, Food and Drug Administration.
Michael P. Fay. Michael A. Proschan. Erica H. Brittain. Ram Tiwari. "Interpreting p-Values and Confidence Intervals Using Well-Calibrated Null Preference Priors." Statist. Sci. 37 (4) 455 - 472, November 2022. https://doi.org/10.1214/21-STS833