Critique of p-Values

Abstract

This paper generalizes the notion of p-value to obtain a system for assessing evidence in favor of an hypothesis. It is not quite a quantification in that evidence is a pair of numbers (the p-value and the p-value with null and alternative interchanged) with evidence for the alternative being claimed when the first number is small and the second is at least moderately large. Traditional significance tests present p-values as a measure of evidence {against} a theory. This usage is rarely called for since scientists usually wish to accept theories (for the time being) not just not reject them; they are more interested in evidence {for} a theory. P-values are not just good or bad for this purpose; their efficacy depends on specifics. We find that a single p-value does not measure evidence for a simple hypothesis relative to a simple alternative, but consideration of both p-values leads to a satisfactory theory. This consideration does not, in general, extend to composite hypotheses since there, best evidence calls for optimization of a bivariate objective function. But in some cases, notably one sided tests for the exponential family, the optimization can be solved, and a single p-value does provide an appealing measure of best evidence for a hypothesis. One possible extension of this theory is proposed and illustrated with a practical safety analysis problem involving the difference of two random variables.