Discussion: View Thread

I think, as I have written before on this list serve, the biggest problem with the p-value is how often it rejects the null hypothesis in a given number of independent trials.  The number of trials until just before an event occurs, under a constant probability of the event occurring, has a geometric distribution - which is highly skewed.  For alpha equal to 0.05, on average, 64% of the time you will reject the null in at least one of any set of 20 independent trials where the null is true.  For a symmetric distribution, the value would be 50%.  I think we tend to think in terms of symmetric rather than skewed distributions and that that is why we struggle so much with the p-value.

Another perspective, which I believe is supported by simulations I've tried, is that p-values, per se, may not be the problem.  Rather, it's that  the textbook hypothesis testing model does not allow for any "thickness" of the null hypothesis.  Is it that surprising (or informative) when, especially for large samples, the data appears (by p-values) to reject that the true parameter is exactly equal to the point-estimate null hypothesis? 

Interestingly, we don't assume that the null is an exact value when we run power calculations to select sample sizes.  In that context, we have to specify a minimum distance from the exact null parameter that we feel is meaningful (practically or clinically, etc.) to detect.  This implies that when the sample result's distance from the null's center is smaller than that distance, the null is being considered "true enough".

As often mentioned against p-values, the probability for or against the sample data under the null hypothesis is not logically the same as probability for or against the null hypothesis given the sample data.  However, there nonetheless a case for a "strong monotonic relationship between the order of magnitude of the p-value and the relative likelihood that the null hypothesis is true".  (Hence the p-value's appeal.)   The trouble is that the variance around that just-mentioned relationship can be huge. (Which is the problem of p-value's).

Here's a rule of thumb that I've encouraged if using a  p-value algorithm to decide whether or not to reject H0:  Presume that (all else being equal):

a) For thick H0’s: (the effective α) > (nominal α);

b) For very thin H0’s: (the effective α) < (nominal α)

This follows because in (a), p-values lead to rejecting too many H0’s that are true (in the "thick" sense) , so the risks of Type I error (i.e. α) are greater than acknowledged; and in (b) p-values fail to reject virtually any H0’s (whether true or false), so the risks of rejecting true H0’s (i.e. α) is less than anticipated "

------------------------------
William Goodman
University of Ontario Institute of Technology
------------------------------