Hi all:
The most blatant misuse of P-values by statisticians is to compare baseline demographic characteristics between treatments in randomized studies. You even see this in the NEJM. It confuses inference to the sample, vs. inference to the target population. No matter the P-value, the actual samples have different distributions. But no matter the P-value, the target populations are one and the same. The P-value is just a very poor metric to make the assessment of imbalanced randomization. The inference is still correct, in the sense that the Pre-experiment null probability of getting a “significant result” is as advertised . If you had an unfortunate randomization, you are talking hindsight, a legitimate concern of course, but not a validity concern. But it livens up the discussion when you do some sensitivity analysis. A minor imbalance in an important factor may have more relevance than a major imbalance in an unimportant one. If you second guess the planned inference, you risk tainting the operating characteristics, point, and interval estimates in your trial. The planned analysis must always be presented, and caveats then discussed.
You can argue these errors (testing null hypotheses you know are true) are harmless, but they encourage overuse of P-values to situations they were never intended for.
Best wishes,
Jon
Jonathan J. Shuster, Ph.D.
Professor,
Department of Health Outcomes and Policy
Director, Biostatistics Epidemiology and Research Design, Clinical and Translational Science Institute
Biostatistician, UF Clinical Research Center
College of Medicine
University of Florida
2004 Mowry Road #2247
Gainesville, FL 32610-0177
(352)-294-5968.
Mobile: (352)-682-0893
shusterj@ufl.edu
http://health-outcomes-policy.ufl.edu/faculty-directory/shuster-jonathan/
Note: This communication may contain information that is legally protected from
unauthorized disclosure. If you are not the intended recipient, please note
that any dissemination, distribution, or copying of this communication (in whole
or in part) is strictly prohibited. If you have received this e-mail in error,
please delete this message and notify the sender by return e-mail.
------------------------------
Jon Shuster
University of Florida
------------------------------
Original Message:
Sent: 07-14-2015 12:21
From: Eric Vance
Subject: Another call to ban P values from science
In addition to a self-organized lunch to discuss how to respond to calls from journals to ban p-values, section members could discuss this issue for 20-25 minutes during the Consulting Section mixer at JSM next month (Tuesday, 5:30-7:30PM, August 11, 2015 in Sheraton Jefferson Room B) and then report out to the whole group during the business part of the meeting.
My quick thoughts:
1. Yes, statisticians are partly to blame for this backlash against statistics and p-values.
2. As statistical consultants and collaborators, we are the main interfacers between scientists and statisticians. We are position to advocate and educate regarding the proper use of statistics in science.