ASA Connect

Check out the response from 60+ statisticians on p-value issue.  https://www.nature.com/articles/s41562-017-0189-z This is published in Nature Human Behavior.  They recommend using p-value 0.005 instead of 0.05.  They make very interesting argument.

Rahul A. Parsa

Iowa State University

Relevant to Dr. Greenland's comment re p value cutoffs, I just wanted to note that results-blind science publishing would end the use of any cutoffs for p values as a criterion for publication.  

If interested, see the references below (also attached) & the abstract below:

Locascio, J. J.  Results blind science publishing.  Basic and Applied Social Psychology, 2017, Vol 39(5), 239-246.      

Locascio, J. J.  Rejoinder to responses to 'Results blind publishing'.  Basic and Applied Social Psychology, 2017, Vol 39(5), 258-261. 

Abstract for "Results Blind Science Publishing":

Problems in science publishing involving publication bias, null hypothesis significance testing (NHST), and irreproducibility of reported results have been widely cited.  Numerous attempts to ameliorate these problems have included statistical methods to assess and correct for publication bias, and recommendation or development of statistical methodologies to replace NHST where some journals have even instituted a policy of banning manuscripts reporting use of NHST.  In an effort to mitigate these problems, a policy of "results blind evaluation" of manuscripts submitted to journals is recommended, in which results reported in manuscripts are given no weight in the decision as to the suitability of the manuscript for publication. Weight would be given exclusively to (a) the judged importance of the research question addressed in the study, typically conveyed in the Introduction section of the manuscript, and (b) the quality of the methodology of the study, including appropriateness of data analysis methods, as reported in the Methods section.  As a practical method of implementing such a policy, a two stage process is suggested whereby the editor initially distributes only the Introduction and Methods sections of a submitted manuscript to reviewers for evaluation and a provisional decision regarding acceptance or rejection for publication is made.  A second stage of review follows in which the complete manuscript is distributed for review but only if the decision of the first stage is for acceptance with no more than minor revision.  

Joseph J. Locascio, Ph.D.,

Assistant Professor of Neurology,

Harvard Medical School;  

Bio-Statistician, Neurology Dept.,  
Memory and Movement Disorders Units, 
Massachusetts Alzheimer's Disease Research Center,
Massachusetts General Hospital (MGH);   

Collaborating Statistician, Neurology Dept.,

Brigham and Women's Hospital, 
Boston, Massachusetts;   
Phone: (617) 724-7192
Email: JLocascio@partners.org 

Frustrating, I know.  I am a proponent of using p-values under the proper circumstances (e.g., planned hypothesis tests).  Here's a way I usually get the "reviewers" to back off.  I ask them which tests they want p-values for, then I count how many there will be, then I tell them the critical p-value will not be 0.05, but rather the Bonferroni corrected p-value (divided by the number of tests performed).  It is an over correction, in many cases, but it brings home the concept that p-values are a measure of the type 1 error, typically something we'd like to keep low, hence the correction.  When the majority of p=0.049s go away and only a few p=0.00001 remain as "significant" perhaps they will better understand that concept.

Good luck,

Susan E. Spruill

Susan E. Spruill, PStat®

Statistical Consultant, President

Applied Statistics and Consulting

828-467-9184 (phone)

Professional Statistician accredited by the American Statistical Association

www.appstatsconsulting.com

Jake,

This is likely an experience that many of us have had, and will continue to have for some time.

Without knowing the specifics of your situation, I can easily imagine what you've described, as I have been in the position myself.  You are certainly correct that p-values are often reported out of habit and without real understanding of what they mean and whether they have any value in a particular situation; you are more than likely correct that the p-values will add little meaningful information to the analysis that you have described, other than satisfying your collaborators who believe that p-values are essential for publication.

As a field, we must continue to work at the problem that many of our clients/collaborators think that p-values are always necessary, and we will.  In the meantime, there are stark realities for some of us in relatively junior positions who are trying to start our career and fearful that we may be branded uncooperative or incompetent should we refuse to provide something for a client/collaborator, even if we know the analytic approach is suboptimal.  My classic approach has been to provide what has been asked with a clearly written explanation of precisely what it does mean (and also what it doesn't mean), in the hopes that they may be persuaded or at least consider what I've written.  I don't believe that it often makes an impression, but I feel better knowing that I've at least expressed my opinion to them while also meeting the obligation I feel to be a cooperative collaborator.

Some others, more accomplished and more secure in their positions, may be able to take a harder stance.

Thanks,

Andrew D. Althouse, PhD

Supervisor of Statistical Projects

UPMC Heart & Vascular Institute

Presbyterian Hospital, Office C701

Phone: 412-802-6811

Email: althousead@upmc.edu

Twitter: @ADAlthousePhD

Website: Faculty Profile

Website: HVI-CBC Profile