This'll raise some people's blood pressures.

Tom Siegfried, the managing editor of Science News, argues that the traditional reliance on certain statistical tests in the life sciences contributes to the irreproducibility of many research findings. Science News, 7/10/2015.

URL: https://www.sciencenews.org/blog/context/top-10-ways-save-science-its-statistical-self

Top 10 ways to save science from its statistical self

------------------------------
Eric Siegel
Biostatistician
Univ of Arkansas for Medical Sciences of Biostatistics
------------------------------

Great... While I understand the fundamental intent, it's advocating a solution to the wrong problem: it's not the tool; it's how it's used. p-value isn't inherently wrong, but in the wrong hands and without the right communication to the audience it can be dangerous. Sure, there might be better way to approach a certain situation, but banning it isn't going to prevent another tool from being misused. Banning kitchen knives isn't going to prevent murderers from using other cutting instruments.

There are several other points I take issues with this piece (as well as some points I actually agree with, just to be clear), and I could totally go on my soapbox here, but I guess I'll shut up now....

------------------------------
Michiko Wolcott
Managing Partner and Principal Consultant
Msight Analytics
------------------------------

A lot of this type of stuff is targetting final analysis without talking about design or appropriate models for a particular design, or even how well a model fits the data. This is partly a result of "cookie cutter" analyses proomoted by menu driven software thats widely available and sometimes abused without an understanding the underlying methods.

The p-value is not inherently wrong but often inappropriately used and abused

So True!  I wonder what this backlash against p-values will do for us as members of the scientific team?  Will we now be viewed as hand-wavers who are attempting to lead scientists astray?  Or will we be viewed as valuable members who can shed light on the results and point the team to better methods, better designs and/or alternative approaches?

This backlash against p-values is OUR fault.  As statisticians we have failed to educate and enforce the proper use of our tools.  This is a good example of why expert systems (tools that allow non-statisticians to get statistical results) will not work.  You can not teach someone how to drive a car and then turn them loose on the road with knowledge of how to interpret the road signs, traffic signals or right-of-ways.

As opposed to banning p-values, scientific journals should require statistical review of all manuscripts for soundness regarding hypotheses, design methods, statistical methodology and interpretation of results.

I have been advocating for more statistical details in scientific journals.  Results will not be reproducible if the information needed to replicate a study are missing.

------------------------------
Susan Spruill
Statistical Consultant
------------------------------

Just another example on the lack of understanding of statistical concepts.

Confidence intervals, are frequently proposed as the "solution" over p-values. A confidence interval contains

     all hypotheses that cannot be rejected at a specific level of significance given the data.  

Is that better than

     the smallest level of significance at which the null hypothesis can be rejected given the data?

To understand either, you need to understand the same concepts. It is the lack of understanding of these concepts that's causing the problem, not focus on different applications of them.

Knut
------------------------------
Knut M. Wittkowski
Head, Dept. Biostatistics, Epidemiology, and Research Design
The Rockefeller University
------------------------------

I'm with Susan on this being on our shoulders. While we can't eradicate the problem--there will always be rogue people--we as a community can and should do more to raise the statistical maturity of the general population. The situation is only going to be exasperated with the proliferation of "self-service" analytics (the BI world is already seeing some effects). So, what can we as both holders of statistical truths and good consultants do? Each of us are in different flavors of consulting--I, for one, am in management consulting doing org & business process design for analytics and analytical capability building work, but often run into the scenario of organizations as a culture putting technology ahead of people, so it essentially requires a cultural change which is not always very easy to accomplish yet easy to screw up.

Would have been a good JSM roundtable topic in hindsight....

------------------------------
Michiko Wolcott
Principal Consultant
Msight Analytics
------------------------------

A round-table is a good idea. Who is available on Monday? (Must be a place somewhere to have lunch together)

Knut

I'm up for it....

------------------------------
Michiko Wolcott
Principal Consultant
Msight Analytics
------------------------------

If a few of you are interested in getting together per this thread, I'd be happy to set something up for 12:30 Monday of JSM.  At this point it isn't easy to schedule a large space, but I can find room for 8-10 people or so.

If you are interested, please email me at ron@amstat.org.  Please do not "reply all" to this message to avoid clogging up this people's inboxes.

Thank you.

Ron
------------------------------
Ron Wasserstein
Executive Director
The American Statistical Association
Promoting the Practice and Profession of Statistics
732 N. Washington St.
Alexandria, VA 22314
703-684-1221 x1860
------------------------------

oh WOW!  I am so happy to see that a round table will be pulled together for this!  I wish I could be there.  Here are a couple of things I am thinking about:

1) where can we start this discussion with non-statisticians (publishers, journal reviewers, scientific meetings, etc?)  We have already had this conversation in the Significance Magazine...but honestly it does not reach enough people.  Publishing a paper in a stats journal will not reach the right audience.

2) what can we, as statisticians, do to educate people on the nuances of different methods (thinking about what was said about the similarities in CIs and p-values)?

3) How can we direct scientists to consider the WHOLE picture when publishing work and not just the "significant" stuff?  This might need to be directed at the journals?

I'm interested to hear how the RT goes, so please keep me posted and let me know if there are things I can do from this end.

------------------------------
Susan Spruill
Statistical Consultant
------------------------------

David, I, as well as Ron Wasserstein, are on the Significance editorial board.  We were acutely aware of the issue when Psychology dumped the p-value.  While Significance's  attention to it is great, this magazine does not reach a wide enough audience.  We need to reach out to the scientific community and appeal to their desire for clarity on the subject of determining statistical significance of their findings.  I like were this thread is going.

Let's keep talking!

------------------------------
Susan Spruill
Statistical Consultant
------------------------------

Hi all:

The most blatant misuse of P-values by statisticians is to compare baseline demographic characteristics between treatments in randomized studies.   You even see this in the NEJM.  It confuses inference to the sample, vs. inference to the target population.  No matter the P-value, the actual samples have different distributions.  But no matter the P-value, the target populations are one and the same.   The P-value is just a very poor metric to make the assessment of imbalanced randomization.  The inference is still correct, in the sense that the Pre-experiment null probability of getting a “significant  result” is as advertised .  If you had an unfortunate randomization, you are talking hindsight, a legitimate concern of course, but not a validity concern.  But it livens up the discussion when you do some sensitivity analysis.  A minor imbalance in an important factor may have more relevance than a major imbalance in an unimportant one.   If you second guess the planned inference, you risk tainting the operating characteristics, point, and interval estimates in your trial.  The planned analysis must always be presented, and caveats then discussed.

You can argue these errors (testing null hypotheses you know are true) are harmless, but they encourage overuse of P-values to situations they were never intended for.

Best wishes,

Jon

Jonathan J. Shuster, Ph.D.

Professor, 

Department of Health Outcomes and Policy

Director, Biostatistics Epidemiology and Research Design, Clinical and Translational Science Institute

Biostatistician, UF Clinical Research Center

College of Medicine

University of Florida

2004 Mowry Road #2247

Gainesville, FL 32610-0177

(352)-294-5968. 

Mobile: (352)-682-0893

shusterj@ufl.edu

http://health-outcomes-policy.ufl.edu/faculty-directory/shuster-jonathan/

Note: This communication may contain information that is legally protected from

unauthorized disclosure.  If you are not the intended recipient, please note

that any dissemination, distribution, or copying of this communication (in whole

or in part) is strictly prohibited.  If you have received this e-mail in error,

please delete this message and notify the sender by return e-mail.

------------------------------
Jon Shuster
University of Florida
------------------------------

This is like the gun control argument: If p-values are outlawed (for statisticians), only the outlaws (non-statisticians) will have p-values!

------------------------------
Martin Lesser, PhD
Biostatistics Unit
Feinstein Institute for Medical Research

------------------------------

And they will generate all their p-values with Excel, too!

------------------------------
Paul Thompson
Director, Methodology and Data Analysis Center
Sanford Research/USD
------------------------------

Completely agree with Dr. Lesser! This statement should be quoted and preserved!

------------------------------
Shahidul Islam, MPH, PStat
Biostatistician
Winthrop University Hospital
Mineola, NY 11590
------------------------------

Funny Martin!  Can we get t-shirts and bumper stickers with that quote?

------------------------------
Susan Spruill
Statistical Consultant
------------------------------

I would definitely buy that!

------------------------------
Rose Calixte
Biostatistician
Winthrop University Hospital
------------------------------

I don't see anything wrong with p-values in isolation. My issue is: what p-value constitutes significance?

Let me propose 2 common scenarios. 1) We want to determine whether there is a non-trivial linear relation between y and x. 2) We want to determine whether there is a significant difference between the means of populations P1 and P2.

There are well-known tests to answer these questions. Assume that all the conditions for the tests' validity are met. Then under classical hypothesis testing, we are more likely to answer questions 1 and 2 affirmatively as the sample size grows larger, because we have sanctified the number .05.

My response to my original question - what p-value constitutes significance - is a moving target. The critical value should decrease as n increases, in my opinion.

------------------------------
Hal Switkay
------------------------------

Back to the article by Tom Siegfried, there are a lot of things where I agree with Gerd Gigeritzer. In the paper Tom referred to, Gerd and Julian state:

"There is no universal method of scientific inference but, rather, a toolbox of useful statistical methods." 

Maybe we can eventually agree that neither p-values nor confidence intervals (nor any one other approach) are to be used universally, but that both have their value in the situation they were developed for.

• When our aim is to assess the magnitude of an effect, confidence intervals are highly appropriate. We want to know the lower bound of the hypotheses that are consistent with the data. 
• When our aim is to decide which of two options to choose, the Neyman-Pearson approach provides us with a strategy that is optimal in the long ran. 
• When we want to generate hypotheses about genetic risk factors, it makes perfect sense to prioritize regions with small p-values, but abusing concepts from confirmatory testing in an exploratory setting, rather than developing statistical methods and decision strategies appropriate for the task has arguably contributed a lot to 10 years of missed opportunities.

Maybe it could help to identify areas where p-values are useful and those where they are not. This could potentially avoid the pendulum of methods from swinging from one extreme (ignoring all results that don't reach a fixed genome-wide significance of 10E-7.5) to the other (preventing p-values from being published).

Knut

I like the guidelines approach too.  But we need to take care that we are not exacerbating the problem with a simple decision-tree like approach that appears to eliminate the need to consult statisticians in the planning, implementing, analysis and publishing of results.

Regarding your last comment on statisticians coming out of "related areas"; I fall into that category.  My educational background is genetics and reproductive physiology.  However, I spent many years honing my statistical skills.  I find that my biological background helps me talk about statistics with my scientific colleagues.  I don't purport to know all there is to know about statistics or the nuances of all the methodologies available, but I can direct them away from "significant" p-values when all other data suggests that we may be bumping up against type I error as opposed to a real finding.  It is a tough job, but usually the realities of scientific probabilities (not statistical) can persuade them to rethink their findings.  We need pure science (e.g. understanding of biological pathways) and we need pure statistics (e.g. mathematical and theoretical concepts).  But there is a huge chasm in between where the majority of applied statisticians and scientists fall.  In this space we need to collaborate.

OK, I'm getting off my soapbox now.

------------------------------
Susan Spruill
Statistical Consultant
------------------------------

Great summary Susan.  I am also an applied statistician and work in that shred space of science and statistics primarily in clinical trials.  It takes patience and persistence to become a trusted colleague of a scientist/physician, but the journey is worth it.  They actually think about planning with you and begin to understand that there is more to statistics than software - the source of the pesky p-values. 

------------------------------
Janet McDougall
President
McDougall Scientific Ltd.
------------------------------

Susan,

My thoughts exactly. Guidelines are a good idea, but for some people they can establish a false sense of security and confidence (pun fully intended). I think the key is to educate people to recognize their limitations and know when to ask for proper help and guidance.

And this definitely isn't limited to "traditional science." Granted, the business end of the world has a rather different criteria when it comes to statistics, but the core issue is the same. And I suspect at the even more fundamental level, this isn't really limited to statistics. It's just that statistics is rather universal in its application and tends to get a bad rap--but this also means that it's easy to be overconfident and there is more opportunity for improper use.

------------------------------
Michiko Wolcott
Principal Consultant
Msight Analytics
------------------------------

Hi, very interesting discussion.

Here is A dirty dozen: twelve p-value misconceptions by S. Goodman http://www.ncbi.nlm.nih.gov/pubmed/18582619

------------------------------
Iryna Lobach
UCSF
------------------------------