ASA Connect

 View Only
Expand all | Collapse all

reporting of p-values in tables

  • 1.  reporting of p-values in tables

    Posted 01-23-2017 13:13

    I have on some occasions received comments from editors or reviewers that p-values in tables should be reported by the asterisk notation. This does not seem to be consistent with ASAs recent statement on p-values (http://www.amstat.org/asa/files/pdfs/P-ValueStatement.pdf), particularly not with third principle ("Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold."). This principle is, I think, equally important in tables as in the main text. In many tables, p-values are stated together with confidence intervals. While the whole confidence distribution is a substitute for a p-value, this is not necessarily the case for a single, often 95%, confidence interval.  In the asterisk notation p-values are only stated with a rather course categorization, and I have some problems with being associated with such a style of reporting. In some cases this may be necessary due to lack of space in a table, but this is not always the case. This requirement is in some cases based on reference to the APA style; perhaps ASA could contact the American Psychological Association to discuss this matter.

    -----

    Tore Wentzel-Larsen
    Regional Center for Child and Adolescent Mental Health, Eastern and Southern Norway
    Norwegian Centre for violence and traumatic stress studies



  • 2.  RE: reporting of p-values in tables

    Posted 01-23-2017 08:56
    Reporting a p-value with an asterisk (such as <=0.05 given a "*", <=0.01 given a "**", etc.) is a common practice for statisticians to report the possible extent for rejection of null hypotheses for non-statisticians.  It is kind of like sticking a finger into someone's eye to draw attention.  In such cases, the statisticians do not make any biological conclusion except for supplying ancillary evidence.  The final conclusion is and should always be made by the biologist ("the experimenter").  I do not see any conflict with the ASA concept regarding this.  The use of both confidence intervals and p-values probably is a redundant practice.

    Ajit K. Thakur, Ph.D.
    Retired Biostatistician and Consultant

    ------------------------------
    Ajit Thakur
    Associate Director
    ------------------------------



  • 3.  RE: reporting of p-values in tables

    Posted 01-24-2017 22:43
    Ajit Thakur said, "The use of both confidence intervals and p-values probably is a redundant practice."

    I must disagree. Confidence intervals give information about the effect size; p-values do not give such information. One can have (especially with a large sample size) a "statistically significant" p-value, but the effect size may be so small as not to be practically significant.

    ------------------------------
    Martha Smith
    University of Texas
    ------------------------------



  • 4.  RE: reporting of p-values in tables

    Posted 01-25-2017 09:52
    The "problem" with confidence intervals is that non-statisticians are often completely incapable of understanding them.  I'll admit my sample is biased: I taught two sections of a required math class for liberal arts majors.  No matter what I did, they just could not grasp the idea of confidence intervals, even when presented with real-world examples of polling margins of error.  Professionally, I have had to present to people who didn't know any statistics, outside of what little they had been taught and could remember, which was often just the mean and maybe the median.

    In my work I normally present standard errors and p-values.  A knowledgeable person can then construct confidence intervals of any probability.  Other users only care whether a variable is a useful explainer of another: the p-value gives a very rough idea of explanatory strength.

    ------------------------------
    Chuck Coleman
    ------------------------------



  • 5.  RE: reporting of p-values in tables

    Posted 01-25-2017 10:47
    What do they think a confidence interval tells them?

    ------------------------------
    Emil M Friedman, PhD
    emilfriedman@gmail.com
    http://www.statisticalconsulting.org
    ------------------------------



  • 6.  RE: reporting of p-values in tables

    Posted 01-26-2017 09:15
    Nothing at all.  They can't interpret the idea of a confidence interval.

    ------------------------------
    Chuck Coleman
    ------------------------------



  • 7.  RE: reporting of p-values in tables

    Posted 01-26-2017 10:21
    If they have no clue regarding confidence intervals how can they possibly understand anything regarding p-values?  Furthermore, if they are that clueless, how can they possibly understand the results of any study that involves variability?

    Perhaps you need to look for other ways to explain confidence intervals to them.  Perhaps some of them are just taking a course because it's required and don't really care about the subject matter.  Maybe a study whose subject catches their interest would help.

    ------------------------------
    Emil M Friedman, PhD
    emilfriedman@gmail.com
    http://www.statisticalconsulting.org
    ------------------------------



  • 8.  RE: reporting of p-values in tables

    Posted 01-26-2017 10:25
    PS, someone who doesn't understand confidence limits probably thinks that 1-p is the probability that the effect is significant and probably doesn't understand the difference between "significant" and "important".

    ------------------------------
    Emil M Friedman, PhD
    emilfriedman@gmail.com
    http://www.statisticalconsulting.org
    ------------------------------



  • 9.  RE: reporting of p-values in tables

    Posted 01-27-2017 10:02
    Perhaps I overstated my case. The way p-values are presented as stars, etc. creates the interpretation that the evidence is "good," "better," etc.  Showing any symbol next to a p-value allows a person to "know" whether the variable has an effect or not, the more stars the better for a "yes" decision. While my students were largely resistant to learning math and statistics, in the real world, most people relying on statistics simply don't have time to learn statistics or don't understand the need to learn the rudiments. This is not only true of statistics, but even more true of economics, but that's another discussion.

    I happened to have a talk last night with a computer graphics designer about the problem of developing a simple visualization to enable people to understand confidence intervals.  I hope I'll have something positive to report.

    ------------------------------
    Chuck Coleman
    ------------------------------



  • 10.  RE: reporting of p-values in tables

    Posted 02-08-2017 16:34
    There was a good graphical explanation of confidence limits in one of the JMP software courses.  We start with a population whose mean we know.  Then we forget we know it and start taking samples.  Each sample generates an upper and lower confidence limit.  Then the CL's are plotted dynamically with a horizontal line representing the true mean.  As you watch, each pair of CL's get added to the plot you see that they randomly vary and that roughly alpha of them fail to include the true mean.

    ------------------------------
    Emil M Friedman, PhD
    emilfriedman@gmail.com
    http://www.statisticalconsulting.org
    ------------------------------



  • 11.  RE: reporting of p-values in tables

    Posted 01-26-2017 08:15

    I like the use of SE and CVs for that matter.  As a matter of fact, I would love to see as many statistical moments as would be allowed, because each tells us a little something about the sample.  However, we all know that it would be numbers over-kill for most journals.  And let's face it, we have helped to develop "lazy" analysis by relying on simple and sometimes singular numbers instead of words.  The p-value is a case in point.  What I would REALLY like journals to start doing is put less emphasis on how the data are presented and MORE emphasis on how the authors interpret the results.  But, do not assume I am for banning p-values or lessening their (or statistics in general) importance.  I want them and other output.  But I also want the authors to talk about what that means in the context of their field.

     

    Every scientific journal should have a statistician on staff as an editor/reviewer.  The problem is, there are not enough of us to go around.

     

    Susan E. Spruill

    Susan E. Spruill, PStat®

    Statistical Consultant, President

    Applied Statistics and Consulting

    828-467-9184 (phone)

    Professional Statistician accredited by the American Statistical Association

    www.appstatsconsulting.com

     






  • 12.  RE: reporting of p-values in tables

    Posted 01-26-2017 08:53
    I totally agree with Dr. Coleman.  I have been practicing bio/applied statistics for almost 50 years- in research environment (NIH) and in the pharmaceutical/chemical industries.  The "effect size" does not make any sense to most of them; whereas, sample size, standard error, mean, and a p-value makes a lot of sense to them. In some cases, such as in bioequivalence trials, equivalence is established using confidence limits. In those cases, the FDA and other such international regulatory organizations clearly ask to have certain type of confidence limit(s) to be calculated in their guidelines.

    Ajit K. Thakur, Ph.D.





  • 13.  RE: reporting of p-values in tables

    Posted 01-26-2017 22:13
     I agree that the difficulty in understanding confidence intervals is a problem. Even when a student succeeds in understanding, they usually don't think they can communicate just what it is to their boss or other "laypeople". So I've been wondering what would be an appropriate explanation that is not so technical as to make the listener turn off, but is still not misleading. I've been considering something like the following, and would be interested to hear what others think  of it, or if they have a better suggetion:

    "We need to remember that our estimate is just that, an estimate, based on the data we have. There are various reasons why it could be off. The confidence interval gives us a rough idea of how much we could be off just based on the fact that different samples may give different estimates. So if our best estimate is 1.2, and the confidence interval is (0.9, 1.5), we're saying that a different sample might have given an estimate anywhere between 0.9 and 1.50  -- but we're pretty confident that the quantity we're trying to estimate is between 0.9 and 1.5."

    ------------------------------
    Martha Smith
    University of Texas
    ------------------------------



  • 14.  RE: reporting of p-values in tables

    Posted 01-23-2017 17:22
    If there is limited space in a table, then the *, **, and *** convention is a way to indicate the approximate size of the p-value.

    Reporting the actual p-value is preferable however, often with p < 0.001 for small values.

    We should avoid reporting "p = 0.034*", and then in the table footer stating that "* p < 0.05". Do we really need a footnote to state that 0.034 < 0.05? Scientists do this because they want to draw attention to significant p-values. But emphasising results is better done with bold font or by sorting the table by p-value (if possible).

    I think the *, **, *** convention does lead to "threshold thinking" and has a certain Orwellian flavour. Much like Newspeak, this convention diminishes people's range of thought. I hope that someone has the courage to smuggle the following into a table footer or figure caption in a manuscript: * good, ** plus good, *** double plus good

    :-)

    ------------------------------
    Stanley E. Lazic, PhD

    Team Leader, Statistics
    AstraZeneca
    Cambridge, UK

    https://stanlazic.github.io/
    ------------------------------



  • 15.  RE: reporting of p-values in tables

    Posted 01-24-2017 09:50
    I believe the confidence interval conveys more information than the p-value.  The confidence contains a mesurement of uncertainty.  Imagine a p-value of 0.06, where the confidence limit of a difference in means, nicks zero but has a right hand tail of positive values.  No significant differences at 0.05, but the confidence gives me pause about strength of evidence.  Same case for marginally significant case.  My practice is to provide both.  The star system likely approximates the significant digits in p-values ( I read it in paper, but don't have the reference available).

    ------------------------------
    Steven Correia
    Environmental analyst (retired)
    Massachusetts Division of Marine Fisheries
    ------------------------------



  • 16.  RE: reporting of p-values in tables

    Posted 01-24-2017 09:06
    My experience is to report the actual p-value to 4 decimal places, e.g. 0.0006 or 0.0493. Very small p-values will be shown as
    "<0.0001" and p-values rounding to 1 will appear as ">0.9999". I avoid the * - system as * can mean anything between 0.049999999... down to 0.010000000... The upper end of the range suggests that if a single value was just a tiny bit different then the p-value might become >=0.05000000000... Of course it could also change to 0.0498 or smaller. 

    One thing I will always avoid is doing something like what appeared in LinkedIn last week. A figure showing statistical results had a p-value of 0.000000000000000000013!!! Come on, how many leading 0s does one need to make his or her point?

    ------------------------------
    Nestor Rohowsky
    President and Principal Consultant
    Integrated Data Consultation Services, Inc.
    ------------------------------



  • 17.  RE: reporting of p-values in tables

    Posted 01-24-2017 10:21
    My preference and practice is as Nester indicated: Present the p-value to 4 decimal places bounded by <0.0001 or >0.9999.  This allows the read to decide if the evidence of rejection (or not) is weak.  As Ajit pointed out asterisks in place of p-values is like a jab in the eye!  I prefer to give the reader the benefit of doubt they can decipher the importance of the statistical results without simplifying a p-value to a symbol.  The only time I would consider it is in graphic where there is an attempt to draw attention to some aspect of the graph.  But even in those cases the actual p-value should be noted.

    As for confidence intervals, I agree, they can be much more meaningful than a simple p-value.  However, we must caution the readers to understand what a confidence interval means.  Regardless of the statistic we use, we should always interpret the results as well.

    ------------------------------
    Susan Spruill
    Statistical Consultant
    ------------------------------



  • 18.  RE: reporting of p-values in tables

    Posted 01-25-2017 02:48
    I Fully agree that the number of digits for P-value should be four. However, for most of the journals the required no. of digits for significant P-value are four digits and for non-significant  P-values it may be just mentioned as P>0.05.

    ------------------------------
    Vanamail Perumal
    Associate Professor of Statistics & Demography
    All India Institute of Medical Sciences
    ------------------------------



  • 19.  RE: reporting of p-values in tables

    Posted 01-26-2017 14:45
    I recommend a look at Cumming's wonderful multimedia video on "The dance of the p value" 


    Dance p 3 Mar09
    YouTube remove preview
    Dance p 3 Mar09
    I use a simulation from my ESCI software to illustrate the enormous variability in the p value, simply because of sampling variability. That's the dance of the p value. Never trust a p value--it's too unreliable! Use estimation, not NHST! Most researchers don't appreciate just how unreliable the p value is!
    View this on YouTube >
     

    ------------------------------
    Dale Berger
    Emeritus Professor of Psychology
    Claremont Graduate University
    statistics website: http://wise.cgu.edu
    ------------------------------



  • 20.  RE: reporting of p-values in tables

    Posted 01-25-2017 02:44
    In meta-analysis it is very valuable to have exact p-values; 0.000000000000000000013 is of course ridiculous; Rohowsky's four decimals will do . In addition, they help to check the correctness of the statistics reported, which is not always the case, see Marjan BakkeJelte M. Wicherts (2011). The (mis)reporting of statistical results in psychology journals. Behavior Research Methods, Volume 43, Issue 3, pp 666–678.

    ------------------------------
    Pieter Kroonenberg
    Emeritus Professor
    Leiden University
    ------------------------------



  • 21.  RE: reporting of p-values in tables

    Posted 01-25-2017 18:59
    As I have learned from the statistical literature, it is better to report a p-value up to 4 decimal places in the tables for the strength of acceptance or rejection.Alsoone can use the p<0.05 when explaining the results/significance in the text. But in genetics studies p-values can be much smaller and are written as, say p= 5×10-6 .

    ------------------------------
    Ramzan Tahir
    Biostatistician
    CHUM Research Center
    ------------------------------



  • 22.  RE: reporting of p-values in tables

    Posted 01-25-2017 19:50
    In my view, in science, p-values are best viewed as a quantitative measure of the strength of evidence against a null hypothesis.  Reporting exact p-values gives members of the scientific community information for making a scientific judgment about the hypotheses based on this study and other related studies.  From that point of view, reporting an exact p-value is preferable to reporting P<0.05.

    Regarding the number of digits to report, I have found it practical to report two non-zero digits to the limit of <0.0001.  Thus, for example, I do not report p=0.5469 or p=0.0342.  Instead I would report p=0.55 and 0.034.

    ------------------------------
    Laurence Magder
    University of Maryland, Baltimore
    ------------------------------



  • 23.  RE: reporting of p-values in tables

    Posted 01-26-2017 08:18

    Thanks for reiterating the "strength of evidence" issue, Laurence.

     

    Susan E. Spruill

    Susan E. Spruill, PStat®

    Statistical Consultant, President

    Applied Statistics and Consulting

    828-467-9184 (phone)

    Professional Statistician accredited by the American Statistical Association

    www.appstatsconsulting.com

     






  • 24.  RE: reporting of p-values in tables

    Posted 01-26-2017 08:21

    Ramzan,

    What is unique about genetics data that warrants reporting such small p-values?  Do you follow a critical rejection of something less than p<0.05?  Just curious.

     

    Susan E. Spruill

    Susan E. Spruill, PStat®

    Statistical Consultant, President

    Applied Statistics and Consulting

    828-467-9184 (phone)

    Professional Statistician accredited by the American Statistical Association

    www.appstatsconsulting.com