ASA Connect

 View Only
Back to eGroups
Expand all | Collapse all

Dealing with claims of fishing?

  • 1.  Dealing with claims of fishing?

    Posted 04-18-2017 20:49

    Hi all,

    My co-author and I just had a paper returned in nursing education telling us that we were fishing for
    something to report because the groups were "too different to compare"

    when comparing between a group that was 88% female and 12% male.

    Our results had:

    Mann-Whitney p = 0.0269, Cohen's d = 0.78  (after multiple comparisons adjustment)

    I'm not really sure how to respond to that.  Any suggestions?


    Joseph Reid
    Assoc. Prof. of Applied Math and Statistics



  • 2.  RE: Dealing with claims of fishing?

    Posted 04-19-2017 01:12

    When you got Mann-Whitney p=0.0269 and Cohen's d = 0.78, (1) what was the quantity you were comparing, (2) what was the group mean of the males, and (3) what was the group mean of the females? Also, do the group means have units, like years or pounds or inches, or are they group means of something such as scores on a survey instrument?

     

    A complaint that the two groups were "too different to compare" does sound mysterious. But it might be that the reviewer wants to suggest that the males and females differ on many factors such as age, height, weight, education level, annual salary, etc, in addition to the quantity you were comparing. If that be the case, then your small p and large d might have arisen from confounding between your quantity and one or more of those other factors, and not from an intrinsic difference between males and females.

    Confidentiality Notice: This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.



    Original Message


  • 3.  RE: Dealing with claims of fishing?

    Posted 04-19-2017 07:21
    You seem to be confusing the Wilcoxon-Mann-Whitney p-value with comparing means. As with using the WMW test to compare medians (which is very common and unfortunately imbedded in our best statistical software, including SAS and base R), this is only valid in the extremely rare case when the location-shift model holds for the density functions of the two groups. The WMW test has a simple and meaningful point estimate and there is a solid confidence interval for it, too. To learn more and to obtain an Rfunc to do it all, see http://rfuncs.weebly.com/wmwodds.html. 


    ------------------------------
    Ralph O'Brien
    Professor of Biostatistics (officially retired; still keenly active)
    Case Western Reserve University
    http://rfuncs.weebly.com/about-ralph-obrien.html
    ------------------------------

    Original Message


  • 4.  RE: Dealing with claims of fishing?

    Posted 04-19-2017 08:56
    There are so many questions surrounding this question.  First, was there an intervention and it just happened the 2 groups were unbalanced by gender?  If so, is it possible the intervention might behalf differently between the sexes?  How about the outcome measure?  Is it gender dependent?  There are a number of ways to account for the effect of gender on the outcome (or interactions with intervention).  Perhaps the reviewers are attempting to make sure you have properly address this. 

    Lastly, when a reviewer says something like "fishing" it is a flag that there is not much else going on in the paper.  Perhaps a number of non-significant outcomes or unexpected results and this one seemed to be the only "beacon" of effect.  You may need to look at how you are presenting it with regards to its importance in the overall experiment.

    ------------------------------
    Susan Spruill
    Statistical Consultant
    ------------------------------

    Original Message


  • 5.  RE: Dealing with claims of fishing?

    Posted 04-19-2017 08:22
    I'm not sure, like the other responders, what you did with the Mann-Whitney, but maybe the reviewers were saying that (with respect to gender) you didn't need a significance test to conclude that your samples came from different populations? Perhaps by "fishing" they meant that you were conducting superfluous significance tests? Not sure, but good luck with the paper.

    ------------------------------
    Evan Blaine, PhD, PStat
    Statistics Program Director
    St. John Fisher College
    Rochester NY
    ------------------------------

    Original Message


  • 6.  RE: Dealing with claims of fishing?

    Posted 04-19-2017 09:45
    You need to tell us a little more of your design.  For example, how many groups do you have that prompts you to use multiple comparison adjustment (such as Bonferroni-type?).  Are you comparing the males versus the females?  If latter is the case, your groups are extremely uneven and the reviewer is probably right is pointing that out.  Distribution-free tests such as Wilcoxon-Mann-Whitney, based on joint ranking suffers from distributional requirement such as at least approximately even sample sizes.  If you have multiple groups, then you should be using the Kruskal-Wallis  nonparametric ANOVA.  That would have similar problem.  You can try exact permutation test as in StatXact but the problem would linger on.  If I were the investigator, I would design the experiment such that I would not have such an extreme sample size problem.  If you just have a group of fish divided into two genders, males and females, then where is the multiplicity comes from?  It just makes comparisons conservative. In other words, the community responding would probably like to know your design.  Thanks.

    Ajit K. Thakur, Ph.D.
    Retired Biostatistician




    Original Message


  • 7.  RE: Dealing with claims of fishing?

    Posted 04-19-2017 11:13
    As already mentioned, the assessment of "fishing" is unlikely to result from the test you have been citing and the reviewer is reacting to something else. Since you refer to "nursing education" I find it likely that you use existing data probably not specifically collected for research purposes. If so, the STROBE statement will give you good guidance about what information to include in a solid manuscript that reports an observational study.  You will find a link to this reporting standard on the Equator Network ( http://www.equator-network.org/ ) that also has links to reporting standards for other study types (CONSORT for randomized trials, PRISMA for systematic reviews etc.)

    STROBE = STrengthening the Reporting of Observational Studies in Epidemiology
    Equator = Enhancing the QUAlity and Transparency Of health Research

    Fridtjof

    ------------------------------
    ________________________________________
    Fridtjof Thomas, Ph.D.
    Associate Professor Division of Biostatistics
    Department of Preventive Medicine
    University of Tennessee Health Science Center
    fthomas4@uthsc.edu
    ------------------------------

    Original Message


  • 8.  RE: Dealing with claims of fishing?

    Posted 04-19-2017 12:18
    Hello Joseph,

    I think if you provided us with a little more information, it would be easier to answer your question.
    Were you comparing two groups, and each of them were composed of 88% female and 12% male?
    Or, what were you comparing?

    If you were measuring variables from this one group, comprised of 88% female and 12% male, then you could perhaps provide evidence that the typical nursing group/class is generally made up of about 88% female and 12% male?

    My best,
    Gretchen
     
    Gretchen A. Donahue



    Original Message


  • 9.  RE: Dealing with claims of fishing?

    Posted 04-19-2017 12:45

    Thanks everyone.  I think I know what is going on now.

    Sample sizes were around 400 under an observational design with non-randomized sampling (obviously not trying to extend the results to something outside of the study.  It's primarily confirmatory analysis.)  And, the primary results were non-significant, which is something the researcher had hoped for (I'm just the statistician).  It wasn't really a matter of sample size causing the problem and tests of distributional shift were reasonable, even with the large difference in sample sizes.  Methodology was comparing results of a generalized partial credit model and determining the number of latent traits were represented within a survey (sample was too small to apply SEM).

    Basically, the test between genders was being used to determine if there were potential confounding factors when answering the primary question.  That is, in order to address the main question, could we use something as simple as ANOVA/ANCOVA (or a non-parametric alternative) or do we need to use a GLM (or possibly a GLMM).  Rather than justifying methodology, I think the wording made it sound like we were searching for anything else that could be significant since our primary results weren't (again, what was hoped for given the study was confirmatory.)



    ------------------------------
    Joseph Reid
    Associate Prof. of Applied Mathematics and Statistics
    Oregon Institute of Technology
    ------------------------------

    Original Message


  • 10.  RE: Dealing with claims of fishing?

    Posted 04-20-2017 11:27
    Hm...  If the "researcher had hoped for" that "the primary results were non-significant" you may want to rethink your approach and/or reporting.  It sounds to me that the correct design would target "equivalence" between the groups, but that is not what I figure you are doing.  A standard test where you don't reject the null does *not* establish that "there is no difference".  If you want to establish the latter you need to define an interval that you accept as "practically equal effects" and you then need an estimate of that difference that is precise enough to make it very likely that the true difference is actually within the range that you defined as "practically equal".  (Non-inferiority trials in medicine deal with that specific problem.)

    Fridtjof

    ------------------------------
    Fridtjof Thomas, Ph.D.
    Associate Professor Division of Biostatistics
    Department of Preventive Medicine
    University of Tennessee Health Science Center
    fthomas4@uthsc.edu
    ------------------------------

    Original Message


  • 11.  RE: Dealing with claims of fishing?

    Posted 04-21-2017 11:00
    Yup, did (not just looking for non-significant p-values, but also very low effect sizes, defined acceptable differences, etc.)  Also using CFA and other methods to compare these results with previous work.  SEM didn't work due to sampling but would have been my first choice.  We didn't have much choice in grouping as the vast majority of nursing students are female and many of the questions they wanted to answer required a proportionally representative sample.  In addition, most of the demographics were just recorded and not used as sampling characteristics since there was no way of knowing ahead of time whether or not they should be adjusted for.  That's kind of one reason we want this particular result in the paper (so future researchers will know that adjustment for gender during sampling is important.)

    ------------------------------
    Joseph Reid
    Associate Prof. of Applied Mathematics and Statistics
    Oregon Institute of Technology
    ------------------------------

    Original Message


Related Content