ASA Connect

Joseph,

I wonder if you might be interested in the related (but by no means identical) concept of blind analysis described in this article [1].

Kind regards,
David

------------------------------
David C. Norris, MD
Precision Methodologies, LLC
Seattle, WA
------------------------------

Original Message:
Sent: 04-10-2017 13:25
From: Joseph Locascio
Subject: Results Blind Manuscript Evaluation

4/10/17

 

Dear ASA Statisticians,

      In connection with occasional comments in ASA Connect about problems pertaining to the over-use/abuse of p values & null hypothesis significance testing (NHST), I'd like to run by the following suggestion to get any opinions. (I don't believe I've ever posted this at ASA Connect before).   

      Some statisticians, including myself, have for years been suggesting that biomed/behavioral science journals employ a "results-blind" evaluation process in deciding whether to publish manuscripts.  I've long felt that this might have beneficial effects in reducing, not only the well-known bias of journals to publish reports with positive results, but also mitigate problems with abuse of p values/NHST. 

      What I mean by "results-blind" manuscript evaluation is as follows:  In deciding whether to publish a manuscript, weight would be given exclusively to (a) the judged importance of the research question addressed in the study, typically conveyed in the Introduction section of the manuscript, and (b) the quality of the methodology of the study, including appropriateness of data analysis methods and adequate statistical power, as reported in the Methods section.  Practically, this "results-blind" process might be implemented by a "two stage evaluation process" whereby the editor initially distributes only the Introduction and Methods sections of a submitted manuscript to reviewers for evaluation.  Only after a provisional decision is made by reviewers for tentative acceptance (or with only minor revision) based on the Introduction & Methods, would the remainder of the paper (Results, Discussion, Abstract) then be distributed to the reviewers for final confirmation of acceptance, & editing suggestions, barring nothing new being discovered in the remainder of the paper that reflects so poorly on the methodology as to contravene the initial acceptance. Journal guidelines to authors would make them fully aware of this practice in reviewing manuscripts.  

      This "two stage evaluation" would have some of the same beneficial effects as "pre-registered" studies, except that it would have the additional benefit of requiring no new burden or change on the part of the authors in submitting manuscripts.  This would also mitigate the problem of researchers not submitting manuscripts for publication because they didn't find significant results, or over-reliance on NHST & p values, or conscious/unconscious cheating to get p values < 0.05, since the authors know their results will have no bearing on determining publication.  Only methodology matters, including their proposed data analysis methods, described & justified in the Methods section.  This would also presumably have a beneficial effect on the "reproducibility crisis" in research. 

      With such a policy, manuscripts would be evaluated on the quality of their methodology, not what they happened to find.  If "no effect" was found, but the methodology was sound, don't we want to know this in order to avoid a positive bias in the literature?   

       I'd be interested in anyone's opinion on this.  Is there something I've missed?  Is it impractical in some way?  Am I being naive about something like this being broadly adopted?  Would it be too difficult politically to implement?  

       Thanks.  

 

Joseph J. Locascio, Ph.D.,

Bio-Statistician,
Neurology Dept.,
Massachusetts General Hospital,
Boston, Massachusetts 
Phone: (617) 724-7192
Email: JLocascio@partners.org        
        

           
"The information transmitted in this email is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this email in error, please contact the sender and delete the material from any computer."

 

 

 

The information in this e-mail is intended only for the person to whom it is
addressed. If you believe this e-mail was sent to you in error and the e-mail
contains patient information, please contact the Partners Compliance HelpLine at
http://www.partners.org/complianceline . If the e-mail was sent to you in error
but does not contain patient information, please contact the sender and properly
dispose of the e-mail.

I commend the motivation, but I'm afraid that authors often highlight results in a paper's introduction and abstract.  That complication makes the implementation of the results-blind process difficult.

A recent paper bySmaldino and McElreath implies that the more journals publish reproduction results, the better the influence of incentives on scientists.

"The natural selection of bad science"
http://rsos.royalsocietypublishing.org/content/3/9/160384

You'd have to have some influence over journal practices to get them to adopt the results-blind process, but I suspect that the influence would be better applied in trying to get them to publish a couple of experiment replications with every issue.

------------------------------
Edward Cashin
Research Scientist II
------------------------------

Original Message:
Sent: 04-10-2017 13:25
From: Joseph Locascio
Subject: Results Blind Manuscript Evaluation

4/10/17

 

Dear ASA Statisticians,

      In connection with occasional comments in ASA Connect about problems pertaining to the over-use/abuse of p values & null hypothesis significance testing (NHST), I'd like to run by the following suggestion to get any opinions. (I don't believe I've ever posted this at ASA Connect before).   

      Some statisticians, including myself, have for years been suggesting that biomed/behavioral science journals employ a "results-blind" evaluation process in deciding whether to publish manuscripts.  I've long felt that this might have beneficial effects in reducing, not only the well-known bias of journals to publish reports with positive results, but also mitigate problems with abuse of p values/NHST. 

      What I mean by "results-blind" manuscript evaluation is as follows:  In deciding whether to publish a manuscript, weight would be given exclusively to (a) the judged importance of the research question addressed in the study, typically conveyed in the Introduction section of the manuscript, and (b) the quality of the methodology of the study, including appropriateness of data analysis methods and adequate statistical power, as reported in the Methods section.  Practically, this "results-blind" process might be implemented by a "two stage evaluation process" whereby the editor initially distributes only the Introduction and Methods sections of a submitted manuscript to reviewers for evaluation.  Only after a provisional decision is made by reviewers for tentative acceptance (or with only minor revision) based on the Introduction & Methods, would the remainder of the paper (Results, Discussion, Abstract) then be distributed to the reviewers for final confirmation of acceptance, & editing suggestions, barring nothing new being discovered in the remainder of the paper that reflects so poorly on the methodology as to contravene the initial acceptance. Journal guidelines to authors would make them fully aware of this practice in reviewing manuscripts.  

      This "two stage evaluation" would have some of the same beneficial effects as "pre-registered" studies, except that it would have the additional benefit of requiring no new burden or change on the part of the authors in submitting manuscripts.  This would also mitigate the problem of researchers not submitting manuscripts for publication because they didn't find significant results, or over-reliance on NHST & p values, or conscious/unconscious cheating to get p values < 0.05, since the authors know their results will have no bearing on determining publication.  Only methodology matters, including their proposed data analysis methods, described & justified in the Methods section.  This would also presumably have a beneficial effect on the "reproducibility crisis" in research. 

      With such a policy, manuscripts would be evaluated on the quality of their methodology, not what they happened to find.  If "no effect" was found, but the methodology was sound, don't we want to know this in order to avoid a positive bias in the literature?   

       I'd be interested in anyone's opinion on this.  Is there something I've missed?  Is it impractical in some way?  Am I being naive about something like this being broadly adopted?  Would it be too difficult politically to implement?  

       Thanks.  

 

Joseph J. Locascio, Ph.D.,

Bio-Statistician,
Neurology Dept.,
Massachusetts General Hospital,
Boston, Massachusetts 
Phone: (617) 724-7192
Email: JLocascio@partners.org        
        

           
"The information transmitted in this email is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this email in error, please contact the sender and delete the material from any computer."

 

 

 

The information in this e-mail is intended only for the person to whom it is
addressed. If you believe this e-mail was sent to you in error and the e-mail
contains patient information, please contact the Partners Compliance HelpLine at
http://www.partners.org/complianceline . If the e-mail was sent to you in error
but does not contain patient information, please contact the sender and properly
dispose of the e-mail.

Original Message------

I commend the motivation, but I'm afraid that authors often highlight results in a paper's introduction and abstract.  That complication makes the implementation of the results-blind process difficult.

A recent paper bySmaldino and McElreath implies that the more journals publish reproduction results, the better the influence of incentives on scientists.

"The natural selection of bad science"
http://rsos.royalsocietypublishing.org/content/3/9/160384

You'd have to have some influence over journal practices to get them to adopt the results-blind process, but I suspect that the influence would be better applied in trying to get them to publish a couple of experiment replications with every issue.

------------------------------
Edward Cashin
Research Scientist II
------------------------------

I agree with all said but for one important detail: The use of power to judge the quality of the study. We all seem to recognize the damage inflicted by NHST, yet many continue to endorse the NHST paradigm via emphasis on power.

Small, "underpowered" studies can contribute high-quality data to the overall pool of evidence, and will lead to no distortion if conducted, reported, presented and interpreted correctly with due accounting for their size. It is the misinterpretation and publication bias attendant to small studies that is the problem; these are not intrinsic quality features of the studies, but rather limitations of the authors and the system in which they publish. 

In some topic areas, especially in which data collection is costly, small studies may contain the bulk of evidence, and patterns across the studies may provide important meta-analytic information as well. To obtain an undistorted picture of this literature it will be essential to remove the pressure of publishing studies based on their results or on features that can be accounted for easily by proper analysis. A precise accounting for sample size and thus power is indeed built in to recommended summaries such as confidence intervals.

Simply mandating confidence intervals is not enough however - they further need correct, direct interpretation as indicators of the precision or information content of the study, rather than as null tests. It is the pressure on investigators to come up with decisive findings that distort interpretations, e.g., claiming evidence of "no effect" when in reality the confidence interval covered every reasonably possible effect from none to important (e.g., see European Journal of Epidemiology 2017, v. 32, p. 87-88 for a startling if sadly typical example). This problem is not addressed by blind analysis.

Smaller studies will include the bulk of so-called "negative" results whose absence from the literature is most commonly cited manifestation of publication bias. Of course journals will prefer larger studies, so any effort to reform review and publication should see to it that there is a repository for studies not seen as important or "sexy" enough to be main articles. This need can be addressed via electronic publication of such studies as brief reports or research letters with details in online appendices, provided those reports are indexed in standard resources (e.g., PubMed, Current Index) to allow identification through online searches.

------------------------------
Sander Greenland
Department of Epidemiology and Department of Statistics
University of California, Los Angeles
------------------------------

Original Message:
Sent: 04-10-2017 13:25
From: Joseph Locascio
Subject: Results Blind Manuscript Evaluation

4/10/17

 

Dear ASA Statisticians,

      In connection with occasional comments in ASA Connect about problems pertaining to the over-use/abuse of p values & null hypothesis significance testing (NHST), I'd like to run by the following suggestion to get any opinions. (I don't believe I've ever posted this at ASA Connect before).   

      Some statisticians, including myself, have for years been suggesting that biomed/behavioral science journals employ a "results-blind" evaluation process in deciding whether to publish manuscripts.  I've long felt that this might have beneficial effects in reducing, not only the well-known bias of journals to publish reports with positive results, but also mitigate problems with abuse of p values/NHST. 

      What I mean by "results-blind" manuscript evaluation is as follows:  In deciding whether to publish a manuscript, weight would be given exclusively to (a) the judged importance of the research question addressed in the study, typically conveyed in the Introduction section of the manuscript, and (b) the quality of the methodology of the study, including appropriateness of data analysis methods and adequate statistical power, as reported in the Methods section.  Practically, this "results-blind" process might be implemented by a "two stage evaluation process" whereby the editor initially distributes only the Introduction and Methods sections of a submitted manuscript to reviewers for evaluation.  Only after a provisional decision is made by reviewers for tentative acceptance (or with only minor revision) based on the Introduction & Methods, would the remainder of the paper (Results, Discussion, Abstract) then be distributed to the reviewers for final confirmation of acceptance, & editing suggestions, barring nothing new being discovered in the remainder of the paper that reflects so poorly on the methodology as to contravene the initial acceptance. Journal guidelines to authors would make them fully aware of this practice in reviewing manuscripts.  

      This "two stage evaluation" would have some of the same beneficial effects as "pre-registered" studies, except that it would have the additional benefit of requiring no new burden or change on the part of the authors in submitting manuscripts.  This would also mitigate the problem of researchers not submitting manuscripts for publication because they didn't find significant results, or over-reliance on NHST & p values, or conscious/unconscious cheating to get p values < 0.05, since the authors know their results will have no bearing on determining publication.  Only methodology matters, including their proposed data analysis methods, described & justified in the Methods section.  This would also presumably have a beneficial effect on the "reproducibility crisis" in research. 

      With such a policy, manuscripts would be evaluated on the quality of their methodology, not what they happened to find.  If "no effect" was found, but the methodology was sound, don't we want to know this in order to avoid a positive bias in the literature?   

       I'd be interested in anyone's opinion on this.  Is there something I've missed?  Is it impractical in some way?  Am I being naive about something like this being broadly adopted?  Would it be too difficult politically to implement?  

       Thanks.  

 

Joseph J. Locascio, Ph.D.,

Bio-Statistician,
Neurology Dept.,
Massachusetts General Hospital,
Boston, Massachusetts 
Phone: (617) 724-7192
Email: JLocascio@partners.org        
        

           
"The information transmitted in this email is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this email in error, please contact the sender and delete the material from any computer."

 

 

 

The information in this e-mail is intended only for the person to whom it is
addressed. If you believe this e-mail was sent to you in error and the e-mail
contains patient information, please contact the Partners Compliance HelpLine at
http://www.partners.org/complianceline . If the e-mail was sent to you in error
but does not contain patient information, please contact the sender and properly
dispose of the e-mail.

Original Message------

I agree with all said but for one important detail: The use of power to judge the quality of the study. We all seem to recognize the damage inflicted by NHST, yet many continue to endorse the NHST paradigm via emphasis on power.

Small, "underpowered" studies can contribute high-quality data to the overall pool of evidence, and will lead to no distortion if conducted, reported, presented and interpreted correctly with due accounting for their size. It is the misinterpretation and publication bias attendant to small studies that is the problem; these are not intrinsic quality features of the studies, but rather limitations of the authors and the system in which they publish. 

In some topic areas, especially in which data collection is costly, small studies may contain the bulk of evidence, and patterns across the studies may provide important meta-analytic information as well. To obtain an undistorted picture of this literature it will be essential to remove the pressure of publishing studies based on their results or on features that can be accounted for easily by proper analysis. A precise accounting for sample size and thus power is indeed built in to recommended summaries such as confidence intervals.

Simply mandating confidence intervals is not enough however - they further need correct, direct interpretation as indicators of the precision or information content of the study, rather than as null tests. It is the pressure on investigators to come up with decisive findings that distort interpretations, e.g., claiming evidence of "no effect" when in reality the confidence interval covered every reasonably possible effect from none to important (e.g., see European Journal of Epidemiology 2017, v. 32, p. 87-88 for a startling if sadly typical example). This problem is not addressed by blind analysis.

Smaller studies will include the bulk of so-called "negative" results whose absence from the literature is most commonly cited manifestation of publication bias. Of course journals will prefer larger studies, so any effort to reform review and publication should see to it that there is a repository for studies not seen as important or "sexy" enough to be main articles. This need can be addressed via electronic publication of such studies as brief reports or research letters with details in online appendices, provided those reports are indexed in standard resources (e.g., PubMed, Current Index) to allow identification through online searches.

------------------------------
Sander Greenland
Department of Epidemiology and Department of Statistics
University of California, Los Angeles
------------------------------

Original Message------

4/10/17

 

Dear ASA Statisticians,

      In connection with occasional comments in ASA Connect about problems pertaining to the over-use/abuse of p values & null hypothesis significance testing (NHST), I'd like to run by the following suggestion to get any opinions. (I don't believe I've ever posted this at ASA Connect before).   

      Some statisticians, including myself, have for years been suggesting that biomed/behavioral science journals employ a "results-blind" evaluation process in deciding whether to publish manuscripts.  I've long felt that this might have beneficial effects in reducing, not only the well-known bias of journals to publish reports with positive results, but also mitigate problems with abuse of p values/NHST. 

      What I mean by "results-blind" manuscript evaluation is as follows:  In deciding whether to publish a manuscript, weight would be given exclusively to (a) the judged importance of the research question addressed in the study, typically conveyed in the Introduction section of the manuscript, and (b) the quality of the methodology of the study, including appropriateness of data analysis methods and adequate statistical power, as reported in the Methods section.  Practically, this "results-blind" process might be implemented by a "two stage evaluation process" whereby the editor initially distributes only the Introduction and Methods sections of a submitted manuscript to reviewers for evaluation.  Only after a provisional decision is made by reviewers for tentative acceptance (or with only minor revision) based on the Introduction & Methods, would the remainder of the paper (Results, Discussion, Abstract) then be distributed to the reviewers for final confirmation of acceptance, & editing suggestions, barring nothing new being discovered in the remainder of the paper that reflects so poorly on the methodology as to contravene the initial acceptance. Journal guidelines to authors would make them fully aware of this practice in reviewing manuscripts.  

      This "two stage evaluation" would have some of the same beneficial effects as "pre-registered" studies, except that it would have the additional benefit of requiring no new burden or change on the part of the authors in submitting manuscripts.  This would also mitigate the problem of researchers not submitting manuscripts for publication because they didn't find significant results, or over-reliance on NHST & p values, or conscious/unconscious cheating to get p values < 0.05, since the authors know their results will have no bearing on determining publication.  Only methodology matters, including their proposed data analysis methods, described & justified in the Methods section.  This would also presumably have a beneficial effect on the "reproducibility crisis" in research. 

      With such a policy, manuscripts would be evaluated on the quality of their methodology, not what they happened to find.  If "no effect" was found, but the methodology was sound, don't we want to know this in order to avoid a positive bias in the literature?   

       I'd be interested in anyone's opinion on this.  Is there something I've missed?  Is it impractical in some way?  Am I being naive about something like this being broadly adopted?  Would it be too difficult politically to implement?  

       Thanks.  

 

Joseph J. Locascio, Ph.D.,

Bio-Statistician,
Neurology Dept.,
Massachusetts General Hospital,
Boston, Massachusetts 
Phone: (617) 724-7192
Email: JLocascio@partners.org        
        

           
"The information transmitted in this email is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this email in error, please contact the sender and delete the material from any computer."

 

 

 

The information in this e-mail is intended only for the person to whom it is
addressed. If you believe this e-mail was sent to you in error and the e-mail
contains patient information, please contact the Partners Compliance HelpLine at
http://www.partners.org/complianceline . If the e-mail was sent to you in error
but does not contain patient information, please contact the sender and properly
dispose of the e-mail.

Original Message------

My first thought is, excellent idea in theory, but cumbersome in practice. But I can see myself being persuaded it might not be as cumbersome as I fear.

 

My second thought is, let's think twice before giving high weight to "the judged importance of the research question addressed in the study". One, even low-importance research deserves to be published if it's well-executed. For example, research that breaks no new ground, but adds to a growing body of results in a particular area. Two, research that strikes me as important may not strike you as important, and vice versa.  

---

Confidentiality Notice: This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.

One difficulty is that surprising results -results inconsistent with conventional theory - might be a chance error, but they might also suggest conventional theory is wrong and in need of revision. Serendipity - the discovery of things not originally planned for in an experience - has a large role in science, especially basic science but often applied science as well.

Of course serendipitously observed results may be mere hypothesis requiring corroboration to confirm. But that doesn't prevent them from being publication worthy.

Jonathan Siegel
Associate Director Clinical Statistics

Sent from my iPhone


Original Message------

4/10/17

 

Dear ASA Statisticians,

      In connection with occasional comments in ASA Connect about problems pertaining to the over-use/abuse of p values & null hypothesis significance testing (NHST), I'd like to run by the following suggestion to get any opinions. (I don't believe I've ever posted this at ASA Connect before).   

      Some statisticians, including myself, have for years been suggesting that biomed/behavioral science journals employ a "results-blind" evaluation process in deciding whether to publish manuscripts.  I've long felt that this might have beneficial effects in reducing, not only the well-known bias of journals to publish reports with positive results, but also mitigate problems with abuse of p values/NHST. 

      What I mean by "results-blind" manuscript evaluation is as follows:  In deciding whether to publish a manuscript, weight would be given exclusively to (a) the judged importance of the research question addressed in the study, typically conveyed in the Introduction section of the manuscript, and (b) the quality of the methodology of the study, including appropriateness of data analysis methods and adequate statistical power, as reported in the Methods section.  Practically, this "results-blind" process might be implemented by a "two stage evaluation process" whereby the editor initially distributes only the Introduction and Methods sections of a submitted manuscript to reviewers for evaluation.  Only after a provisional decision is made by reviewers for tentative acceptance (or with only minor revision) based on the Introduction & Methods, would the remainder of the paper (Results, Discussion, Abstract) then be distributed to the reviewers for final confirmation of acceptance, & editing suggestions, barring nothing new being discovered in the remainder of the paper that reflects so poorly on the methodology as to contravene the initial acceptance. Journal guidelines to authors would make them fully aware of this practice in reviewing manuscripts.  

      This "two stage evaluation" would have some of the same beneficial effects as "pre-registered" studies, except that it would have the additional benefit of requiring no new burden or change on the part of the authors in submitting manuscripts.  This would also mitigate the problem of researchers not submitting manuscripts for publication because they didn't find significant results, or over-reliance on NHST & p values, or conscious/unconscious cheating to get p values < 0.05, since the authors know their results will have no bearing on determining publication.  Only methodology matters, including their proposed data analysis methods, described & justified in the Methods section.  This would also presumably have a beneficial effect on the "reproducibility crisis" in research. 

      With such a policy, manuscripts would be evaluated on the quality of their methodology, not what they happened to find.  If "no effect" was found, but the methodology was sound, don't we want to know this in order to avoid a positive bias in the literature?   

       I'd be interested in anyone's opinion on this.  Is there something I've missed?  Is it impractical in some way?  Am I being naive about something like this being broadly adopted?  Would it be too difficult politically to implement?  

       Thanks.  

 

Joseph J. Locascio, Ph.D.,

Bio-Statistician,
Neurology Dept.,
Massachusetts General Hospital,
Boston, Massachusetts 
Phone: (617) 724-7192
Email: JLocascio@partners.org        
        

           
"The information transmitted in this email is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this email in error, please contact the sender and delete the material from any computer."

 

 

 

The information in this e-mail is intended only for the person to whom it is
addressed. If you believe this e-mail was sent to you in error and the e-mail
contains patient information, please contact the Partners Compliance HelpLine at
http://www.partners.org/complianceline . If the e-mail was sent to you in error
but does not contain patient information, please contact the sender and properly
dispose of the e-mail.

Confirming what is already well-accepted is often not worth publishing, but a solid refutation of current thinking should be published.

------------------------------
Emil M Friedman, PhD
emilfriedman@gmail.com
http://www.statisticalconsulting.org
------------------------------

Original Message:
Sent: 04-10-2017 13:25
From: Joseph Locascio
Subject: Results Blind Manuscript Evaluation

4/10/17

 

Dear ASA Statisticians,

      In connection with occasional comments in ASA Connect about problems pertaining to the over-use/abuse of p values & null hypothesis significance testing (NHST), I'd like to run by the following suggestion to get any opinions. (I don't believe I've ever posted this at ASA Connect before).   

      Some statisticians, including myself, have for years been suggesting that biomed/behavioral science journals employ a "results-blind" evaluation process in deciding whether to publish manuscripts.  I've long felt that this might have beneficial effects in reducing, not only the well-known bias of journals to publish reports with positive results, but also mitigate problems with abuse of p values/NHST. 

      What I mean by "results-blind" manuscript evaluation is as follows:  In deciding whether to publish a manuscript, weight would be given exclusively to (a) the judged importance of the research question addressed in the study, typically conveyed in the Introduction section of the manuscript, and (b) the quality of the methodology of the study, including appropriateness of data analysis methods and adequate statistical power, as reported in the Methods section.  Practically, this "results-blind" process might be implemented by a "two stage evaluation process" whereby the editor initially distributes only the Introduction and Methods sections of a submitted manuscript to reviewers for evaluation.  Only after a provisional decision is made by reviewers for tentative acceptance (or with only minor revision) based on the Introduction & Methods, would the remainder of the paper (Results, Discussion, Abstract) then be distributed to the reviewers for final confirmation of acceptance, & editing suggestions, barring nothing new being discovered in the remainder of the paper that reflects so poorly on the methodology as to contravene the initial acceptance. Journal guidelines to authors would make them fully aware of this practice in reviewing manuscripts.  

      This "two stage evaluation" would have some of the same beneficial effects as "pre-registered" studies, except that it would have the additional benefit of requiring no new burden or change on the part of the authors in submitting manuscripts.  This would also mitigate the problem of researchers not submitting manuscripts for publication because they didn't find significant results, or over-reliance on NHST & p values, or conscious/unconscious cheating to get p values < 0.05, since the authors know their results will have no bearing on determining publication.  Only methodology matters, including their proposed data analysis methods, described & justified in the Methods section.  This would also presumably have a beneficial effect on the "reproducibility crisis" in research. 

      With such a policy, manuscripts would be evaluated on the quality of their methodology, not what they happened to find.  If "no effect" was found, but the methodology was sound, don't we want to know this in order to avoid a positive bias in the literature?   

       I'd be interested in anyone's opinion on this.  Is there something I've missed?  Is it impractical in some way?  Am I being naive about something like this being broadly adopted?  Would it be too difficult politically to implement?  

       Thanks.  

 

Joseph J. Locascio, Ph.D.,

Bio-Statistician,
Neurology Dept.,
Massachusetts General Hospital,
Boston, Massachusetts 
Phone: (617) 724-7192
Email: JLocascio@partners.org        
        

           
"The information transmitted in this email is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this email in error, please contact the sender and delete the material from any computer."

 

 

 

The information in this e-mail is intended only for the person to whom it is
addressed. If you believe this e-mail was sent to you in error and the e-mail
contains patient information, please contact the Partners Compliance HelpLine at
http://www.partners.org/complianceline . If the e-mail was sent to you in error
but does not contain patient information, please contact the sender and properly
dispose of the e-mail.

Interesting, good to see these ideas all, but when you discuss "... replication studies or exploratory research..." and "clinical utility," are you concentrating only on papers with statistical methodology in support of physical experiments, say a medical study, for example, but not all kinds of statistical papers, in all journals and proceedings, or am I missing something?   It does not seem that one size fits all, nor that it should. 
    
If the idea is to break up institutionalized narrow thinking, this certainly is worth considering.  Thanks.

------------------------------
James Knaub
Lead Mathematical Statistician
Retired
------------------------------

Original Message:
Sent: 04-10-2017 13:25
From: Joseph Locascio
Subject: Results Blind Manuscript Evaluation

4/10/17

 

Dear ASA Statisticians,

      In connection with occasional comments in ASA Connect about problems pertaining to the over-use/abuse of p values & null hypothesis significance testing (NHST), I'd like to run by the following suggestion to get any opinions. (I don't believe I've ever posted this at ASA Connect before).   

      Some statisticians, including myself, have for years been suggesting that biomed/behavioral science journals employ a "results-blind" evaluation process in deciding whether to publish manuscripts.  I've long felt that this might have beneficial effects in reducing, not only the well-known bias of journals to publish reports with positive results, but also mitigate problems with abuse of p values/NHST. 

      What I mean by "results-blind" manuscript evaluation is as follows:  In deciding whether to publish a manuscript, weight would be given exclusively to (a) the judged importance of the research question addressed in the study, typically conveyed in the Introduction section of the manuscript, and (b) the quality of the methodology of the study, including appropriateness of data analysis methods and adequate statistical power, as reported in the Methods section.  Practically, this "results-blind" process might be implemented by a "two stage evaluation process" whereby the editor initially distributes only the Introduction and Methods sections of a submitted manuscript to reviewers for evaluation.  Only after a provisional decision is made by reviewers for tentative acceptance (or with only minor revision) based on the Introduction & Methods, would the remainder of the paper (Results, Discussion, Abstract) then be distributed to the reviewers for final confirmation of acceptance, & editing suggestions, barring nothing new being discovered in the remainder of the paper that reflects so poorly on the methodology as to contravene the initial acceptance. Journal guidelines to authors would make them fully aware of this practice in reviewing manuscripts.  

      This "two stage evaluation" would have some of the same beneficial effects as "pre-registered" studies, except that it would have the additional benefit of requiring no new burden or change on the part of the authors in submitting manuscripts.  This would also mitigate the problem of researchers not submitting manuscripts for publication because they didn't find significant results, or over-reliance on NHST & p values, or conscious/unconscious cheating to get p values < 0.05, since the authors know their results will have no bearing on determining publication.  Only methodology matters, including their proposed data analysis methods, described & justified in the Methods section.  This would also presumably have a beneficial effect on the "reproducibility crisis" in research. 

      With such a policy, manuscripts would be evaluated on the quality of their methodology, not what they happened to find.  If "no effect" was found, but the methodology was sound, don't we want to know this in order to avoid a positive bias in the literature?   

       I'd be interested in anyone's opinion on this.  Is there something I've missed?  Is it impractical in some way?  Am I being naive about something like this being broadly adopted?  Would it be too difficult politically to implement?  

       Thanks.  

 

Joseph J. Locascio, Ph.D.,

Bio-Statistician,
Neurology Dept.,
Massachusetts General Hospital,
Boston, Massachusetts 
Phone: (617) 724-7192
Email: JLocascio@partners.org        
        

           
"The information transmitted in this email is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this email in error, please contact the sender and delete the material from any computer."

 

 

 

The information in this e-mail is intended only for the person to whom it is
addressed. If you believe this e-mail was sent to you in error and the e-mail
contains patient information, please contact the Partners Compliance HelpLine at
http://www.partners.org/complianceline . If the e-mail was sent to you in error
but does not contain patient information, please contact the sender and properly
dispose of the e-mail.

Original Message------

Interesting, good to see these ideas all, but when you discuss "... replication studies or exploratory research..." and "clinical utility," are you concentrating only on papers with statistical methodology in support of physical experiments, say a medical study, for example, but not all kinds of statistical papers, in all journals and proceedings, or am I missing something?   It does not seem that one size fits all, nor that it should. 
    
If the idea is to break up institutionalized narrow thinking, this certainly is worth considering.  Thanks.

------------------------------
James Knaub
Lead Mathematical Statistician
Retired
------------------------------