(updated 11-29-2017)
The ASA was recently asked how it is addressing reproducible research. Responding to the inquiry provided a useful exercise to evaluate what we had done and what we should be doing. We are sharing our thoughts here to ask what we might have missed and to ask for input on what we should be doing. Please share your comments with me through email or in the comments space below.
While JASA has established publication standards to improve research quality and reproducibility and Amstat News published a column by Keith A. Baggerly and Donald A. Berry making the case that journals have a key role to play in making research reproducible, the ASA efforts have been more focused on how more statistical awareness and engagement of statisticians will improve reproducibility across the scientific disciplines. In other words, statistics makes the science better, as we like to say.
We know members of the statistical community have been very active in reproducible research with a goal to improve science. We knew we couldn’t begin to list the extensive work by you but we invite you to also share your work in the comments section of this blog entry.
Here are some of the actions the ASA has been doing to help improve the practice of statistics and thereby improve reproducibility:
- P-Value statement (open access), with more than 220,000 views as of November, 2017
- Symposium on Statistical Inference (slides freely available at the link), October, 2017
- Upcoming open access special issue of The American Statistician: Statistical Inference in the 21st Century: A World Beyond P<0.05
- Recommendations to Funding Agencies for Supporting Reproducible Research, a January 2017 document written by many ASA members
- Statistical Issues Seen in Non-Statistics Proposals, a January 2017 document of the ASA Committee on Funded Research (CFR) to help those seeking federal funding make their proposals more statistically robust.
- Ten Simple Rules for Effective Statistical Practice, a 2016 Journal of Computational Biology publication by statistical community leaders (led by Nancy Reid) that has nearly 200,000 views as of November 2017.
- Five Statisticians Share Insights for Applicants and Reviewers, Peer Review Notes, September, 2016, https://www.csr.nih.gov/CSRPRP/2016/09/statisticians-share-insights-for-applicants-and-reviewers/. An interview with the NIH Center for Scientific Review organized by the ASA CFR
- Working with Science magazine for creation of its Statistics Board of Reviewing Editors
- Similar work with Nature
- Similar work with APS journals
- The ASA CFR organized a 2017 Invited JSM Session featuring top Science, Nature, and NIH officials titled How Funding Agencies and Journals Are Encouraging Reproducible Research and the Role of Statisticians.
- Hosted a Teaching reproducible research webinar in 2016
- Over the last three years, the ASA Committee on Publications has taken a couple of important steps to encourage reproducibility and replication of research in statistics. First, each of the ASA journals has now modified its submission policy to request authors to submit their data and code before any article can be published. Authors are asked to indicate if they will do so and to explain if they are not able to do this. Journal editors are encouraged to take this into account in their decision making. Second, as noted above, the Application and Case Studies Section of the Journal of the American Statistical Association has begun a pilot program with associate editors / reviewers focused on reproducibility/replication. The data analysis results from each article in the section are reviewed by an individual focused completely on the ability of the results to be replicated.
We were also really pleased to see the National Academies Committee on Applied and Theoretical Statistics hos its 2015 workshop, Statistical Challenges in Assessing and Fostering the Reproducibility of Scientific Results. And of course we were gratified by the involvement of statisticians in this 2017 NSF workshops
I look forward to your thoughts.
Ron