[2/10/15 update: See also this February Amstat News article: GAO to FBI: More Focus on Statistics]
The U.S. Government Accountability Office (GAO) today released its report on the scientific approaches in the FBI investigation of the 2001 anthrax attacks. The GAO summary of the report, which is titled, "ANTHRAX: Agency Approaches to Validation and Statistical Analyses Could Be Improved," states the genetic tests conducted by the four FBI contractors "were generally scientifically verified and validated, and met the FBI's criteria" but they also found considerable variation in the approaches used by the four contractors, which could have been avoided. "The GAO found that the FBI lacked a comprehensive approach—or framework—that could have ensured standardization of the testing process." The summary also provides the GAO recommendation "that the FBI develop a framework for validation and statistical approaches for future investigations," with which the FBI agreed.
The GAO asked the ASA Ad-Hoc Advisory Committee on
Forensic Science for its assistance in reviewing statistical aspects of the GAO's planned methodology for their investigation and to review a draft of the final report. The role of the ASA committee is described in the final report:
Finally, we asked a subcommittee of the American Statistical Association’s (ASA) Ad-hoc Advisory Committee on Forensic Science for its review and comment on the statistical aspects of a draft of our report. The subcommittee provided us with detailed comments that expressed general agreement with the statistical aspects of the draft, suggested changes to terminology related to the frequency with which microbial properties are present in a population, and suggested appropriate caveats and limitations to analyses we conducted. We incorporated these comments as appropriate throughout the report. (p. 65)
The GAO report also notes the FBI Laboratory's consultation of the full ASA forensic science committee. In a section headed, "The FBI Is Addressing the Need for Formal Statistical Expertise In Future Investigations," the report reads
An FBI official told us that since the 2009 NRC report, the FBI has been
building formal forensic statistical expertise both internally and externally. For example, he said that the FBI laboratory division had created an internal statistical working group to examine the FBI’s statistical needs in its forensic methods. The group included a professor of statistics visiting for 6 months to examine the statistical questions related to patterns, such as fingerprints, and also other science, such as chemistry and explosives. Additionally, the FBI has established a working relationship with members of the American Statistical Association’s Ad-Hoc Advisory Committee on Forensic Science in order to discuss its statistical capacity. The FBI has also worked with other agencies to identify areas of statistical research needed for future investigations.
To read about more about ASA forensic science committee's work on Forensic Science reform, see this web page, the 11/7/14 ASA Community blog entry, Statisticians Well-Represented on Forensic Science Reform Oversight Body, and the many blog entries linked therein.
To see the NYT's article on the report by William Broad, see Inquiry in Anthrax Mailings Had Gaps, Report Says.
See other ASA Science Policy blog entries. For ASA science policy updates, follow @ASA_SciPol on Twitter.