The American Statistical Association’s Scientific and Public Affairs advisory committee is releasing guidance on the technical language included in the Family First Prevention Services Act. Last July, the U.S. House of Representatives passed the FFPS Act (H.R. 5456), a bill designed to address problems faced by foster youth, their families, and their care takers, by unanimous consent. This bipartisan bill has also been introduced in the Senate, where it is expected to be approved after the election. The bill seems widely supported—over 400 prominent child advocacy and health organizations have expressed their support and urged movement by the end of this Congress in a letter to leadership in September.
The bill appears to have been strongly influenced by evidence-based policymaking, as it reauthorizes existing evidence-based programs through FY2017 and provides new evidence-based programs focusing on areas like mental health and substance abuse to families in need. While the ASA certainly supports evidence-based policymaking in general, some of the technical language in this particular bill may present some problems. In particular, the phrase “conventional standards of statistical significance” appears several times throughout the bill. This particular wording was likely intended to convey a sense of robust analysis; however, ASA’s SPA advisory committee felt that the language encourages a specific type of statistical analyses at the expense of others that may be more appropriate. To that end, the committee is releasing a statement on the bill. The statement is intended to serve as guidance to legislators in hopes that the technical language in future bills is not as artificially restrictive. Recommendations for replacement phrases that would make clear that modern and powerful statistical methods could be brought to bear in evaluation are also included. ASA’s Science Policy staff has also been in contact with Hill staffers and some of the bill’s authors to bring awareness to the potential problem. Many were unaware that the language might be regarded as restrictive or overly narrow. Even though we are unable to pursue official actions because of the advanced stage of the bill in the legislative process, most seemed sympathetic and open to further discussion.
Part of the response to the bill was driven by the same motivation behind the recent ASA statement on p-values. Last March, after months of concern over the broad use of the p-value as a definitive test of statistical significance in recent decades, the ASA issued a statement on p-values and their context, process, and purpose. This statement, which has been widely read by both the statistical and larger scientific community, contains the first explicit recommendations on statistical methodology and use in the ASA’s 177 year history. In part, the p-value statement reads, “Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold.” There are many other factors to consider when making such decisions, such as experimental design, model assumptions, sample size, and effect size. Some of these factors are mentioned in the FFPS Act, but none carry such explicit meaning as statistical significance. For these reasons, the ASA Scientific and Public Affairs Advisory Committee wished to make clear the wealth of statistical advances that could be brought to the table as evidence in effective evidence-based policymaking.