I think there’s a serious difficulty here.
Not long ago, practices of this sort were quite common in many parts of academia. They and similar practices are still common in many areas of commerce. For example, Nicholas Taleb wrote that it is common for financial companies to start large numbers of funds and shut down the unprofitable ones. Since the only funds whose five-year profitability gets reported are the ones in existence for five years, survivorship bias means reported five-year profitability consistently exceeds what an investor could expect to achieve. This is routine, customary, universally accepted, considered covered by the standard disclaimer ( “Past results do not predict future performance.”)
The legal standard to prove fraud has historically been quite high, with the burden on the plaintiffs to prove falsity. The null hypothesis is essentially that challenged statements are true. The burden’s direction is the opposite of scientific settings, the pharmaceutical industry, etc., where claims are presumed false absent evidence.
I think the social and legal context requires us to exercise care and explain things carefully in attempting to communicate often unintuitive concepts to others and effect social change. I don’t, in all candor, believe that laughing at others’ alleged stupidity is going to help uschange things.
The New York Times ran an article a few weeks ago about Professor Amy Cuddy, formerly of Harvard, who used methods which were common enough a few years ago but recently came to be discredited as introducing potential for bias. I think we would agree that Professor Cuddy’s methods were indeed susceptible to the criticism given them, yet this does not make her a fraud. Orderly change requires education and voluntary acceptance based on reason and persuasion. Attempting to introduce change by insult, shaming, authority, and force may be the way of some political circles, but it is not the way of science, even when we are convinced we are right and others wrong. In addition, I think the judge has a point that too low a standard for fraud could risk dampening legitimate scientific debate, or having courts police matters better decided by peer review.
It wouldn’t be unreasonable, if the case is appealed, for statisticians to attempt to clarify the issue for the appeal and better explain the reasoning. Whatever the utility of subgroup analysis in the abstract, if a large enough number of subgroups get analyzed it becomes highly likely that one or more subgroups will be found significant by chance. The number of subgroups involved in the case was large. And it might help to clarify that the issue is not whether the product works in a subgroup. The judge quite correctly found that there is no evidence the product doesn’t work. The question is whether the trial “proves” the product works. And here we need to be careful very about wording and nuance. The trial results might indeed suggest the product works in the indicated subgroups. They merely don’t prove it.
I think the judge’s concern about practices long accepted as legitimate suddenly becoming illegal or regarded as fraud overnight is a fair one, as the Amy Cuddy situation illustrates. I also agree care is required to prevent legitimate academic argument from being squelched and general free speech from being infringed. I think these are concerns we ought to take seriously and attempt to address. It might be appropriate to try to move the needle in an incremental, nuanced way, for example limiting claims of outright fraud as a matter for courts rather than peer review to where an especially large number of subgroups are evaluated or mention of their existence is omitted.
But in any event, I think we have to be careful about how we communicate about things like this, and consider how what we do affects others. I don’t think laughing at others’ supposed stupidity is going to help anyone.
Jonathan Siegel
Associate Director Clinical Statistics
Sent from my iPad
Original Message------
Following text is copied from the following URL. Dumbing down here!
http://www.fdalawblog.net/2017/10/district-court-dismisses-ftc-lawsuit-regarding-marketers-of-prevagen-ftc-failed-to-carry-burden/
=== Copied Text Starts Here ===
District Court Dismisses FTC Lawsuit Regarding Marketers of Prevagen; FTC Failed to Carry Burden
October 18, 2017By Riëtte van Laack -In January 2017, the Federal Trade Commission (FTC) and New York Attorney General (collectively, "Plaintiffs") announced a lawsuit charging that Quincy Bioscience, related entities, and two individuals (collectively "Defendants"), the marketers of the dietary supplement Prevagen, made false and unsubstantiated claims that Prevagen improved memory. Challenged claims included "Prevagen improves memory," Prevagen "has been clinically shown to improve memory," and the claim that "A landmark double-blind and placebo controlled trial demonstrated Prevagen improved short-term memory, learning, and delayed recall over 90 days." Plaintiffs alleged that Defendants relied on a single study that failed to show that Prevagen works better that a placebo on any measure of cognition.
Defendants moved to dismiss the complaint on several grounds, including the argument that Plaintiffs' Complaint failed to adequately allege that the advertising claims for Prevagen violate the FTC Act. On September 28, 2017, the District Court of the Southern District of New York granted the motion.
As described in the decision, Defendants claims were based on a clinical study that met the "gold standard," i.e., a double blind, placebo controlled human clinical study using objective outcome measures using 218 subjects." Plaintiffs did not take issue with the design of the study. However, they did take issue with the Defendants' analyses of that study in support of its marketing claims. Namely, after the study failed to show a statistically significant improvement in the experimental group over the placebo group, Defendants conducted a "number" (more than 30) post hoc analyses of the results, looking at data for smaller subgroups. Doing so, they did find some statistically significant differences. These statistically significant results provided the basis for Defendants' challenged marketing claims. Plaintiffs argued that this post hoc subgroup analysis did not constitute valid support for the claim because the post hoc analyses increased the probability of finding a significant improvement in a subgroup for one of the parameters. They argued that '[g]iven the sheer number of comparisons run and the fact that they were post hoc, the few positive findings on isolated tasks for small subgroups of the study population do not provide reliable evidence of a treatment effect.' In other words, according to the Plaintiffs, the finding of a significant difference was merely the result of the number of analyses; if you throw a dice often enough, you will get the result you want.
The Court rejected Plaintiffs' arguments, and concluded that their challenge to Defendants' substantiation was theoretical. They had no evidence that the claim was not supported, but only showed there was a possibility that the study results did not support Defendants' claims. This mere possibility rather than plausibility did not entitle Plaintiffs to relief.
Not surprising, FTC has not given much publicity to this loss. Plaintiffs may be considering whether to appeal the decision. We will be monitoring further developments.
=== Copied Text Ends Here ===
The title of this post was inspired by this joke:
"A man is wandering through a street market and comes to a booth with a
sign "Smart Pills - $10 a Bottle". He reasons that he could stand to be
a bit smarter so he gives $10 to the seller. He then decides to take
one immediately and pops one of the little brown pills in his mouth. He
exclaims "Hey, these are rabbit t*rds!", to which the seller replies
"See, you're getting smarter already!""
--