ASA Connect

Some time ago, we discussed the question whether reporting results of a post-hoc subgroup analysis without clearly indicating the nature of the analysis performed and the exploratory nature of the results is illegal and whether it is or should be made a crime.

At the time, my view was that, understanding that the practice can be highly misleading and lead people to action based on error and noise, the practice is unfortunately so entrenched that as a pragmatic matter it would probably need to be criminalized explicitly. We are positively inundated with chance phenomena presented as meaningful, and with subgroups presented as significant when they are a actually selected from large frames of them in a manner which goes unreported.  To pick one example of many, just look at how financial funds are rated. We see lots of funds with good-looking 5-year performance ratings. But the only funds that receive 5-year performance rates are those that have survived 5 years. And the only funds that survive 5 years are those that have performed reasonably well, quite possibly by chance.  We never see the funds that are closed. 

While I don't know the outcome of the upcoming elections, it is possible that we will have a Congress interested in revisiting and clarifying the trade practice laws.

If so, this might be the sort of thing that the ASA should take a policy position on. 

I understand that the ASA does not like to take positions on statistical practice. But there are certain practices that are so egregious that explicit clarification of the law might be in everyone's interest. 

Some time ago, we discussed the question whether reporting results of a post-hoc subgroup analysis without clearly indicating the nature of the analysis performed and the exploratory nature of the results is illegal and whether it is or should be made a crime.

At the time, my view was that, understanding that the practice can be highly misleading and lead people to action based on error and noise, the practice is unfortunately so entrenched that as a pragmatic matter it would probably need to be criminalized explicitly. We are positively inundated with chance phenomena presented as meaningful, and with subgroups presented as significant when they are a actually selected from large frames of them in a manner which goes unreported.  To pick one example of many, just look at how financial funds are rated. We see lots of funds with good-looking 5-year performance ratings. But the only funds that receive 5-year performance rates are those that have survived 5 years. And the only funds that survive 5 years are those that have performed reasonably well, quite possibly by chance.  We never see the funds that are closed. 

While I don't know the outcome of the upcoming elections, it is possible that we will have a Congress interested in revisiting and clarifying the trade practice laws.

If so, this might be the sort of thing that the ASA should take a policy position on. 

I understand that the ASA does not like to take positions on statistical practice. But there are certain practices that are so egregious that explicit clarification of the law might be in everyone's interest. 

Some time ago, we discussed the question whether reporting results of a post-hoc subgroup analysis without clearly indicating the nature of the analysis performed and the exploratory nature of the results is illegal and whether it is or should be made a crime.

At the time, my view was that, understanding that the practice can be highly misleading and lead people to action based on error and noise, the practice is unfortunately so entrenched that as a pragmatic matter it would probably need to be criminalized explicitly. We are positively inundated with chance phenomena presented as meaningful, and with subgroups presented as significant when they are a actually selected from large frames of them in a manner which goes unreported.  To pick one example of many, just look at how financial funds are rated. We see lots of funds with good-looking 5-year performance ratings. But the only funds that receive 5-year performance rates are those that have survived 5 years. And the only funds that survive 5 years are those that have performed reasonably well, quite possibly by chance.  We never see the funds that are closed. 

While I don't know the outcome of the upcoming elections, it is possible that we will have a Congress interested in revisiting and clarifying the trade practice laws.

If so, this might be the sort of thing that the ASA should take a policy position on. 

I understand that the ASA does not like to take positions on statistical practice. But there are certain practices that are so egregious that explicit clarification of the law might be in everyone's interest. 

To clarify, the illegal act would be, not performing post-box subgroup analyses, but starting with a large number of groups, selecting the small number that come out as statistically significant, and then making claims based on representing those as the only ones that were looked at.

The FTC currently has a legal case based on a theory that the practice represents an unfair and deceptive trade practice. In FTC v. Quincy Bioscience Holding Company, the FTC is claiming that after a failed study, Quincy Bioscience conducted post-god analyses of over 30 subgroups, found no statistical significant beneficial effect in the vast majority of these subgroups, but the results showed "a few isolated subgroups and tasks" in which the study showed statistically significant improvement. According to the complaint, the company then conducted a marketing campaign representing these findings as the results of the study, I.e. the study showed that the product improved memory in the specific subgroups and tasks for which the post-box analysis had shown statistically significant results.

Originally, criminal charges were brought. But the judge dismissed the charges on grounds that, basically, post-hoc subgroup analysis is so commonly done as not to make clear beyond a reasonable doubt that it is criminally misleading. The FTC then brought a civil suit,  which is pending. 

What do we make of all this? The case should make clear that it is not simply the act of doing post-hoc subgroup analyses that's at issue. The problem is representing those analyses as the study's results, as if they had been the study's purpose all along. These conditions should clearly insulate liability anyone who acknowledges both the post-hoc nature of the analyses and the existence of a larger set of things they looked at from which the significant results were cherry-picked. 

The case also makes clear that there is no clear law on the subject, and the courts are in debate about what to do about it. 

Perhaps the ASA should have a position.

The 2nd Circuit decided an appeal on narrow grounds and sent the issue back to the trial court.

As this is a legal matter, I'd better reiterate that the views here are mine alone and not those of my employer.

The FTC's argument, which explains why it thinks this type of representation represents an unfair and deceptive trade practice, is here:

https://www.ftc.gov/system/files/documents/cases/quincy_bioscience_ca2_ftc_brief_special_appendix_2018-0228.pdf

Some time ago, we discussed the question whether reporting results of a post-hoc subgroup analysis without clearly indicating the nature of the analysis performed and the exploratory nature of the results is illegal and whether it is or should be made a crime.

At the time, my view was that, understanding that the practice can be highly misleading and lead people to action based on error and noise, the practice is unfortunately so entrenched that as a pragmatic matter it would probably need to be criminalized explicitly. We are positively inundated with chance phenomena presented as meaningful, and with subgroups presented as significant when they are a actually selected from large frames of them in a manner which goes unreported.  To pick one example of many, just look at how financial funds are rated. We see lots of funds with good-looking 5-year performance ratings. But the only funds that receive 5-year performance rates are those that have survived 5 years. And the only funds that survive 5 years are those that have performed reasonably well, quite possibly by chance.  We never see the funds that are closed. 

While I don't know the outcome of the upcoming elections, it is possible that we will have a Congress interested in revisiting and clarifying the trade practice laws.

If so, this might be the sort of thing that the ASA should take a policy position on. 

I understand that the ASA does not like to take positions on statistical practice. But there are certain practices that are so egregious that explicit clarification of the law might be in everyone's interest. 

To clarify, the illegal act would be, not performing post-box subgroup analyses, but starting with a large number of groups, selecting the small number that come out as statistically significant, and then making claims based on representing those as the only ones that were looked at.

The FTC currently has a legal case based on a theory that the practice represents an unfair and deceptive trade practice. In FTC v. Quincy Bioscience Holding Company, the FTC is claiming that after a failed study, Quincy Bioscience conducted post-god analyses of over 30 subgroups, found no statistical significant beneficial effect in the vast majority of these subgroups, but the results showed "a few isolated subgroups and tasks" in which the study showed statistically significant improvement. According to the complaint, the company then conducted a marketing campaign representing these findings as the results of the study, I.e. the study showed that the product improved memory in the specific subgroups and tasks for which the post-box analysis had shown statistically significant results.

Originally, criminal charges were brought. But the judge dismissed the charges on grounds that, basically, post-hoc subgroup analysis is so commonly done as not to make clear beyond a reasonable doubt that it is criminally misleading. The FTC then brought a civil suit,  which is pending. 

What do we make of all this? The case should make clear that it is not simply the act of doing post-hoc subgroup analyses that's at issue. The problem is representing those analyses as the study's results, as if they had been the study's purpose all along. These conditions should clearly insulate liability anyone who acknowledges both the post-hoc nature of the analyses and the existence of a larger set of things they looked at from which the significant results were cherry-picked. 

The case also makes clear that there is no clear law on the subject, and the courts are in debate about what to do about it. 

Perhaps the ASA should have a position.

The 2nd Circuit decided an appeal on narrow grounds and sent the issue back to the trial court.

As this is a legal matter, I'd better reiterate that the views here are mine alone and not those of my employer.

The FTC's argument, which explains why it thinks this type of representation represents an unfair and deceptive trade practice, is here:

https://www.ftc.gov/system/files/documents/cases/quincy_bioscience_ca2_ftc_brief_special_appendix_2018-0228.pdf

Some time ago, we discussed the question whether reporting results of a post-hoc subgroup analysis without clearly indicating the nature of the analysis performed and the exploratory nature of the results is illegal and whether it is or should be made a crime.

At the time, my view was that, understanding that the practice can be highly misleading and lead people to action based on error and noise, the practice is unfortunately so entrenched that as a pragmatic matter it would probably need to be criminalized explicitly. We are positively inundated with chance phenomena presented as meaningful, and with subgroups presented as significant when they are a actually selected from large frames of them in a manner which goes unreported.  To pick one example of many, just look at how financial funds are rated. We see lots of funds with good-looking 5-year performance ratings. But the only funds that receive 5-year performance rates are those that have survived 5 years. And the only funds that survive 5 years are those that have performed reasonably well, quite possibly by chance.  We never see the funds that are closed. 

While I don't know the outcome of the upcoming elections, it is possible that we will have a Congress interested in revisiting and clarifying the trade practice laws.

If so, this might be the sort of thing that the ASA should take a policy position on. 

I understand that the ASA does not like to take positions on statistical practice. But there are certain practices that are so egregious that explicit clarification of the law might be in everyone's interest. 

To clarify, the illegal act would be, not performing post-box subgroup analyses, but starting with a large number of groups, selecting the small number that come out as statistically significant, and then making claims based on representing those as the only ones that were looked at.

The FTC currently has a legal case based on a theory that the practice represents an unfair and deceptive trade practice. In FTC v. Quincy Bioscience Holding Company, the FTC is claiming that after a failed study, Quincy Bioscience conducted post-god analyses of over 30 subgroups, found no statistical significant beneficial effect in the vast majority of these subgroups, but the results showed "a few isolated subgroups and tasks" in which the study showed statistically significant improvement. According to the complaint, the company then conducted a marketing campaign representing these findings as the results of the study, I.e. the study showed that the product improved memory in the specific subgroups and tasks for which the post-box analysis had shown statistically significant results.

Originally, criminal charges were brought. But the judge dismissed the charges on grounds that, basically, post-hoc subgroup analysis is so commonly done as not to make clear beyond a reasonable doubt that it is criminally misleading. The FTC then brought a civil suit,  which is pending. 

What do we make of all this? The case should make clear that it is not simply the act of doing post-hoc subgroup analyses that's at issue. The problem is representing those analyses as the study's results, as if they had been the study's purpose all along. These conditions should clearly insulate liability anyone who acknowledges both the post-hoc nature of the analyses and the existence of a larger set of things they looked at from which the significant results were cherry-picked. 

The case also makes clear that there is no clear law on the subject, and the courts are in debate about what to do about it. 

Perhaps the ASA should have a position.

The 2nd Circuit decided an appeal on narrow grounds and sent the issue back to the trial court.

As this is a legal matter, I'd better reiterate that the views here are mine alone and not those of my employer.

The FTC's argument, which explains why it thinks this type of representation represents an unfair and deceptive trade practice, is here:

https://www.ftc.gov/system/files/documents/cases/quincy_bioscience_ca2_ftc_brief_special_appendix_2018-0228.pdf

Some time ago, we discussed the question whether reporting results of a post-hoc subgroup analysis without clearly indicating the nature of the analysis performed and the exploratory nature of the results is illegal and whether it is or should be made a crime.

At the time, my view was that, understanding that the practice can be highly misleading and lead people to action based on error and noise, the practice is unfortunately so entrenched that as a pragmatic matter it would probably need to be criminalized explicitly. We are positively inundated with chance phenomena presented as meaningful, and with subgroups presented as significant when they are a actually selected from large frames of them in a manner which goes unreported.  To pick one example of many, just look at how financial funds are rated. We see lots of funds with good-looking 5-year performance ratings. But the only funds that receive 5-year performance rates are those that have survived 5 years. And the only funds that survive 5 years are those that have performed reasonably well, quite possibly by chance.  We never see the funds that are closed. 

While I don't know the outcome of the upcoming elections, it is possible that we will have a Congress interested in revisiting and clarifying the trade practice laws.

If so, this might be the sort of thing that the ASA should take a policy position on. 

I understand that the ASA does not like to take positions on statistical practice. But there are certain practices that are so egregious that explicit clarification of the law might be in everyone's interest. 

Some time ago, we discussed the question whether reporting results of a post-hoc subgroup analysis without clearly indicating the nature of the analysis performed and the exploratory nature of the results is illegal and whether it is or should be made a crime.

At the time, my view was that, understanding that the practice can be highly misleading and lead people to action based on error and noise, the practice is unfortunately so entrenched that as a pragmatic matter it would probably need to be criminalized explicitly. We are positively inundated with chance phenomena presented as meaningful, and with subgroups presented as significant when they are a actually selected from large frames of them in a manner which goes unreported.  To pick one example of many, just look at how financial funds are rated. We see lots of funds with good-looking 5-year performance ratings. But the only funds that receive 5-year performance rates are those that have survived 5 years. And the only funds that survive 5 years are those that have performed reasonably well, quite possibly by chance.  We never see the funds that are closed. 

While I don't know the outcome of the upcoming elections, it is possible that we will have a Congress interested in revisiting and clarifying the trade practice laws.

If so, this might be the sort of thing that the ASA should take a policy position on. 

I understand that the ASA does not like to take positions on statistical practice. But there are certain practices that are so egregious that explicit clarification of the law might be in everyone's interest. 

Some time ago, we discussed the question whether reporting results of a post-hoc subgroup analysis without clearly indicating the nature of the analysis performed and the exploratory nature of the results is illegal and whether it is or should be made a crime.

At the time, my view was that, understanding that the practice can be highly misleading and lead people to action based on error and noise, the practice is unfortunately so entrenched that as a pragmatic matter it would probably need to be criminalized explicitly. We are positively inundated with chance phenomena presented as meaningful, and with subgroups presented as significant when they are a actually selected from large frames of them in a manner which goes unreported.  To pick one example of many, just look at how financial funds are rated. We see lots of funds with good-looking 5-year performance ratings. But the only funds that receive 5-year performance rates are those that have survived 5 years. And the only funds that survive 5 years are those that have performed reasonably well, quite possibly by chance.  We never see the funds that are closed. 

While I don't know the outcome of the upcoming elections, it is possible that we will have a Congress interested in revisiting and clarifying the trade practice laws.

If so, this might be the sort of thing that the ASA should take a policy position on. 

I understand that the ASA does not like to take positions on statistical practice. But there are certain practices that are so egregious that explicit clarification of the law might be in everyone's interest. 

Some time ago, we discussed the question whether reporting results of a post-hoc subgroup analysis without clearly indicating the nature of the analysis performed and the exploratory nature of the results is illegal and whether it is or should be made a crime.

At the time, my view was that, understanding that the practice can be highly misleading and lead people to action based on error and noise, the practice is unfortunately so entrenched that as a pragmatic matter it would probably need to be criminalized explicitly. We are positively inundated with chance phenomena presented as meaningful, and with subgroups presented as significant when they are a actually selected from large frames of them in a manner which goes unreported.  To pick one example of many, just look at how financial funds are rated. We see lots of funds with good-looking 5-year performance ratings. But the only funds that receive 5-year performance rates are those that have survived 5 years. And the only funds that survive 5 years are those that have performed reasonably well, quite possibly by chance.  We never see the funds that are closed. 

While I don't know the outcome of the upcoming elections, it is possible that we will have a Congress interested in revisiting and clarifying the trade practice laws.

If so, this might be the sort of thing that the ASA should take a policy position on. 

I understand that the ASA does not like to take positions on statistical practice. But there are certain practices that are so egregious that explicit clarification of the law might be in everyone's interest. 

Some time ago, we discussed the question whether reporting results of a post-hoc subgroup analysis without clearly indicating the nature of the analysis performed and the exploratory nature of the results is illegal and whether it is or should be made a crime.

At the time, my view was that, understanding that the practice can be highly misleading and lead people to action based on error and noise, the practice is unfortunately so entrenched that as a pragmatic matter it would probably need to be criminalized explicitly. We are positively inundated with chance phenomena presented as meaningful, and with subgroups presented as significant when they are a actually selected from large frames of them in a manner which goes unreported.  To pick one example of many, just look at how financial funds are rated. We see lots of funds with good-looking 5-year performance ratings. But the only funds that receive 5-year performance rates are those that have survived 5 years. And the only funds that survive 5 years are those that have performed reasonably well, quite possibly by chance.  We never see the funds that are closed. 

While I don't know the outcome of the upcoming elections, it is possible that we will have a Congress interested in revisiting and clarifying the trade practice laws.

If so, this might be the sort of thing that the ASA should take a policy position on. 

I understand that the ASA does not like to take positions on statistical practice. But there are certain practices that are so egregious that explicit clarification of the law might be in everyone's interest.