ASA Connect

Donald Berry
Feb 16, 2018 

I'm very interested in communicating statistical results to courts, and communicating statistical ideas and conclusions to people more generally. I've testified in many cases in a variety of areas. I'll give two examples, one that I wrote about and the other in which I testified, although in an unusual way.

The first is O.J. Simpson's murder trial. It's the most important courtroom example in the U.S. where statistics and communicating statistics played critical roles, although in less than a positive way. Internationally, in my view, the most important cases regarding the importance of communicating statistics are the Sally Clark case in the U.K. and Lucia de Berk in the Netherlands. (Google them.) As regards the former, Peter Donnelly's rendition is arguably the most elegant TED talk that exists: <https://www.ted.com/talks/peter_donnelly_shows_how_stats_fool_juries/transcript>

I wrote an article (Berry DA. DNA, Statistics and the Simpson Case. Chance 7(1994)(4):9-12) in advance of the Simpson trial. The article did not predict the result, of course, but it did try to illuminate the ability for courts to understand numbers and what they mean. And that turned out to be a major, possibly defining issue in the actual trial. Here's one of the paragraphs from my article:

Bruce Weir was the statistician for the prosecution in the actual Simpson trial. In his testimony on a Friday he presented some calculations of prevalence of DNA bands in the general population. Pushed by the defense, over the weekend he realized he had made a mistake. He had forgotten a factor of 2. It's the Hardy-Weinberg 2. For example, the probability that a couple's two children are a boy and a girl is approximately ½ x ½ = ¼, right? Well, only if the boy-girl question is something like, the older is a boy and the younger is a girl. If it's just boy-girl in any order then the right answer is twice ¼ or ½. Nothing is statistics is simpler than this. We all make mistakes similar to Weir's, but we usually catch them before we announce them. On Monday Weir testified to his mistake. For example, when he had calculated 1/1600 for the blood on the famous glove being Simpson's, it should have been 1/800.

The defense attorneys had a field day with Weir's mistake. They were unrelenting. They focused on his reliability and not so much on the fact that the jury had been presented with two numbers for the same quantity. See the report in the next day's LA Times: <http://articles.latimes.com/1995-06-27/local/me-17506_1_dna-analysis>.

As in the earlier example, both match probabilities, 1/800 and 1/1600, are incriminating. But the defense attorneys were successful in turning the evidence on its head and worked to convince the jury that they couldn't believe any of it. Apparently, they succeeded.

The other example is athlete doping. In 2011 I represented an Estonian Olympic gold medalist in cross-country skiing at the Court of Arbitration for Sport (CAS) in Lausanne, Switzerland. (Pro bono, by the way.) He had been banned from sport for three years for using human growth hormone in out-of-competition training. Tests had been carried out by the World Anti-Doping Agency (WADA). He appealed the ban to the CAS. For the first time, the ban was overturned. And, as clearly stated in the CAS ruling, the exclusive reason was statistics. (There has been one other ban overturned by CAS since that time. That was a Russian runner whose sample was tested beyond the statute of limitations.)

The CAS hearings are different from most hearings. The CAS agreed with the athlete's request that I be allowed to sit with the CAS hearing panel and ask questions of witnesses from both sides. Krista Fischer and I chronicled our experiences in this article: (Fischer K, Berry DA. Statisticians introduce science to international doping agency: The Andrus Veerpalu case. Chance 27(2014)(3):10-16.) Krista is one of a handful of statisticians in Estonia, the smallest Baltic country at 1.3 million people.

At the hearing the WADA statistician presented the way he determined the decision limit (DL), the measurement above which the athlete is determined to be a doper. He had fewer than 200 cases that had been collected from standard competitions. So some of these values might have been from dopers. So he deleted the highest values! Then he fit a log-normal distribution and found the DL to be the 99.99 percentile of the distribution. So he (and WADA) claimed a false-positive rate for the test of 1 in 10,000.

There were the two critical problems with this method that I pointed out at the hearing ,,, two critical "communications." First I asked the WADA statistician how he could conclude that only 1 in 10,000 non-dopers would have a higher value of the DL when his sample contained fewer than 200 people.

I also posed the other critical problem with his analysis-his rejection of "outliers"-as a question:

At the hearing, Berry asked the statistician representing WADA the following rhetorical question: "So you're saying that you cut off the tail of the distribution and then claimed that it had a small tail, is that correct?"

I could tell by follow-up questions of this witness that one particular member of the CAS really got it … he understood what I was saying. So the fact that the athlete's ban was lifted may have had more to do with this person being on the CAS panel and not my asking these questions.

If you want further reading on the statistical issues of doping: Berry DA. Commentary: The science of doping. Nature 454(2008):692-693. Up until the day the article went to press its title was "The science of doping … or lack thereof." But Nature's editors dropped the last three words from the title under threat of a law suit by WADA. Despite my raking WADA over the coals in this article and in the Chance article with Krista Fischer, I've never been contacted by them regarding how to improve their science. After the Veerpalu debacle WADA did hire some local (mediocre) statisticians in Montreal to endorse what they're doing and, surprise!, they did endorse what WADA had been doing.

Also, here's a news article regarding Veerpalu: <https://www.outsideonline.com/1925761/whats-wrong-world-anti-doping-agency>

If you're in the mood for a bit of humor, a lot of funny things happen in court … but nobody laughs! Here's the funniest anecdote from my courtroom experience. It was the 1980s. I was testifying (also pro bono) for the Attorney General of Minnesota. The State was prosecuting the defendant for operating a marketing business that was in effect a pyramid scheme with a product being sold only to cover up the scam to avoid the laws against pyramids. There was only one other witness for the State, someone from the Attorney General's office who testified that the scam was in fact operating in Minnesota. But the room was packed, mostly with lawyers from the "big city." The judge was Esther Tomljanovich. I was on the stand sitting next to her. The State's attorney began asking questions of me. One question elicited "I object" from one of the big city lawyers. He then explained his objection, talking to me! He said the question didn't deal with mathematics or statistics, which were my areas of competence. He launched into a detailed explanation and not incidentally was doing mathematics and statistics along the way … which ironically happened not to be among his competences. And he finished up addressing me saying, "And therefore it's not mathematics or statistics, is it sir?" Now, I knew place. I turned to the judge. She looked at me and said, "Do you want to rule on this or should I?"

That's the story. Remember I said this was the 1980s. But the rest of the story is kinda cute. She started to say things, looking alternatively at the suitably red-faced lawyer and me. It became clear that she didn't know what to rule. So I spoke up, asking her if it would help for me to say what I understood the State's attorney to be asking. She said "Yes." So I did. And she replied, "Objection overruled!"

I suppose the real "rest of the story" is the result of the case: the State won.

Donald Berry
Feb 16, 2018 

I'm very interested in communicating statistical results to courts, and communicating statistical ideas and conclusions to people more generally. I've testified in many cases in a variety of areas. I'll give two examples, one that I wrote about and the other in which I testified, although in an unusual way.

The first is O.J. Simpson's murder trial. It's the most important courtroom example in the U.S. where statistics and communicating statistics played critical roles, although in less than a positive way. Internationally, in my view, the most important cases regarding the importance of communicating statistics are the Sally Clark case in the U.K. and Lucia de Berk in the Netherlands. (Google them.) As regards the former, Peter Donnelly's rendition is arguably the most elegant TED talk that exists: <https://www.ted.com/talks/peter_donnelly_shows_how_stats_fool_juries/transcript>

I wrote an article (Berry DA. DNA, Statistics and the Simpson Case. Chance 7(1994)(4):9-12) in advance of the Simpson trial. The article did not predict the result, of course, but it did try to illuminate the ability for courts to understand numbers and what they mean. And that turned out to be a major, possibly defining issue in the actual trial. Here's one of the paragraphs from my article:

Bruce Weir was the statistician for the prosecution in the actual Simpson trial. In his testimony on a Friday he presented some calculations of prevalence of DNA bands in the general population. Pushed by the defense, over the weekend he realized he had made a mistake. He had forgotten a factor of 2. It's the Hardy-Weinberg 2. For example, the probability that a couple's two children are a boy and a girl is approximately ½ x ½ = ¼, right? Well, only if the boy-girl question is something like, the older is a boy and the younger is a girl. If it's just boy-girl in any order then the right answer is twice ¼ or ½. Nothing is statistics is simpler than this. We all make mistakes similar to Weir's, but we usually catch them before we announce them. On Monday Weir testified to his mistake. For example, when he had calculated 1/1600 for the blood on the famous glove being Simpson's, it should have been 1/800.

The defense attorneys had a field day with Weir's mistake. They were unrelenting. They focused on his reliability and not so much on the fact that the jury had been presented with two numbers for the same quantity. See the report in the next day's LA Times: <http://articles.latimes.com/1995-06-27/local/me-17506_1_dna-analysis>.

As in the earlier example, both match probabilities, 1/800 and 1/1600, are incriminating. But the defense attorneys were successful in turning the evidence on its head and worked to convince the jury that they couldn't believe any of it. Apparently, they succeeded.

The other example is athlete doping. In 2011 I represented an Estonian Olympic gold medalist in cross-country skiing at the Court of Arbitration for Sport (CAS) in Lausanne, Switzerland. (Pro bono, by the way.) He had been banned from sport for three years for using human growth hormone in out-of-competition training. Tests had been carried out by the World Anti-Doping Agency (WADA). He appealed the ban to the CAS. For the first time, the ban was overturned. And, as clearly stated in the CAS ruling, the exclusive reason was statistics. (There has been one other ban overturned by CAS since that time. That was a Russian runner whose sample was tested beyond the statute of limitations.)

The CAS hearings are different from most hearings. The CAS agreed with the athlete's request that I be allowed to sit with the CAS hearing panel and ask questions of witnesses from both sides. Krista Fischer and I chronicled our experiences in this article: (Fischer K, Berry DA. Statisticians introduce science to international doping agency: The Andrus Veerpalu case. Chance 27(2014)(3):10-16.) Krista is one of a handful of statisticians in Estonia, the smallest Baltic country at 1.3 million people.

At the hearing the WADA statistician presented the way he determined the decision limit (DL), the measurement above which the athlete is determined to be a doper. He had fewer than 200 cases that had been collected from standard competitions. So some of these values might have been from dopers. So he deleted the highest values! Then he fit a log-normal distribution and found the DL to be the 99.99 percentile of the distribution. So he (and WADA) claimed a false-positive rate for the test of 1 in 10,000.

There were the two critical problems with this method that I pointed out at the hearing ,,, two critical "communications." First I asked the WADA statistician how he could conclude that only 1 in 10,000 non-dopers would have a higher value of the DL when his sample contained fewer than 200 people.

I also posed the other critical problem with his analysis-his rejection of "outliers"-as a question:

At the hearing, Berry asked the statistician representing WADA the following rhetorical question: "So you're saying that you cut off the tail of the distribution and then claimed that it had a small tail, is that correct?"

I could tell by follow-up questions of this witness that one particular member of the CAS really got it … he understood what I was saying. So the fact that the athlete's ban was lifted may have had more to do with this person being on the CAS panel and not my asking these questions.

If you want further reading on the statistical issues of doping: Berry DA. Commentary: The science of doping. Nature 454(2008):692-693. Up until the day the article went to press its title was "The science of doping … or lack thereof." But Nature's editors dropped the last three words from the title under threat of a law suit by WADA. Despite my raking WADA over the coals in this article and in the Chance article with Krista Fischer, I've never been contacted by them regarding how to improve their science. After the Veerpalu debacle WADA did hire some local (mediocre) statisticians in Montreal to endorse what they're doing and, surprise!, they did endorse what WADA had been doing.

Also, here's a news article regarding Veerpalu: <https://www.outsideonline.com/1925761/whats-wrong-world-anti-doping-agency>

If you're in the mood for a bit of humor, a lot of funny things happen in court … but nobody laughs! Here's the funniest anecdote from my courtroom experience. It was the 1980s. I was testifying (also pro bono) for the Attorney General of Minnesota. The State was prosecuting the defendant for operating a marketing business that was in effect a pyramid scheme with a product being sold only to cover up the scam to avoid the laws against pyramids. There was only one other witness for the State, someone from the Attorney General's office who testified that the scam was in fact operating in Minnesota. But the room was packed, mostly with lawyers from the "big city." The judge was Esther Tomljanovich. I was on the stand sitting next to her. The State's attorney began asking questions of me. One question elicited "I object" from one of the big city lawyers. He then explained his objection, talking to me! He said the question didn't deal with mathematics or statistics, which were my areas of competence. He launched into a detailed explanation and not incidentally was doing mathematics and statistics along the way … which ironically happened not to be among his competences. And he finished up addressing me saying, "And therefore it's not mathematics or statistics, is it sir?" Now, I knew place. I turned to the judge. She looked at me and said, "Do you want to rule on this or should I?"

That's the story. Remember I said this was the 1980s. But the rest of the story is kinda cute. She started to say things, looking alternatively at the suitably red-faced lawyer and me. It became clear that she didn't know what to rule. So I spoke up, asking her if it would help for me to say what I understood the State's attorney to be asking. She said "Yes." So I did. And she replied, "Objection overruled!"

I suppose the real "rest of the story" is the result of the case: the State won.

A recent essay of Mary Gray's is: ADJUSTING THE ODDS, Chance 30(2), May 2017, http://chance.amstat.org/2017/04/adjusting-the-odds/ It focuses on matters "equity" and "disparate impact," claims about which are decided a thousand times more often in the human resources departments and "diversity and equity" offices of corporations, universities, etc. than in the courts.

Here is one curious excerpt from Gray's essay:

"Litigation is expensive, so banding a group together in what is termed a "class action" case is a useful technique. Unfortunately, because of court decisions, this has become increasingly difficult in recent years. However, even in the case of a single complainant, statistical evidence of similar discriminatory conduct may be helpful; DOZENS, IF NOT HUNDREDS, OF REGRESSION STUDIES OF SALARIES IN HIGHER EDUCATION HAVE SHOWN THAT WOMEN, ON THE AVERAGE, ARE PAID LESS THAN MEN EVEN WHEN MEASURABLE CHARACTERISTICS ARE THE SAME.

And here's a question. If her essay had been subjected to rigorous peer review by experts in this matter, would the sentence I've put in all CAPS been defensible? One sees this sort of claim often in blogs, op eds, non-peer reviewed articles, etc. but I believe it to be false. If ANYONE KNOWS OF A SINGLE STUDY DONE in the last few decades that, taking into account all the major variables known to justifiably determine salary levels of academics, found evidence of sex discrimination, I'd be interested to know of it? Ditto for claims of sex discrimination in hiring.

One of the few outfits that does not tolerate shoddy or superficial statistics on these matters (or a high reliance on anecdotes) is the Cornell Institute for Women in Science (https://www.human.cornell.edu/hd/research/labs/ciws/home). They not long ago had a paper in PNAS that found women applicants for academic positions get preferential consideration.

Similar close analysis of 30 hiring decisions in my own department and of earlier overheated claims in the University of California found no credible statistical evidence of bias. A review of these can be found in:

RACE, SEX, AND FACULTY SEARCHES, DEPARTMENT OF BIOLOGY, SDSU, 1988-2002, WITH COMMENTARY ON POLICIES AND ACTIONS OF THE SDSU ADMINISTRATION. San Diego State University, San Diego CA. 16 pp., 2003 (Now available as: NAS Article, National Association of Scholars, New York NY, September 11, 2017.)

Statisticians who want to get a black belt by helping win big court cases on these issues, might first aspire to getting their green belt by whistle-blowing on the almost universal misuse of statistical data in their own universities.

This is a scary subject given how much confusion there is in the statistics profession and literature over relatively simple statistical issues, such as the meaning of P, the need to fix alphas, when, if ever, to use 1-tailed tests,etc.

When Lombardi and I prepared our monograph on misuse of 1-tailed tests (Lombardi, C.M. and S.H. Hurlbert, 2009. Misprescription and misuse of one-tailed tests. Austral Ecology 34:447-468, Appendix -- ATTACHED) we had to omit, for sake of brevity, the section below, which is now about ten years out of date. Feel free to borrow from, everyone!

ABUSE IN THE COURTS

            Medical research is not the only area where poor decisions have been made as a result of an obsession with the 0.05 level of significance and inappropriate use of 1-tailed tests.   Confusion over 1-tailed testing seems to have had and be having dramatic real consequences for plaintiffs and defendants in the U.S. legal system. Datko (1989) gives an illuminating review of how this issue has influenced litigation on employment discrimination under the 1964 Civil Rights Act of the U.S. Congress. That act "forbids the making of decisions regarding employment on the basis of race, sex, religion and several other factors." Statistical evidence frequently is a major factor determining the disposition of claims brought under this act.

            Let us first list some of the disparate findings by various courts as to when 1- and 2-tailed tests are admissible and then critique the logic of some of Datko's own recommendations.

            One court or another has held: 1) that chance could be ruled out only when "the observed result was two or three standard deviations from the average event"; 2) that 2-tailed tests are more appropriate than 1-tailed in discrimination cases; 3) that 1-tailed tests are appropriate when there is "independent" evidence of intent to discriminate; and 4) that 1-tailed tests can be justified simply because the plaintiff claims to have suffered discrimination. That the courts seem completely befuddled by the issue is perhaps not surprising given that statisticians collectively are as well.

            Datko's review gives a good description of a bad state of affairs. He makes some legitimate points,   but three of his key premises or recommendations are unacceptable. Thus we do not believe he has provided the clarification he had hoped to. First, he believes that  should be fixed and claims that "Most courts and social scientists believe that the minimum level of significance should be .05", citing Hayes (1981) and a specific court case as his authorities. It is a bit vague, but he gives the impression that some cases have indeed been decided on the basis of P values of 0.05 or slightly less. In such legal settings, decisions as to what value of P should be regarded as "conclusive evidence" would seem to be a matter of legal, not statistical, principle. Just as a judgment against a defendant requires stronger evidence in a criminal than in a civil case (e.g. Gray 1993), so the critical level of P should vary according to the nature or import of the fact being tried. That is, 0.05 might be sufficient to establish a minor or subsidiary fact, but it would seem a very loose criterion for determining, solely from statistical evidence, the fact of discrimination itself . After all, if there were, say, 100,000 corporations in the U.S. and if not one were to discriminate against females in employment, one could in fact "prove", using an  = 0.05 that about 5,000 of them do discriminate against females. Wouldn't that make a jolly pile of fees, awards and penalties!   A P value in the neighborhood of 0.05 is perhaps a reasonable basis for reaching a tentative scientific conclusion or for concluding a result is worth publishing, but it would seem a dangerous basis for any highly consequential and non-tentative legal decisions, civil or criminal.

Additionally, we would argue that P values should be considered only in conjunction with the apparent magnitude of the alleged discrimination, as measured by some appropriate index. In recognition that important variables not easily quantified are neglected in such analyses and that the data gathered and methods used also have their uncertainties, it would make sense to require that this apparent magnitude exceed a certain level, before purely statistical evidence of discrimination be judged conclusive. At one point the U.S. government "agency charged with the administration of Title VII, propounded the so-called four-fifths rule: a selection rate for minorities less than four-fifths of the highest selection rate is taken to constitute evidence of discrimination. Courts were never very enamored of this rule, and statisticians soon convinced them to substitute a measure of statistical significance" (Gray 1993). It is not clear why both P values and effect sizes cannot be taken into account.

            A second fault in Datko's analysis is that he accepts the idea that  be set at 0.05 regardless of the tailedness of the test. This makes no sense, as we have argued earlier. It is the same error made by those who claim that simply by switching from 2- to 1-tailed tests medical experiments can be made less costly and ethically superior, by using fewer patients, with no diminution in the force of the evidence .

            Thirdly, Datko argues, as have some courts, that 1-tailed tests should or can be used if "independent evidence" or "independent proof" of discrimination exists, whereas otherwise a 2-tailed test is appropriate. This again is based on the incorrect premise that the conclusiveness of the evidence, as indicated by a P value, can be judged independently of whether a test was 1- or 2-tailed. At least where the probability distribution of the test statistic is symmetric under the null hypothesis, the force of the evidence is completely unaffected by the tailedness of the test. We also note that the statistical evidence and the other evidence cannot be regarded as logically "independent" when both the decision to carry out a test and the nature of the test selected are determined by the "other" evidence. To his credit, Datko rejects the idea that the claim of a plaintiff is by itself sufficient grounds to justify a 1-tailed test. Unfortunately a well known statistics text for lawyers (Finkelstein and Levin 1990:125) does advocate that idea.

            Despite our disagreements with him, we concur with Datko's (1989) conclusion that this "issue is ripe for hearing and resolution at the U.S. Supreme Court level. The guidelines for the use of statistics in employment discrimination settings is an important topic since statistics are generally crucial and, on occasion, the only evidence produced by plaintiffs to prove discrimination."

Datko, D. 1989. Should the tail wag the dog? One- versus two-tailed statistical tests in Title VII employment discrimination litigation. Capital University Law Review 18:445-461.

Gray, M.W. 1993. Can statistics tell us what we do not want to hear? The case of complex salary structures. Statistical Science 8:144-179.