ASA Connect

Below is an item just published involving the statistical concepts addressed in materials I have posted here frequently, including my letters to ASA of October 8, 2015[1] and July 25, 2016.[2]

“Misunderstanding of Statistics Confounds Analyses of Criminal Justice Issues in Baltimore and Voter ID Issues in Texas and North Carolina,” Federalist Society Blog (Oct. 3, 2016)

http://www.fed-soc.org/blog/detail/misunderstanding-of-statistics-confounds-analyses-of-criminal-justice-issues-in-baltimore-and-voter-id-issues-in-texas-and-north-carolina

The item explains that (a) contrary to the belief reflected in the Department of Justice report on police practices in Baltimore, MD, generally reducing adverse criminal justice outcomes will tend to increase the proportions African Americans make up of persons experiencing those outcomes and (b) contrary to the beliefs reflected in court decisions involving voter IDs requirements, less stringent requirements will tend to produce larger relative racial differences in rates of failing to meet the requirements than more stringent ones.

The latter point should be borne in mind by those intending to participate in the voter ID studies discussed on ASA website.

1. http://jpscanlan.com/images/Letter_to_American_Statistical_Association_Oct._8,_2015_.pdf
2. http://www.jpscanlan.com/images/Letter_to_American_Statistical_Association_July_25,_2016_.pdf

Below is an item just published involving the statistical concepts addressed in materials I have posted here frequently, including my letters to ASA of October 8, 2015[1] and July 25, 2016.[2]

“Misunderstanding of Statistics Confounds Analyses of Criminal Justice Issues in Baltimore and Voter ID Issues in Texas and North Carolina,” Federalist Society Blog (Oct. 3, 2016)

http://www.fed-soc.org/blog/detail/misunderstanding-of-statistics-confounds-analyses-of-criminal-justice-issues-in-baltimore-and-voter-id-issues-in-texas-and-north-carolina

The item explains that (a) contrary to the belief reflected in the Department of Justice report on police practices in Baltimore, MD, generally reducing adverse criminal justice outcomes will tend to increase the proportions African Americans make up of persons experiencing those outcomes and (b) contrary to the beliefs reflected in court decisions involving voter IDs requirements, less stringent requirements will tend to produce larger relative racial differences in rates of failing to meet the requirements than more stringent ones.

The latter point should be borne in mind by those intending to participate in the voter ID studies discussed on ASA website.

1. http://jpscanlan.com/images/Letter_to_American_Statistical_Association_Oct._8,_2015_.pdf
2. http://www.jpscanlan.com/images/Letter_to_American_Statistical_Association_July_25,_2016_.pdf

Below is an item just published involving the statistical concepts addressed in materials I have posted here frequently, including my letters to ASA of October 8, 2015[1] and July 25, 2016.[2]

“Misunderstanding of Statistics Confounds Analyses of Criminal Justice Issues in Baltimore and Voter ID Issues in Texas and North Carolina,” Federalist Society Blog (Oct. 3, 2016)

http://www.fed-soc.org/blog/detail/misunderstanding-of-statistics-confounds-analyses-of-criminal-justice-issues-in-baltimore-and-voter-id-issues-in-texas-and-north-carolina

The item explains that (a) contrary to the belief reflected in the Department of Justice report on police practices in Baltimore, MD, generally reducing adverse criminal justice outcomes will tend to increase the proportions African Americans make up of persons experiencing those outcomes and (b) contrary to the beliefs reflected in court decisions involving voter IDs requirements, less stringent requirements will tend to produce larger relative racial differences in rates of failing to meet the requirements than more stringent ones.

The latter point should be borne in mind by those intending to participate in the voter ID studies discussed on ASA website.

1. http://jpscanlan.com/images/Letter_to_American_Statistical_Association_Oct._8,_2015_.pdf
2. http://www.jpscanlan.com/images/Letter_to_American_Statistical_Association_July_25,_2016_.pdf

Do we spend enough time in our introductory courses on interpreting proportion data? We typically present inferences for one proportion and the difference between two proportions and let it go at that. However when proportion data can be reported in a way as to make good outcomes look bad and bad outcomes look good, as Mr. Scanlan has pointed out on many issues of importance to society, then it is up to us to let our students know about it. Giving a name to a phenomenon, as Tukey often did, may help focus attention on the issue. Let me suggest "Scanlan's paradox" as a possibility.

Below is an item just published involving the statistical concepts addressed in materials I have posted here frequently, including my letters to ASA of October 8, 2015[1] and July 25, 2016.[2]

“Misunderstanding of Statistics Confounds Analyses of Criminal Justice Issues in Baltimore and Voter ID Issues in Texas and North Carolina,” Federalist Society Blog (Oct. 3, 2016)

http://www.fed-soc.org/blog/detail/misunderstanding-of-statistics-confounds-analyses-of-criminal-justice-issues-in-baltimore-and-voter-id-issues-in-texas-and-north-carolina

The item explains that (a) contrary to the belief reflected in the Department of Justice report on police practices in Baltimore, MD, generally reducing adverse criminal justice outcomes will tend to increase the proportions African Americans make up of persons experiencing those outcomes and (b) contrary to the beliefs reflected in court decisions involving voter IDs requirements, less stringent requirements will tend to produce larger relative racial differences in rates of failing to meet the requirements than more stringent ones.

The latter point should be borne in mind by those intending to participate in the voter ID studies discussed on ASA website.

1. http://jpscanlan.com/images/Letter_to_American_Statistical_Association_Oct._8,_2015_.pdf
2. http://www.jpscanlan.com/images/Letter_to_American_Statistical_Association_July_25,_2016_.pdf

Thanks for these comments.  A few responsive observations. 

Regarding Ed Gracely’s point:   I am not making an error by discussing relative rather than absolute differences in rates.  My article is about what the Department of Justice and courts do.  Whether the DOJ and the courts are making an error is another matter. 

But in arguing the greater importance of absolute differences with regard to appraisal of a demographic difference, one must be mindful of the ways absolute differences tend to be affected by the frequency of an outcome, as I explained in a Chance editorial more than decade ago, [1] and as I have explained in scores of places since then.  See, e.g., references 2-4.  In fact, my October 8, 2015 letter to the ASA mentions absolute differences 74 times.  See especially the discussion in reference 3 (at 337-339) about the way that reliance on absolute differences to measure healthcare disparities led Massachusetts to unwisely include a disparities element in its Medicaid pay-for-performance programs and to do so in a way that is likely to result in increased healthcare disparities (regardless of how they are measured). 

Thus, the failure to understand that the two relative differences tend to change in opposite directions as the prevalence of an outcomes (and the fact that so many policies are based on the mistaken belief that reducing the frequency of an outcome will tend to reduce rather increase relative differences) are merely manifestations of the larger failure to recognize patterns by which all standard measures tend to be affected by the frequency of an outcome and implications of those patterns.  

The key consideration for those thinking about broader issues (rather than the particular subject of my article) is that anything said about demographic difference is misleading if it fails to address the way the particular measure employed tends to be affected by the prevalence of the outcome.   

Regarding David Norris’s points:   It may be that many people recognize shortcoming of reliance on large relative differences regarding rare outcome (though they seem all to do so without recognizing that the difference will tend to increase as the outcome becomes even rarer).  But the fact remains that billions of dollars are spent each year studying relative difference in favorable or adverse outcomes (as to numerous issues in the law and the social and medical science) including by highly reputed research institutions, and extremely important policy decisions and law enforcement policies are based on perceptions about relative differences.  All this occurs without any attention to the way the relative difference the observer happens to be examining tends to be affected by the prevalence of an outcome.  And, of course, in cases when the adverse outcome is examined, any understanding an observer may have about effects of the prevalence of an outcome typically is the opposite of reality. 

(Similar issues, of course, apply to reliance on absolute differences, though in the case of absolute differences interpretations cannot turn on whether one examines the favorable or the adverse outcome and there is no widespread mistaken expectation as to the effect of the frequency of an outcome.)  

Mistaken reliance on relative differences predominates even in epidemiology where subgroup effects are identified on the basis a departure from constant relative effect across different baseline rates (an approach that is illogical as well as unsound, as I discuss in reference 3 (at 339-34) and the October 2015 ASA letter (at 12-13). 

In the view of most observers (including me), the number-needed-to treat (NNT), which is a function of the absolute effect of a factor on the outcome rate at issue, is the most important consideration in treatment decisions (though that is still a different matter from whether the absolute difference is a sound measure of association).  But as discussed in the introductory material to reference 5, it is standard practice, as recommended, for example, by Oxford's Center for Evidence Based Medicine and BMJ Clinical Evidence, to employ the relative effect observed in a clinical trial to calculate the NNT in a situation involving different baseline rates from that involved in the trial.  (Both of those entities rejected my suggestion that they modify this guidance, though the UK entity called Patient (patient.info) did appreciatively accept my suggested modifications.)  See also references 6 to 8. 

Not being a statistician, I am not a good person to distinguish between arithmetic and statistics.  It’s all mathematics to me.  But the patterns I describe do not have to do with things like randomness or sampling error (save in that such factors will affect the consistency with which one will observe the patterns I describe).  Rather, the patterns I describe have to do with the fact that, in situations such as that reflected in Figure 1 of reference 9, as an outcome is increasingly restricted toward either end of the overall distribution each standard measure tends to change in a particular way.  One cannot draw conclusions about processes (or about the strength of the associations reflected by a pair of outcome rates) based on those standard measures without distinguishing the extent to which observed patterns are functions of the prevalence of the outcome and the extent to which they reflect something about underlying forces. 

It may be that there are better ways, either as to tone or substance, of articulating my points than found, say, in my October 8, 2015, letter to ASA (or the letters to other entities collected in reference 10 or the workshops collected in reference 11).  I always welcome suggestions.  And I have received some useful ones (including from Dr. Norris).  But it is best when the are focused on particular illustrations that I do use.

Regarding James Higgins’ kind comments about “Scanlan’s paradox”:  In the 2006 Chance editorial I used the term “heuristic rule X” or “HRX” to describe the pattern whereby the two relative difference tend to change in opposite as the prevalence of an outcome changes.  That caused researchers in the UK to term the pattern “Scanlan’s rule.”  That prompted me to create a web page by that name with numerous subpages.  So it is fair to say I promote the usage, though, I hope, not too aggessively.  But a problem with the usage or like shorthand terms for the pattern of relative differences is that the usage may detract from appreciation of the patterns by which absolute differences and odds ratios tend to be affected by the prevalence of an outcome. So there are really two rules, which in workshops I principally refer to as Interpretive Rules 1 and 2 (though the second is pretty complicated).   This is matter of increasing importance as more and more researchers rely on absolute differences to measure healthcare disparities (and various other things that used to be discussed in terms of relative differences) without recognizing the patterns by which absolute differences tend to change as the frequency of an outcome changes – or while mistakenly thinking that because the absolute difference is unaffected by which outcome one examines, it effectively addresses the problems I identify with the two relative differences. 

Finally, a couple of observations about the relationship between the two relative differences and the absolute difference:  As the frequency of an outcome changes, the absolute difference tends to change in the same direction as the smaller relative difference (while the difference measured by the odds ratio tends to change in the opposite direction of the absolute difference and the same direction as the larger relative difference).  See figure 3 (at 22) of the October 2015 ASA letter.  Since in most contexts other than employment testing observers relying on relative differences commonly rely on the larger of the two relative differences (often without any thought to a second relative difference), there is a systematic tendency for observers relying on relative differences to reach opposite conclusion about such things as directions of change over time from observers relying on the absolute difference. 

As discussed in reference 2 and 3 and the ASA October 2015 letter, recognition that the absolute difference and the relative difference the observer happens to be looking at have changed in opposite directions is increasing.  But it is still not widespread.  Many people analyzing demographic differences still seem entirely unaware that  measure other than the one they are using can (much less commonly will or in fact would do so in their study) yield a different results about direction of changes over time from the direction they have identified.  See references 12 and 13 for seemingly extreme situations though they may actually differ little from the standard, as discussed in the concluding pages of reference 3). 

But even where observers have recognized that a relative difference and the absolute difference can yield (or have yielded) different conclusions about such things as directions of changes over time, they have invariably done so without seeming to realize that there is a second relative difference.  This has been the case even though, while all measures may change in the same direction (thus indicating a true change in the strength of the forces causing the outcome rates to differ), when a mentioned relative difference and the absolute difference have changed in different directions, the unmentioned relative difference will necessarily have changed in the opposite direction of the mentioned relative difference and the same direction as the absolute difference.  

1. “Can We Actually Measure Health Disparities?,” Chance (Spring 2006)http://www.jpscanlan.com/images/Can_We_Actually_Measure_Health_Disparities.pdf 

2. “The Mismeasure of Health Disparities,” Journal of Public Health Management and Practice (July/Aug. 2016)http://www.jpscanlan.com/images/The_Mismeasure_of_Health_Disparities_JPHMP_2016_.pdf

3. “Race and Mortality Revisited,” Society (July/Aug. 2014) http://jpscanlan.com/images/Race_and_Mortality_Revisited.pdf

4. “Misunderstanding of Statistics Leads to Misguided Law Enforcement Policies,” Amstat News (Dec. 2012)http://magazine.amstat.org/blog/2012/12/01/misguided-law-enforcement/

5. Subgroup Effects subpage of the Scanlan’s Rule page of jpscanlan.com  http://jpscanlan.com/images/Subgroup_Effects_-_Interaction.pdf

6. Re: The number needed to treat: a clinically useful measure of treatment effect. BMJ (Nov. 21, 2011) http://www.bmj.com/rapid-response/2011/11/21/re-number-needed-treat-clinically-useful-measure-treatment-effect

7. Assumption of constant relative risk reductions across different baseline rates is unsound. CMAJ (Mar. 12, 2012) http://www.cmaj.ca/content/171/4/353.short/reply#cmaj_el_690714
8. Re: Progress research strategy (PROGRESS) 4: Stratified medicine research.  BMJ. (Feb. 25, 2013) http://www.bmj.com/content/346/bmj.e5793/rr/632884
9. “Divining Difference,” Chance (Fall 1994) http://jpscanlan.com/images/Divining_Difference.pdf

10. http://www.jpscanlan.com/measurementletters.html
11. http://jpscanlan.com/publications/conferencepresentations.html
12. http://jpscanlan.com/measuringhealthdisp/spuriouscontradictions.html
13. http://jpscanlan.com/measuringhealthdisp/ahrqsvanderbiltreport.html

Do we spend enough time in our introductory courses on interpreting proportion data? We typically present inferences for one proportion and the difference between two proportions and let it go at that. However when proportion data can be reported in a way as to make good outcomes look bad and bad outcomes look good, as Mr. Scanlan has pointed out on many issues of importance to society, then it is up to us to let our students know about it. Giving a name to a phenomenon, as Tukey often did, may help focus attention on the issue. Let me suggest "Scanlan's paradox" as a possibility.

Below is an item just published involving the statistical concepts addressed in materials I have posted here frequently, including my letters to ASA of October 8, 2015[1] and July 25, 2016.[2]

“Misunderstanding of Statistics Confounds Analyses of Criminal Justice Issues in Baltimore and Voter ID Issues in Texas and North Carolina,” Federalist Society Blog (Oct. 3, 2016)

http://www.fed-soc.org/blog/detail/misunderstanding-of-statistics-confounds-analyses-of-criminal-justice-issues-in-baltimore-and-voter-id-issues-in-texas-and-north-carolina

The item explains that (a) contrary to the belief reflected in the Department of Justice report on police practices in Baltimore, MD, generally reducing adverse criminal justice outcomes will tend to increase the proportions African Americans make up of persons experiencing those outcomes and (b) contrary to the beliefs reflected in court decisions involving voter IDs requirements, less stringent requirements will tend to produce larger relative racial differences in rates of failing to meet the requirements than more stringent ones.

The latter point should be borne in mind by those intending to participate in the voter ID studies discussed on ASA website.

1. http://jpscanlan.com/images/Letter_to_American_Statistical_Association_Oct._8,_2015_.pdf
2. http://www.jpscanlan.com/images/Letter_to_American_Statistical_Association_July_25,_2016_.pdf

Mr Scanlan:

I'll take up your challenge to lay out an approach, in terms of the sort of problem you typically address, which might be more fruitful than lobbying the ASA to instruct the President of the United States (!) on an arithmetical principle.

Typically, it seems your problem settings involve matters of social injustice, and common ways of measuring and responding to them. I suggest you try the following:

1. From among the social injustice situations you have addressed in the past, select a case where you enjoy some common ground with your opponents. This would mean at least (i) agreeing that a social injustice does exist, and (ii) sharing your opponents' conviction that a policy response is morally desirable.
2. Describe the causal factors in said social injustice, and clarify points where you and your opponents agree and disagree in this regard. If there are substantial disagreements about questions of causality, then you might revert to Step 1 and find a different social injustice situation. Or you might just note your points of disagreement but concede the causal picture to your opponents 'for the sake of argument' -- i.e., to enable you to proceed with the development of your arithmetical argument. 
3. Having thus found common ground with your opponents on moral and causal matters, proceed to demonstrate the distinct policy scenarios entailed by your and your opponents' separate arithmetical proclivities. The strongest argument here would persuade your opponents that they would achieve morally superior results by abandoning relative risks in favor of absolute risks as guides of policy and measures of progress against injustice. If you make your argument well, your opponents will evaulate the policy consequences of their own arithmetic, compare the conequences of yours, and realize that yours result in morally superior results.

If you should find you cannot get past Step 1, then you will have discovered that your true grievance is not of an arithmetical nature after all, but of an entirely different sort. In that case, you might do well to abandon an arithmetical principle ('HRX') that has enjoyed no apparent substantive formal development over several decades, as a sterile distraction from the more fundamental argument you ought to be advancing perhaps along philosophical lines.

One of the strongest signs of a sterile theory is that it cannot support technical development. I have no doubt that HRX could be formalized quite readily, and am baffled that you have not pursued this course. If you don't do so in the next month or so, I may take up the matter myself sometime before the end of the year--either as a blog post, or something I submit to Chance or Significance. It is not inconceivable that such development will leave your HRX principle in a state that "puts it beyond use" (as they say in Northern Ireland) for the applications you seem to have been pursuing.

To be sure, I am not dismissing the argument you have been making as utterly uninteresting. I am however deeply suspicious of its failure to develop formally and evolve technically. The sheer amount of verbiage involved is astounding for so straightforward a point.

Kind regards,

Mr Scanlan:

I'll take up your challenge to lay out an approach, in terms of the sort of problem you typically address, which might be more fruitful than lobbying the ASA to instruct the President of the United States (!) on an arithmetical principle.

Typically, it seems your problem settings involve matters of social injustice, and common ways of measuring and responding to them. I suggest you try the following:

1. From among the social injustice situations you have addressed in the past, select a case where you enjoy some common ground with your opponents. This would mean at least (i) agreeing that a social injustice does exist, and (ii) sharing your opponents' conviction that a policy response is morally desirable.
2. Describe the causal factors in said social injustice, and clarify points where you and your opponents agree and disagree in this regard. If there are substantial disagreements about questions of causality, then you might revert to Step 1 and find a different social injustice situation. Or you might just note your points of disagreement but concede the causal picture to your opponents 'for the sake of argument' -- i.e., to enable you to proceed with the development of your arithmetical argument. 
3. Having thus found common ground with your opponents on moral and causal matters, proceed to demonstrate the distinct policy scenarios entailed by your and your opponents' separate arithmetical proclivities. The strongest argument here would persuade your opponents that they would achieve morally superior results by abandoning relative risks in favor of absolute risks as guides of policy and measures of progress against injustice. If you make your argument well, your opponents will evaulate the policy consequences of their own arithmetic, compare the conequences of yours, and realize that yours result in morally superior results.

If you should find you cannot get past Step 1, then you will have discovered that your true grievance is not of an arithmetical nature after all, but of an entirely different sort. In that case, you might do well to abandon an arithmetical principle ('HRX') that has enjoyed no apparent substantive formal development over several decades, as a sterile distraction from the more fundamental argument you ought to be advancing perhaps along philosophical lines.

One of the strongest signs of a sterile theory is that it cannot support technical development. I have no doubt that HRX could be formalized quite readily, and am baffled that you have not pursued this course. If you don't do so in the next month or so, I may take up the matter myself sometime before the end of the year--either as a blog post, or something I submit to Chance or Significance. It is not inconceivable that such development will leave your HRX principle in a state that "puts it beyond use" (as they say in Northern Ireland) for the applications you seem to have been pursuing.

To be sure, I am not dismissing the argument you have been making as utterly uninteresting. I am however deeply suspicious of its failure to develop formally and evolve technically. The sheer amount of verbiage involved is astounding for so straightforward a point.

Kind regards,

Thanks for these comments.  A few responsive observations. 

Regarding Ed Gracely’s point:   I am not making an error by discussing relative rather than absolute differences in rates.  My article is about what the Department of Justice and courts do.  Whether the DOJ and the courts are making an error is another matter. 

But in arguing the greater importance of absolute differences with regard to appraisal of a demographic difference, one must be mindful of the ways absolute differences tend to be affected by the frequency of an outcome, as I explained in a Chance editorial more than decade ago, [1] and as I have explained in scores of places since then.  See, e.g., references 2-4.  In fact, my October 8, 2015 letter to the ASA mentions absolute differences 74 times.  See especially the discussion in reference 3 (at 337-339) about the way that reliance on absolute differences to measure healthcare disparities led Massachusetts to unwisely include a disparities element in its Medicaid pay-for-performance programs and to do so in a way that is likely to result in increased healthcare disparities (regardless of how they are measured). 

Thus, the failure to understand that the two relative differences tend to change in opposite directions as the prevalence of an outcomes (and the fact that so many policies are based on the mistaken belief that reducing the frequency of an outcome will tend to reduce rather increase relative differences) are merely manifestations of the larger failure to recognize patterns by which all standard measures tend to be affected by the frequency of an outcome and implications of those patterns.  

The key consideration for those thinking about broader issues (rather than the particular subject of my article) is that anything said about demographic difference is misleading if it fails to address the way the particular measure employed tends to be affected by the prevalence of the outcome.   

Regarding David Norris’s points:   It may be that many people recognize shortcoming of reliance on large relative differences regarding rare outcome (though they seem all to do so without recognizing that the difference will tend to increase as the outcome becomes even rarer).  But the fact remains that billions of dollars are spent each year studying relative difference in favorable or adverse outcomes (as to numerous issues in the law and the social and medical science) including by highly reputed research institutions, and extremely important policy decisions and law enforcement policies are based on perceptions about relative differences.  All this occurs without any attention to the way the relative difference the observer happens to be examining tends to be affected by the prevalence of an outcome.  And, of course, in cases when the adverse outcome is examined, any understanding an observer may have about effects of the prevalence of an outcome typically is the opposite of reality. 

(Similar issues, of course, apply to reliance on absolute differences, though in the case of absolute differences interpretations cannot turn on whether one examines the favorable or the adverse outcome and there is no widespread mistaken expectation as to the effect of the frequency of an outcome.)  

Mistaken reliance on relative differences predominates even in epidemiology where subgroup effects are identified on the basis a departure from constant relative effect across different baseline rates (an approach that is illogical as well as unsound, as I discuss in reference 3 (at 339-34) and the October 2015 ASA letter (at 12-13). 

In the view of most observers (including me), the number-needed-to treat (NNT), which is a function of the absolute effect of a factor on the outcome rate at issue, is the most important consideration in treatment decisions (though that is still a different matter from whether the absolute difference is a sound measure of association).  But as discussed in the introductory material to reference 5, it is standard practice, as recommended, for example, by Oxford's Center for Evidence Based Medicine and BMJ Clinical Evidence, to employ the relative effect observed in a clinical trial to calculate the NNT in a situation involving different baseline rates from that involved in the trial.  (Both of those entities rejected my suggestion that they modify this guidance, though the UK entity called Patient (patient.info) did appreciatively accept my suggested modifications.)  See also references 6 to 8. 

Not being a statistician, I am not a good person to distinguish between arithmetic and statistics.  It’s all mathematics to me.  But the patterns I describe do not have to do with things like randomness or sampling error (save in that such factors will affect the consistency with which one will observe the patterns I describe).  Rather, the patterns I describe have to do with the fact that, in situations such as that reflected in Figure 1 of reference 9, as an outcome is increasingly restricted toward either end of the overall distribution each standard measure tends to change in a particular way.  One cannot draw conclusions about processes (or about the strength of the associations reflected by a pair of outcome rates) based on those standard measures without distinguishing the extent to which observed patterns are functions of the prevalence of the outcome and the extent to which they reflect something about underlying forces. 

It may be that there are better ways, either as to tone or substance, of articulating my points than found, say, in my October 8, 2015, letter to ASA (or the letters to other entities collected in reference 10 or the workshops collected in reference 11).  I always welcome suggestions.  And I have received some useful ones (including from Dr. Norris).  But it is best when the are focused on particular illustrations that I do use.

Regarding James Higgins’ kind comments about “Scanlan’s paradox”:  In the 2006 Chance editorial I used the term “heuristic rule X” or “HRX” to describe the pattern whereby the two relative difference tend to change in opposite as the prevalence of an outcome changes.  That caused researchers in the UK to term the pattern “Scanlan’s rule.”  That prompted me to create a web page by that name with numerous subpages.  So it is fair to say I promote the usage, though, I hope, not too aggessively.  But a problem with the usage or like shorthand terms for the pattern of relative differences is that the usage may detract from appreciation of the patterns by which absolute differences and odds ratios tend to be affected by the prevalence of an outcome. So there are really two rules, which in workshops I principally refer to as Interpretive Rules 1 and 2 (though the second is pretty complicated).   This is matter of increasing importance as more and more researchers rely on absolute differences to measure healthcare disparities (and various other things that used to be discussed in terms of relative differences) without recognizing the patterns by which absolute differences tend to change as the frequency of an outcome changes – or while mistakenly thinking that because the absolute difference is unaffected by which outcome one examines, it effectively addresses the problems I identify with the two relative differences. 

Finally, a couple of observations about the relationship between the two relative differences and the absolute difference:  As the frequency of an outcome changes, the absolute difference tends to change in the same direction as the smaller relative difference (while the difference measured by the odds ratio tends to change in the opposite direction of the absolute difference and the same direction as the larger relative difference).  See figure 3 (at 22) of the October 2015 ASA letter.  Since in most contexts other than employment testing observers relying on relative differences commonly rely on the larger of the two relative differences (often without any thought to a second relative difference), there is a systematic tendency for observers relying on relative differences to reach opposite conclusion about such things as directions of change over time from observers relying on the absolute difference. 

As discussed in reference 2 and 3 and the ASA October 2015 letter, recognition that the absolute difference and the relative difference the observer happens to be looking at have changed in opposite directions is increasing.  But it is still not widespread.  Many people analyzing demographic differences still seem entirely unaware that  measure other than the one they are using can (much less commonly will or in fact would do so in their study) yield a different results about direction of changes over time from the direction they have identified.  See references 12 and 13 for seemingly extreme situations though they may actually differ little from the standard, as discussed in the concluding pages of reference 3). 

But even where observers have recognized that a relative difference and the absolute difference can yield (or have yielded) different conclusions about such things as directions of changes over time, they have invariably done so without seeming to realize that there is a second relative difference.  This has been the case even though, while all measures may change in the same direction (thus indicating a true change in the strength of the forces causing the outcome rates to differ), when a mentioned relative difference and the absolute difference have changed in different directions, the unmentioned relative difference will necessarily have changed in the opposite direction of the mentioned relative difference and the same direction as the absolute difference.  

1. “Can We Actually Measure Health Disparities?,” Chance (Spring 2006)http://www.jpscanlan.com/images/Can_We_Actually_Measure_Health_Disparities.pdf 

2. “The Mismeasure of Health Disparities,” Journal of Public Health Management and Practice (July/Aug. 2016)http://www.jpscanlan.com/images/The_Mismeasure_of_Health_Disparities_JPHMP_2016_.pdf

3. “Race and Mortality Revisited,” Society (July/Aug. 2014) http://jpscanlan.com/images/Race_and_Mortality_Revisited.pdf

4. “Misunderstanding of Statistics Leads to Misguided Law Enforcement Policies,” Amstat News (Dec. 2012)http://magazine.amstat.org/blog/2012/12/01/misguided-law-enforcement/

5. Subgroup Effects subpage of the Scanlan’s Rule page of jpscanlan.com  http://jpscanlan.com/images/Subgroup_Effects_-_Interaction.pdf

6. Re: The number needed to treat: a clinically useful measure of treatment effect. BMJ (Nov. 21, 2011) http://www.bmj.com/rapid-response/2011/11/21/re-number-needed-treat-clinically-useful-measure-treatment-effect

7. Assumption of constant relative risk reductions across different baseline rates is unsound. CMAJ (Mar. 12, 2012) http://www.cmaj.ca/content/171/4/353.short/reply#cmaj_el_690714
8. Re: Progress research strategy (PROGRESS) 4: Stratified medicine research.  BMJ. (Feb. 25, 2013) http://www.bmj.com/content/346/bmj.e5793/rr/632884
9. “Divining Difference,” Chance (Fall 1994) http://jpscanlan.com/images/Divining_Difference.pdf

10. http://www.jpscanlan.com/measurementletters.html
11. http://jpscanlan.com/publications/conferencepresentations.html
12. http://jpscanlan.com/measuringhealthdisp/spuriouscontradictions.html
13. http://jpscanlan.com/measuringhealthdisp/ahrqsvanderbiltreport.html

Do we spend enough time in our introductory courses on interpreting proportion data? We typically present inferences for one proportion and the difference between two proportions and let it go at that. However when proportion data can be reported in a way as to make good outcomes look bad and bad outcomes look good, as Mr. Scanlan has pointed out on many issues of importance to society, then it is up to us to let our students know about it. Giving a name to a phenomenon, as Tukey often did, may help focus attention on the issue. Let me suggest "Scanlan's paradox" as a possibility.

Below is an item just published involving the statistical concepts addressed in materials I have posted here frequently, including my letters to ASA of October 8, 2015[1] and July 25, 2016.[2]

“Misunderstanding of Statistics Confounds Analyses of Criminal Justice Issues in Baltimore and Voter ID Issues in Texas and North Carolina,” Federalist Society Blog (Oct. 3, 2016)

http://www.fed-soc.org/blog/detail/misunderstanding-of-statistics-confounds-analyses-of-criminal-justice-issues-in-baltimore-and-voter-id-issues-in-texas-and-north-carolina

The item explains that (a) contrary to the belief reflected in the Department of Justice report on police practices in Baltimore, MD, generally reducing adverse criminal justice outcomes will tend to increase the proportions African Americans make up of persons experiencing those outcomes and (b) contrary to the beliefs reflected in court decisions involving voter IDs requirements, less stringent requirements will tend to produce larger relative racial differences in rates of failing to meet the requirements than more stringent ones.

The latter point should be borne in mind by those intending to participate in the voter ID studies discussed on ASA website.

1. http://jpscanlan.com/images/Letter_to_American_Statistical_Association_Oct._8,_2015_.pdf
2. http://www.jpscanlan.com/images/Letter_to_American_Statistical_Association_July_25,_2016_.pdf

Mr Scanlan:

I'll take up your challenge to lay out an approach, in terms of the sort of problem you typically address, which might be more fruitful than lobbying the ASA to instruct the President of the United States (!) on an arithmetical principle.

Typically, it seems your problem settings involve matters of social injustice, and common ways of measuring and responding to them. I suggest you try the following:

1. From among the social injustice situations you have addressed in the past, select a case where you enjoy some common ground with your opponents. This would mean at least (i) agreeing that a social injustice does exist, and (ii) sharing your opponents' conviction that a policy response is morally desirable.
2. Describe the causal factors in said social injustice, and clarify points where you and your opponents agree and disagree in this regard. If there are substantial disagreements about questions of causality, then you might revert to Step 1 and find a different social injustice situation. Or you might just note your points of disagreement but concede the causal picture to your opponents 'for the sake of argument' -- i.e., to enable you to proceed with the development of your arithmetical argument. 
3. Having thus found common ground with your opponents on moral and causal matters, proceed to demonstrate the distinct policy scenarios entailed by your and your opponents' separate arithmetical proclivities. The strongest argument here would persuade your opponents that they would achieve morally superior results by abandoning relative risks in favor of absolute risks as guides of policy and measures of progress against injustice. If you make your argument well, your opponents will evaulate the policy consequences of their own arithmetic, compare the conequences of yours, and realize that yours result in morally superior results.

If you should find you cannot get past Step 1, then you will have discovered that your true grievance is not of an arithmetical nature after all, but of an entirely different sort. In that case, you might do well to abandon an arithmetical principle ('HRX') that has enjoyed no apparent substantive formal development over several decades, as a sterile distraction from the more fundamental argument you ought to be advancing perhaps along philosophical lines.

One of the strongest signs of a sterile theory is that it cannot support technical development. I have no doubt that HRX could be formalized quite readily, and am baffled that you have not pursued this course. If you don't do so in the next month or so, I may take up the matter myself sometime before the end of the year--either as a blog post, or something I submit to Chance or Significance. It is not inconceivable that such development will leave your HRX principle in a state that "puts it beyond use" (as they say in Northern Ireland) for the applications you seem to have been pursuing.

To be sure, I am not dismissing the argument you have been making as utterly uninteresting. I am however deeply suspicious of its failure to develop formally and evolve technically. The sheer amount of verbiage involved is astounding for so straightforward a point.

Kind regards,

------------------------------
David C. Norris, MD
David Norris Consulting, LLC
Seattle, WA

Original Message:
Sent: 10-04-2016 21:05
From: James Scanlan
Subject: Disparate impact of Baltimore police practices and voter ID laws

Thanks for these comments.  A few responsive observations. 

Regarding Ed Gracely’s point:   I am not making an error by discussing relative rather than absolute differences in rates.  My article is about what the Department of Justice and courts do.  Whether the DOJ and the courts are making an error is another matter. 

But in arguing the greater importance of absolute differences with regard to appraisal of a demographic difference, one must be mindful of the ways absolute differences tend to be affected by the frequency of an outcome, as I explained in a Chance editorial more than decade ago, [1] and as I have explained in scores of places since then.  See, e.g., references 2-4.  In fact, my October 8, 2015 letter to the ASA mentions absolute differences 74 times.  See especially the discussion in reference 3 (at 337-339) about the way that reliance on absolute differences to measure healthcare disparities led Massachusetts to unwisely include a disparities element in its Medicaid pay-for-performance programs and to do so in a way that is likely to result in increased healthcare disparities (regardless of how they are measured). 

Thus, the failure to understand that the two relative differences tend to change in opposite directions as the prevalence of an outcomes (and the fact that so many policies are based on the mistaken belief that reducing the frequency of an outcome will tend to reduce rather increase relative differences) are merely manifestations of the larger failure to recognize patterns by which all standard measures tend to be affected by the frequency of an outcome and implications of those patterns.  

The key consideration for those thinking about broader issues (rather than the particular subject of my article) is that anything said about demographic difference is misleading if it fails to address the way the particular measure employed tends to be affected by the prevalence of the outcome.   

Regarding David Norris’s points:   It may be that many people recognize shortcoming of reliance on large relative differences regarding rare outcome (though they seem all to do so without recognizing that the difference will tend to increase as the outcome becomes even rarer).  But the fact remains that billions of dollars are spent each year studying relative difference in favorable or adverse outcomes (as to numerous issues in the law and the social and medical science) including by highly reputed research institutions, and extremely important policy decisions and law enforcement policies are based on perceptions about relative differences.  All this occurs without any attention to the way the relative difference the observer happens to be examining tends to be affected by the prevalence of an outcome.  And, of course, in cases when the adverse outcome is examined, any understanding an observer may have about effects of the prevalence of an outcome typically is the opposite of reality. 

(Similar issues, of course, apply to reliance on absolute differences, though in the case of absolute differences interpretations cannot turn on whether one examines the favorable or the adverse outcome and there is no widespread mistaken expectation as to the effect of the frequency of an outcome.)  

Mistaken reliance on relative differences predominates even in epidemiology where subgroup effects are identified on the basis a departure from constant relative effect across different baseline rates (an approach that is illogical as well as unsound, as I discuss in reference 3 (at 339-34) and the October 2015 ASA letter (at 12-13). 

In the view of most observers (including me), the number-needed-to treat (NNT), which is a function of the absolute effect of a factor on the outcome rate at issue, is the most important consideration in treatment decisions (though that is still a different matter from whether the absolute difference is a sound measure of association).  But as discussed in the introductory material to reference 5, it is standard practice, as recommended, for example, by Oxford's Center for Evidence Based Medicine and BMJ Clinical Evidence, to employ the relative effect observed in a clinical trial to calculate the NNT in a situation involving different baseline rates from that involved in the trial.  (Both of those entities rejected my suggestion that they modify this guidance, though the UK entity called Patient (patient.info) did appreciatively accept my suggested modifications.)  See also references 6 to 8. 

Not being a statistician, I am not a good person to distinguish between arithmetic and statistics.  It’s all mathematics to me.  But the patterns I describe do not have to do with things like randomness or sampling error (save in that such factors will affect the consistency with which one will observe the patterns I describe).  Rather, the patterns I describe have to do with the fact that, in situations such as that reflected in Figure 1 of reference 9, as an outcome is increasingly restricted toward either end of the overall distribution each standard measure tends to change in a particular way.  One cannot draw conclusions about processes (or about the strength of the associations reflected by a pair of outcome rates) based on those standard measures without distinguishing the extent to which observed patterns are functions of the prevalence of the outcome and the extent to which they reflect something about underlying forces. 

It may be that there are better ways, either as to tone or substance, of articulating my points than found, say, in my October 8, 2015, letter to ASA (or the letters to other entities collected in reference 10 or the workshops collected in reference 11).  I always welcome suggestions.  And I have received some useful ones (including from Dr. Norris).  But it is best when the are focused on particular illustrations that I do use.

Regarding James Higgins’ kind comments about “Scanlan’s paradox”:  In the 2006 Chance editorial I used the term “heuristic rule X” or “HRX” to describe the pattern whereby the two relative difference tend to change in opposite as the prevalence of an outcome changes.  That caused researchers in the UK to term the pattern “Scanlan’s rule.”  That prompted me to create a web page by that name with numerous subpages.  So it is fair to say I promote the usage, though, I hope, not too aggessively.  But a problem with the usage or like shorthand terms for the pattern of relative differences is that the usage may detract from appreciation of the patterns by which absolute differences and odds ratios tend to be affected by the prevalence of an outcome. So there are really two rules, which in workshops I principally refer to as Interpretive Rules 1 and 2 (though the second is pretty complicated).   This is matter of increasing importance as more and more researchers rely on absolute differences to measure healthcare disparities (and various other things that used to be discussed in terms of relative differences) without recognizing the patterns by which absolute differences tend to change as the frequency of an outcome changes – or while mistakenly thinking that because the absolute difference is unaffected by which outcome one examines, it effectively addresses the problems I identify with the two relative differences. 

Finally, a couple of observations about the relationship between the two relative differences and the absolute difference:  As the frequency of an outcome changes, the absolute difference tends to change in the same direction as the smaller relative difference (while the difference measured by the odds ratio tends to change in the opposite direction of the absolute difference and the same direction as the larger relative difference).  See figure 3 (at 22) of the October 2015 ASA letter.  Since in most contexts other than employment testing observers relying on relative differences commonly rely on the larger of the two relative differences (often without any thought to a second relative difference), there is a systematic tendency for observers relying on relative differences to reach opposite conclusion about such things as directions of change over time from observers relying on the absolute difference. 

As discussed in reference 2 and 3 and the ASA October 2015 letter, recognition that the absolute difference and the relative difference the observer happens to be looking at have changed in opposite directions is increasing.  But it is still not widespread.  Many people analyzing demographic differences still seem entirely unaware that  measure other than the one they are using can (much less commonly will or in fact would do so in their study) yield a different results about direction of changes over time from the direction they have identified.  See references 12 and 13 for seemingly extreme situations though they may actually differ little from the standard, as discussed in the concluding pages of reference 3). 

But even where observers have recognized that a relative difference and the absolute difference can yield (or have yielded) different conclusions about such things as directions of changes over time, they have invariably done so without seeming to realize that there is a second relative difference.  This has been the case even though, while all measures may change in the same direction (thus indicating a true change in the strength of the forces causing the outcome rates to differ), when a mentioned relative difference and the absolute difference have changed in different directions, the unmentioned relative difference will necessarily have changed in the opposite direction of the mentioned relative difference and the same direction as the absolute difference.  

1. “Can We Actually Measure Health Disparities?,” Chance (Spring 2006)http://www.jpscanlan.com/images/Can_We_Actually_Measure_Health_Disparities.pdf 

2. “The Mismeasure of Health Disparities,” Journal of Public Health Management and Practice (July/Aug. 2016)http://www.jpscanlan.com/images/The_Mismeasure_of_Health_Disparities_JPHMP_2016_.pdf

3. “Race and Mortality Revisited,” Society (July/Aug. 2014) http://jpscanlan.com/images/Race_and_Mortality_Revisited.pdf

4. “Misunderstanding of Statistics Leads to Misguided Law Enforcement Policies,” Amstat News (Dec. 2012)http://magazine.amstat.org/blog/2012/12/01/misguided-law-enforcement/

5. Subgroup Effects subpage of the Scanlan’s Rule page of jpscanlan.com  http://jpscanlan.com/images/Subgroup_Effects_-_Interaction.pdf

6. Re: The number needed to treat: a clinically useful measure of treatment effect. BMJ (Nov. 21, 2011) http://www.bmj.com/rapid-response/2011/11/21/re-number-needed-treat-clinically-useful-measure-treatment-effect

7. Assumption of constant relative risk reductions across different baseline rates is unsound. CMAJ (Mar. 12, 2012) http://www.cmaj.ca/content/171/4/353.short/reply#cmaj_el_690714
8. Re: Progress research strategy (PROGRESS) 4: Stratified medicine research.  BMJ. (Feb. 25, 2013) http://www.bmj.com/content/346/bmj.e5793/rr/632884
9. “Divining Difference,” Chance (Fall 1994) http://jpscanlan.com/images/Divining_Difference.pdf

10. http://www.jpscanlan.com/measurementletters.html
11. http://jpscanlan.com/publications/conferencepresentations.html
12. http://jpscanlan.com/measuringhealthdisp/spuriouscontradictions.html
13. http://jpscanlan.com/measuringhealthdisp/ahrqsvanderbiltreport.html

------------------------------
James Scanlan
James P. Scanlan Attorney At Law

Original Message:
Sent: 10-04-2016 14:14
From: James Higgins
Subject: Disparate impact of Baltimore police practices and voter ID laws

Do we spend enough time in our introductory courses on interpreting proportion data? We typically present inferences for one proportion and the difference between two proportions and let it go at that. However when proportion data can be reported in a way as to make good outcomes look bad and bad outcomes look good, as Mr. Scanlan has pointed out on many issues of importance to society, then it is up to us to let our students know about it. Giving a name to a phenomenon, as Tukey often did, may help focus attention on the issue. Let me suggest "Scanlan's paradox" as a possibility.

------------------------------
James Higgins
Kansas State Univ

Original Message:
Sent: 10-03-2016 13:07
From: James Scanlan
Subject: Disparate impact of Baltimore police practices and voter ID laws

Below is an item just published involving the statistical concepts addressed in materials I have posted here frequently, including my letters to ASA of October 8, 2015[1] and July 25, 2016.[2]

“Misunderstanding of Statistics Confounds Analyses of Criminal Justice Issues in Baltimore and Voter ID Issues in Texas and North Carolina,” Federalist Society Blog (Oct. 3, 2016)

http://www.fed-soc.org/blog/detail/misunderstanding-of-statistics-confounds-analyses-of-criminal-justice-issues-in-baltimore-and-voter-id-issues-in-texas-and-north-carolina

The item explains that (a) contrary to the belief reflected in the Department of Justice report on police practices in Baltimore, MD, generally reducing adverse criminal justice outcomes will tend to increase the proportions African Americans make up of persons experiencing those outcomes and (b) contrary to the beliefs reflected in court decisions involving voter IDs requirements, less stringent requirements will tend to produce larger relative racial differences in rates of failing to meet the requirements than more stringent ones.

The latter point should be borne in mind by those intending to participate in the voter ID studies discussed on ASA website.

1. http://jpscanlan.com/images/Letter_to_American_Statistical_Association_Oct._8,_2015_.pdf
2. http://www.jpscanlan.com/images/Letter_to_American_Statistical_Association_July_25,_2016_.pdf

------------------------------
James Scanlan
James P. Scanlan Attorney At Law
------------------------------

All:

To transcend the squalor of this weekend's politics, I eschewed the news and sought to do something positive for our democracy. Accordingly, I have produced a Shiny app https://dnc-llc.shinyapps.io/ScanlanRule/ that I believe fully demystifies the 'Scanlan Rule'. [I feel compelled to note that the 'dnc' in the URL stands for 'David Norris Consulting', not the other 'DNC'! My aim here is to develop an objective, nonpartisan argument.]

On working this through, I have concluded that the Scanlan Rule actually comprises two ideas.

1. One part of the Scanlan Rule is the trivial, purely 'arithmetical' idea that for a rare adverse outcome, the group-wise proportions not experiencing the outcome will be close to unity, as will the ratio of these proportions. In the context of the Veasey v Abbott case (in connection with which Mr. Scanlan submitted an amicus curiae brief mentioned earlier in this thread), it would seem that this idea directs our attention to a ratio such as 91.9% : 98% = 0.938, for consideration alongside a ratio like 8.1% : 2% = 4. (Here, 8.1% and 2% are the estimated fractions of registered African-American and Anglo voters, respectively, who lacked ID meeting the requirements of Texas SB 14.) Thus, alongside a statement like "African-American voters are 4 times as likely to be disenfranchised by SB 14," we might be instructed to note, "but African-Americans achieve nearly 94% of the enfranchisement of Anglos under this bill."
2. The second part is, I will grant Mr. Scanlan, a nontrivial claim. This is his claim regarding the limit of group outcome ratios as an adverse outcome becomes vanishingly rare. While nontrivial, this claim is also not true in general. Moreover, even when it is true it appears no more morally urgent than is the 'arithmetical' portion of Scanlan's Rule. The Shiny app linked above deals with this 'limit clause' of the Scanlan Rule by scaling the [adverse] outcome according to quantiles of a reference group. This allows the outcomes of other groups to be plotted as cumulative distributions relative to a 45-degree reference-group line. A uniformly disadvantaged group will lie above the reference group's 45-degree line, while a uniformly advantaged group (relative to the reference) will lie strictly below. One can also readily conceive of groups with 'mixed' experience, as e.g. with immigrant groups whose 1st generation may be held back by low capital (including social capital like language skills) but whose later generations are propelled by 'work-ethic'. (Similar, but generationally inverted, phenomena have been described in terms of 'health paradoxes' of immigrant groups whose later generations adopt unhealthy American dietary and other social norms.) The only constraint on comparison-group curves is that they be monotone increasing and run through points (0,0) and (1,1). The app implements a 2-parameter family of such curves, with sliders controlling the slopes at endpoints (0,0) and (1,1). While this family hardly exhausts the possibilities, it does permit enough free play to demonstrate cases where the 'limit clause' of the Scanlan Rule does not hold in general; try it yourself!

Each population group in the app has a 'Scanlan Limit' relative to the reference group. Because the app's diagram shows this reference group as a 45-degree line, each comparison group's 'Scanlan Limit' appears on the figure as the slope of its curve at (0,0).

Mr. Scanlan's discussions typically suggest a view of policy in terms of threshold-setting. While Mr. Scanlan usually discusses policy-setting relative to the whole-population rate of the outcome, I believe the app demonstrates that discussing it relative to a reference group's rate aids in the exposition. (Certainly, in the social justice applications of the Scanlan Rule, the comparisons will often be framed relative to such a reference group, presumed to enjoy a certain standard of justice that it is desired to extend to relatively disadvantaged groups.)

I trust that this brief exposition will suffice for the statistical audience on this ASA Community. Time permitting, I will write a blog post aiming to make this more accessible to a nontechnical audience, and also more directly 'debunking' the Scanlan Rule as an argument against opposition to demographically targeted voting-rights restrictions and similar abuses.

Kind regards,

Mr Scanlan:

I'll take up your challenge to lay out an approach, in terms of the sort of problem you typically address, which might be more fruitful than lobbying the ASA to instruct the President of the United States (!) on an arithmetical principle.

Typically, it seems your problem settings involve matters of social injustice, and common ways of measuring and responding to them. I suggest you try the following:

1. From among the social injustice situations you have addressed in the past, select a case where you enjoy some common ground with your opponents. This would mean at least (i) agreeing that a social injustice does exist, and (ii) sharing your opponents' conviction that a policy response is morally desirable.
2. Describe the causal factors in said social injustice, and clarify points where you and your opponents agree and disagree in this regard. If there are substantial disagreements about questions of causality, then you might revert to Step 1 and find a different social injustice situation. Or you might just note your points of disagreement but concede the causal picture to your opponents 'for the sake of argument' -- i.e., to enable you to proceed with the development of your arithmetical argument. 
3. Having thus found common ground with your opponents on moral and causal matters, proceed to demonstrate the distinct policy scenarios entailed by your and your opponents' separate arithmetical proclivities. The strongest argument here would persuade your opponents that they would achieve morally superior results by abandoning relative risks in favor of absolute risks as guides of policy and measures of progress against injustice. If you make your argument well, your opponents will evaulate the policy consequences of their own arithmetic, compare the conequences of yours, and realize that yours result in morally superior results.

If you should find you cannot get past Step 1, then you will have discovered that your true grievance is not of an arithmetical nature after all, but of an entirely different sort. In that case, you might do well to abandon an arithmetical principle ('HRX') that has enjoyed no apparent substantive formal development over several decades, as a sterile distraction from the more fundamental argument you ought to be advancing perhaps along philosophical lines.

One of the strongest signs of a sterile theory is that it cannot support technical development. I have no doubt that HRX could be formalized quite readily, and am baffled that you have not pursued this course. If you don't do so in the next month or so, I may take up the matter myself sometime before the end of the year--either as a blog post, or something I submit to Chance or Significance. It is not inconceivable that such development will leave your HRX principle in a state that "puts it beyond use" (as they say in Northern Ireland) for the applications you seem to have been pursuing.

To be sure, I am not dismissing the argument you have been making as utterly uninteresting. I am however deeply suspicious of its failure to develop formally and evolve technically. The sheer amount of verbiage involved is astounding for so straightforward a point.

Kind regards,

Thanks for these comments.  A few responsive observations. 

Regarding Ed Gracely’s point:   I am not making an error by discussing relative rather than absolute differences in rates.  My article is about what the Department of Justice and courts do.  Whether the DOJ and the courts are making an error is another matter. 

But in arguing the greater importance of absolute differences with regard to appraisal of a demographic difference, one must be mindful of the ways absolute differences tend to be affected by the frequency of an outcome, as I explained in a Chance editorial more than decade ago, [1] and as I have explained in scores of places since then.  See, e.g., references 2-4.  In fact, my October 8, 2015 letter to the ASA mentions absolute differences 74 times.  See especially the discussion in reference 3 (at 337-339) about the way that reliance on absolute differences to measure healthcare disparities led Massachusetts to unwisely include a disparities element in its Medicaid pay-for-performance programs and to do so in a way that is likely to result in increased healthcare disparities (regardless of how they are measured). 

Thus, the failure to understand that the two relative differences tend to change in opposite directions as the prevalence of an outcomes (and the fact that so many policies are based on the mistaken belief that reducing the frequency of an outcome will tend to reduce rather increase relative differences) are merely manifestations of the larger failure to recognize patterns by which all standard measures tend to be affected by the frequency of an outcome and implications of those patterns.  

The key consideration for those thinking about broader issues (rather than the particular subject of my article) is that anything said about demographic difference is misleading if it fails to address the way the particular measure employed tends to be affected by the prevalence of the outcome.   

Regarding David Norris’s points:   It may be that many people recognize shortcoming of reliance on large relative differences regarding rare outcome (though they seem all to do so without recognizing that the difference will tend to increase as the outcome becomes even rarer).  But the fact remains that billions of dollars are spent each year studying relative difference in favorable or adverse outcomes (as to numerous issues in the law and the social and medical science) including by highly reputed research institutions, and extremely important policy decisions and law enforcement policies are based on perceptions about relative differences.  All this occurs without any attention to the way the relative difference the observer happens to be examining tends to be affected by the prevalence of an outcome.  And, of course, in cases when the adverse outcome is examined, any understanding an observer may have about effects of the prevalence of an outcome typically is the opposite of reality. 

(Similar issues, of course, apply to reliance on absolute differences, though in the case of absolute differences interpretations cannot turn on whether one examines the favorable or the adverse outcome and there is no widespread mistaken expectation as to the effect of the frequency of an outcome.)  

Mistaken reliance on relative differences predominates even in epidemiology where subgroup effects are identified on the basis a departure from constant relative effect across different baseline rates (an approach that is illogical as well as unsound, as I discuss in reference 3 (at 339-34) and the October 2015 ASA letter (at 12-13). 

In the view of most observers (including me), the number-needed-to treat (NNT), which is a function of the absolute effect of a factor on the outcome rate at issue, is the most important consideration in treatment decisions (though that is still a different matter from whether the absolute difference is a sound measure of association).  But as discussed in the introductory material to reference 5, it is standard practice, as recommended, for example, by Oxford's Center for Evidence Based Medicine and BMJ Clinical Evidence, to employ the relative effect observed in a clinical trial to calculate the NNT in a situation involving different baseline rates from that involved in the trial.  (Both of those entities rejected my suggestion that they modify this guidance, though the UK entity called Patient (patient.info) did appreciatively accept my suggested modifications.)  See also references 6 to 8. 

Not being a statistician, I am not a good person to distinguish between arithmetic and statistics.  It’s all mathematics to me.  But the patterns I describe do not have to do with things like randomness or sampling error (save in that such factors will affect the consistency with which one will observe the patterns I describe).  Rather, the patterns I describe have to do with the fact that, in situations such as that reflected in Figure 1 of reference 9, as an outcome is increasingly restricted toward either end of the overall distribution each standard measure tends to change in a particular way.  One cannot draw conclusions about processes (or about the strength of the associations reflected by a pair of outcome rates) based on those standard measures without distinguishing the extent to which observed patterns are functions of the prevalence of the outcome and the extent to which they reflect something about underlying forces. 

It may be that there are better ways, either as to tone or substance, of articulating my points than found, say, in my October 8, 2015, letter to ASA (or the letters to other entities collected in reference 10 or the workshops collected in reference 11).  I always welcome suggestions.  And I have received some useful ones (including from Dr. Norris).  But it is best when the are focused on particular illustrations that I do use.

Regarding James Higgins’ kind comments about “Scanlan’s paradox”:  In the 2006 Chance editorial I used the term “heuristic rule X” or “HRX” to describe the pattern whereby the two relative difference tend to change in opposite as the prevalence of an outcome changes.  That caused researchers in the UK to term the pattern “Scanlan’s rule.”  That prompted me to create a web page by that name with numerous subpages.  So it is fair to say I promote the usage, though, I hope, not too aggessively.  But a problem with the usage or like shorthand terms for the pattern of relative differences is that the usage may detract from appreciation of the patterns by which absolute differences and odds ratios tend to be affected by the prevalence of an outcome. So there are really two rules, which in workshops I principally refer to as Interpretive Rules 1 and 2 (though the second is pretty complicated).   This is matter of increasing importance as more and more researchers rely on absolute differences to measure healthcare disparities (and various other things that used to be discussed in terms of relative differences) without recognizing the patterns by which absolute differences tend to change as the frequency of an outcome changes – or while mistakenly thinking that because the absolute difference is unaffected by which outcome one examines, it effectively addresses the problems I identify with the two relative differences. 

Finally, a couple of observations about the relationship between the two relative differences and the absolute difference:  As the frequency of an outcome changes, the absolute difference tends to change in the same direction as the smaller relative difference (while the difference measured by the odds ratio tends to change in the opposite direction of the absolute difference and the same direction as the larger relative difference).  See figure 3 (at 22) of the October 2015 ASA letter.  Since in most contexts other than employment testing observers relying on relative differences commonly rely on the larger of the two relative differences (often without any thought to a second relative difference), there is a systematic tendency for observers relying on relative differences to reach opposite conclusion about such things as directions of change over time from observers relying on the absolute difference. 

As discussed in reference 2 and 3 and the ASA October 2015 letter, recognition that the absolute difference and the relative difference the observer happens to be looking at have changed in opposite directions is increasing.  But it is still not widespread.  Many people analyzing demographic differences still seem entirely unaware that  measure other than the one they are using can (much less commonly will or in fact would do so in their study) yield a different results about direction of changes over time from the direction they have identified.  See references 12 and 13 for seemingly extreme situations though they may actually differ little from the standard, as discussed in the concluding pages of reference 3). 

But even where observers have recognized that a relative difference and the absolute difference can yield (or have yielded) different conclusions about such things as directions of changes over time, they have invariably done so without seeming to realize that there is a second relative difference.  This has been the case even though, while all measures may change in the same direction (thus indicating a true change in the strength of the forces causing the outcome rates to differ), when a mentioned relative difference and the absolute difference have changed in different directions, the unmentioned relative difference will necessarily have changed in the opposite direction of the mentioned relative difference and the same direction as the absolute difference.  

1. “Can We Actually Measure Health Disparities?,” Chance (Spring 2006)http://www.jpscanlan.com/images/Can_We_Actually_Measure_Health_Disparities.pdf 

2. “The Mismeasure of Health Disparities,” Journal of Public Health Management and Practice (July/Aug. 2016)http://www.jpscanlan.com/images/The_Mismeasure_of_Health_Disparities_JPHMP_2016_.pdf

3. “Race and Mortality Revisited,” Society (July/Aug. 2014) http://jpscanlan.com/images/Race_and_Mortality_Revisited.pdf

4. “Misunderstanding of Statistics Leads to Misguided Law Enforcement Policies,” Amstat News (Dec. 2012)http://magazine.amstat.org/blog/2012/12/01/misguided-law-enforcement/

5. Subgroup Effects subpage of the Scanlan’s Rule page of jpscanlan.com  http://jpscanlan.com/images/Subgroup_Effects_-_Interaction.pdf

6. Re: The number needed to treat: a clinically useful measure of treatment effect. BMJ (Nov. 21, 2011) http://www.bmj.com/rapid-response/2011/11/21/re-number-needed-treat-clinically-useful-measure-treatment-effect

7. Assumption of constant relative risk reductions across different baseline rates is unsound. CMAJ (Mar. 12, 2012) http://www.cmaj.ca/content/171/4/353.short/reply#cmaj_el_690714
8. Re: Progress research strategy (PROGRESS) 4: Stratified medicine research.  BMJ. (Feb. 25, 2013) http://www.bmj.com/content/346/bmj.e5793/rr/632884
9. “Divining Difference,” Chance (Fall 1994) http://jpscanlan.com/images/Divining_Difference.pdf

10. http://www.jpscanlan.com/measurementletters.html
11. http://jpscanlan.com/publications/conferencepresentations.html
12. http://jpscanlan.com/measuringhealthdisp/spuriouscontradictions.html
13. http://jpscanlan.com/measuringhealthdisp/ahrqsvanderbiltreport.html

Do we spend enough time in our introductory courses on interpreting proportion data? We typically present inferences for one proportion and the difference between two proportions and let it go at that. However when proportion data can be reported in a way as to make good outcomes look bad and bad outcomes look good, as Mr. Scanlan has pointed out on many issues of importance to society, then it is up to us to let our students know about it. Giving a name to a phenomenon, as Tukey often did, may help focus attention on the issue. Let me suggest "Scanlan's paradox" as a possibility.

Below is an item just published involving the statistical concepts addressed in materials I have posted here frequently, including my letters to ASA of October 8, 2015[1] and July 25, 2016.[2]

“Misunderstanding of Statistics Confounds Analyses of Criminal Justice Issues in Baltimore and Voter ID Issues in Texas and North Carolina,” Federalist Society Blog (Oct. 3, 2016)

http://www.fed-soc.org/blog/detail/misunderstanding-of-statistics-confounds-analyses-of-criminal-justice-issues-in-baltimore-and-voter-id-issues-in-texas-and-north-carolina

The item explains that (a) contrary to the belief reflected in the Department of Justice report on police practices in Baltimore, MD, generally reducing adverse criminal justice outcomes will tend to increase the proportions African Americans make up of persons experiencing those outcomes and (b) contrary to the beliefs reflected in court decisions involving voter IDs requirements, less stringent requirements will tend to produce larger relative racial differences in rates of failing to meet the requirements than more stringent ones.

The latter point should be borne in mind by those intending to participate in the voter ID studies discussed on ASA website.

1. http://jpscanlan.com/images/Letter_to_American_Statistical_Association_Oct._8,_2015_.pdf
2. http://www.jpscanlan.com/images/Letter_to_American_Statistical_Association_July_25,_2016_.pdf

Dr. Norris:

A few observations regarding your post of October 10, 2016.

1. At issue is the pattern, inherent in other than highly irregular risk distributions, whereby the rarer an outcome the greater tends to be the relative difference in experiencing it and the smaller tends to be the relative difference in avoiding it. The patterns exist across the distribution and not merely at the tails of the distributions.

It seems that earlier you were treating increasingly large relative differences as an outcome becomes rare as trivial, while now you are treating the increasingly small relative differences in the corresponding opposite outcome as that outcome becomes very common as trivial. In any case, I am not sure that it matters whether one treats either pattern (really the same pattern, see October 2015 ASA letter [1] (at 14 n.22) and slide 51 of the 2015 UMass Medical seminar[2]) as a trivial arithmetical pattern or an important statistical one.

What seems important to me, however, is that virtually no one analyzing group differences in outcome rates seems to understand the effects of the prevalence of an outcome on measures of group differences in outcome rates. Also, important federal policies are based on the belief that reducing the frequency of outcomes like arrest, mortgage rejection, or suspension from school will tend to reduce relative differences in rates of experiencing the outcome. That is, the government’s belief is the exact opposite of reality. At least one federal statute also reflects that belief. See “Race and Mortality Revisited,” Society (July/Aug. 2014) (at 342).[3] And the government monitors the fairness of practices on the basis of the size of relative differences in adverse outcome at the same time that it encourages modification to practices that tend to increase those differences. So complying with government encouragements to reduce adverse outcomes tends to increase, not decrease, the chances that the government will sue one for discrimination.

Also important to understand is that one cannot appraise any demographic differences or how policies affect those differences without understanding how the measures employed (including the absolute difference between rates) tend to be affected by the frequency of an outcome. That, however, may be a bit afield of your instant post, save in that one must understand that no moral question is involved. Rather, the question is what approach most accurately quantifies the differences in the circumstances of two groups reflected by their rates of experiencing some favorable or adverse outcomes.

2. Your post notes that there is mention in the thread that I filed an amicus curia brief in connection with the case of Veasey v. Abbott. I cannot find that mention. In any event, I did not file a brief in the case. I did file an amicus curiae brief [4] in Texas Department of Housing and Community Development v. The Inclusive Communities Project, Inc. (Nov. 17, 2014) and mentioned that brief in the article that I posted at the beginning of this thread. Presumably that is what you’re thinking about.

3. Skipping down a bit, your post observes that I “usually discuss[ ] policy-setting relative to the whole-population rate of the outcome” and suggests that your app shows that it is better to discuss the matter relative to the reference group’s rate. Assuming that I understand this point correctly, it would be more accurate to say, rather than that I “usually” discuss the matter with reference to the whole-population rate, that I “never” do it that way. When using test score data I always discuss the matter using the reference group rate as the benchmark for overall prevalence of an outcome on prevalence. See, e.g., the Harvard University letter (at 5) [5] and the 2008 ICHPS oral (at 3).[6] (See also Section A.8 of the Scanlan’s Rule page, [7]which explains why, though I speak of overall prevalence for shorthand, I actually focus on the rates of the group’s being compared without any interest in an overall rate ) That is what “cutoffs defined by the AG fail rate” means in Figures 1 through 3 of the October 2015 ASA letter or the scores of like figures in letters and conference presentations. See, e.g., Figures 1 through 3 of the UMass seminar. That is also what I meant by “[I]f the cutoff were lowered to the point where 95% of [the advantaged group] passes the test,” in the December 2012 Amstat News column.[8]

Sometimes, without any reference to overall rates, I track the patterns according to objective indicators such as income level reflected by ratios of the poverty line (as in Table 1 and the figures in the 2006 Chance editorial [9] and Figure P.1 (slide 88) of the UMass seminar) or credit score (as in the figures in the March 4, 2013 letter to the Federal Reserve Board and its Appendix).[10] But the comparison is always between the disadvantaged group’s rate and advantaged group’s rate (save, of course, when explaining the impossibility of analyzing group differences based on a comparison of the proportion a group makes up of persons potentially experiencing an outcome and the proportion it comprises of persons actually experiencing the outcome, as in the September 12, 2016 letter to the Antioch Unified School District).[11]

4. Turning to the app you created, I first note that it is akin to what I had in mind when in an email of July 26, I stated that “I wanted to be able to move a cursor across the x-axis of an illustration like Figure 1 (slide 23) of [the UMass seminar] and [have] the two ratios change as I move the cursor.” That figure, however, is based on distributions where the means differ by one standard deviation and my illustrations of changes in measures are commonly based on distributions where the means differ by half a standard deviation (as in Figure 1 of “Divining Difference” Chance (Fall 1994)).[12] An illustration like that produced by your app – but showing the patterns of changing rate ratios as one moves the cursor (with the larger figure in the numerator, see October 2015 ASA letter at 10 n.15) would be very useful for illustrating to policy makers that their beliefs about effects of generally reducing adverse outcome rates on relative differences in rates of experiencing those outcomes are incorrect. I hope you will consider creating such an app.

Otherwise, however, I do not yet understand this illustration or the specifications underlying it at all well enough to know whether I think it has value or is pertinent to issue I address. Apparently you regard it as is showing that the pattern I describe (or what you perceive to be one of two patterns I describe, or some pattern that you divine from my illustrations though I do not actually mention) is “not true in general.” By that, I assume you mean “is not always true” rather “is generally not true.” I do not know whether such point is merely contradicting some supposed claim of mine that I say the patterns I describe will always be observed (see, October 2015 ASA letter (at 27) and the numerous other places going back to 1992 [13] where I explain that tendencies are tendencies) in which case the illustration would not be of value. Possibly, you are saying something else and that something else is something that I have not considered. But I am a long way from understanding that and whether there is a possibility that a point you make or demonstrate affects the points I make about the need to understand patterns by which measures tend to be affected by the prevalence of an outcome.

I can say, however, in my view, the discussion of a very disadvantaged group and a somewhat disadvantaged group has nothing to do with the points I make about appraising differences in the circumstances of two groups reflected by their outcomes rates. Thus, I would suggest sticking with an advantaged and disadvantaged group for purposes of any point you are making about patterns I describe.

1. http://jpscanlan.com/images/Letter_to_American_Statistical_Association_Oct._8,_2015_.pdf
2. http://jpscanlan.com/images/Univ_Mass_Medical_School_Seminar_Nov._18,_2015_.pdf
3. http://jpscanlan.com/images/Race_and_Mortality_Revisited.pdf
4. http://jpscanlan.com/images/Scanlan_amicus_brief_in_Texas_Dpt_of_Housing_case.pdf
5. http://jpscanlan.com/images/Harvard_University_Measurement_Letter.pdf
6. http://www.jpscanlan.com/images/2008_ICHPS_Oral.pdf
7. http://magazine.amstat.org/blog/2012/12/01/misguided-law-enforcement/
8. http://magazine.amstat.org/blog/2012/12/01/misguided-law-enforcement/
9. http://www.jpscanlan.com/images/Can_We_Actually_Measure_Health_Disparities.pdf
10. http://jpscanlan.com/images/Federal_Reserve_Board_Letter_with_Appendix.pdf
11. http://www.jpscanlan.com/images/Letter_to_Antioch_Unified_School_District_Sept._12,_2016_.pdf
12. http://jpscanlan.com/images/Divining_Difference.pdf
13. http://jpscanlan.com/images/American_Banker_4-27-92.pdf

All:

To transcend the squalor of this weekend's politics, I eschewed the news and sought to do something positive for our democracy. Accordingly, I have produced a Shiny app https://dnc-llc.shinyapps.io/ScanlanRule/ that I believe fully demystifies the 'Scanlan Rule'. [I feel compelled to note that the 'dnc' in the URL stands for 'David Norris Consulting', not the other 'DNC'! My aim here is to develop an objective, nonpartisan argument.]

On working this through, I have concluded that the Scanlan Rule actually comprises two ideas.

1. One part of the Scanlan Rule is the trivial, purely 'arithmetical' idea that for a rare adverse outcome, the group-wise proportions not experiencing the outcome will be close to unity, as will the ratio of these proportions. In the context of the Veasey v Abbott case (in connection with which Mr. Scanlan submitted an amicus curiae brief mentioned earlier in this thread), it would seem that this idea directs our attention to a ratio such as 91.9% : 98% = 0.938, for consideration alongside a ratio like 8.1% : 2% = 4. (Here, 8.1% and 2% are the estimated fractions of registered African-American and Anglo voters, respectively, who lacked ID meeting the requirements of Texas SB 14.) Thus, alongside a statement like "African-American voters are 4 times as likely to be disenfranchised by SB 14," we might be instructed to note, "but African-Americans achieve nearly 94% of the enfranchisement of Anglos under this bill."
2. The second part is, I will grant Mr. Scanlan, a nontrivial claim. This is his claim regarding the limit of group outcome ratios as an adverse outcome becomes vanishingly rare. While nontrivial, this claim is also not true in general. Moreover, even when it is true it appears no more morally urgent than is the 'arithmetical' portion of Scanlan's Rule. The Shiny app linked above deals with this 'limit clause' of the Scanlan Rule by scaling the [adverse] outcome according to quantiles of a reference group. This allows the outcomes of other groups to be plotted as cumulative distributions relative to a 45-degree reference-group line. A uniformly disadvantaged group will lie above the reference group's 45-degree line, while a uniformly advantaged group (relative to the reference) will lie strictly below. One can also readily conceive of groups with 'mixed' experience, as e.g. with immigrant groups whose 1st generation may be held back by low capital (including social capital like language skills) but whose later generations are propelled by 'work-ethic'. (Similar, but generationally inverted, phenomena have been described in terms of 'health paradoxes' of immigrant groups whose later generations adopt unhealthy American dietary and other social norms.) The only constraint on comparison-group curves is that they be monotone increasing and run through points (0,0) and (1,1). The app implements a 2-parameter family of such curves, with sliders controlling the slopes at endpoints (0,0) and (1,1). While this family hardly exhausts the possibilities, it does permit enough free play to demonstrate cases where the 'limit clause' of the Scanlan Rule does not hold in general; try it yourself!

Each population group in the app has a 'Scanlan Limit' relative to the reference group. Because the app's diagram shows this reference group as a 45-degree line, each comparison group's 'Scanlan Limit' appears on the figure as the slope of its curve at (0,0).

Mr. Scanlan's discussions typically suggest a view of policy in terms of threshold-setting. While Mr. Scanlan usually discusses policy-setting relative to the whole-population rate of the outcome, I believe the app demonstrates that discussing it relative to a reference group's rate aids in the exposition. (Certainly, in the social justice applications of the Scanlan Rule, the comparisons will often be framed relative to such a reference group, presumed to enjoy a certain standard of justice that it is desired to extend to relatively disadvantaged groups.)

I trust that this brief exposition will suffice for the statistical audience on this ASA Community. Time permitting, I will write a blog post aiming to make this more accessible to a nontechnical audience, and also more directly 'debunking' the Scanlan Rule as an argument against opposition to demographically targeted voting-rights restrictions and similar abuses.

Kind regards,

Mr Scanlan:

I'll take up your challenge to lay out an approach, in terms of the sort of problem you typically address, which might be more fruitful than lobbying the ASA to instruct the President of the United States (!) on an arithmetical principle.

Typically, it seems your problem settings involve matters of social injustice, and common ways of measuring and responding to them. I suggest you try the following:

1. From among the social injustice situations you have addressed in the past, select a case where you enjoy some common ground with your opponents. This would mean at least (i) agreeing that a social injustice does exist, and (ii) sharing your opponents' conviction that a policy response is morally desirable.
2. Describe the causal factors in said social injustice, and clarify points where you and your opponents agree and disagree in this regard. If there are substantial disagreements about questions of causality, then you might revert to Step 1 and find a different social injustice situation. Or you might just note your points of disagreement but concede the causal picture to your opponents 'for the sake of argument' -- i.e., to enable you to proceed with the development of your arithmetical argument. 
3. Having thus found common ground with your opponents on moral and causal matters, proceed to demonstrate the distinct policy scenarios entailed by your and your opponents' separate arithmetical proclivities. The strongest argument here would persuade your opponents that they would achieve morally superior results by abandoning relative risks in favor of absolute risks as guides of policy and measures of progress against injustice. If you make your argument well, your opponents will evaulate the policy consequences of their own arithmetic, compare the conequences of yours, and realize that yours result in morally superior results.

If you should find you cannot get past Step 1, then you will have discovered that your true grievance is not of an arithmetical nature after all, but of an entirely different sort. In that case, you might do well to abandon an arithmetical principle ('HRX') that has enjoyed no apparent substantive formal development over several decades, as a sterile distraction from the more fundamental argument you ought to be advancing perhaps along philosophical lines.

One of the strongest signs of a sterile theory is that it cannot support technical development. I have no doubt that HRX could be formalized quite readily, and am baffled that you have not pursued this course. If you don't do so in the next month or so, I may take up the matter myself sometime before the end of the year--either as a blog post, or something I submit to Chance or Significance. It is not inconceivable that such development will leave your HRX principle in a state that "puts it beyond use" (as they say in Northern Ireland) for the applications you seem to have been pursuing.

To be sure, I am not dismissing the argument you have been making as utterly uninteresting. I am however deeply suspicious of its failure to develop formally and evolve technically. The sheer amount of verbiage involved is astounding for so straightforward a point.

Kind regards,

Thanks for these comments.  A few responsive observations. 

Regarding Ed Gracely’s point:   I am not making an error by discussing relative rather than absolute differences in rates.  My article is about what the Department of Justice and courts do.  Whether the DOJ and the courts are making an error is another matter. 

But in arguing the greater importance of absolute differences with regard to appraisal of a demographic difference, one must be mindful of the ways absolute differences tend to be affected by the frequency of an outcome, as I explained in a Chance editorial more than decade ago, [1] and as I have explained in scores of places since then.  See, e.g., references 2-4.  In fact, my October 8, 2015 letter to the ASA mentions absolute differences 74 times.  See especially the discussion in reference 3 (at 337-339) about the way that reliance on absolute differences to measure healthcare disparities led Massachusetts to unwisely include a disparities element in its Medicaid pay-for-performance programs and to do so in a way that is likely to result in increased healthcare disparities (regardless of how they are measured). 

Thus, the failure to understand that the two relative differences tend to change in opposite directions as the prevalence of an outcomes (and the fact that so many policies are based on the mistaken belief that reducing the frequency of an outcome will tend to reduce rather increase relative differences) are merely manifestations of the larger failure to recognize patterns by which all standard measures tend to be affected by the frequency of an outcome and implications of those patterns.  

The key consideration for those thinking about broader issues (rather than the particular subject of my article) is that anything said about demographic difference is misleading if it fails to address the way the particular measure employed tends to be affected by the prevalence of the outcome.   

Regarding David Norris’s points:   It may be that many people recognize shortcoming of reliance on large relative differences regarding rare outcome (though they seem all to do so without recognizing that the difference will tend to increase as the outcome becomes even rarer).  But the fact remains that billions of dollars are spent each year studying relative difference in favorable or adverse outcomes (as to numerous issues in the law and the social and medical science) including by highly reputed research institutions, and extremely important policy decisions and law enforcement policies are based on perceptions about relative differences.  All this occurs without any attention to the way the relative difference the observer happens to be examining tends to be affected by the prevalence of an outcome.  And, of course, in cases when the adverse outcome is examined, any understanding an observer may have about effects of the prevalence of an outcome typically is the opposite of reality. 

(Similar issues, of course, apply to reliance on absolute differences, though in the case of absolute differences interpretations cannot turn on whether one examines the favorable or the adverse outcome and there is no widespread mistaken expectation as to the effect of the frequency of an outcome.)  

Mistaken reliance on relative differences predominates even in epidemiology where subgroup effects are identified on the basis a departure from constant relative effect across different baseline rates (an approach that is illogical as well as unsound, as I discuss in reference 3 (at 339-34) and the October 2015 ASA letter (at 12-13). 

In the view of most observers (including me), the number-needed-to treat (NNT), which is a function of the absolute effect of a factor on the outcome rate at issue, is the most important consideration in treatment decisions (though that is still a different matter from whether the absolute difference is a sound measure of association).  But as discussed in the introductory material to reference 5, it is standard practice, as recommended, for example, by Oxford's Center for Evidence Based Medicine and BMJ Clinical Evidence, to employ the relative effect observed in a clinical trial to calculate the NNT in a situation involving different baseline rates from that involved in the trial.  (Both of those entities rejected my suggestion that they modify this guidance, though the UK entity called Patient (patient.info) did appreciatively accept my suggested modifications.)  See also references 6 to 8. 

Not being a statistician, I am not a good person to distinguish between arithmetic and statistics.  It’s all mathematics to me.  But the patterns I describe do not have to do with things like randomness or sampling error (save in that such factors will affect the consistency with which one will observe the patterns I describe).  Rather, the patterns I describe have to do with the fact that, in situations such as that reflected in Figure 1 of reference 9, as an outcome is increasingly restricted toward either end of the overall distribution each standard measure tends to change in a particular way.  One cannot draw conclusions about processes (or about the strength of the associations reflected by a pair of outcome rates) based on those standard measures without distinguishing the extent to which observed patterns are functions of the prevalence of the outcome and the extent to which they reflect something about underlying forces. 

It may be that there are better ways, either as to tone or substance, of articulating my points than found, say, in my October 8, 2015, letter to ASA (or the letters to other entities collected in reference 10 or the workshops collected in reference 11).  I always welcome suggestions.  And I have received some useful ones (including from Dr. Norris).  But it is best when the are focused on particular illustrations that I do use.

Regarding James Higgins’ kind comments about “Scanlan’s paradox”:  In the 2006 Chance editorial I used the term “heuristic rule X” or “HRX” to describe the pattern whereby the two relative difference tend to change in opposite as the prevalence of an outcome changes.  That caused researchers in the UK to term the pattern “Scanlan’s rule.”  That prompted me to create a web page by that name with numerous subpages.  So it is fair to say I promote the usage, though, I hope, not too aggessively.  But a problem with the usage or like shorthand terms for the pattern of relative differences is that the usage may detract from appreciation of the patterns by which absolute differences and odds ratios tend to be affected by the prevalence of an outcome. So there are really two rules, which in workshops I principally refer to as Interpretive Rules 1 and 2 (though the second is pretty complicated).   This is matter of increasing importance as more and more researchers rely on absolute differences to measure healthcare disparities (and various other things that used to be discussed in terms of relative differences) without recognizing the patterns by which absolute differences tend to change as the frequency of an outcome changes – or while mistakenly thinking that because the absolute difference is unaffected by which outcome one examines, it effectively addresses the problems I identify with the two relative differences. 

Finally, a couple of observations about the relationship between the two relative differences and the absolute difference:  As the frequency of an outcome changes, the absolute difference tends to change in the same direction as the smaller relative difference (while the difference measured by the odds ratio tends to change in the opposite direction of the absolute difference and the same direction as the larger relative difference).  See figure 3 (at 22) of the October 2015 ASA letter.  Since in most contexts other than employment testing observers relying on relative differences commonly rely on the larger of the two relative differences (often without any thought to a second relative difference), there is a systematic tendency for observers relying on relative differences to reach opposite conclusion about such things as directions of change over time from observers relying on the absolute difference. 

As discussed in reference 2 and 3 and the ASA October 2015 letter, recognition that the absolute difference and the relative difference the observer happens to be looking at have changed in opposite directions is increasing.  But it is still not widespread.  Many people analyzing demographic differences still seem entirely unaware that  measure other than the one they are using can (much less commonly will or in fact would do so in their study) yield a different results about direction of changes over time from the direction they have identified.  See references 12 and 13 for seemingly extreme situations though they may actually differ little from the standard, as discussed in the concluding pages of reference 3). 

But even where observers have recognized that a relative difference and the absolute difference can yield (or have yielded) different conclusions about such things as directions of changes over time, they have invariably done so without seeming to realize that there is a second relative difference.  This has been the case even though, while all measures may change in the same direction (thus indicating a true change in the strength of the forces causing the outcome rates to differ), when a mentioned relative difference and the absolute difference have changed in different directions, the unmentioned relative difference will necessarily have changed in the opposite direction of the mentioned relative difference and the same direction as the absolute difference.  

1. “Can We Actually Measure Health Disparities?,” Chance (Spring 2006)http://www.jpscanlan.com/images/Can_We_Actually_Measure_Health_Disparities.pdf 

2. “The Mismeasure of Health Disparities,” Journal of Public Health Management and Practice (July/Aug. 2016)http://www.jpscanlan.com/images/The_Mismeasure_of_Health_Disparities_JPHMP_2016_.pdf

3. “Race and Mortality Revisited,” Society (July/Aug. 2014) http://jpscanlan.com/images/Race_and_Mortality_Revisited.pdf

4. “Misunderstanding of Statistics Leads to Misguided Law Enforcement Policies,” Amstat News (Dec. 2012)http://magazine.amstat.org/blog/2012/12/01/misguided-law-enforcement/

5. Subgroup Effects subpage of the Scanlan’s Rule page of jpscanlan.com  http://jpscanlan.com/images/Subgroup_Effects_-_Interaction.pdf

6. Re: The number needed to treat: a clinically useful measure of treatment effect. BMJ (Nov. 21, 2011) http://www.bmj.com/rapid-response/2011/11/21/re-number-needed-treat-clinically-useful-measure-treatment-effect

7. Assumption of constant relative risk reductions across different baseline rates is unsound. CMAJ (Mar. 12, 2012) http://www.cmaj.ca/content/171/4/353.short/reply#cmaj_el_690714
8. Re: Progress research strategy (PROGRESS) 4: Stratified medicine research.  BMJ. (Feb. 25, 2013) http://www.bmj.com/content/346/bmj.e5793/rr/632884
9. “Divining Difference,” Chance (Fall 1994) http://jpscanlan.com/images/Divining_Difference.pdf

10. http://www.jpscanlan.com/measurementletters.html
11. http://jpscanlan.com/publications/conferencepresentations.html
12. http://jpscanlan.com/measuringhealthdisp/spuriouscontradictions.html
13. http://jpscanlan.com/measuringhealthdisp/ahrqsvanderbiltreport.html

Do we spend enough time in our introductory courses on interpreting proportion data? We typically present inferences for one proportion and the difference between two proportions and let it go at that. However when proportion data can be reported in a way as to make good outcomes look bad and bad outcomes look good, as Mr. Scanlan has pointed out on many issues of importance to society, then it is up to us to let our students know about it. Giving a name to a phenomenon, as Tukey often did, may help focus attention on the issue. Let me suggest "Scanlan's paradox" as a possibility.

Below is an item just published involving the statistical concepts addressed in materials I have posted here frequently, including my letters to ASA of October 8, 2015[1] and July 25, 2016.[2]

“Misunderstanding of Statistics Confounds Analyses of Criminal Justice Issues in Baltimore and Voter ID Issues in Texas and North Carolina,” Federalist Society Blog (Oct. 3, 2016)

http://www.fed-soc.org/blog/detail/misunderstanding-of-statistics-confounds-analyses-of-criminal-justice-issues-in-baltimore-and-voter-id-issues-in-texas-and-north-carolina

The item explains that (a) contrary to the belief reflected in the Department of Justice report on police practices in Baltimore, MD, generally reducing adverse criminal justice outcomes will tend to increase the proportions African Americans make up of persons experiencing those outcomes and (b) contrary to the beliefs reflected in court decisions involving voter IDs requirements, less stringent requirements will tend to produce larger relative racial differences in rates of failing to meet the requirements than more stringent ones.

The latter point should be borne in mind by those intending to participate in the voter ID studies discussed on ASA website.

1. http://jpscanlan.com/images/Letter_to_American_Statistical_Association_Oct._8,_2015_.pdf
2. http://www.jpscanlan.com/images/Letter_to_American_Statistical_Association_July_25,_2016_.pdf

Dr. Norris:

A few observations regarding your post of October 10, 2016.

1. At issue is the pattern, inherent in other than highly irregular risk distributions, whereby the rarer an outcome the greater tends to be the relative difference in experiencing it and the smaller tends to be the relative difference in avoiding it. The patterns exist across the distribution and not merely at the tails of the distributions.

It seems that earlier you were treating increasingly large relative differences as an outcome becomes rare as trivial, while now you are treating the increasingly small relative differences in the corresponding opposite outcome as that outcome becomes very common as trivial. In any case, I am not sure that it matters whether one treats either pattern (really the same pattern, see October 2015 ASA letter [1] (at 14 n.22) and slide 51 of the 2015 UMass Medical seminar[2]) as a trivial arithmetical pattern or an important statistical one.

What seems important to me, however, is that virtually no one analyzing group differences in outcome rates seems to understand the effects of the prevalence of an outcome on measures of group differences in outcome rates. Also, important federal policies are based on the belief that reducing the frequency of outcomes like arrest, mortgage rejection, or suspension from school will tend to reduce relative differences in rates of experiencing the outcome. That is, the government’s belief is the exact opposite of reality. At least one federal statute also reflects that belief. See “Race and Mortality Revisited,” Society (July/Aug. 2014) (at 342).[3] And the government monitors the fairness of practices on the basis of the size of relative differences in adverse outcome at the same time that it encourages modification to practices that tend to increase those differences. So complying with government encouragements to reduce adverse outcomes tends to increase, not decrease, the chances that the government will sue one for discrimination.

Also important to understand is that one cannot appraise any demographic differences or how policies affect those differences without understanding how the measures employed (including the absolute difference between rates) tend to be affected by the frequency of an outcome. That, however, may be a bit afield of your instant post, save in that one must understand that no moral question is involved. Rather, the question is what approach most accurately quantifies the differences in the circumstances of two groups reflected by their rates of experiencing some favorable or adverse outcomes.

2. Your post notes that there is mention in the thread that I filed an amicus curia brief in connection with the case of Veasey v. Abbott. I cannot find that mention. In any event, I did not file a brief in the case. I did file an amicus curiae brief [4] in Texas Department of Housing and Community Development v. The Inclusive Communities Project, Inc. (Nov. 17, 2014) and mentioned that brief in the article that I posted at the beginning of this thread. Presumably that is what you’re thinking about.

3. Skipping down a bit, your post observes that I “usually discuss[ ] policy-setting relative to the whole-population rate of the outcome” and suggests that your app shows that it is better to discuss the matter relative to the reference group’s rate. Assuming that I understand this point correctly, it would be more accurate to say, rather than that I “usually” discuss the matter with reference to the whole-population rate, that I “never” do it that way. When using test score data I always discuss the matter using the reference group rate as the benchmark for overall prevalence of an outcome on prevalence. See, e.g., the Harvard University letter (at 5) [5] and the 2008 ICHPS oral (at 3).[6] (See also Section A.8 of the Scanlan’s Rule page, [7]which explains why, though I speak of overall prevalence for shorthand, I actually focus on the rates of the group’s being compared without any interest in an overall rate ) That is what “cutoffs defined by the AG fail rate” means in Figures 1 through 3 of the October 2015 ASA letter or the scores of like figures in letters and conference presentations. See, e.g., Figures 1 through 3 of the UMass seminar. That is also what I meant by “[I]f the cutoff were lowered to the point where 95% of [the advantaged group] passes the test,” in the December 2012 Amstat News column.[8]

Sometimes, without any reference to overall rates, I track the patterns according to objective indicators such as income level reflected by ratios of the poverty line (as in Table 1 and the figures in the 2006 Chance editorial [9] and Figure P.1 (slide 88) of the UMass seminar) or credit score (as in the figures in the March 4, 2013 letter to the Federal Reserve Board and its Appendix).[10] But the comparison is always between the disadvantaged group’s rate and advantaged group’s rate (save, of course, when explaining the impossibility of analyzing group differences based on a comparison of the proportion a group makes up of persons potentially experiencing an outcome and the proportion it comprises of persons actually experiencing the outcome, as in the September 12, 2016 letter to the Antioch Unified School District).[11]

4. Turning to the app you created, I first note that it is akin to what I had in mind when in an email of July 26, I stated that “I wanted to be able to move a cursor across the x-axis of an illustration like Figure 1 (slide 23) of [the UMass seminar] and [have] the two ratios change as I move the cursor.” That figure, however, is based on distributions where the means differ by one standard deviation and my illustrations of changes in measures are commonly based on distributions where the means differ by half a standard deviation (as in Figure 1 of “Divining Difference” Chance (Fall 1994)).[12] An illustration like that produced by your app – but showing the patterns of changing rate ratios as one moves the cursor (with the larger figure in the numerator, see October 2015 ASA letter at 10 n.15) would be very useful for illustrating to policy makers that their beliefs about effects of generally reducing adverse outcome rates on relative differences in rates of experiencing those outcomes are incorrect. I hope you will consider creating such an app.

Otherwise, however, I do not yet understand this illustration or the specifications underlying it at all well enough to know whether I think it has value or is pertinent to issue I address. Apparently you regard it as is showing that the pattern I describe (or what you perceive to be one of two patterns I describe, or some pattern that you divine from my illustrations though I do not actually mention) is “not true in general.” By that, I assume you mean “is not always true” rather “is generally not true.” I do not know whether such point is merely contradicting some supposed claim of mine that I say the patterns I describe will always be observed (see, October 2015 ASA letter (at 27) and the numerous other places going back to 1992 [13] where I explain that tendencies are tendencies) in which case the illustration would not be of value. Possibly, you are saying something else and that something else is something that I have not considered. But I am a long way from understanding that and whether there is a possibility that a point you make or demonstrate affects the points I make about the need to understand patterns by which measures tend to be affected by the prevalence of an outcome.

I can say, however, in my view, the discussion of a very disadvantaged group and a somewhat disadvantaged group has nothing to do with the points I make about appraising differences in the circumstances of two groups reflected by their outcomes rates. Thus, I would suggest sticking with an advantaged and disadvantaged group for purposes of any point you are making about patterns I describe.

1. http://jpscanlan.com/images/Letter_to_American_Statistical_Association_Oct._8,_2015_.pdf
2. http://jpscanlan.com/images/Univ_Mass_Medical_School_Seminar_Nov._18,_2015_.pdf
3. http://jpscanlan.com/images/Race_and_Mortality_Revisited.pdf
4. http://jpscanlan.com/images/Scanlan_amicus_brief_in_Texas_Dpt_of_Housing_case.pdf
5. http://jpscanlan.com/images/Harvard_University_Measurement_Letter.pdf
6. http://www.jpscanlan.com/images/2008_ICHPS_Oral.pdf
7. http://magazine.amstat.org/blog/2012/12/01/misguided-law-enforcement/
8. http://magazine.amstat.org/blog/2012/12/01/misguided-law-enforcement/
9. http://www.jpscanlan.com/images/Can_We_Actually_Measure_Health_Disparities.pdf
10. http://jpscanlan.com/images/Federal_Reserve_Board_Letter_with_Appendix.pdf
11. http://www.jpscanlan.com/images/Letter_to_Antioch_Unified_School_District_Sept._12,_2016_.pdf
12. http://jpscanlan.com/images/Divining_Difference.pdf
13. http://jpscanlan.com/images/American_Banker_4-27-92.pdf

All:

To transcend the squalor of this weekend's politics, I eschewed the news and sought to do something positive for our democracy. Accordingly, I have produced a Shiny app https://dnc-llc.shinyapps.io/ScanlanRule/ that I believe fully demystifies the 'Scanlan Rule'. [I feel compelled to note that the 'dnc' in the URL stands for 'David Norris Consulting', not the other 'DNC'! My aim here is to develop an objective, nonpartisan argument.]

On working this through, I have concluded that the Scanlan Rule actually comprises two ideas.

1. One part of the Scanlan Rule is the trivial, purely 'arithmetical' idea that for a rare adverse outcome, the group-wise proportions not experiencing the outcome will be close to unity, as will the ratio of these proportions. In the context of the Veasey v Abbott case (in connection with which Mr. Scanlan submitted an amicus curiae brief mentioned earlier in this thread), it would seem that this idea directs our attention to a ratio such as 91.9% : 98% = 0.938, for consideration alongside a ratio like 8.1% : 2% = 4. (Here, 8.1% and 2% are the estimated fractions of registered African-American and Anglo voters, respectively, who lacked ID meeting the requirements of Texas SB 14.) Thus, alongside a statement like "African-American voters are 4 times as likely to be disenfranchised by SB 14," we might be instructed to note, "but African-Americans achieve nearly 94% of the enfranchisement of Anglos under this bill."
2. The second part is, I will grant Mr. Scanlan, a nontrivial claim. This is his claim regarding the limit of group outcome ratios as an adverse outcome becomes vanishingly rare. While nontrivial, this claim is also not true in general. Moreover, even when it is true it appears no more morally urgent than is the 'arithmetical' portion of Scanlan's Rule. The Shiny app linked above deals with this 'limit clause' of the Scanlan Rule by scaling the [adverse] outcome according to quantiles of a reference group. This allows the outcomes of other groups to be plotted as cumulative distributions relative to a 45-degree reference-group line. A uniformly disadvantaged group will lie above the reference group's 45-degree line, while a uniformly advantaged group (relative to the reference) will lie strictly below. One can also readily conceive of groups with 'mixed' experience, as e.g. with immigrant groups whose 1st generation may be held back by low capital (including social capital like language skills) but whose later generations are propelled by 'work-ethic'. (Similar, but generationally inverted, phenomena have been described in terms of 'health paradoxes' of immigrant groups whose later generations adopt unhealthy American dietary and other social norms.) The only constraint on comparison-group curves is that they be monotone increasing and run through points (0,0) and (1,1). The app implements a 2-parameter family of such curves, with sliders controlling the slopes at endpoints (0,0) and (1,1). While this family hardly exhausts the possibilities, it does permit enough free play to demonstrate cases where the 'limit clause' of the Scanlan Rule does not hold in general; try it yourself!

Each population group in the app has a 'Scanlan Limit' relative to the reference group. Because the app's diagram shows this reference group as a 45-degree line, each comparison group's 'Scanlan Limit' appears on the figure as the slope of its curve at (0,0).

Mr. Scanlan's discussions typically suggest a view of policy in terms of threshold-setting. While Mr. Scanlan usually discusses policy-setting relative to the whole-population rate of the outcome, I believe the app demonstrates that discussing it relative to a reference group's rate aids in the exposition. (Certainly, in the social justice applications of the Scanlan Rule, the comparisons will often be framed relative to such a reference group, presumed to enjoy a certain standard of justice that it is desired to extend to relatively disadvantaged groups.)

I trust that this brief exposition will suffice for the statistical audience on this ASA Community. Time permitting, I will write a blog post aiming to make this more accessible to a nontechnical audience, and also more directly 'debunking' the Scanlan Rule as an argument against opposition to demographically targeted voting-rights restrictions and similar abuses.

Kind regards,

Mr Scanlan:

I'll take up your challenge to lay out an approach, in terms of the sort of problem you typically address, which might be more fruitful than lobbying the ASA to instruct the President of the United States (!) on an arithmetical principle.

Typically, it seems your problem settings involve matters of social injustice, and common ways of measuring and responding to them. I suggest you try the following:

1. From among the social injustice situations you have addressed in the past, select a case where you enjoy some common ground with your opponents. This would mean at least (i) agreeing that a social injustice does exist, and (ii) sharing your opponents' conviction that a policy response is morally desirable.
2. Describe the causal factors in said social injustice, and clarify points where you and your opponents agree and disagree in this regard. If there are substantial disagreements about questions of causality, then you might revert to Step 1 and find a different social injustice situation. Or you might just note your points of disagreement but concede the causal picture to your opponents 'for the sake of argument' -- i.e., to enable you to proceed with the development of your arithmetical argument. 
3. Having thus found common ground with your opponents on moral and causal matters, proceed to demonstrate the distinct policy scenarios entailed by your and your opponents' separate arithmetical proclivities. The strongest argument here would persuade your opponents that they would achieve morally superior results by abandoning relative risks in favor of absolute risks as guides of policy and measures of progress against injustice. If you make your argument well, your opponents will evaulate the policy consequences of their own arithmetic, compare the conequences of yours, and realize that yours result in morally superior results.

If you should find you cannot get past Step 1, then you will have discovered that your true grievance is not of an arithmetical nature after all, but of an entirely different sort. In that case, you might do well to abandon an arithmetical principle ('HRX') that has enjoyed no apparent substantive formal development over several decades, as a sterile distraction from the more fundamental argument you ought to be advancing perhaps along philosophical lines.

One of the strongest signs of a sterile theory is that it cannot support technical development. I have no doubt that HRX could be formalized quite readily, and am baffled that you have not pursued this course. If you don't do so in the next month or so, I may take up the matter myself sometime before the end of the year--either as a blog post, or something I submit to Chance or Significance. It is not inconceivable that such development will leave your HRX principle in a state that "puts it beyond use" (as they say in Northern Ireland) for the applications you seem to have been pursuing.

To be sure, I am not dismissing the argument you have been making as utterly uninteresting. I am however deeply suspicious of its failure to develop formally and evolve technically. The sheer amount of verbiage involved is astounding for so straightforward a point.

Kind regards,

Thanks for these comments.  A few responsive observations. 

Regarding Ed Gracely’s point:   I am not making an error by discussing relative rather than absolute differences in rates.  My article is about what the Department of Justice and courts do.  Whether the DOJ and the courts are making an error is another matter. 

But in arguing the greater importance of absolute differences with regard to appraisal of a demographic difference, one must be mindful of the ways absolute differences tend to be affected by the frequency of an outcome, as I explained in a Chance editorial more than decade ago, [1] and as I have explained in scores of places since then.  See, e.g., references 2-4.  In fact, my October 8, 2015 letter to the ASA mentions absolute differences 74 times.  See especially the discussion in reference 3 (at 337-339) about the way that reliance on absolute differences to measure healthcare disparities led Massachusetts to unwisely include a disparities element in its Medicaid pay-for-performance programs and to do so in a way that is likely to result in increased healthcare disparities (regardless of how they are measured). 

Thus, the failure to understand that the two relative differences tend to change in opposite directions as the prevalence of an outcomes (and the fact that so many policies are based on the mistaken belief that reducing the frequency of an outcome will tend to reduce rather increase relative differences) are merely manifestations of the larger failure to recognize patterns by which all standard measures tend to be affected by the frequency of an outcome and implications of those patterns.  

The key consideration for those thinking about broader issues (rather than the particular subject of my article) is that anything said about demographic difference is misleading if it fails to address the way the particular measure employed tends to be affected by the prevalence of the outcome.   

Regarding David Norris’s points:   It may be that many people recognize shortcoming of reliance on large relative differences regarding rare outcome (though they seem all to do so without recognizing that the difference will tend to increase as the outcome becomes even rarer).  But the fact remains that billions of dollars are spent each year studying relative difference in favorable or adverse outcomes (as to numerous issues in the law and the social and medical science) including by highly reputed research institutions, and extremely important policy decisions and law enforcement policies are based on perceptions about relative differences.  All this occurs without any attention to the way the relative difference the observer happens to be examining tends to be affected by the prevalence of an outcome.  And, of course, in cases when the adverse outcome is examined, any understanding an observer may have about effects of the prevalence of an outcome typically is the opposite of reality. 

(Similar issues, of course, apply to reliance on absolute differences, though in the case of absolute differences interpretations cannot turn on whether one examines the favorable or the adverse outcome and there is no widespread mistaken expectation as to the effect of the frequency of an outcome.)  

Mistaken reliance on relative differences predominates even in epidemiology where subgroup effects are identified on the basis a departure from constant relative effect across different baseline rates (an approach that is illogical as well as unsound, as I discuss in reference 3 (at 339-34) and the October 2015 ASA letter (at 12-13). 

In the view of most observers (including me), the number-needed-to treat (NNT), which is a function of the absolute effect of a factor on the outcome rate at issue, is the most important consideration in treatment decisions (though that is still a different matter from whether the absolute difference is a sound measure of association).  But as discussed in the introductory material to reference 5, it is standard practice, as recommended, for example, by Oxford's Center for Evidence Based Medicine and BMJ Clinical Evidence, to employ the relative effect observed in a clinical trial to calculate the NNT in a situation involving different baseline rates from that involved in the trial.  (Both of those entities rejected my suggestion that they modify this guidance, though the UK entity called Patient (patient.info) did appreciatively accept my suggested modifications.)  See also references 6 to 8. 

Not being a statistician, I am not a good person to distinguish between arithmetic and statistics.  It’s all mathematics to me.  But the patterns I describe do not have to do with things like randomness or sampling error (save in that such factors will affect the consistency with which one will observe the patterns I describe).  Rather, the patterns I describe have to do with the fact that, in situations such as that reflected in Figure 1 of reference 9, as an outcome is increasingly restricted toward either end of the overall distribution each standard measure tends to change in a particular way.  One cannot draw conclusions about processes (or about the strength of the associations reflected by a pair of outcome rates) based on those standard measures without distinguishing the extent to which observed patterns are functions of the prevalence of the outcome and the extent to which they reflect something about underlying forces. 

It may be that there are better ways, either as to tone or substance, of articulating my points than found, say, in my October 8, 2015, letter to ASA (or the letters to other entities collected in reference 10 or the workshops collected in reference 11).  I always welcome suggestions.  And I have received some useful ones (including from Dr. Norris).  But it is best when the are focused on particular illustrations that I do use.

Regarding James Higgins’ kind comments about “Scanlan’s paradox”:  In the 2006 Chance editorial I used the term “heuristic rule X” or “HRX” to describe the pattern whereby the two relative difference tend to change in opposite as the prevalence of an outcome changes.  That caused researchers in the UK to term the pattern “Scanlan’s rule.”  That prompted me to create a web page by that name with numerous subpages.  So it is fair to say I promote the usage, though, I hope, not too aggessively.  But a problem with the usage or like shorthand terms for the pattern of relative differences is that the usage may detract from appreciation of the patterns by which absolute differences and odds ratios tend to be affected by the prevalence of an outcome. So there are really two rules, which in workshops I principally refer to as Interpretive Rules 1 and 2 (though the second is pretty complicated).   This is matter of increasing importance as more and more researchers rely on absolute differences to measure healthcare disparities (and various other things that used to be discussed in terms of relative differences) without recognizing the patterns by which absolute differences tend to change as the frequency of an outcome changes – or while mistakenly thinking that because the absolute difference is unaffected by which outcome one examines, it effectively addresses the problems I identify with the two relative differences. 

Finally, a couple of observations about the relationship between the two relative differences and the absolute difference:  As the frequency of an outcome changes, the absolute difference tends to change in the same direction as the smaller relative difference (while the difference measured by the odds ratio tends to change in the opposite direction of the absolute difference and the same direction as the larger relative difference).  See figure 3 (at 22) of the October 2015 ASA letter.  Since in most contexts other than employment testing observers relying on relative differences commonly rely on the larger of the two relative differences (often without any thought to a second relative difference), there is a systematic tendency for observers relying on relative differences to reach opposite conclusion about such things as directions of change over time from observers relying on the absolute difference. 

As discussed in reference 2 and 3 and the ASA October 2015 letter, recognition that the absolute difference and the relative difference the observer happens to be looking at have changed in opposite directions is increasing.  But it is still not widespread.  Many people analyzing demographic differences still seem entirely unaware that  measure other than the one they are using can (much less commonly will or in fact would do so in their study) yield a different results about direction of changes over time from the direction they have identified.  See references 12 and 13 for seemingly extreme situations though they may actually differ little from the standard, as discussed in the concluding pages of reference 3). 

But even where observers have recognized that a relative difference and the absolute difference can yield (or have yielded) different conclusions about such things as directions of changes over time, they have invariably done so without seeming to realize that there is a second relative difference.  This has been the case even though, while all measures may change in the same direction (thus indicating a true change in the strength of the forces causing the outcome rates to differ), when a mentioned relative difference and the absolute difference have changed in different directions, the unmentioned relative difference will necessarily have changed in the opposite direction of the mentioned relative difference and the same direction as the absolute difference.  

1. “Can We Actually Measure Health Disparities?,” Chance (Spring 2006)http://www.jpscanlan.com/images/Can_We_Actually_Measure_Health_Disparities.pdf 

2. “The Mismeasure of Health Disparities,” Journal of Public Health Management and Practice (July/Aug. 2016)http://www.jpscanlan.com/images/The_Mismeasure_of_Health_Disparities_JPHMP_2016_.pdf

3. “Race and Mortality Revisited,” Society (July/Aug. 2014) http://jpscanlan.com/images/Race_and_Mortality_Revisited.pdf

4. “Misunderstanding of Statistics Leads to Misguided Law Enforcement Policies,” Amstat News (Dec. 2012)http://magazine.amstat.org/blog/2012/12/01/misguided-law-enforcement/

5. Subgroup Effects subpage of the Scanlan’s Rule page of jpscanlan.com  http://jpscanlan.com/images/Subgroup_Effects_-_Interaction.pdf

6. Re: The number needed to treat: a clinically useful measure of treatment effect. BMJ (Nov. 21, 2011) http://www.bmj.com/rapid-response/2011/11/21/re-number-needed-treat-clinically-useful-measure-treatment-effect

7. Assumption of constant relative risk reductions across different baseline rates is unsound. CMAJ (Mar. 12, 2012) http://www.cmaj.ca/content/171/4/353.short/reply#cmaj_el_690714
8. Re: Progress research strategy (PROGRESS) 4: Stratified medicine research.  BMJ. (Feb. 25, 2013) http://www.bmj.com/content/346/bmj.e5793/rr/632884
9. “Divining Difference,” Chance (Fall 1994) http://jpscanlan.com/images/Divining_Difference.pdf

10. http://www.jpscanlan.com/measurementletters.html
11. http://jpscanlan.com/publications/conferencepresentations.html
12. http://jpscanlan.com/measuringhealthdisp/spuriouscontradictions.html
13. http://jpscanlan.com/measuringhealthdisp/ahrqsvanderbiltreport.html

Do we spend enough time in our introductory courses on interpreting proportion data? We typically present inferences for one proportion and the difference between two proportions and let it go at that. However when proportion data can be reported in a way as to make good outcomes look bad and bad outcomes look good, as Mr. Scanlan has pointed out on many issues of importance to society, then it is up to us to let our students know about it. Giving a name to a phenomenon, as Tukey often did, may help focus attention on the issue. Let me suggest "Scanlan's paradox" as a possibility.

Below is an item just published involving the statistical concepts addressed in materials I have posted here frequently, including my letters to ASA of October 8, 2015[1] and July 25, 2016.[2]

“Misunderstanding of Statistics Confounds Analyses of Criminal Justice Issues in Baltimore and Voter ID Issues in Texas and North Carolina,” Federalist Society Blog (Oct. 3, 2016)

http://www.fed-soc.org/blog/detail/misunderstanding-of-statistics-confounds-analyses-of-criminal-justice-issues-in-baltimore-and-voter-id-issues-in-texas-and-north-carolina

The item explains that (a) contrary to the belief reflected in the Department of Justice report on police practices in Baltimore, MD, generally reducing adverse criminal justice outcomes will tend to increase the proportions African Americans make up of persons experiencing those outcomes and (b) contrary to the beliefs reflected in court decisions involving voter IDs requirements, less stringent requirements will tend to produce larger relative racial differences in rates of failing to meet the requirements than more stringent ones.

The latter point should be borne in mind by those intending to participate in the voter ID studies discussed on ASA website.

1. http://jpscanlan.com/images/Letter_to_American_Statistical_Association_Oct._8,_2015_.pdf
2. http://www.jpscanlan.com/images/Letter_to_American_Statistical_Association_July_25,_2016_.pdf

All:

To transcend the squalor of this weekend's politics, I eschewed the news and sought to do something positive for our democracy. Accordingly, I have produced a Shiny app https://dnc-llc.shinyapps.io/ScanlanRule/ that I believe fully demystifies the 'Scanlan Rule'. [I feel compelled to note that the 'dnc' in the URL stands for 'David Norris Consulting', not the other 'DNC'! My aim here is to develop an objective, nonpartisan argument.]

On working this through, I have concluded that the Scanlan Rule actually comprises two ideas.

1. One part of the Scanlan Rule is the trivial, purely 'arithmetical' idea that for a rare adverse outcome, the group-wise proportions not experiencing the outcome will be close to unity, as will the ratio of these proportions. In the context of the Veasey v Abbott case (in connection with which Mr. Scanlan submitted an amicus curiae brief mentioned earlier in this thread), it would seem that this idea directs our attention to a ratio such as 91.9% : 98% = 0.938, for consideration alongside a ratio like 8.1% : 2% = 4. (Here, 8.1% and 2% are the estimated fractions of registered African-American and Anglo voters, respectively, who lacked ID meeting the requirements of Texas SB 14.) Thus, alongside a statement like "African-American voters are 4 times as likely to be disenfranchised by SB 14," we might be instructed to note, "but African-Americans achieve nearly 94% of the enfranchisement of Anglos under this bill."
2. The second part is, I will grant Mr. Scanlan, a nontrivial claim. This is his claim regarding the limit of group outcome ratios as an adverse outcome becomes vanishingly rare. While nontrivial, this claim is also not true in general. Moreover, even when it is true it appears no more morally urgent than is the 'arithmetical' portion of Scanlan's Rule. The Shiny app linked above deals with this 'limit clause' of the Scanlan Rule by scaling the [adverse] outcome according to quantiles of a reference group. This allows the outcomes of other groups to be plotted as cumulative distributions relative to a 45-degree reference-group line. A uniformly disadvantaged group will lie above the reference group's 45-degree line, while a uniformly advantaged group (relative to the reference) will lie strictly below. One can also readily conceive of groups with 'mixed' experience, as e.g. with immigrant groups whose 1st generation may be held back by low capital (including social capital like language skills) but whose later generations are propelled by 'work-ethic'. (Similar, but generationally inverted, phenomena have been described in terms of 'health paradoxes' of immigrant groups whose later generations adopt unhealthy American dietary and other social norms.) The only constraint on comparison-group curves is that they be monotone increasing and run through points (0,0) and (1,1). The app implements a 2-parameter family of such curves, with sliders controlling the slopes at endpoints (0,0) and (1,1). While this family hardly exhausts the possibilities, it does permit enough free play to demonstrate cases where the 'limit clause' of the Scanlan Rule does not hold in general; try it yourself!

Each population group in the app has a 'Scanlan Limit' relative to the reference group. Because the app's diagram shows this reference group as a 45-degree line, each comparison group's 'Scanlan Limit' appears on the figure as the slope of its curve at (0,0).

Mr. Scanlan's discussions typically suggest a view of policy in terms of threshold-setting. While Mr. Scanlan usually discusses policy-setting relative to the whole-population rate of the outcome, I believe the app demonstrates that discussing it relative to a reference group's rate aids in the exposition. (Certainly, in the social justice applications of the Scanlan Rule, the comparisons will often be framed relative to such a reference group, presumed to enjoy a certain standard of justice that it is desired to extend to relatively disadvantaged groups.)

I trust that this brief exposition will suffice for the statistical audience on this ASA Community. Time permitting, I will write a blog post aiming to make this more accessible to a nontechnical audience, and also more directly 'debunking' the Scanlan Rule as an argument against opposition to demographically targeted voting-rights restrictions and similar abuses.

Kind regards,

------------------------------
David C. Norris, MD
David Norris Consulting, LLC
Seattle, WA

Original Message:
Sent: 10-05-2016 15:43
From: David Norris
Subject: Disparate impact of Baltimore police practices and voter ID laws

Mr Scanlan:

I'll take up your challenge to lay out an approach, in terms of the sort of problem you typically address, which might be more fruitful than lobbying the ASA to instruct the President of the United States (!) on an arithmetical principle.

Typically, it seems your problem settings involve matters of social injustice, and common ways of measuring and responding to them. I suggest you try the following:

1. From among the social injustice situations you have addressed in the past, select a case where you enjoy some common ground with your opponents. This would mean at least (i) agreeing that a social injustice does exist, and (ii) sharing your opponents' conviction that a policy response is morally desirable.
2. Describe the causal factors in said social injustice, and clarify points where you and your opponents agree and disagree in this regard. If there are substantial disagreements about questions of causality, then you might revert to Step 1 and find a different social injustice situation. Or you might just note your points of disagreement but concede the causal picture to your opponents 'for the sake of argument' -- i.e., to enable you to proceed with the development of your arithmetical argument. 
3. Having thus found common ground with your opponents on moral and causal matters, proceed to demonstrate the distinct policy scenarios entailed by your and your opponents' separate arithmetical proclivities. The strongest argument here would persuade your opponents that they would achieve morally superior results by abandoning relative risks in favor of absolute risks as guides of policy and measures of progress against injustice. If you make your argument well, your opponents will evaulate the policy consequences of their own arithmetic, compare the conequences of yours, and realize that yours result in morally superior results.

If you should find you cannot get past Step 1, then you will have discovered that your true grievance is not of an arithmetical nature after all, but of an entirely different sort. In that case, you might do well to abandon an arithmetical principle ('HRX') that has enjoyed no apparent substantive formal development over several decades, as a sterile distraction from the more fundamental argument you ought to be advancing perhaps along philosophical lines.

One of the strongest signs of a sterile theory is that it cannot support technical development. I have no doubt that HRX could be formalized quite readily, and am baffled that you have not pursued this course. If you don't do so in the next month or so, I may take up the matter myself sometime before the end of the year--either as a blog post, or something I submit to Chance or Significance. It is not inconceivable that such development will leave your HRX principle in a state that "puts it beyond use" (as they say in Northern Ireland) for the applications you seem to have been pursuing.

To be sure, I am not dismissing the argument you have been making as utterly uninteresting. I am however deeply suspicious of its failure to develop formally and evolve technically. The sheer amount of verbiage involved is astounding for so straightforward a point.

Kind regards,

------------------------------
David C. Norris, MD
David Norris Consulting, LLC
Seattle, WA

Original Message:
Sent: 10-04-2016 21:05
From: James Scanlan
Subject: Disparate impact of Baltimore police practices and voter ID laws

Thanks for these comments.  A few responsive observations. 

Regarding Ed Gracely’s point:   I am not making an error by discussing relative rather than absolute differences in rates.  My article is about what the Department of Justice and courts do.  Whether the DOJ and the courts are making an error is another matter. 

But in arguing the greater importance of absolute differences with regard to appraisal of a demographic difference, one must be mindful of the ways absolute differences tend to be affected by the frequency of an outcome, as I explained in a Chance editorial more than decade ago, [1] and as I have explained in scores of places since then.  See, e.g., references 2-4.  In fact, my October 8, 2015 letter to the ASA mentions absolute differences 74 times.  See especially the discussion in reference 3 (at 337-339) about the way that reliance on absolute differences to measure healthcare disparities led Massachusetts to unwisely include a disparities element in its Medicaid pay-for-performance programs and to do so in a way that is likely to result in increased healthcare disparities (regardless of how they are measured). 

Thus, the failure to understand that the two relative differences tend to change in opposite directions as the prevalence of an outcomes (and the fact that so many policies are based on the mistaken belief that reducing the frequency of an outcome will tend to reduce rather increase relative differences) are merely manifestations of the larger failure to recognize patterns by which all standard measures tend to be affected by the frequency of an outcome and implications of those patterns.  

The key consideration for those thinking about broader issues (rather than the particular subject of my article) is that anything said about demographic difference is misleading if it fails to address the way the particular measure employed tends to be affected by the prevalence of the outcome.   

Regarding David Norris’s points:   It may be that many people recognize shortcoming of reliance on large relative differences regarding rare outcome (though they seem all to do so without recognizing that the difference will tend to increase as the outcome becomes even rarer).  But the fact remains that billions of dollars are spent each year studying relative difference in favorable or adverse outcomes (as to numerous issues in the law and the social and medical science) including by highly reputed research institutions, and extremely important policy decisions and law enforcement policies are based on perceptions about relative differences.  All this occurs without any attention to the way the relative difference the observer happens to be examining tends to be affected by the prevalence of an outcome.  And, of course, in cases when the adverse outcome is examined, any understanding an observer may have about effects of the prevalence of an outcome typically is the opposite of reality. 

(Similar issues, of course, apply to reliance on absolute differences, though in the case of absolute differences interpretations cannot turn on whether one examines the favorable or the adverse outcome and there is no widespread mistaken expectation as to the effect of the frequency of an outcome.)  

Mistaken reliance on relative differences predominates even in epidemiology where subgroup effects are identified on the basis a departure from constant relative effect across different baseline rates (an approach that is illogical as well as unsound, as I discuss in reference 3 (at 339-34) and the October 2015 ASA letter (at 12-13). 

In the view of most observers (including me), the number-needed-to treat (NNT), which is a function of the absolute effect of a factor on the outcome rate at issue, is the most important consideration in treatment decisions (though that is still a different matter from whether the absolute difference is a sound measure of association).  But as discussed in the introductory material to reference 5, it is standard practice, as recommended, for example, by Oxford's Center for Evidence Based Medicine and BMJ Clinical Evidence, to employ the relative effect observed in a clinical trial to calculate the NNT in a situation involving different baseline rates from that involved in the trial.  (Both of those entities rejected my suggestion that they modify this guidance, though the UK entity called Patient (patient.info) did appreciatively accept my suggested modifications.)  See also references 6 to 8. 

Not being a statistician, I am not a good person to distinguish between arithmetic and statistics.  It’s all mathematics to me.  But the patterns I describe do not have to do with things like randomness or sampling error (save in that such factors will affect the consistency with which one will observe the patterns I describe).  Rather, the patterns I describe have to do with the fact that, in situations such as that reflected in Figure 1 of reference 9, as an outcome is increasingly restricted toward either end of the overall distribution each standard measure tends to change in a particular way.  One cannot draw conclusions about processes (or about the strength of the associations reflected by a pair of outcome rates) based on those standard measures without distinguishing the extent to which observed patterns are functions of the prevalence of the outcome and the extent to which they reflect something about underlying forces. 

It may be that there are better ways, either as to tone or substance, of articulating my points than found, say, in my October 8, 2015, letter to ASA (or the letters to other entities collected in reference 10 or the workshops collected in reference 11).  I always welcome suggestions.  And I have received some useful ones (including from Dr. Norris).  But it is best when the are focused on particular illustrations that I do use.

Regarding James Higgins’ kind comments about “Scanlan’s paradox”:  In the 2006 Chance editorial I used the term “heuristic rule X” or “HRX” to describe the pattern whereby the two relative difference tend to change in opposite as the prevalence of an outcome changes.  That caused researchers in the UK to term the pattern “Scanlan’s rule.”  That prompted me to create a web page by that name with numerous subpages.  So it is fair to say I promote the usage, though, I hope, not too aggessively.  But a problem with the usage or like shorthand terms for the pattern of relative differences is that the usage may detract from appreciation of the patterns by which absolute differences and odds ratios tend to be affected by the prevalence of an outcome. So there are really two rules, which in workshops I principally refer to as Interpretive Rules 1 and 2 (though the second is pretty complicated).   This is matter of increasing importance as more and more researchers rely on absolute differences to measure healthcare disparities (and various other things that used to be discussed in terms of relative differences) without recognizing the patterns by which absolute differences tend to change as the frequency of an outcome changes – or while mistakenly thinking that because the absolute difference is unaffected by which outcome one examines, it effectively addresses the problems I identify with the two relative differences. 

Finally, a couple of observations about the relationship between the two relative differences and the absolute difference:  As the frequency of an outcome changes, the absolute difference tends to change in the same direction as the smaller relative difference (while the difference measured by the odds ratio tends to change in the opposite direction of the absolute difference and the same direction as the larger relative difference).  See figure 3 (at 22) of the October 2015 ASA letter.  Since in most contexts other than employment testing observers relying on relative differences commonly rely on the larger of the two relative differences (often without any thought to a second relative difference), there is a systematic tendency for observers relying on relative differences to reach opposite conclusion about such things as directions of change over time from observers relying on the absolute difference. 

As discussed in reference 2 and 3 and the ASA October 2015 letter, recognition that the absolute difference and the relative difference the observer happens to be looking at have changed in opposite directions is increasing.  But it is still not widespread.  Many people analyzing demographic differences still seem entirely unaware that  measure other than the one they are using can (much less commonly will or in fact would do so in their study) yield a different results about direction of changes over time from the direction they have identified.  See references 12 and 13 for seemingly extreme situations though they may actually differ little from the standard, as discussed in the concluding pages of reference 3). 

But even where observers have recognized that a relative difference and the absolute difference can yield (or have yielded) different conclusions about such things as directions of changes over time, they have invariably done so without seeming to realize that there is a second relative difference.  This has been the case even though, while all measures may change in the same direction (thus indicating a true change in the strength of the forces causing the outcome rates to differ), when a mentioned relative difference and the absolute difference have changed in different directions, the unmentioned relative difference will necessarily have changed in the opposite direction of the mentioned relative difference and the same direction as the absolute difference.  

1. “Can We Actually Measure Health Disparities?,” Chance (Spring 2006)http://www.jpscanlan.com/images/Can_We_Actually_Measure_Health_Disparities.pdf 

2. “The Mismeasure of Health Disparities,” Journal of Public Health Management and Practice (July/Aug. 2016)http://www.jpscanlan.com/images/The_Mismeasure_of_Health_Disparities_JPHMP_2016_.pdf

3. “Race and Mortality Revisited,” Society (July/Aug. 2014) http://jpscanlan.com/images/Race_and_Mortality_Revisited.pdf

4. “Misunderstanding of Statistics Leads to Misguided Law Enforcement Policies,” Amstat News (Dec. 2012)http://magazine.amstat.org/blog/2012/12/01/misguided-law-enforcement/

5. Subgroup Effects subpage of the Scanlan’s Rule page of jpscanlan.com  http://jpscanlan.com/images/Subgroup_Effects_-_Interaction.pdf

6. Re: The number needed to treat: a clinically useful measure of treatment effect. BMJ (Nov. 21, 2011) http://www.bmj.com/rapid-response/2011/11/21/re-number-needed-treat-clinically-useful-measure-treatment-effect

7. Assumption of constant relative risk reductions across different baseline rates is unsound. CMAJ (Mar. 12, 2012) http://www.cmaj.ca/content/171/4/353.short/reply#cmaj_el_690714
8. Re: Progress research strategy (PROGRESS) 4: Stratified medicine research.  BMJ. (Feb. 25, 2013) http://www.bmj.com/content/346/bmj.e5793/rr/632884
9. “Divining Difference,” Chance (Fall 1994) http://jpscanlan.com/images/Divining_Difference.pdf

10. http://www.jpscanlan.com/measurementletters.html
11. http://jpscanlan.com/publications/conferencepresentations.html
12. http://jpscanlan.com/measuringhealthdisp/spuriouscontradictions.html
13. http://jpscanlan.com/measuringhealthdisp/ahrqsvanderbiltreport.html

------------------------------
James Scanlan
James P. Scanlan Attorney At Law

Original Message:
Sent: 10-04-2016 14:14
From: James Higgins
Subject: Disparate impact of Baltimore police practices and voter ID laws

Do we spend enough time in our introductory courses on interpreting proportion data? We typically present inferences for one proportion and the difference between two proportions and let it go at that. However when proportion data can be reported in a way as to make good outcomes look bad and bad outcomes look good, as Mr. Scanlan has pointed out on many issues of importance to society, then it is up to us to let our students know about it. Giving a name to a phenomenon, as Tukey often did, may help focus attention on the issue. Let me suggest "Scanlan's paradox" as a possibility.

------------------------------
James Higgins
Kansas State Univ

Original Message:
Sent: 10-03-2016 13:07
From: James Scanlan
Subject: Disparate impact of Baltimore police practices and voter ID laws

Below is an item just published involving the statistical concepts addressed in materials I have posted here frequently, including my letters to ASA of October 8, 2015[1] and July 25, 2016.[2]

“Misunderstanding of Statistics Confounds Analyses of Criminal Justice Issues in Baltimore and Voter ID Issues in Texas and North Carolina,” Federalist Society Blog (Oct. 3, 2016)

http://www.fed-soc.org/blog/detail/misunderstanding-of-statistics-confounds-analyses-of-criminal-justice-issues-in-baltimore-and-voter-id-issues-in-texas-and-north-carolina

The item explains that (a) contrary to the belief reflected in the Department of Justice report on police practices in Baltimore, MD, generally reducing adverse criminal justice outcomes will tend to increase the proportions African Americans make up of persons experiencing those outcomes and (b) contrary to the beliefs reflected in court decisions involving voter IDs requirements, less stringent requirements will tend to produce larger relative racial differences in rates of failing to meet the requirements than more stringent ones.

The latter point should be borne in mind by those intending to participate in the voter ID studies discussed on ASA website.

1. http://jpscanlan.com/images/Letter_to_American_Statistical_Association_Oct._8,_2015_.pdf
2. http://www.jpscanlan.com/images/Letter_to_American_Statistical_Association_July_25,_2016_.pdf

------------------------------
James Scanlan
James P. Scanlan Attorney At Law
------------------------------

Below is an item just published involving the statistical concepts addressed in materials I have posted here frequently, including my letters to ASA of October 8, 2015[1] and July 25, 2016.[2]

“Misunderstanding of Statistics Confounds Analyses of Criminal Justice Issues in Baltimore and Voter ID Issues in Texas and North Carolina,” Federalist Society Blog (Oct. 3, 2016)

http://www.fed-soc.org/blog/detail/misunderstanding-of-statistics-confounds-analyses-of-criminal-justice-issues-in-baltimore-and-voter-id-issues-in-texas-and-north-carolina

The item explains that (a) contrary to the belief reflected in the Department of Justice report on police practices in Baltimore, MD, generally reducing adverse criminal justice outcomes will tend to increase the proportions African Americans make up of persons experiencing those outcomes and (b) contrary to the beliefs reflected in court decisions involving voter IDs requirements, less stringent requirements will tend to produce larger relative racial differences in rates of failing to meet the requirements than more stringent ones.

The latter point should be borne in mind by those intending to participate in the voter ID studies discussed on ASA website.

1. http://jpscanlan.com/images/Letter_to_American_Statistical_Association_Oct._8,_2015_.pdf
2. http://www.jpscanlan.com/images/Letter_to_American_Statistical_Association_July_25,_2016_.pdf

Mr. Scanlon's letters are one reason I keep checking this website. Fascinating reasoning.  

------------------------------
Timothy Marvin

Original Message:
Sent: 10-03-2016 13:07
From: James Scanlan
Subject: Disparate impact of Baltimore police practices and voter ID laws

Below is an item just published involving the statistical concepts addressed in materials I have posted here frequently, including my letters to ASA of October 8, 2015[1] and July 25, 2016.[2]

“Misunderstanding of Statistics Confounds Analyses of Criminal Justice Issues in Baltimore and Voter ID Issues in Texas and North Carolina,” Federalist Society Blog (Oct. 3, 2016)

http://www.fed-soc.org/blog/detail/misunderstanding-of-statistics-confounds-analyses-of-criminal-justice-issues-in-baltimore-and-voter-id-issues-in-texas-and-north-carolina

The item explains that (a) contrary to the belief reflected in the Department of Justice report on police practices in Baltimore, MD, generally reducing adverse criminal justice outcomes will tend to increase the proportions African Americans make up of persons experiencing those outcomes and (b) contrary to the beliefs reflected in court decisions involving voter IDs requirements, less stringent requirements will tend to produce larger relative racial differences in rates of failing to meet the requirements than more stringent ones.

The latter point should be borne in mind by those intending to participate in the voter ID studies discussed on ASA website.

1. http://jpscanlan.com/images/Letter_to_American_Statistical_Association_Oct._8,_2015_.pdf
2. http://www.jpscanlan.com/images/Letter_to_American_Statistical_Association_July_25,_2016_.pdf

------------------------------
James Scanlan
James P. Scanlan Attorney At Law
------------------------------

I think Dr. Norris is arrogantly and unfairly dismissive of
Scanlan's work. What Norris appears to have produced
with his app, is what in signal detection theory would be
called an ROC curve, which plots the hits against the false
alarms; under a standard normal shift model with
the hit distribution right shifted to the false alarm
distribution, the hits and false alarms are upper tail
probability coordinate points. This is not what Scanlan is
concerned about.
What Scanlan is interested in, although he
uses neither the notation nor the language of probability theory, which makes
his work sometimes hard to parse, is the
ratio of these tail areas, or risk ratios, and how these ratios change as the argument, x,
changes. Risk ratio and their use are standard in say epidemiology, where both
upper tail and lower tail ratios can be of focus. This is an area of Scanlan's concern.
Norris appears to have ignored
any literature or knowledge of work in the area.
If he wants more rigor than Scanlan has provided,
he can consult Lambert and Subramanian, Social Choice and Welfare, 2014, 43, 565-576. There
he will find a stated but unproven theorem (p. 569) for positive random variables, which addresses
Scanlan's rule. Note the stated theorem does not apply to normal densities.

------------------------------
Hoben Thomas
Pennsylvania State University

Original Message:
Sent: 10-05-2016 13:15
From: Timothy Marvin
Subject: Disparate impact of Baltimore police practices and voter ID laws

Mr. Scanlon's letters are one reason I keep checking this website. Fascinating reasoning.  

------------------------------
Timothy Marvin

Original Message:
Sent: 10-03-2016 13:07
From: James Scanlan
Subject: Disparate impact of Baltimore police practices and voter ID laws

Below is an item just published involving the statistical concepts addressed in materials I have posted here frequently, including my letters to ASA of October 8, 2015[1] and July 25, 2016.[2]

“Misunderstanding of Statistics Confounds Analyses of Criminal Justice Issues in Baltimore and Voter ID Issues in Texas and North Carolina,” Federalist Society Blog (Oct. 3, 2016)

http://www.fed-soc.org/blog/detail/misunderstanding-of-statistics-confounds-analyses-of-criminal-justice-issues-in-baltimore-and-voter-id-issues-in-texas-and-north-carolina

The item explains that (a) contrary to the belief reflected in the Department of Justice report on police practices in Baltimore, MD, generally reducing adverse criminal justice outcomes will tend to increase the proportions African Americans make up of persons experiencing those outcomes and (b) contrary to the beliefs reflected in court decisions involving voter IDs requirements, less stringent requirements will tend to produce larger relative racial differences in rates of failing to meet the requirements than more stringent ones.

The latter point should be borne in mind by those intending to participate in the voter ID studies discussed on ASA website.

1. http://jpscanlan.com/images/Letter_to_American_Statistical_Association_Oct._8,_2015_.pdf
2. http://www.jpscanlan.com/images/Letter_to_American_Statistical_Association_July_25,_2016_.pdf

------------------------------
James Scanlan
James P. Scanlan Attorney At Law
------------------------------

I thank Hoben Thomas for alerting me to the possibility that, due to its superficial resemblance to an ROC plot, the figure in my Shiny app https://dnc-llc.shinyapps.io/ScanlanRule/ might be misinterpreted as one. I believe that anyone who spends a few minutes actually using the app, however, will readily appreciate that the semantics of its 'Scanlan plot' are quite distinct from those of ROC plots. For example, a grounding concept of the ROC plot is the existence of a true binary state (say, aircraft/bird in an air defense radar or disease/not-disease in a clinical testing scenario) which it is desired to 'detect' by applying a cutoff to a continuous 'signal'. I do not believe there are such 'true state', 'detection' and 'signal' concepts in the Scanlan Rule, nor can any such construct be meaningfully extracted from the app's 'Scanlan plot'.

The abstract of the Lambert & Subramanian paper reads as follows: 

"Demographic disparities between the rates of occurrence of an adverse economic outcome can be observed to be increasing even as general social improvements supposedly lead towards the elimination of the adverse outcome in question. Scanlan (Chance 19(2):47–51, 2006) noticed this tendency and developed a ‘heuristic rule’ to explain it. In this paper, we explore the issue analytically, providing a criterion from stochastic ordering theory under which one of two demographic groups can be considered disadvantaged and the other advantaged, and showing that Scanlan’s heuristic obtains as a rigorous finding in such cases. Normative implications and appropriate social policy are discussed."

While applying 'stochastic ordering theory' to this question would now seem unnecessary in light of the highly accessible geometric intuition presented in the app, I am indeed interested to learn what "normative implications" might be drawn and clearly articulated by Lambert & Subramanian.

------------------------------
David C. Norris, MD
David Norris Consulting, LLC
Seattle, WA

Original Message:
Sent: 10-11-2016 13:41
From: Hoben Thomas
Subject: Disparate impact of Baltimore police practices and voter ID laws

I think Dr. Norris is arrogantly and unfairly dismissive of
Scanlan's work. What Norris appears to have produced
with his app, is what in signal detection theory would be
called an ROC curve, which plots the hits against the false
alarms; under a standard normal shift model with
the hit distribution right shifted to the false alarm
distribution, the hits and false alarms are upper tail
probability coordinate points. This is not what Scanlan is
concerned about.
What Scanlan is interested in, although he
uses neither the notation nor the language of probability theory, which makes
his work sometimes hard to parse, is the
ratio of these tail areas, or risk ratios, and how these ratios change as the argument, x,
changes. Risk ratio and their use are standard in say epidemiology, where both
upper tail and lower tail ratios can be of focus. This is an area of Scanlan's concern.
Norris appears to have ignored
any literature or knowledge of work in the area.
If he wants more rigor than Scanlan has provided,
he can consult Lambert and Subramanian, Social Choice and Welfare, 2014, 43, 565-576. There
he will find a stated but unproven theorem (p. 569) for positive random variables, which addresses
Scanlan's rule. Note the stated theorem does not apply to normal densities.

------------------------------
Hoben Thomas
Pennsylvania State University

Original Message:
Sent: 10-05-2016 13:15
From: Timothy Marvin
Subject: Disparate impact of Baltimore police practices and voter ID laws

Mr. Scanlon's letters are one reason I keep checking this website. Fascinating reasoning.  

------------------------------
Timothy Marvin

Original Message:
Sent: 10-03-2016 13:07
From: James Scanlan
Subject: Disparate impact of Baltimore police practices and voter ID laws

Below is an item just published involving the statistical concepts addressed in materials I have posted here frequently, including my letters to ASA of October 8, 2015[1] and July 25, 2016.[2]

“Misunderstanding of Statistics Confounds Analyses of Criminal Justice Issues in Baltimore and Voter ID Issues in Texas and North Carolina,” Federalist Society Blog (Oct. 3, 2016)

http://www.fed-soc.org/blog/detail/misunderstanding-of-statistics-confounds-analyses-of-criminal-justice-issues-in-baltimore-and-voter-id-issues-in-texas-and-north-carolina

The item explains that (a) contrary to the belief reflected in the Department of Justice report on police practices in Baltimore, MD, generally reducing adverse criminal justice outcomes will tend to increase the proportions African Americans make up of persons experiencing those outcomes and (b) contrary to the beliefs reflected in court decisions involving voter IDs requirements, less stringent requirements will tend to produce larger relative racial differences in rates of failing to meet the requirements than more stringent ones.

The latter point should be borne in mind by those intending to participate in the voter ID studies discussed on ASA website.

1. http://jpscanlan.com/images/Letter_to_American_Statistical_Association_Oct._8,_2015_.pdf
2. http://www.jpscanlan.com/images/Letter_to_American_Statistical_Association_July_25,_2016_.pdf

------------------------------
James Scanlan
James P. Scanlan Attorney At Law
------------------------------

I thank Peter Kenny for his effort to 'bridge the gap' here in a most helpful way, and for valuable feedback on the app.

1. I have updated the app to more clearly illustrate how the 'Scanlan limit' and 'Elite ratio' sliders parametrize the 'Scanlan curves' of the comparison groups. The precise manner in which the curves are plotted out is irrelevant, although I'm glad to share the code. The important point is that these are proper CDFs, being monotone increasing and passing through points (0,0) and (1,1). The app now shows fine dotted tangent lines at the endpoints of the comparison-group CDFs. (Please consider that a more sophisticated user interface would dispense with sliders, and instead allow direct manipulation of these tangent lines.) 
2. The focus Peter Kenny places on the 'uniform' (dis)advantage concept has helped me to realize that this word is not at all right. Accordingly, I have changed the default labels to "Strictly (dis)advantaged". To be quite formal about what I mean about these concepts, I'll say simply that the 'Scanlan curve' for a strictly disadvantaged group lies strictly above the reference-group line on the open interval (0,1). Thus, for any 'policy setting' that creates an adverse outcome for R% of the reference group and D% of a strictly disadvantaged group, we have D>R strictly. (And vice versa of course for a group that is strictly advantaged relative to the reference group with respect to the given policy question.)
3. A counterexample to (i.e., disproof of) the 'weak form' of the Scanlan Rule can be had by loading the app with the defaults, and sliding the log₁₀('Scanlan limit') slider for the strictly disadvantaged group down to 0.1 or 0.0; the group will remain strictly disadvantaged according to the above definition, yet the convexity at (0,0) will disprove the 'Scanlan rule'.
4. Peter Kenny's expressed hope that the conversation can now move to the particulars of particular cases is most salutary. One reason I coined the term 'Scanlan limit' is to provide a means for challenging Mr. Scanlan on an objective basis in particular cases. We might now ask Mr. Scanlan for his estimate of the Scanlan limit in each particular case he engages with. (Anyone who takes his shifted normal densities at face value holds that the Scanlan limit for a disadvantaged group is typically +∞, which I plan to demonstrate is a most untenable position.) In any case, now that my app has made a geometric expression of the Scanlan Rule potentially accessible to a wide audience—and has shown this 'rule' in fact to be a fallacy—the Scanlan Rule can no longer be advanced as if it were a universal law governing "reality". Instead, the Scanlan Rule can now be advanced only as a 'rule of thumb', with its applicability in each specific case being debatable on the particulars. I predict most such debates will prove fatal for the Scanlan Rule.

Kind regards,

------------------------------
David C. Norris, MD
David Norris Consulting, LLC
Seattle, WA

Original Message:
Sent: 10-13-2016 05:52
From: Peter Kenny
Subject: Disparate impact of Baltimore police practices and voter ID laws

I thank Peter Kenny for his effort to 'bridge the gap' here in a most helpful way, and for valuable feedback on the app.

1. I have updated the app to more clearly illustrate how the 'Scanlan limit' and 'Elite ratio' sliders parametrize the 'Scanlan curves' of the comparison groups. The precise manner in which the curves are plotted out is irrelevant, although I'm glad to share the code. The important point is that these are proper CDFs, being monotone increasing and passing through points (0,0) and (1,1). The app now shows fine dotted tangent lines at the endpoints of the comparison-group CDFs. (Please consider that a more sophisticated user interface would dispense with sliders, and instead allow direct manipulation of these tangent lines.) 
2. The focus Peter Kenny places on the 'uniform' (dis)advantage concept has helped me to realize that this word is not at all right. Accordingly, I have changed the default labels to "Strictly (dis)advantaged". To be quite formal about what I mean about these concepts, I'll say simply that the 'Scanlan curve' for a strictly disadvantaged group lies strictly above the reference-group line on the open interval (0,1). Thus, for any 'policy setting' that creates an adverse outcome for R% of the reference group and D% of a strictly disadvantaged group, we have D>R strictly. (And vice versa of course for a group that is strictly advantaged relative to the reference group with respect to the given policy question.)
3. A counterexample to (i.e., disproof of) the 'weak form' of the Scanlan Rule can be had by loading the app with the defaults, and sliding the log₁₀('Scanlan limit') slider for the strictly disadvantaged group down to 0.1 or 0.0; the group will remain strictly disadvantaged according to the above definition, yet the convexity at (0,0) will disprove the 'Scanlan rule'.
4. Peter Kenny's expressed hope that the conversation can now move to the particulars of particular cases is most salutary. One reason I coined the term 'Scanlan limit' is to provide a means for challenging Mr. Scanlan on an objective basis in particular cases. We might now ask Mr. Scanlan for his estimate of the Scanlan limit in each particular case he engages with. (Anyone who takes his shifted normal densities at face value holds that the Scanlan limit for a disadvantaged group is typically +∞, which I plan to demonstrate is a most untenable position.) In any case, now that my app has made a geometric expression of the Scanlan Rule potentially accessible to a wide audience—and has shown this 'rule' in fact to be a fallacy—the Scanlan Rule can no longer be advanced as if it were a universal law governing "reality". Instead, the Scanlan Rule can now be advanced only as a 'rule of thumb', with its applicability in each specific case being debatable on the particulars. I predict most such debates will prove fatal for the Scanlan Rule.

Kind regards,

------------------------------
David C. Norris, MD
David Norris Consulting, LLC
Seattle, WA

Original Message:
Sent: 10-13-2016 05:52
From: Peter Kenny
Subject: Disparate impact of Baltimore police practices and voter ID laws

Dear Peter Kenny,

Thank you again for your constructive efforts to promote a properly scientific dialogue.

1. I do thank you for highlighting the L&S citation, and intend to respond directly when I have had the opportunity to obtain and review the article. I look forward in particular to 3 instructive benefits from the article:
   • Comparing the L&S definition of 'disadvantage' (based, I presume, on 'stochastic dominance theory') against my own definition of 'strict disadvantage'
   • Comparing their sense of what the 'Scanlan Rule' is or might be against my own sense of it
   • Evaluating what I hope will be their clearly articulated normative consequences of the 'Scanlan Rule'
2. In the 'Scanlan plot' of my app, R(x) ≡ x by construction. Thus the 'Scanlan ratio' which you denote D(x)/R(x) is actually D(x)/x, the slope of the line drawn from (0,0) to the intersection of the 'Scanlan policy line' with the 'Scanlan curve' of the group in question. This geometric objectification of a key construct in the Scanlan Rule is a major aim of my formulation. I have updated the app now to show this line. Note in particular how this line approaches the 'Scanlan limit' (dotted) line as the policy setting approaches zero. Thank you again for helpful feedback on the app!
3. I would disagree with you as to the existence of a claim of universality. The claim of a "tendency, inherent in other than highly irregular risk distributions" would in any professional technical discourse be interpreted as a claim of a generic property. By advancing his case in the (social-)scientific literature and on ASA Connect, Mr. Scanlan has invited scrutiny by the standards of such discourse.
4. I would be inclined to invert the spirit of your point #4, and to place the onus upon Mr. Scanlan to defend his claims empirically. You seem (correct me if I am wrong here) to be seriously doubting the empirical falsifiability of Mr. Scanlan's doctrine "in individual cases". I concur; it seems quite likely to me that the Scanlan doctrine is not falsifiable—i.e. that it fails the demarcation criterion Karl Popper advanced as a distinction between science and non-science. It is really up to Mr. Scanlan to demonstrate otherwise.

Kind regards,

Dear Peter Kenny,

Having at last obtained the L&S article [1], I can confirm that their interpretation of the Scanlan Rule corresponds to mine, and that their mathematical treatment was equivalent, if somewhat less elementary. The 'likelihood ratio ordering' they introduce from 'stochastic dominance theory' corresponds precisely to the concavity condition I identified earlier in this thread as equivalent to the Scanlan Rule:

...

Lambert & Subramanian's composition FA ° FD^-1 in fact accomplishes (by less elementary means) the same effect as rescaling their utility Y without loss of generality to convert F_A to a uniform distribution. That, in turn, is equivalent to my use of quantiles of a reference population in place of the measure Y. That reference appears in my Shiny app as the 45° line. Of note, their Theorem 2 provides the basis for this trick: 

------------------------------
David C. Norris, MD
David Norris Consulting, LLC
Seattle, WA

Original Message:
Sent: 10-14-2016 17:21
From: Peter Kenny
Subject: Disparate impact of Baltimore police practices and voter ID laws

Jonathan Siegel, thank you for initiating a healthy new phase of the discussion—one focused on interpretation and normative implications. I'll reply just briefly, pointwise:

1. Your parenthetical remark under #1 would be an argument in favor of my choice of axes, with (typically disadvantaged) comparison groups on the vertical, and the (typically relatively advantaged) reference group on the horizontal axis.
2. I'd like to pick up and extend your point #2 with specific reference to the 'voter ID' laws mentioned in the title of this thread. If the idiom of the 'Scanlan policy line' is applicable here, then we need to suppose we can conceive a continuum of 'voter ID' policy implementations, running from less strict (slide the line to the left) to more strict (slide the line to the right). In this setting, a convex (upward) 'Scanlan curve' for African-Americans in the Texas case would imply that mere tinkering with the severity of voter ID requirements is pointless, and that voter ID legislation must be abandoned altogether as inherently racially discriminatory.

Kind regards,

The reductio ad absurdum that Jonathan Siegel notes was intentional, if not entirely overt. I did preface my reductio with the proviso, "If the idiom of the 'Scanlan policy line' is applicable here..." It seems appropriate here to offer a preview of how I may proceed to argue the latter parts of my planned multi-part blog post on this issue. The idiom of attaching '(dis)advantage' to groups themselves, while drawing a single 'policy line' constitutes, in my view, the divide-by-zero error at the root of the 'Scanlan fallacy'.

Or perhaps begging the question is the better term. Why not draw two policy lines, explicitly showing the biased nature of the policy? Would we speak of the 'disadvantage' of African-Americans with respect to Jim Crow laws? (Clearly not; I almost feel offended having written that sentence myself, frankly.) To the extent that policy discussions can articulate the causal basis for the discriminatory nature of Baltimore's policing or Texas's voter ID law, then we will be able to draw those 2 lines and explicitly address the discriminatory bias separating them.

The whole matter seems to me fundamentally connected with Judea Pearl's causal-statistical distinction. Arguing these critically important matters on a purely statistical basis (to say nothing of a purely geometrical one!) puts us in a treacherous 'causality-free zone' where 'anything goes'. Which is dangerous for our democracy, I think.

As I'm now departing from this thread, I'll offer the following tools to those who remain:

• This discussion of the 'baloney detection kit' [sic!] from Ch 13 of Carl Sagan's dying love-letter to humanity, The Demon-Haunted World: Science as a Candle in the Dark.
• An interesting discussion of Immunizing Strategies & Epistemic Defense Mechanisms employed by pseudoscience, in which section 3.1.2 seems of particular relevance. The opening quote from Frank Cioffi explains some part of my interest in this matter.

Many thanks to everybody for the excellent exchange!

Kind regards,

------------------------------
David C. Norris, MD
David Norris Consulting, LLC
Seattle, WA

Original Message:
Sent: 10-20-2016 07:09
From: Jonathan Siegel
Subject: Disparate impact of Baltimore police practices and voter ID laws

I appreciate David Norris's reply.

I think a difficulty with taking the "inherently discriminatory" argument too far is that under the interpretation, any government action can be characterized as "inherently discriminatory." For example, if, the government provides education or healthcare, than under the theory ambient societal discrimination will result in the advantaged group benefiting from education etc. more than the disadvantaged group. Should we conclude that education, health care, etc. are inherently discriminatory programs and, to avoid causing disparate impact resulting from unequal effect, the government should never provide them.

Perhaps the State of Virginia, which briefly abolished public education to avoid complying with desegregation, was presciently progressive. If one does nothing at all, one certainly doesn't discriminate. And a legal regime which drives government to do nothing to avoid doing harm is not necessarily helpful to the disadvantaged group.

I think this logical "reductio ad absurdem" consequence - that taken to its logical limit it results in making doing nothing the only way to avoid doing harm and hence the only legally permissible course - is the best argument for not making the prevalence ratio approach ones sole measure of outcome. It seems intuitively obvious that from the point of view of the disadvantaged group a large absolute benefit may often be well worth a small disparate impact at the tails.

Jonathan Siegel

Sent from my iPhone