ASA Connect

Attached is an outline/sketch of comments I intend to submit in response to the Request for Comments for the Commission on Evidence-Based Policymaking.  Comments will address issues I raised in letters to ASA dated October 8, 2015,[1] and July 25, 2016.[2]  Also attached is a memo summarizing certain points in the outline and explaining why I am broadly circulating the outline well in advance of the November 14, 2016 due date.  

Key points include (a) problems in analyses of demographic differences that fail to consider ways measures employed tend to be affected by the prevalence of an outcome; (a) problems in subgroup analyses premised on expectation that absent such effect one will observe a constant relative effect across different baseline rates; (c) the illogic of the rate ratio as a measure of association; (d) the unsoundness of analyses of demographic differences based on comparisons of the proportions a group comprises of persons potentially and actually experiencing an outcome; (e) the government’s mistaken belief that reducing the frequency of adverse lending, discipline, criminal justice, and employment outcomes tends to reduce relative differences in rates of experiencing the outcomes; (f) unsoundness of analyses of discrimination issues based on data pertaining solely to persons who accepted some outcome or situation. The outline as it evolves into formal comments may be found here: http://www.jpscanlan.com/images/Outline_of_Comments_to_CEBP.pdf

I am posting this on ASA Connect and to the sections of which I am a member.  Apologies for the duplication.

1.   http://jpscanlan.com/images/Letter_to_American_Statistical_Association_Oct._8,_2015_.pdf

2..  http://www.jpscanlan.com/images/Letter_to_American_Statistical_Association_July_25,_2016_.pdf

Attachment(s)

Transmittal Memo to ASA Members.pdf   151 KB 1 version

Outline of Comments to CEBP.pdf   314 KB 1 version

A comment to Mr. Scanlan and others: the issue here is an instance of the sometimes non-intuitive effects of heterogeneous probabilities and/or confounding variables across (sub)populations. Such things are quite common, Simpson's paradox for example, but are hard for people to understand. See for example,

Vaupel and Yashin (1985), Heterogeneity's ruses: Some surprising effects of selection on population dynamics, The American Statistician 39,176-185. There are plenty of other papers that deal with "heterogeneity's ruses."

In my view, Simpson’s paradox and the points made by Vaupel and Yashin raise issues rather different from those I raise.  Simpson’s paradox involves situations like the following (to use a variation on a situation in which I first encountered the paradox about 37 years ago):  An employer’s suburban facility has 200 applicants,  of whom 90% are white and 10% are black, for 100 jobs; the facility hires 90 white and 10 blacks.  The employer's central city facility has 500 applicants, of whom 50% are white and 50% are black, for 100 jobs; it hires 50 whites and 50 blacks.  The white and black selection rates are identical in each setting.  But aggregated figures show selection rates of 33% for whites and 22% for blacks.  Variations on the paradox exist where the black selection rate is higher than the white selection rate in both settings, but overall selection rates are lower for blacks than whites.

By contrast, issues I raise go to, say, the situation where in the suburban setting we observe hiring of 92 white and 8 blacks (selection rates of 51% for whites and 40% for blacks) and, assuming the forces causing black and white rates to differ are the same in the suburbs as in the city (or that that is what we want to figure out), whether we should expect similar pattern of relative differences in the central city.  And I maintain that, assuming the forces are the same, we would expect the relative difference in selection to be larger, but relative differences in rejection rates to be smaller, in the central city (where the selection rates are lower but rejection rates are higher).  My point (c) goes to the fact that it is impossible to observe the same relative difference in selection rates in the central city facility as in the suburban facility while at the same time observing the same relative difference in rejection rates in the two settings.

See reference 1 with regard to why we would expect larger relative racial differences in public school suspension,  but smaller relative differences in avoiding suspension, in suburbs (where suspension are comparatively uncommon and avoiding suspension is comparatively common) than in central cities.  Table 8 of reference 2 (at 342) illustrates the same point with DOE data on racial differences in pre-school suspensions (and avoidance of suspensions) compared with K12.    

Similarly, issues I raise with regard to my second point (mislettered as a second (a)) go to why we should expect factor like having high income to cause (and why we in fact observe) larger proportionate decreases in mortgage rejection rates for whites than for blacks but larger proportionate increases in mortgage approval rates for blacks than whites (see ref. 2  at 340-341 ) or larger proportionate decreases in dichotomized poor self-rated health for whites than blacks but larger proportionate increases in dichotomized good self-rated health for blacks than whites.  See figure 5 (slide 66) of reference 3 (an interesting illustration, in my view, because the second part of the illustration is simply the first part of the illustration turned upside down).  See figure 7 of ref. 2 (at 341) for an example how having a criminal record reduces favorable outcome rates proportionately more for blacks than whites while increasing adverse outcome rates proportionately more for white than blacks.  See generally references 4 and 5.

I have only scanned the Vaupel and Yashin 1987 article.  But it seems to me that the points they are making about misperceptions regarding things like declining hazard ratios with age relate to selection effects.   I believe the role of selection effects in patterns of changes in relative differences concerning mortality and survival over time (and certain other measurement problems) can be avoided by examining patterns of mortality/survival to various ages (as discussed in references 6-9).  Issues I raise go the fact that we will commonly observe that the higher the age (and hence the higher are the rates of failing to reach it) the smaller tend to be relative differences in failing to reach to it and the larger tend to be the relative differences in reaching it, as shown in references 7-9.  (Crossover issues at very high ages are another matter.)  

Further, while selection effects will contribute to declining relative differences in mortality within age categories (e.g., between age 50 and 60 compared with between ages 60 and 70), such effects will tend to counter the pattern of larger relative differences in survival within higher age groups than lower age groups.  Yet we commonly observe increasing relative differences in survival within higher age groups. See within age group illustrations in references 7-9 and Table 6 of reference 10.  That suggests that, even apart from selection effects, we are observing  a real pattern of decreasing relative differences in the outcome that is becoming more common (mortality) albeit one somewhat enhanced by selection effects.  That is not to say we are actually observing heterogeneity in effects by age group.   Rather, the factor that I discuss as leading to a spurious heterogeneity reflected in decreasing relative differences in mortality by age groups is different from the factor Vaupel and Yashin discuss as leading to the same spurious heterogeneity.  Again, I have not carefully read their paper yet.

1. http://jpscanlan.com/disciplinedisparities/suburbandisparities.html
2. http://jpscanlan.com/images/Race_and_Mortality_Revisited.pdf
3. http://jpscanlan.com/images/Univ_Mass_Medical_School_Seminar_Nov._18,_2015_.pdf
4. http://jpscanlan.com/measuringhealthdisp/reportingheterogeneity.html
5. http://jpscanlan.com/scanlansrule/subgroupeffectsnc.html
6. http://jpscanlan.com/measuringhealthdisp/cohortconsiderations.html
7. http://jpscanlan.com/scanlansrule/lifetableillustrations.html
8.  http://jpscanlan.com/images/LIFE_TABLE_INFORMATION.pdf

9. http://jpscanlan.com/scanlansrule/interactionsbyage.html
10. http://jpscanlan.com/images/Scanlan_JP_NDS_Presentation_2R.ppt

In the transmittal attached to the original post of this thread, I suggested I would have a draft of comments for the Commission on Evidence-Based Policymaking  sufficiently in advance of the November 14 deadline to enable members to provide comments to me or to submit comments to the Commission on the points I raised.  But I did not have a draft I was comfortable with prior to November 14.  Turned out, however, that the deadline was extended until December 14. 

I went ahead and submitted my comments, a copy of which may be found here. 

http://www.jpscanlan.com/images/Comments_of_J_Scanlan_for_Comm_on_Evidence-Based_Policymaking_Nov._14,_2016_.pdf

The Introduction (pages 1-8) and Recommendations (pages 45-46) will give you their gist.

Over the next month, I may be correcting the comments,  supplementing them to focus on discrete issues, and encouraging others to comment on the issues they raise.  The cover of the online version will indicate whether it has corrections and reference any pertinent updates.