ASA Connect

Dear Statistics Colleagues:

My dataset contains people with diabetes at a community health clinic.  The treatment group is being compared to diabetes patients who received usual care at the clinic.  Propensity score adjustment is being used because the groups differ significantly in blood sugar and age, and because the treatment and control groups were not based on a random sample.

I tried two methods of propensity score (1) adjustment: stratification with 5 strata and (2) ATE (Average Treatment Effect) weighting.  The diagnostics indicated that stratification did a good job of balancing blood sugar and age between the two groups, but the ATE weighting produced results in which blood sugar and age had the opposite distribution as the original dataset.  I.E., instead of Xbar_group1>Xbar_group2, Xbar_group1<Xbar_group2.  So, the stratification results look reasonal to me, but the ATE weighting results do not.

Does anyone know of articles that focus on conditions when it's better to use ATE weights or stratification?


------------------------------
Brandy Sinco, BS, MA, MS
Statistician and Programmer/Analyst
------------------------------

I'm not too familiar with causal models, but you might want to look at these:

1. Lunceford, Jared K., and Marie Davidian. "Stratification and weighting via the propensity score in estimation of causal treatment effects: a comparative study." Statistics in medicine23.19 (2004): 2937-2960.
2. Hade EM, Lu B. Bias associated with using the estimated propensity score as a regression covariate. Stat Med. 2013;33(1):74–87. doi:10.1002/sim.5884

I believe the first more directly addresses your question, whereas the second looks at a larger variety of approaches.

------------------------------
David Klemish
Ph.D. Candidate
Duke University
------------------------------

Original Message:
Sent: 04-12-2019 08:34
From: Brandy Sinco
Subject: Propensity Score References

Dear Statistics Colleagues:

My dataset contains people with diabetes at a community health clinic.  The treatment group is being compared to diabetes patients who received usual care at the clinic.  Propensity score adjustment is being used because the groups differ significantly in blood sugar and age, and because the treatment and control groups were not based on a random sample.

I tried two methods of propensity score (1) adjustment: stratification with 5 strata and (2) ATE (Average Treatment Effect) weighting.  The diagnostics indicated that stratification did a good job of balancing blood sugar and age between the two groups, but the ATE weighting produced results in which blood sugar and age had the opposite distribution as the original dataset.  I.E., instead of Xbar_group1>Xbar_group2, Xbar_group1<Xbar_group2.  So, the stratification results look reasonal to me, but the ATE weighting results do not.

Does anyone know of articles that focus on conditions when it's better to use ATE weights or stratification?


------------------------------
Brandy Sinco, BS, MA, MS
Statistician and Programmer/Analyst
------------------------------

Brandy,

If you use R and have data on more than ~1,000 diabetic patients, I highly recommend you try my LocalControlStrategy package for analysis of (observational) cross-sectional data. (The package named simply LocalControl might be more appropriate if your data are longitudinal or the fraction of "treated" patients is relatively small, say < 10%.) LC strategy stresses modern "statistical thinking" and is highly visual ...but also computes some p-values for unsupervised (and nonparametric) learning. Anyway, my package provides many references, as does my web site http://localcontrolstatistics.org 

If your "ATE" adjustment is "inverse propensity weighting" but differs from the formulation of Lunceford & Davidian, you probably have not implemented it correctly. LC strategy provides unbiased estimates of local treatment effect-sizes within clusters (= blocks = subgroups) of diabetic patients who are relatively well-matched on their "other" baseline x-covariates (potential confounders.)

Good Luck!

------------------------------
Bob Obenchain
Principal Consultant
Risk Benefit Statistics LLC
------------------------------

Original Message:
Sent: 04-12-2019 08:34
From: Brandy Sinco
Subject: Propensity Score References

Dear Statistics Colleagues:

My dataset contains people with diabetes at a community health clinic.  The treatment group is being compared to diabetes patients who received usual care at the clinic.  Propensity score adjustment is being used because the groups differ significantly in blood sugar and age, and because the treatment and control groups were not based on a random sample.

I tried two methods of propensity score (1) adjustment: stratification with 5 strata and (2) ATE (Average Treatment Effect) weighting.  The diagnostics indicated that stratification did a good job of balancing blood sugar and age between the two groups, but the ATE weighting produced results in which blood sugar and age had the opposite distribution as the original dataset.  I.E., instead of Xbar_group1>Xbar_group2, Xbar_group1<Xbar_group2.  So, the stratification results look reasonal to me, but the ATE weighting results do not.

Does anyone know of articles that focus on conditions when it's better to use ATE weights or stratification?


------------------------------
Brandy Sinco, BS, MA, MS
Statistician and Programmer/Analyst
------------------------------

1. Make sure your stratification approach and your weighting approach are estimating the same causal estimand. You've said that you created weights to estimate ATE (so your weights must be t/p + (1-t)/(1-p), correct?). Make sure you also weight each stratum by the total number of cases so that your stratification estimator is also estimating ATE. That way you know you're comparing apples to apples. I've seen cases where an analyst creates weights for ATE but then weights the strata by the number of treatment cases in each stratum (which produces an ATT estimate) and then wonder why they look so different.

2. Good performance out of estimators based on propensity score weights depends on having a good propensity score estimator. Traditional logistic regression is usually a poor choice. Kang and Schafer (2007) is a case in point. They show really bad results with weights. Their demonstration shows that the traditional logistic regression model (linear on the log odds scale) does a poor job at the edges. So if you're getting weird results with weights, best bet is that weighting is fine but your propensity score model is not very good. Have a look at the RAND tutorials (https://www.rand.org/statistics/twang/tutorials.html) for a good strategy for working with propensity score weights... which I find generally do outperform other PS methods when done well.

------------------------------
Greg Ridgeway
Associate Professor
University of Pennsylvania
------------------------------

Original Message:
Sent: 04-12-2019 08:34
From: Brandy Sinco
Subject: Propensity Score References

Dear Statistics Colleagues:

My dataset contains people with diabetes at a community health clinic.  The treatment group is being compared to diabetes patients who received usual care at the clinic.  Propensity score adjustment is being used because the groups differ significantly in blood sugar and age, and because the treatment and control groups were not based on a random sample.

I tried two methods of propensity score (1) adjustment: stratification with 5 strata and (2) ATE (Average Treatment Effect) weighting.  The diagnostics indicated that stratification did a good job of balancing blood sugar and age between the two groups, but the ATE weighting produced results in which blood sugar and age had the opposite distribution as the original dataset.  I.E., instead of Xbar_group1>Xbar_group2, Xbar_group1<Xbar_group2.  So, the stratification results look reasonal to me, but the ATE weighting results do not.

Does anyone know of articles that focus on conditions when it's better to use ATE weights or stratification?


------------------------------
Brandy Sinco, BS, MA, MS
Statistician and Programmer/Analyst
------------------------------

Brandy,

It is possible that bias is introduced when using propensity score weighing. I would suggest you to look at the following paper for issues that can arise when using this technique: 
1. Freedman, D. A., & Berk, R. A. (2008). Weighting Regressions by Propensity Scores. Evaluation Review, 32(4), 392–409. https://doi.org/10.1177/0193841X08317586

------------------------------
Stephen Parry
Statistical Consultant
CSCU
------------------------------

Original Message:
Sent: 04-12-2019 08:34
From: Brandy Sinco
Subject: Propensity Score References

Dear Statistics Colleagues:

My dataset contains people with diabetes at a community health clinic.  The treatment group is being compared to diabetes patients who received usual care at the clinic.  Propensity score adjustment is being used because the groups differ significantly in blood sugar and age, and because the treatment and control groups were not based on a random sample.

I tried two methods of propensity score (1) adjustment: stratification with 5 strata and (2) ATE (Average Treatment Effect) weighting.  The diagnostics indicated that stratification did a good job of balancing blood sugar and age between the two groups, but the ATE weighting produced results in which blood sugar and age had the opposite distribution as the original dataset.  I.E., instead of Xbar_group1>Xbar_group2, Xbar_group1<Xbar_group2.  So, the stratification results look reasonal to me, but the ATE weighting results do not.

Does anyone know of articles that focus on conditions when it's better to use ATE weights or stratification?


------------------------------
Brandy Sinco, BS, MA, MS
Statistician and Programmer/Analyst
------------------------------