ASA Connect

How would you conduct propensity score analysis for an observational set of data (two non-randomized treatment groups and multiple baseline characteristics), but we want to be blinded to the outcome.  Would we just stop after calculation of the propensity scores?

------------------------------
Michelle Secic
------------------------------

One cannot estimate propensity scores without somewhat breaking the blind by revealing which of the experimental units are in the same treatment cohort. If the cohorts are of different expected size (established, standard treatment vs new-to-the-market treatment) this info almost surely breaks the blind. But what you can do without breaking the blind is better than what most propensity score estimates typically achieve. You can cluster the experimental units in the confounding X-space of their baseline covariate values. If the clustering is hierarchical, you can predetermine the entire tree rather than having to decide (blinded) exactly how many clusters you will use in any final (unblinded) analysis. In the limit as clusters become very small, cluster membership becomes a "balancing score" as fine as or finer than the TRUE propensity score. On the other hand, very small clusters can turn out to be uninformative (contain only treated units or only control units), so the number of clusters actually used needs to be determined after unblinding (like the calipers on propensity estimates in traditional binning approaches.) Of course, rather large clusters can also be uninformative, but this usually means that the cluster is outside of the X-space "common support" of the treatment cohorts. The general approach I am describing is called LOCAL CONTROL and considerable information about this analysis strategy can be found online at localcontrolstatistics.org. Good luck in you research ...and try to keep the dimension of you X-space small and limited to the few baseline covariates that subject matter experts would consider most important for predicting the y-outcome(s) or treatment choice.   

------------------------------
Robert Obenchain
Principal Consultant
Risk Benefit Statistics

Original Message:
Sent: 12-18-2015 15:33
From: Michelle Secic
Subject: propensity score analysis - blinded to outcome

How would you conduct propensity score analysis for an observational set of data (two non-randomized treatment groups and multiple baseline characteristics), but we want to be blinded to the outcome.  Would we just stop after calculation of the propensity scores?

------------------------------
Michelle Secic
------------------------------

I agree with the comments of R Obenchain. Note, additionally, that the notion of "breaking the blind" can be done selectively. If a person who is not involved with the trial is given the information, and uses it to calculate the propensity scores, the advantages can be gained while keeping most or all of directly involved personnel still blinded. It's not all-or-nothing. In many trials, specific components of the process are done blinded while others are aware of condition. In trials involving evaluation, for instance, evaluation can be done by blinded evaluators, while the rest of the personnel may be fully informed about condition. 

------------------------------
Paul Thompson
Director, Methodology and Data Analysis Center
Sanford Research/USD

Original Message:
Sent: 12-21-2015 04:32
From: Robert Obenchain
Subject: propensity score analysis - blinded to outcome

One cannot estimate propensity scores without somewhat breaking the blind by revealing which of the experimental units are in the same treatment cohort. If the cohorts are of different expected size (established, standard treatment vs new-to-the-market treatment) this info almost surely breaks the blind. But what you can do without breaking the blind is better than what most propensity score estimates typically achieve. You can cluster the experimental units in the confounding X-space of their baseline covariate values. If the clustering is hierarchical, you can predetermine the entire tree rather than having to decide (blinded) exactly how many clusters you will use in any final (unblinded) analysis. In the limit as clusters become very small, cluster membership becomes a "balancing score" as fine as or finer than the TRUE propensity score. On the other hand, very small clusters can turn out to be uninformative (contain only treated units or only control units), so the number of clusters actually used needs to be determined after unblinding (like the calipers on propensity estimates in traditional binning approaches.) Of course, rather large clusters can also be uninformative, but this usually means that the cluster is outside of the X-space "common support" of the treatment cohorts. The general approach I am describing is called LOCAL CONTROL and considerable information about this analysis strategy can be found online at localcontrolstatistics.org. Good luck in you research ...and try to keep the dimension of you X-space small and limited to the few baseline covariates that subject matter experts would consider most important for predicting the y-outcome(s) or treatment choice.   

------------------------------
Robert Obenchain
Principal Consultant
Risk Benefit Statistics

You might consider this method if your data set is large

Obenchain RL, Young SS. (2013) Advancing statistical thinking in health care research. Journal of Statistical Theory and Practice 7, 456-469.

------------------------------
Sidney Young
Retired

Original Message:
Sent: 12-18-2015 15:33
From: Michelle Secic
Subject: propensity score analysis - blinded to outcome

How would you conduct propensity score analysis for an observational set of data (two non-randomized treatment groups and multiple baseline characteristics), but we want to be blinded to the outcome.  Would we just stop after calculation of the propensity scores?

------------------------------
Michelle Secic
------------------------------

Hi Michelle, 

What you didn't mention in your post is WHY it is important for you to be blinded to the outcome at this stage of your statistical analyses. 

Is it because you would like to compute the propensity scores now and fit the outcome model(s) later on? 

How far along in the data collection process are you? Have the data on your outcome variables been fully collected or are you still collecting some of these data?

If you are still collecting data on your outcome variables, what is the motivating factor for initiating the propensity score computation at this stage? Is that factor driven by time constraints or by clinical considerations? 

What does the study protocol stipulates about when the propensity score calculation should be made in relation to the data collection schedule? 

More importantly, does the study protocol mention anything about what would happen if the distributions of propensity scores in the two groups do not overlap (meaning that the groups are not comparable with respect to the observed baseline characteristics)?

Thanks, 

Isabella

------------------------------
Isabella Ghement
Ghement Statistical Consulting Company Ltd.
isabella@ghement.ca

Original Message:
Sent: 12-18-2015 15:33
From: Michelle Secic
Subject: propensity score analysis - blinded to outcome

How would you conduct propensity score analysis for an observational set of data (two non-randomized treatment groups and multiple baseline characteristics), but we want to be blinded to the outcome.  Would we just stop after calculation of the propensity scores?

------------------------------
Michelle Secic
------------------------------

Given the circumstances I'm not sure why you would feel need for this to be blind.  but you could have the propensity analysis done by a third party with the score made available to you only when you are ready to begin the final study analysis.  That would keep you blinded as to the propensity score.

------------------------------
David Mangen

Original Message:
Sent: 12-18-2015 15:33
From: Michelle Secic
Subject: propensity score analysis - blinded to outcome

How would you conduct propensity score analysis for an observational set of data (two non-randomized treatment groups and multiple baseline characteristics), but we want to be blinded to the outcome.  Would we just stop after calculation of the propensity scores?

------------------------------
Michelle Secic
------------------------------

Typically, when someone says they have conducted a "propensity score analysis ... blinded to the outcome" what they mean is they have calculated the propensity scores (PS) without looking at the outcome data at all. Then they match or weight using those PS, evaluating the quality of the matching/weighting - again without looking at the outcome data. If the quality of the matching/weighting is poor, e.g. the standardized differences for some important covariates are large, then the PS model and matching/weighting process can be revised without any fear of biasing the study. The researchers are safe to iteratively revise their matching/weighting procedure because they have blinded themselves to the outcome. They can't be accused of cherry picking their cohort based on the exposure's estimated effect on the outcome. Only after the researchers have agreed on the final PS model and matched/weighted cohort and set that in stone, do they allow themselves to see and analyze the outcome.

------------------------------
Robert Alan Greevy, Jr, PhD
Associate Professor of Biostatistics
Director, Health Services Research Biostatistics
Vanderbilt University School of Medicine
biostat.mc.vanderbilt.edu/RobertGreevy

Original Message:
Sent: 12-18-2015 15:33
From: Michelle Secic
Subject: propensity score analysis - blinded to outcome

How would you conduct propensity score analysis for an observational set of data (two non-randomized treatment groups and multiple baseline characteristics), but we want to be blinded to the outcome.  Would we just stop after calculation of the propensity scores?

------------------------------
Michelle Secic
------------------------------