Discussion: View Thread

Dear all,

I am reviewing a Statistical Analysis Plan for a clinical trial and I found that study center is included automatically as covariate in the ANOVA model for the primary analysis. There are about 20 study centers, which will add in 19 indicator center variables. I personally don't think that center should be added in the model as covariate unless there is significance evidence to do so (e.g. significant center by treatment effect). Based on my experience, in primary analysis I usually don't specify to include center as a covariate, but as a sensitivity analysis I would study the interaction effect of study center and if found to be significant, an alternative model might be used (e.g. treat center as a random effect in the model).

My question to you is: what is your experience when dealing with multi-center clinical trials? Do you usually add Center as a covariate in your primary analysis (model)? If so, what is the reason (gain) to add center as a covariate? Any thoughts would be appreciated.

Thanks very much!
-------------------------------------------
[Caiyan] [Li]

-------------------------------------------

Although I do not work in clinical trials, in other experimental fields it would be used as an independent variable (design factor) with its interactions rather than just its main effect.   It could reduce the size of the error term (residuals). Finding that some centers did not follow the general pattern would signal the need to double check data entry, follow up activity, etc.


-------------------------------------------
Arthur Kendall
Social Research Consultants
-------------------------------------------

ICH E-9 (Section 3.2) states

The  main treatment effect may be investigated first using a model which allows for centre differences, but does not include a term for treatment-by-centre  interaction.  If  the   treatment   effect  is  homogeneous  across centres, the routine inclusion of interaction terms in the model reduces the efficiency of the test for the main effects. In the presence  of true heterogeneity of treatment effects, the interpretation of the main treatment effect is controversial.

The question becomes one of interpretation. I know many statisticians who don't include site in the model for the primary analysis and interpret "may be investigated" to be secondary

-------------------------------------------
David Bristol
Statistical Consulting Services
-------------------------------------------

I respectfully disagree with David's simultaneous reply in one respect.  He also is essentially suggesting a meta-analytic approach.  But even with a treatment by center interaction, the main effect still has interpretability. If this was not the case, meta-analysis would not reside at the very top of the "evidence pyramid". If you google this, you will see many versions, but they all put meta-analysis at the top, and there are few good applications of fixed effects.

Jon
-------------------------------------------
Jon Shuster
University of Florida
-------------------------------------------







Original Message:
Sent: 12-02-2011 12:59
From: David Bristol
Subject: Covariate adjustment: should we include study center as a covariate?


ICH E-9 (Section 3.2) states

The  main treatment effect may be investigated first using a model which allows for centre differences, but does not include a term for treatment-by-centre  interaction.  If  the   treatment   effect  is  homogeneous  across centres, the routine inclusion of interaction terms in the model reduces the efficiency of the test for the main effects. In the presence  of true heterogeneity of treatment effects, the interpretation of the main treatment effect is controversial.

The question becomes one of interpretation. I know many statisticians who don't include site in the model for the primary analysis and interpret "may be investigated" to be secondary

-------------------------------------------
David Bristol
Statistical Consulting Services
-------------------------------------------

A nice paper on this is by Paul P. Gallo, Practical Issues in Linear Models Analyses in Mulicenter Clinical Trials in the Biometrical Report, Vol. 6, No. 2, Summer 1998.

-------------------------------------------
Rocco Brunelle
Senior Statistician
Bowsher Brunelle Smith LLC
-------------------------------------------

Original Message:
Sent: 12-02-2011 13:10
From: Jon Shuster
Subject: Covariate adjustment: should we include study center as a covariate?


I respectfully disagree with David's simultaneous reply in one respect.  He also is essentially suggesting a meta-analytic approach.  But even with a treatment by center interaction, the main effect still has interpretability. If this was not the case, meta-analysis would not reside at the very top of the "evidence pyramid". If you google this, you will see many versions, but they all put meta-analysis at the top, and there are few good applications of fixed effects.

Jon
-------------------------------------------
Jon Shuster
University of Florida
-------------------------------------------

I dislike calling a multicenter clinical trial a meta analysis.  I think the term meta analysis should be reserved for combining information from multiple distinct studies.  A clincal trial is a single study.  We certainly have something similar to a meta analysis as each center has a dsitinct set of patients with independent data sets.  But the protocols are identical, the data collected is the same and the statistical plan is based on the combination of data from all sites.  Sample size requirements are usually based on power requirements for endpoints that are analyzed on the combined data.

In clinical trials regulated by FDA, the FDA is very concerned about variable treatment effect in multicenter trials and similarly and perhaps more importantly for country/region in international trials.  This was a topic of a Princeton-Trenton Chapter seminar just this week. Josh Chen gave a presentation on it and he is a member of a pharma working group on the issue.  In the regulatory setting this is very important because the regulatory authorities need to decide on the approval of the drug/treatment in the country that they represent.

-------------------------------------------
Michael Chernick
Director of Biostatistical Services
Lankenau Institute for Medical Research
-------------------------------------------







Original Message:
Sent: 12-02-2011 13:10
From: Jon Shuster
Subject: Covariate adjustment: should we include study center as a covariate?


I respectfully disagree with David's simultaneous reply in one respect.  He also is essentially suggesting a meta-analytic approach.  But even with a treatment by center interaction, the main effect still has interpretability. If this was not the case, meta-analysis would not reside at the very top of the "evidence pyramid". If you google this, you will see many versions, but they all put meta-analysis at the top, and there are few good applications of fixed effects.

Jon
-------------------------------------------
Jon Shuster
University of Florida
-------------------------------------------

Thanks all for the nice discussion.

I agree with Michael that a prospective multi-center clinical trial (majority trials are multi-center) shouldn't be considered as meta-analysis, which usually refers to retrospectively analyzing data from different similar trials.

I was fortunate to work in the FDA for couple years and I understand what Michael is saying below. Yes, the regulatory agency is very concerned with the heterogeniety across regions (US vs. OUS). But is it the same with study centers in the same region (e.g. all US centers)? From my limited experience, many companies don't include center in the primary analysis. Maybe it varies depending on the company's usual practice?

Michael, what was the recommendation from the seminar you mentioned? Is it possible for us to view the seminar materials?

Thanks a lot!
-------------------------------------------
[Caiyan] [Li]

-------------------------------------------







Original Message:
Sent: 12-02-2011 13:45
From: Michael Chernick
Subject: Covariate adjustment: should we include study center as a covariate?

I dislike calling a multicenter clinical trial a meta analysis.  I think the term meta analysis should be reserved for combining information from multiple distinct studies.  A clincal trial is a single study.  We certainly have something similar to a meta analysis as each center has a dsitinct set of patients with independent data sets.  But the protocols are identical, the data collected is the same and the statistical plan is based on the combination of data from all sites.  Sample size requirements are usually based on power requirements for endpoints that are analyzed on the combined data.

In clinical trials regulated by FDA, the FDA is very concerned about variable treatment effect in multicenter trials and similarly and perhaps more importantly for country/region in international trials.  This was a topic of a Princeton-Trenton Chapter seminar just this week. Josh Chen gave a presentation on it and he is a member of a pharma working group on the issue.  In the regulatory setting this is very important because the regulatory authorities need to decide on the approval of the drug/treatment in the country that they represent.

-------------------------------------------
Michael Chernick
Director of Biostatistical Services
Lankenau Institute for Medical Research
-------------------------------------------







Original Message:
Sent: 12-02-2011 13:10
From: Jon Shuster
Subject: Covariate adjustment: should we include study center as a covariate?


I respectfully disagree with David's simultaneous reply in one respect.  He also is essentially suggesting a meta-analytic approach.  But even with a treatment by center interaction, the main effect still has interpretability. If this was not the case, meta-analysis would not reside at the very top of the "evidence pyramid". If you google this, you will see many versions, but they all put meta-analysis at the top, and there are few good applications of fixed effects.

Jon
-------------------------------------------
Jon Shuster
University of Florida
-------------------------------------------

Josh spoke about consistency across regions and discussed examples of trials where there were inconsistencies across regions.  Part of the discussion included reasons for inconsistencies across regions and methods to test for inconsistencies.  Countries like Japan want to require a high percentage of the trial patients come from their country so that they can judge the drug based solely on their own data.  Empirical Bayes shrinkage estimators were suggested as a means to obtain an adjusted treatment effect by region.  The slides will be available to the Chapter members on the Chapter site.  Non-chapter members could probably contact Josh if they are interested in his slides.

Regarding center effects, the FDA is tending to require the companies to test for such effects in ANOVA models.  While this is not as great of a concern as region differences it is getting much more attention these days.
-------------------------------------------
Michael Chernick
Director of Biostatistical Services
Lankenau Institute for Medical Research
-------------------------------------------







Original Message:
Sent: 12-02-2011 13:59
From: Caiyan Li
Subject: Covariate adjustment: should we include study center as a covariate?


Thanks all for the nice discussion.

I agree with Michael that a prospective multi-center clinical trial (majority trials are multi-center) shouldn't be considered as meta-analysis, which usually refers to retrospectively analyzing data from different similar trials.

I was fortunate to work in the FDA for couple years and I understand what Michael is saying below. Yes, the regulatory agency is very concerned with the heterogeniety across regions (US vs. OUS). But is it the same with study centers in the same region (e.g. all US centers)? From my limited experience, many companies don't include center in the primary analysis. Maybe it varies depending on the company's usual practice?

Michael, what was the recommendation from the seminar you mentioned? Is it possible for us to view the seminar materials?

Thanks a lot!
-------------------------------------------
[Caiyan] [Li]

-------------------------------------------







Original Message:
Sent: 12-02-2011 13:45
From: Michael Chernick
Subject: Covariate adjustment: should we include study center as a covariate?

I dislike calling a multicenter clinical trial a meta analysis.  I think the term meta analysis should be reserved for combining information from multiple distinct studies.  A clincal trial is a single study.  We certainly have something similar to a meta analysis as each center has a dsitinct set of patients with independent data sets.  But the protocols are identical, the data collected is the same and the statistical plan is based on the combination of data from all sites.  Sample size requirements are usually based on power requirements for endpoints that are analyzed on the combined data.

In clinical trials regulated by FDA, the FDA is very concerned about variable treatment effect in multicenter trials and similarly and perhaps more importantly for country/region in international trials.  This was a topic of a Princeton-Trenton Chapter seminar just this week. Josh Chen gave a presentation on it and he is a member of a pharma working group on the issue.  In the regulatory setting this is very important because the regulatory authorities need to decide on the approval of the drug/treatment in the country that they represent.

-------------------------------------------
Michael Chernick
Director of Biostatistical Services
Lankenau Institute for Medical Research
-------------------------------------------

I do not think one size fits all.  And meta-analysis may be just termnology, as the basics are exactly per a random effects trial, especially if you have subject level data.  But there has to be rationale for fixed vs. random effects in both multicenter trials or on meta-analysis.  There is no diagnostic test for heterogeneity, as the possible conclusions are (1) there is significant heterogeneity or (2) heterogeneityis oinclnclusive.  You can never prove homogeneity beyond a reasonable doubt.  Aslo, one never accounts for what influence such a diagnostic test has on the power and type I error.  People just assume they came down the right branch of the tree, whereas these tests are highly error prone.

So if it is a drug trial with very tight eligibility and objective evaluation, fixed effects seem fine.  If it is a device trial, where surgery is involved, it should be random effects.

Other trials fall in the cracks, and to be conservative, I would need to be convinced at the design phase that fixed effects are acceptable before I planned on a fixed basis.  Random effects will generally have more uncertainty in the global effect size estimate, but have a wider validity, since it is not wrong to use random effects when there are really fixed effects.

Jon

-------------------------------------------
Jon Shuster
University of Florida
-------------------------------------------







Original Message:
Sent: 12-02-2011 13:59
From: Caiyan Li
Subject: Covariate adjustment: should we include study center as a covariate?


Thanks all for the nice discussion.

I agree with Michael that a prospective multi-center clinical trial (majority trials are multi-center) shouldn't be considered as meta-analysis, which usually refers to retrospectively analyzing data from different similar trials.

I was fortunate to work in the FDA for couple years and I understand what Michael is saying below. Yes, the regulatory agency is very concerned with the heterogeniety across regions (US vs. OUS). But is it the same with study centers in the same region (e.g. all US centers)? From my limited experience, many companies don't include center in the primary analysis. Maybe it varies depending on the company's usual practice?

Michael, what was the recommendation from the seminar you mentioned? Is it possible for us to view the seminar materials?

Thanks a lot!
-------------------------------------------
[Caiyan] [Li]

-------------------------------------------







Original Message:
Sent: 12-02-2011 13:45
From: Michael Chernick
Subject: Covariate adjustment: should we include study center as a covariate?

I dislike calling a multicenter clinical trial a meta analysis.  I think the term meta analysis should be reserved for combining information from multiple distinct studies.  A clincal trial is a single study.  We certainly have something similar to a meta analysis as each center has a dsitinct set of patients with independent data sets.  But the protocols are identical, the data collected is the same and the statistical plan is based on the combination of data from all sites.  Sample size requirements are usually based on power requirements for endpoints that are analyzed on the combined data.

In clinical trials regulated by FDA, the FDA is very concerned about variable treatment effect in multicenter trials and similarly and perhaps more importantly for country/region in international trials.  This was a topic of a Princeton-Trenton Chapter seminar just this week. Josh Chen gave a presentation on it and he is a member of a pharma working group on the issue.  In the regulatory setting this is very important because the regulatory authorities need to decide on the approval of the drug/treatment in the country that they represent.

-------------------------------------------
Michael Chernick
Director of Biostatistical Services
Lankenau Institute for Medical Research
-------------------------------------------

I agree about the fixed versus random effects argument that you raise.  As regarding testing for differences in mean effects I don't understand at all what you are saying.  An ANOVA testing for center difference does exact what we want.  All statistical tests of course carry uncertainty with the conclusions.  A treatment by center interaction can also be tested for in an ANOVA.

-------------------------------------------
Michael Chernick
Director of Biostatistical Services
Lankenau Institute for Medical Research
-------------------------------------------







Original Message:
Sent: 12-02-2011 14:46
From: Jon Shuster
Subject: Covariate adjustment: should we include study center as a covariate?

I do not think one size fits all.  And meta-analysis may be just termnology, as the basics are exactly per a random effects trial, especially if you have subject level data.  But there has to be rationale for fixed vs. random effects in both multicenter trials or on meta-analysis.  There is no diagnostic test for heterogeneity, as the possible conclusions are (1) there is significant heterogeneity or (2) heterogeneityis oinclnclusive.  You can never prove homogeneity beyond a reasonable doubt.  Aslo, one never accounts for what influence such a diagnostic test has on the power and type I error.  People just assume they came down the right branch of the tree, whereas these tests are highly error prone.

So if it is a drug trial with very tight eligibility and objective evaluation, fixed effects seem fine.  If it is a device trial, where surgery is involved, it should be random effects.

Other trials fall in the cracks, and to be conservative, I would need to be convinced at the design phase that fixed effects are acceptable before I planned on a fixed basis.  Random effects will generally have more uncertainty in the global effect size estimate, but have a wider validity, since it is not wrong to use random effects when there are really fixed effects.

Jon

-------------------------------------------
Jon Shuster
University of Florida
-------------------------------------------







Original Message:
Sent: 12-02-2011 13:59
From: Caiyan Li
Subject: Covariate adjustment: should we include study center as a covariate?


Thanks all for the nice discussion.

I agree with Michael that a prospective multi-center clinical trial (majority trials are multi-center) shouldn't be considered as meta-analysis, which usually refers to retrospectively analyzing data from different similar trials.

I was fortunate to work in the FDA for couple years and I understand what Michael is saying below. Yes, the regulatory agency is very concerned with the heterogeniety across regions (US vs. OUS). But is it the same with study centers in the same region (e.g. all US centers)? From my limited experience, many companies don't include center in the primary analysis. Maybe it varies depending on the company's usual practice?

Michael, what was the recommendation from the seminar you mentioned? Is it possible for us to view the seminar materials?

Thanks a lot!
-------------------------------------------
[Caiyan] [Li]

-------------------------------------------







Original Message:
Sent: 12-02-2011 13:45
From: Michael Chernick
Subject: Covariate adjustment: should we include study center as a covariate?

I dislike calling a multicenter clinical trial a meta analysis.  I think the term meta analysis should be reserved for combining information from multiple distinct studies.  A clincal trial is a single study.  We certainly have something similar to a meta analysis as each center has a dsitinct set of patients with independent data sets.  But the protocols are identical, the data collected is the same and the statistical plan is based on the combination of data from all sites.  Sample size requirements are usually based on power requirements for endpoints that are analyzed on the combined data.

In clinical trials regulated by FDA, the FDA is very concerned about variable treatment effect in multicenter trials and similarly and perhaps more importantly for country/region in international trials.  This was a topic of a Princeton-Trenton Chapter seminar just this week. Josh Chen gave a presentation on it and he is a member of a pharma working group on the issue.  In the regulatory setting this is very important because the regulatory authorities need to decide on the approval of the drug/treatment in the country that they represent.

-------------------------------------------
Michael Chernick
Director of Biostatistical Services
Lankenau Institute for Medical Research
-------------------------------------------

Yes, it is always possible to test for center difference and/or treatment by center interaction in models.
But our main interest here is the treatment effect (compared to the control), not the center effect. Usually, we would tend to include those variables that are expected to be predictive in the response, such as treatment, baseline of the response, patient characteristics. When we design the trial, we generally would make the assumption that centers are "similar" in the conduct of the trial so that we can make conclusions based on the combined data. Of course, this assumption is strong and may be violated, that is why we usually conduct sensitivity analyses to assess this assumption. If there is strong evidence show that this assumption is voilated, the result from the primary analysis (where center is not included as a covariate) might be challenged in the interpretation. Anyway, my point still is: I would not include center automatically in the primary analysis where the initial efficacy evaluation is bases on, but instead include it in additional analyses to include center. In the study result report, I would include both results (with or without center), but make conclusions based on the primary model without center as covariate unless there is significant interaction effect in the model with center.

-------------------------------------------
[Caiyan] [Li]

-------------------------------------------







Original Message:
Sent: 12-02-2011 15:02
From: Michael Chernick
Subject: Covariate adjustment: should we include study center as a covariate?

I agree about the fixed versus random effects argument that you raise.  As regarding testing for differences in mean effects I don't understand at all what you are saying.  An ANOVA testing for center difference does exact what we want.  All statistical tests of course carry uncertainty with the conclusions.  A treatment by center interaction can also be tested for in an ANOVA.

-------------------------------------------
Michael Chernick
Director of Biostatistical Services
Lankenau Institute for Medical Research
-------------------------------------------







Original Message:
Sent: 12-02-2011 14:46
From: Jon Shuster
Subject: Covariate adjustment: should we include study center as a covariate?

I do not think one size fits all.  And meta-analysis may be just termnology, as the basics are exactly per a random effects trial, especially if you have subject level data.  But there has to be rationale for fixed vs. random effects in both multicenter trials or on meta-analysis.  There is no diagnostic test for heterogeneity, as the possible conclusions are (1) there is significant heterogeneity or (2) heterogeneityis oinclnclusive.  You can never prove homogeneity beyond a reasonable doubt.  Aslo, one never accounts for what influence such a diagnostic test has on the power and type I error.  People just assume they came down the right branch of the tree, whereas these tests are highly error prone.

So if it is a drug trial with very tight eligibility and objective evaluation, fixed effects seem fine.  If it is a device trial, where surgery is involved, it should be random effects.

Other trials fall in the cracks, and to be conservative, I would need to be convinced at the design phase that fixed effects are acceptable before I planned on a fixed basis.  Random effects will generally have more uncertainty in the global effect size estimate, but have a wider validity, since it is not wrong to use random effects when there are really fixed effects.

Jon

-------------------------------------------
Jon Shuster
University of Florida
-------------------------------------------







Original Message:
Sent: 12-02-2011 13:59
From: Caiyan Li
Subject: Covariate adjustment: should we include study center as a covariate?


Thanks all for the nice discussion.

I agree with Michael that a prospective multi-center clinical trial (majority trials are multi-center) shouldn't be considered as meta-analysis, which usually refers to retrospectively analyzing data from different similar trials.

I was fortunate to work in the FDA for couple years and I understand what Michael is saying below. Yes, the regulatory agency is very concerned with the heterogeniety across regions (US vs. OUS). But is it the same with study centers in the same region (e.g. all US centers)? From my limited experience, many companies don't include center in the primary analysis. Maybe it varies depending on the company's usual practice?

Michael, what was the recommendation from the seminar you mentioned? Is it possible for us to view the seminar materials?

Thanks a lot!
-------------------------------------------
[Caiyan] [Li]

-------------------------------------------

Yes, I think what you say you do is what is commonly done.  What I said about treatment effect applies to the difference between the mean effect for the treatment and the mean effect for the treatment.  The reason for looking at a center effect is to make sure it is not influencing the magnitude of the treatment effect.
-------------------------------------------
Michael Chernick
Director of Biostatistical Services
Lankenau Institute for Medical Research
-------------------------------------------







Original Message:
Sent: 12-02-2011 15:16
From: Caiyan Li
Subject: Covariate adjustment: should we include study center as a covariate?

Yes, it is always possible to test for center difference and/or treatment by center interaction in models.
But our main interest here is the treatment effect (compared to the control), not the center effect. Usually, we would tend to include those variables that are expected to be predictive in the response, such as treatment, baseline of the response, patient characteristics. When we design the trial, we generally would make the assumption that centers are "similar" in the conduct of the trial so that we can make conclusions based on the combined data. Of course, this assumption is strong and may be violated, that is why we usually conduct sensitivity analyses to assess this assumption. If there is strong evidence show that this assumption is voilated, the result from the primary analysis (where center is not included as a covariate) might be challenged in the interpretation. Anyway, my point still is: I would not include center automatically in the primary analysis where the initial efficacy evaluation is bases on, but instead include it in additional analyses to include center. In the study result report, I would include both results (with or without center), but make conclusions based on the primary model without center as covariate unless there is significant interaction effect in the model with center.

-------------------------------------------
[Caiyan] [Li]

-------------------------------------------







Original Message:
Sent: 12-02-2011 15:02
From: Michael Chernick
Subject: Covariate adjustment: should we include study center as a covariate?

I agree about the fixed versus random effects argument that you raise.  As regarding testing for differences in mean effects I don't understand at all what you are saying.  An ANOVA testing for center difference does exact what we want.  All statistical tests of course carry uncertainty with the conclusions.  A treatment by center interaction can also be tested for in an ANOVA.

-------------------------------------------
Michael Chernick
Director of Biostatistical Services
Lankenau Institute for Medical Research
-------------------------------------------







Original Message:
Sent: 12-02-2011 14:46
From: Jon Shuster
Subject: Covariate adjustment: should we include study center as a covariate?

I do not think one size fits all.  And meta-analysis may be just termnology, as the basics are exactly per a random effects trial, especially if you have subject level data.  But there has to be rationale for fixed vs. random effects in both multicenter trials or on meta-analysis.  There is no diagnostic test for heterogeneity, as the possible conclusions are (1) there is significant heterogeneity or (2) heterogeneityis oinclnclusive.  You can never prove homogeneity beyond a reasonable doubt.  Aslo, one never accounts for what influence such a diagnostic test has on the power and type I error.  People just assume they came down the right branch of the tree, whereas these tests are highly error prone.

So if it is a drug trial with very tight eligibility and objective evaluation, fixed effects seem fine.  If it is a device trial, where surgery is involved, it should be random effects.

Other trials fall in the cracks, and to be conservative, I would need to be convinced at the design phase that fixed effects are acceptable before I planned on a fixed basis.  Random effects will generally have more uncertainty in the global effect size estimate, but have a wider validity, since it is not wrong to use random effects when there are really fixed effects.

Jon

-------------------------------------------
Jon Shuster
University of Florida
-------------------------------------------







Original Message:
Sent: 12-02-2011 13:59
From: Caiyan Li
Subject: Covariate adjustment: should we include study center as a covariate?


Thanks all for the nice discussion.

I agree with Michael that a prospective multi-center clinical trial (majority trials are multi-center) shouldn't be considered as meta-analysis, which usually refers to retrospectively analyzing data from different similar trials.

I was fortunate to work in the FDA for couple years and I understand what Michael is saying below. Yes, the regulatory agency is very concerned with the heterogeniety across regions (US vs. OUS). But is it the same with study centers in the same region (e.g. all US centers)? From my limited experience, many companies don't include center in the primary analysis. Maybe it varies depending on the company's usual practice?

Michael, what was the recommendation from the seminar you mentioned? Is it possible for us to view the seminar materials?

Thanks a lot!
-------------------------------------------
[Caiyan] [Li]

-------------------------------------------

-------------------------------------------
George Milliken
-------------------------------------------
This is an age old discussion.  In the world of Animal health, they consider the locations or clinics as a random effect which is really considered clinics as a blocking factor in the study.  This also relates to the inference space issue in that when clinics are considered as a random effect then one can make inferences to all clinics the set of clinics represent, which is called a broad inference space.  In the world of human health they have considered clinics as a fixed effect, which is bothersome to me as when clinics are considered as a fixed effect you can only really draw statistical inferences relating to those clinics in the study, which is called a narrow inference space (Many reseachers ignore this aspect).  In the case where clinics are considered as a fixed effect the treatments are compared by using the within clinic variability or patient to patient variability within a clinic.  When clinics are considered as a random effect, the clinic by treatment interaction is used as an error term for comparing treatments which in essence allows for the broad inference space.  The problem is that a lot of clinical trials involve two treatments and a few clinics, like 10 so the clinic by treatment interaction has only 9 degree of freedom (many studies involve 3 locations providing 2 df to compare treatments).  Where as if these 10 clinics have 10 patients each the residual degrees of freedom will be 90 minus what ever covariates are in the model (not considering clinic as a covariate).  Or if there are 2 clinics with 100 patients each there would be 198 df instead of 1df for comparing treatments.  I think you can see why one likes to consider clinics as fixed. The within clinic variability is generally smaller than the between clinic variability, so one can find more significant differences among the treatments when clinics are considered as a fixed effect (more df and smaller error term) than when it is considered as a random effect (fewer df and larger error term). 
So the main issue is whether you want to make a narrow inference about the treatments or make a broad inference about the treatments.  My phlosophy is that we want to make a broad inference so we can determine the effect of treatments across the population of clinics.






Original Message:
Sent: 12-02-2011 15:02
From: Michael Chernick
Subject: Covariate adjustment: should we include study center as a covariate?

I agree about the fixed versus random effects argument that you raise.  As regarding testing for differences in mean effects I don't understand at all what you are saying.  An ANOVA testing for center difference does exact what we want.  All statistical tests of course carry uncertainty with the conclusions.  A treatment by center interaction can also be tested for in an ANOVA.

-------------------------------------------
Michael Chernick
Director of Biostatistical Services
Lankenau Institute for Medical Research
-------------------------------------------







Original Message:
Sent: 12-02-2011 14:46
From: Jon Shuster
Subject: Covariate adjustment: should we include study center as a covariate?

I do not think one size fits all.  And meta-analysis may be just termnology, as the basics are exactly per a random effects trial, especially if you have subject level data.  But there has to be rationale for fixed vs. random effects in both multicenter trials or on meta-analysis.  There is no diagnostic test for heterogeneity, as the possible conclusions are (1) there is significant heterogeneity or (2) heterogeneityis oinclnclusive.  You can never prove homogeneity beyond a reasonable doubt.  Aslo, one never accounts for what influence such a diagnostic test has on the power and type I error.  People just assume they came down the right branch of the tree, whereas these tests are highly error prone.

So if it is a drug trial with very tight eligibility and objective evaluation, fixed effects seem fine.  If it is a device trial, where surgery is involved, it should be random effects.

Other trials fall in the cracks, and to be conservative, I would need to be convinced at the design phase that fixed effects are acceptable before I planned on a fixed basis.  Random effects will generally have more uncertainty in the global effect size estimate, but have a wider validity, since it is not wrong to use random effects when there are really fixed effects.

Jon

-------------------------------------------
Jon Shuster
University of Florida
-------------------------------------------







Original Message:
Sent: 12-02-2011 13:59
From: Caiyan Li
Subject: Covariate adjustment: should we include study center as a covariate?


Thanks all for the nice discussion.

I agree with Michael that a prospective multi-center clinical trial (majority trials are multi-center) shouldn't be considered as meta-analysis, which usually refers to retrospectively analyzing data from different similar trials.

I was fortunate to work in the FDA for couple years and I understand what Michael is saying below. Yes, the regulatory agency is very concerned with the heterogeniety across regions (US vs. OUS). But is it the same with study centers in the same region (e.g. all US centers)? From my limited experience, many companies don't include center in the primary analysis. Maybe it varies depending on the company's usual practice?

Michael, what was the recommendation from the seminar you mentioned? Is it possible for us to view the seminar materials?

Thanks a lot!
-------------------------------------------
[Caiyan] [Li]

-------------------------------------------

I think this is a controversial issue.  The use of a covariate is not quite the same as stratification, which has information on both the main effect and treatment by center interaction.  But the stratified analysis can be more powerful than either igniroing center or using center as a covariate if center has good predictive value for response differences (interaction) or even if they have differing effects.    The analogy is whether to condict a fixed or random effects meta-analysis.  The covariate approach is a fixed effects meta-analysis whereas the stratified approach is a random effects meta-analysis.

Jon

-------------------------------------------
Jon Shuster
University of Florida
-------------------------------------------

Perhaps list members are just using different dialects of statistics.

Nominal level variables in some disciplines would not be called covariates.  The levels of a nominal level variables can either be used as fixed effects (the set of values are the population of relevant levels) or random (the levels are acted on as if a randomly chosen set of values from the population).

It seems to me that fixed effects from experiment oriented disciplines are (at least very much) the same thing as stratification in survey oriented disciplines. Likewise, random effects form experiment oriented disciplines are (at least very much) the same thing as  cluster effects in survey oriented disciplines.

The only reason I wouldn't spontaneously think of a multi center clinical clinical trial as a meta analysis is that it is part of the analysis a priori, whereas in meta analysis the centers would be thought of as a group after the studies were done.  Both could have fixed and random effects. (And of course the operational definitions would be much closer to being the same when the group of centers were identified a priori in a clinical trial.)

-------------------------------------------
Arthur Kendall
Social Research Consultants
-------------------------------------------







Original Message:
Sent: 12-02-2011 13:05
From: Jon Shuster
Subject: Covariate adjustment: should we include study center as a covariate?


I think this is a controversial issue.  The use of a covariate is not quite the same as stratification, which has information on both the main effect and treatment by center interaction.  But the stratified analysis can be more powerful than either igniroing center or using center as a covariate if center has good predictive value for response differences (interaction) or even if they have differing effects.    The analogy is whether to condict a fixed or random effects meta-analysis.  The covariate approach is a fixed effects meta-analysis whereas the stratified approach is a random effects meta-analysis.

Jon

-------------------------------------------
Jon Shuster
University of Florida
-------------------------------------------

What I've seen in many industry studies that get reviewed at my academic institution is, they have Study Center as a stratification factor in the randomization scheme, and if the primary efficacy variable is binary or time-to-event, then they do a Cochran-Mantel-Haenszel test or stratified log-rank test, respectively, with Study Center as the stratification factor in both instances.  I've never seen Study Center be an explicit effect that people worry about in a primary efficacy analysis.  I think the equivalent in an ANOVA context would be to make Study Center a random effect in a mixed model.

Which leads me to wonder: could it be that the purpose of including Study Center as a covariate is simply to make it a blocking factor in an analysis that uses (for example) SAS Proc GLM instead of SAS Proc Mixed? 

-------------------------------------------
Eric Siegel
Biostatistician
Univ of Arkansas for Medical Sciences
-------------------------------------------