Discussion: View Thread

 [Full Disclosure: I'm not a biostatistician (I know, I know, how the heck do you get a job with the title of "Statistician" in an AMC and *not* be a biostatistician?  Long story, but not so weird nowadays...).]

Trying to understand why a national non-profit organization would publish to its members Risk of Mortality models which:
o  have half of the estimated coefficients (way) not statistically significant (e.g., 15 out of 30 for one model);
o  include no measures of goodness of fit (e.g., C statistic);
o  no indication of degree of correlation among the factors in the models;
o  no list of candidate factors considered in developing the models.

As far as I can tell, these models *may* be OK for estimating the Expected Mortality Rates used in comparing to Observed Rates. The O/E and (O - E) values are then used to "flag" (or not) the member organization - based on what appear to me to be arbitrarily chosen "cut-offs."

The docs are wanting to use the Models to a priori "rank" patients for treatment, or to determine what treatments to consider to lower patients' risk, but I'm thinking no way - it's all about correlation, not causation.

Need this group's help in clarifying my thinking...thanks!  :-)

-------------------------------------------
Wayne Fischer
Statistician
University of Texas Medical Branch
-------------------------------------------

Wayne:

Is there some historical or administrative reason that a particular set of predictor variables
might be included, even if many of them don't have significant coefficients in the current model?
For example, someone might have done some work back in, say, the 70s that produced the current
set of explanatory variables to be included, and as a result they are always included when a new
model is done.  (This is also a backhanded way to think you don't need to publish any overall
fit statistics, if you're going to always use the same variables regardless.) The cut-offs you mention
could be historical as well.

Another possible reason to include "extra" variables is to make the variance of the fitted dependent variable
as small as possible (by driving down whatever the moral equivalent of R-squared is),
even if it means that you're including some variables that are correlated with
each other and thus decreasing the reliability of the fitted coefficients. I don't like this tack, since it leads
to black-box models that are over-fitted and data-dependent, but others' mileage may vary, especially
if "we've always included this set of variables in our models." But I think your instincts are right for this situation:
you might be able to do something with the fitted value (E), but you can't rely on "looking under the hood"
at the individual coefficients to help determine a course of treatment. In particular, you can have the nasty
situation that among a set of correlated variables, the real explanatory variable loses out in the
luck-of-the-draw "battle" for significance to one of its proxies (e.g., if smoking incidence is highly correlated
with gender, and smoking is the genuine risk factor, but gender "wins" the battle and thus leads a clinician
into thinking, "well, gender has a marginally significant coefficient and smoking does not, so let's
differentiate treatment by gender").

Hope this helps,

>>Kathy

-------------------------------------------
Katherine Godfrey
-------------------------------------------

Thanks, Kathy - that helps a lot.  :-)

The non-profit org "re-generates" the models every six months using all members' data, has been doing so for (several? many?) years, and I've only been involved in looking at their approach since Feb.  But the latest of the two models we're interested in, and their versions from six months ago, are different enough to make me wonder about your thought that some variable have been "carried along" historically.  I talked with one of their statisticians who said that the non-significant variables were included because "they have face validity or members deemed them clinically important."

[More disclosure: Most of my modeling work has been in the engineering and physical sciences, so I may be biased...]  I'd never before heard of those reasons for justifying inclusion of variables.  In fact, my experience has been that when someone has argued for inclusion of a variable(s), many times it turned out not to be so - using Designed Experiments, no less!

So I've ben reading a lot about "Variable Selection" in the social sciences and healthcare.  Hosmer & Lemeshow put it this way:

"Epidemiologic methodologists suggest including all clinically and intuitively relevant variables in the model, regardless of their 'statistical significance.' "  This gives me a queasy feeling...who decides "clinical relevance"..."intuitive relevance"...and based on what criteria?  Shouldn't we let the data speak for themselves?

H&L do go on to point out that "The major problem with this approach is that the model may be 'overfit,' producing numerically unstable estimates.  ...typically characterized by unrealistically large estimated coefficients and/or estimated standard errors."  Seems to me that alone argues *against* loading up a model with "clinically and intuitively relevant" variables.

Anyone else with some thoughts?  Thanks!   :-)

-------------------------------------------
Wayne Fischer
Statistician
University of Texas Medical Branch
-------------------------------------------



Original Message:
Sent: 08-20-2014 23:21
From: Katherine Godfrey
Subject: Interpretation and Use of Risk of Mortality Models

Wayne:

Is there some historical or administrative reason that a particular set of predictor variables
might be included, even if many of them don't have significant coefficients in the current model?
For example, someone might have done some work back in, say, the 70s that produced the current
set of explanatory variables to be included, and as a result they are always included when a new
model is done.  (This is also a backhanded way to think you don't need to publish any overall
fit statistics, if you're going to always use the same variables regardless.) The cut-offs you mention
could be historical as well.

Another possible reason to include "extra" variables is to make the variance of the fitted dependent variable
as small as possible (by driving down whatever the moral equivalent of R-squared is),
even if it means that you're including some variables that are correlated with
each other and thus decreasing the reliability of the fitted coefficients. I don't like this tack, since it leads
to black-box models that are over-fitted and data-dependent, but others' mileage may vary, especially
if "we've always included this set of variables in our models." But I think your instincts are right for this situation:
you might be able to do something with the fitted value (E), but you can't rely on "looking under the hood"
at the individual coefficients to help determine a course of treatment. In particular, you can have the nasty
situation that among a set of correlated variables, the real explanatory variable loses out in the
luck-of-the-draw "battle" for significance to one of its proxies (e.g., if smoking incidence is highly correlated
with gender, and smoking is the genuine risk factor, but gender "wins" the battle and thus leads a clinician
into thinking, "well, gender has a marginally significant coefficient and smoking does not, so let's
differentiate treatment by gender").

Hope this helps,

>>Kathy

-------------------------------------------
Katherine Godfrey
-------------------------------------------

Hosmer & Lemeshow are correct that this is typical practice in epidemiology. Extra variables are included in the model for at least 2 reasons: (1) clinically or theoretically relevant variables need to be tested for significance, otherwise no one will believe the results of the model; (2) in observational studies, you would include variables that were part of the sampling scheme or case-control matching in order to capture (as best you can) factors used in the study design (not needed in designed experiments). However, after including these extra variables in the first model, I recommend paring back the variables to those most significant that still give a good fit, reporting some model comparison measure, e.g., AIC. Then you can say to the non-statistical client "See, I included the variables you thought were relevant, but only these few were really needed.". ------------------------------------------- Linda Pickle StatNet Consulting, LLC -------------------------------------------
Original Message: Sent: 08-21-2014 08:57 From: Wayne Fischer Subject: Interpretation and Use of Risk of Mortality Models Thanks, Kathy - that helps a lot. :-) The non-profit org "re-generates" the models every six months using all members' data, has been doing so for (several? many?) years, and I've only been involved in looking at their approach since Feb. But the latest of the two models we're interested in, and their versions from six months ago, are different enough to make me wonder about your thought that some variable have been "carried along" historically. I talked with one of their statisticians who said that the non-significant variables were included because "they have face validity or members deemed them clinically important." [More disclosure: Most of my modeling work has been in the engineering and physical sciences, so I may be biased...] I'd never before heard of those reasons for justifying inclusion of variables. In fact, my experience has been that when someone has argued for inclusion of a variable(s), many times it turned out not to be so - using Designed Experiments, no less! So I've ben reading a lot about "Variable Selection" in the social sciences and healthcare. Hosmer & Lemeshow put it this way: "Epidemiologic methodologists suggest including all clinically and intuitively relevant variables in the model, regardless of their 'statistical significance.' " This gives me a queasy feeling...who decides "clinical relevance"..."intuitive relevance"...and based on what criteria? Shouldn't we let the data speak for themselves? H&L do go on to point out that "The major problem with this approach is that the model may be 'overfit,' producing numerically unstable estimates. ...typically characterized by unrealistically large estimated coefficients and/or estimated standard errors." Seems to me that alone argues *against* loading up a model with "clinically and intuitively relevant" variables. Anyone else with some thoughts? Thanks! :-) ------------------------------------------- Wayne Fischer Statistician University of Texas Medical Branch -------------------------------------------
Original Message: Sent: 08-20-2014 23:21 From: Katherine Godfrey Subject: Interpretation and Use of Risk of Mortality Models Wayne: Is there some historical or administrative reason that a particular set of predictor variables might be included, even if many of them don't have significant coefficients in the current model? For example, someone might have done some work back in, say, the 70s that produced the current set of explanatory variables to be included, and as a result they are always included when a new model is done. (This is also a backhanded way to think you don't need to publish any overall fit statistics, if you're going to always use the same variables regardless.) The cut-offs you mention could be historical as well. Another possible reason to include "extra" variables is to make the variance of the fitted dependent variable as small as possible (by driving down whatever the moral equivalent of R-squared is), even if it means that you're including some variables that are correlated with each other and thus decreasing the reliability of the fitted coefficients. I don't like this tack, since it leads to black-box models that are over-fitted and data-dependent, but others' mileage may vary, especially if "we've always included this set of variables in our models." But I think your instincts are right for this situation: you might be able to do something with the fitted value (E), but you can't rely on "looking under the hood" at the individual coefficients to help determine a course of treatment. In particular, you can have the nasty situation that among a set of correlated variables, the real explanatory variable loses out in the luck-of-the-draw "battle" for significance to one of its proxies (e.g., if smoking incidence is highly correlated with gender, and smoking is the genuine risk factor, but gender "wins" the battle and thus leads a clinician into thinking, "well, gender has a marginally significant coefficient and smoking does not, so let's differentiate treatment by gender"). Hope this helps, >>Kathy ------------------------------------------- Katherine Godfrey -------------------------------------------
Original Message: Sent: 08-20-2014 18:05 From: Wayne Fischer Subject: Interpretation and Use of Risk of Mortality Models [Full Disclosure: I'm not a biostatistician (I know, I know, how the heck do you get a job with the title of "Statistician" in an AMC and *not* be a biostatistician? Long story, but not so weird nowadays...).] Trying to understand why a national non-profit organization would publish to its members Risk of Mortality models which: o have half of the estimated coefficients (way) not statistically significant (e.g., 15 out of 30 for one model); o include no measures of goodness of fit (e.g., C statistic); o no indication of degree of correlation among the factors in the models; o no list of candidate factors considered in developing the models. As far as I can tell, these models *may* be OK for estimating the Expected Mortality Rates used in comparing to Observed Rates. The O/E and (O - E) values are then used to "flag" (or not) the member organization - based on what appear to me to be arbitrarily chosen "cut-offs." The docs are wanting to use the Models to a priori "rank" patients for treatment, or to determine what treatments to consider to lower patients' risk, but I'm thinking no way - it's all about correlation, not causation. Need this group's help in clarifying my thinking...thanks! :-) ------------------------------------------- Wayne Fischer Statistician University of Texas Medical Branch -------------------------------------------

Hi all,

-------------------------------------------
Ehsan Motazedi, MSc.
Department of Biostatistics
Erasmus MC, Rotterdam
-------------------------------------------



Original Message:
Sent: 08-21-2014 09:28
From: Linda Pickle
Subject: Interpretation and Use of Risk of Mortality Models

Hosmer & Lemeshow are correct that this is typical practice in epidemiology. Extra variables are included in the model for at least 2 reasons: (1) clinically or theoretically relevant variables need to be tested for significance, otherwise no one will believe the results of the model; (2) in observational studies, you would include variables that were part of the sampling scheme or case-control matching in order to capture (as best you can) factors used in the study design (not needed in designed experiments). However, after including these extra variables in the first model, I recommend paring back the variables to those most significant that still give a good fit, reporting some model comparison measure, e.g., AIC. Then you can say to the non-statistical client "See, I included the variables you thought were relevant, but only these few were really needed.".

-------------------------------------------
Linda Pickle
StatNet Consulting, LLC
-------------------------------------------

Thanks, Ehsan.

This is an instance of your second type of problem: trying to estimate the Expected Risk of Mortality.  But, as I've said, we are given no overall goodness of fit statistics - neither absolute nor comparative (C statistic, AIC, BIC, etc.)

The use of the models is to estimate the Expected Risk of Mortality, not rank patients per se.  The Expected Risk (E) is then compared to the Observed Mortality Rate (O) two ways: the ratio O/E and the difference (O - E).  Those values are then compared to "cut-off" values (arbitrarily chosen, I think) to decide whether a member org gets flagged as performing significantly worse than expected.

-------------------------------------------
Wayne Fischer
Statistician
University of Texas Medical Branch
-------------------------------------------



Original Message:
Sent: 08-21-2014 10:50
From: Ehsan Motazedi
Subject: Interpretation and Use of Risk of Mortality Models

Hi all,

Based upon my experience in epidemiology, there are actually two types of problems:

1) You are interested in a specific condition, e.g. maternal weight, and would like to test the significance of its effect on another condition, e.g. child IQ. Especially in observational studies, you usually include other covariates in the model, as long as you do not cause the mentioned instability in the estimates, to increase precision and hopefully the power. In this case the significance of the other covariates rarely matters (unless for instance you are interested in their effect modification or want to detect mediation), and sometimes you even would like to see their effect is not significant to rule out accidental bias.

2) Your aim is to build a model with predictive power. In that case it certainly matters that you do not include a variable unless it improves the predictive power markedly. Backward/Forward selection, AIC, cross validation, VIF, ridge regression, Bayes Factor, etc. are some of the tools that you could then apply for variable selection. 

So if the non-profit org wants to rank the patients with that model, I believe it is better to build a pruned model; and if some variable is clinically indispensable but not chosen by the selection procedure, I think it might be better to build a separate classifier for that and then make a decision by combining the two classifier scores/classifications. 

-------------------------------------------
Ehsan Motazedi, MSc.
Department of Biostatistics
Erasmus MC, Rotterdam
-------------------------------------------



Original Message:
Sent: 08-21-2014 09:28
From: Linda Pickle
Subject: Interpretation and Use of Risk of Mortality Models

Hosmer & Lemeshow are correct that this is typical practice in epidemiology. Extra variables are included in the model for at least 2 reasons: (1) clinically or theoretically relevant variables need to be tested for significance, otherwise no one will believe the results of the model; (2) in observational studies, you would include variables that were part of the sampling scheme or case-control matching in order to capture (as best you can) factors used in the study design (not needed in designed experiments). However, after including these extra variables in the first model, I recommend paring back the variables to those most significant that still give a good fit, reporting some model comparison measure, e.g., AIC. Then you can say to the non-statistical client "See, I included the variables you thought were relevant, but only these few were really needed.".

-------------------------------------------
Linda Pickle
StatNet Consulting, LLC
-------------------------------------------

Please remove me from this discussion thread.

-------------------------------------------
Kimberly Hockman
Senior Consultant
DuPont Company
-------------------------------------------



Original Message:
Sent: 08-21-2014 13:27
From: Wayne Fischer
Subject: Interpretation and Use of Risk of Mortality Models

Thanks, Ehsan.

This is an instance of your second type of problem: trying to estimate the Expected Risk of Mortality.  But, as I've said, we are given no overall goodness of fit statistics - neither absolute nor comparative (C statistic, AIC, BIC, etc.)

The use of the models is to estimate the Expected Risk of Mortality, not rank patients per se.  The Expected Risk (E) is then compared to the Observed Mortality Rate (O) two ways: the ratio O/E and the difference (O - E).  Those values are then compared to "cut-off" values (arbitrarily chosen, I think) to decide whether a member org gets flagged as performing significantly worse than expected.

-------------------------------------------
Wayne Fischer
Statistician
University of Texas Medical Branch
-------------------------------------------



Original Message:
Sent: 08-21-2014 10:50
From: Ehsan Motazedi
Subject: Interpretation and Use of Risk of Mortality Models

Hi all,

Based upon my experience in epidemiology, there are actually two types of problems:

1) You are interested in a specific condition, e.g. maternal weight, and would like to test the significance of its effect on another condition, e.g. child IQ. Especially in observational studies, you usually include other covariates in the model, as long as you do not cause the mentioned instability in the estimates, to increase precision and hopefully the power. In this case the significance of the other covariates rarely matters (unless for instance you are interested in their effect modification or want to detect mediation), and sometimes you even would like to see their effect is not significant to rule out accidental bias.

2) Your aim is to build a model with predictive power. In that case it certainly matters that you do not include a variable unless it improves the predictive power markedly. Backward/Forward selection, AIC, cross validation, VIF, ridge regression, Bayes Factor, etc. are some of the tools that you could then apply for variable selection. 

So if the non-profit org wants to rank the patients with that model, I believe it is better to build a pruned model; and if some variable is clinically indispensable but not chosen by the selection procedure, I think it might be better to build a separate classifier for that and then make a decision by combining the two classifier scores/classifications. 

-------------------------------------------
Ehsan Motazedi, MSc.
Department of Biostatistics
Erasmus MC, Rotterdam
-------------------------------------------



Original Message:
Sent: 08-21-2014 09:28
From: Linda Pickle
Subject: Interpretation and Use of Risk of Mortality Models

Hosmer & Lemeshow are correct that this is typical practice in epidemiology. Extra variables are included in the model for at least 2 reasons: (1) clinically or theoretically relevant variables need to be tested for significance, otherwise no one will believe the results of the model; (2) in observational studies, you would include variables that were part of the sampling scheme or case-control matching in order to capture (as best you can) factors used in the study design (not needed in designed experiments). However, after including these extra variables in the first model, I recommend paring back the variables to those most significant that still give a good fit, reporting some model comparison measure, e.g., AIC. Then you can say to the non-statistical client "See, I included the variables you thought were relevant, but only these few were really needed.".

-------------------------------------------
Linda Pickle
StatNet Consulting, LLC
-------------------------------------------

Wayne,

Looks like the model is used for some adminstrative purpose (to flag an organization).  This may be another reason for considering the "super set" of all admissible predictors to avoid any one organization crying foul "we would have done better if you had included such and such predictor in the model".


Nagaraj

-------------------------------------------
Nagaraj Neerchal
Professor and Chair
UMBC
-------------------------------------------



Original Message:
Sent: 08-21-2014 13:29
From: Kimberly Hockman
Subject: Interpretation and Use of Risk of Mortality Models

Please remove me from this discussion thread.

-------------------------------------------
Kimberly Hockman
Senior Consultant
DuPont Company
-------------------------------------------



Original Message:
Sent: 08-21-2014 13:27
From: Wayne Fischer
Subject: Interpretation and Use of Risk of Mortality Models

Thanks, Ehsan.

This is an instance of your second type of problem: trying to estimate the Expected Risk of Mortality.  But, as I've said, we are given no overall goodness of fit statistics - neither absolute nor comparative (C statistic, AIC, BIC, etc.)

The use of the models is to estimate the Expected Risk of Mortality, not rank patients per se.  The Expected Risk (E) is then compared to the Observed Mortality Rate (O) two ways: the ratio O/E and the difference (O - E).  Those values are then compared to "cut-off" values (arbitrarily chosen, I think) to decide whether a member org gets flagged as performing significantly worse than expected.

-------------------------------------------
Wayne Fischer
Statistician
University of Texas Medical Branch
-------------------------------------------



Original Message:
Sent: 08-21-2014 10:50
From: Ehsan Motazedi
Subject: Interpretation and Use of Risk of Mortality Models

Hi all,

Based upon my experience in epidemiology, there are actually two types of problems:

1) You are interested in a specific condition, e.g. maternal weight, and would like to test the significance of its effect on another condition, e.g. child IQ. Especially in observational studies, you usually include other covariates in the model, as long as you do not cause the mentioned instability in the estimates, to increase precision and hopefully the power. In this case the significance of the other covariates rarely matters (unless for instance you are interested in their effect modification or want to detect mediation), and sometimes you even would like to see their effect is not significant to rule out accidental bias.

2) Your aim is to build a model with predictive power. In that case it certainly matters that you do not include a variable unless it improves the predictive power markedly. Backward/Forward selection, AIC, cross validation, VIF, ridge regression, Bayes Factor, etc. are some of the tools that you could then apply for variable selection. 

So if the non-profit org wants to rank the patients with that model, I believe it is better to build a pruned model; and if some variable is clinically indispensable but not chosen by the selection procedure, I think it might be better to build a separate classifier for that and then make a decision by combining the two classifier scores/classifications. 

-------------------------------------------
Ehsan Motazedi, MSc.
Department of Biostatistics
Erasmus MC, Rotterdam
-------------------------------------------

Thanks, Linda.

Regarding "paring back the variables to those most significant that still give a good fit," as I said, no fit measures - absolute or comparative (C statistic, AIC, BIC) - are given.  And if paring back occurred, would it be usual that half of the retained coefficients have p-values > 0.10 - ranging up to > 0.8?

And, as I also said, we are not given a list of all the variables considered initially in developing the models...

-------------------------------------------
Wayne Fischer
Statistician
University of Texas Medical Branch
-------------------------------------------



Original Message:
Sent: 08-21-2014 09:28
From: Linda Pickle
Subject: Interpretation and Use of Risk of Mortality Models

Hosmer & Lemeshow are correct that this is typical practice in epidemiology. Extra variables are included in the model for at least 2 reasons: (1) clinically or theoretically relevant variables need to be tested for significance, otherwise no one will believe the results of the model; (2) in observational studies, you would include variables that were part of the sampling scheme or case-control matching in order to capture (as best you can) factors used in the study design (not needed in designed experiments). However, after including these extra variables in the first model, I recommend paring back the variables to those most significant that still give a good fit, reporting some model comparison measure, e.g., AIC. Then you can say to the non-statistical client "See, I included the variables you thought were relevant, but only these few were really needed.".

-------------------------------------------
Linda Pickle
StatNet Consulting, LLC
-------------------------------------------

Was this based on observational (aka happenstance) data?  Perhaps its an example of something I once called "brainless regression". 
-------------------------------------------
The views expressed on this Web site/blog are mine alone and do not necessarily reflect the view of my employer.
-------------------------------------------
Emil M Friedman, PhD
emil.friedman@alum.mit.edu (forwards to day job)
emilfrie@alumni.princeton.edu (home)
http://www.statisticalconsulting.org
-------------------------------------------

Thanks, Emil.

Yes, observational data - from all the member organizations.  I'm pretty sure it's *not* "brainless" regression (I've seen that, too, in my carer) - the non-profit uses PhD statistician(s).

But it appears to me an attitude (unintentional perhaps) of "Trust us - this is a good model."

-------------------------------------------
Wayne Fischer
Statistician
University of Texas Medical Branch
-------------------------------------------



Original Message:
Sent: 08-21-2014 10:37
From: Emil Friedman
Subject: Interpretation and Use of Risk of Mortality Models

Was this based on observational (aka happenstance) data?  Perhaps its an example of something I once called "brainless regression". 
-------------------------------------------
The views expressed on this Web site/blog are mine alone and do not necessarily reflect the view of my employer.
-------------------------------------------
Emil M Friedman, PhD
emil.friedman@alum.mit.edu (forwards to day job)
emilfrie@alumni.princeton.edu (home)
http://www.statisticalconsulting.org
-------------------------------------------

Just a quick comment. When you include a bunch of variables in a model to perform risk adjustment, stepwise selection of the variables is bad because it leads to residual confounding. There are several alternatives, but the simplest, if you have enough data (keep in mind the rule of 15), is to include any variable which is biologically plausible and don't remove anything, even if the p-value is large. So a risk adjusted model where ALL of the risk factors have p-values less than 0.05 is actually a bad model. As far as the other things (e.g., no measure of goodness of fit), it may page limits or a desire to keep things at a very non-technical level. ------------------------------------------- Stephen Simon Independent Statistical Consultant P. Mean Consulting -------------------------------------------
Original Message: Sent: 08-20-2014 18:05 From: Wayne Fischer Subject: Interpretation and Use of Risk of Mortality Models [Full Disclosure: I'm not a biostatistician (I know, I know, how the heck do you get a job with the title of "Statistician" in an AMC and *not* be a biostatistician? Long story, but not so weird nowadays...).] Trying to understand why a national non-profit organization would publish to its members Risk of Mortality models which: o have half of the estimated coefficients (way) not statistically significant (e.g., 15 out of 30 for one model); o include no measures of goodness of fit (e.g., C statistic); o no indication of degree of correlation among the factors in the models; o no list of candidate factors considered in developing the models. As far as I can tell, these models *may* be OK for estimating the Expected Mortality Rates used in comparing to Observed Rates. The O/E and (O - E) values are then used to "flag" (or not) the member organization - based on what appear to me to be arbitrarily chosen "cut-offs." The docs are wanting to use the Models to a priori "rank" patients for treatment, or to determine what treatments to consider to lower patients' risk, but I'm thinking no way - it's all about correlation, not causation. Need this group's help in clarifying my thinking...thanks! :-) ------------------------------------------- Wayne Fischer Statistician University of Texas Medical Branch -------------------------------------------

Thanks, Stephen.  

(1)  We are not told the method used to build the models.
(2)  What is "residual confounding?"
(3)  The amount of data should not be a concern since the models are based on using all members' data.
(4)  Given the well-documented issues due to overfitting, isn't a policy of including any variable that is biologically plausible as problematical as only including those with p less than 0.05?


-------------------------------------------
Wayne Fischer
Statistician
University of Texas Medical Branch
-------------------------------------------



Original Message:
Sent: 08-22-2014 00:19
From: Stephen Simon
Subject: Interpretation and Use of Risk of Mortality Models


Just a quick comment. When you include a bunch of variables in a model to perform risk adjustment, stepwise selection of the variables is bad because it leads to residual confounding. There are several alternatives, but the simplest, if you have enough data (keep in mind the rule of 15), is to include any variable which is biologically plausible and don't remove anything, even if the p-value is large. So a risk adjusted model where ALL of the risk factors have p-values less than 0.05 is actually a bad model.

As far as the other things (e.g., no measure of goodness of fit), it may page limits or a desire to keep things at a very non-technical level.
-------------------------------------------
Stephen Simon
Independent Statistical Consultant
P. Mean Consulting
-------------------------------------------

"Residual confounding" is the confounding that hasn't been adjusted for in the experiment or its analysis.
It seems to be a term most frequently encountered in epidemiological contexts, particularly with
regard to observational studies (which would include the fitting exercise Wayne's been looking at).

http://sphweb.bumc.bu.edu/otlt/MPH-Modules/BS/BS704-EP713_Confounding-EM/BS704-EP713_Confounding-EM4.html

Stepwise regression techniques run the risk of picking the "wrong" variable(s) out of set of confounded ones (for example,
in the scenario I posited in my earlier post, you might have ended up with a model that kept the gender variable and dumped
the smoking one).

It occurs to me that there's another reason to cast a wide net in this sort of model fitting: it allows you the chance
to spot changes in the interplay among the variables of interest.  Sticking with the gender and smoking pairing, consider
the relationship between gender and smoking over the last 100 years:

100 years ago: very few women smoke, most men do
75 years ago: smoking incidence is more comparable among younger men and women (but lower for women),
but not among older women and men
50 years ago: smoking incidence is more equal across genders, although women are still less likely to smoke than men
25 years ago: smoking incidence is lower overall, and much more equal between the genders (it's dropped more for men than women)
today: overall incidence has dropped even more, this time more for women than men (that is, a random smoker was more
likely to be a women either today or 50 years ago than 25 years ago).

So, if you'd done a study back 100 years ago, when essentially only men smoked (thus completely confounding
gender and smoking), you could have produced a pruned model with gender but not smoking.  And you would have
missed the opportunity to switch to smoking (or adjust for the interaction between smoking and gender and age) as
time went on.

http://www.infoplease.com/ipa/A0762370.html
http://www.cdc.gov/tobacco/data_statistics/fact_sheets/adult_data/cig_smoking/
http://www.lung.org/finding-cures/our-research/trend-reports/Tobacco-Trend-Report.pdf (Table 4)

But that doesn't solve the problem of the confounding decreasing the reliability of the fitted estimates for
a variable in a set of confounded variables.

>>Kathy


-------------------------------------------
Katherine Godfrey
-------------------------------------------



Original Message:
Sent: 08-22-2014 09:50
From: Wayne Fischer
Subject: Interpretation and Use of Risk of Mortality Models

Thanks, Stephen.  

(1)  We are not told the method used to build the models.
(2)  What is "residual confounding?"
(3)  The amount of data should not be a concern since the models are based on using all members' data.
(4)  Given the well-documented issues due to overfitting, isn't a policy of including any variable that is biologically plausible as problematical as only including those with p less than 0.05?


-------------------------------------------
Wayne Fischer
Statistician
University of Texas Medical Branch
-------------------------------------------

I think there might be a solution for "residual confounding", by calculating VIF for each excluded variable regressing on the included predictors, and changing the model by replacing/adding the suspected confounders into the model and see the change in the fitness measure. But in Wayne's model no goodness of fit measure is available. 

However, as the P-values are available, so must be the confidence intervals for each regression coefficient. Then you may calculate the confidence interval for your response (assume it was risk) by Delta method. I think it would be a reasonable way to use this model. 

-------------------------------------------
Ehsan Motazedi,MSc.
Department of Biostatistics
Erasmus MC, Rotterdam
-------------------------------------------



Original Message:
Sent: 08-22-2014 11:13
From: Katherine Godfrey
Subject: Interpretation and Use of Risk of Mortality Models

"Residual confounding" is the confounding that hasn't been adjusted for in the experiment or its analysis.
It seems to be a term most frequently encountered in epidemiological contexts, particularly with
regard to observational studies (which would include the fitting exercise Wayne's been looking at).

http://sphweb.bumc.bu.edu/otlt/MPH-Modules/BS/BS704-EP713_Confounding-EM/BS704-EP713_Confounding-EM4.html

Stepwise regression techniques run the risk of picking the "wrong" variable(s) out of set of confounded ones (for example,
in the scenario I posited in my earlier post, you might have ended up with a model that kept the gender variable and dumped
the smoking one).

It occurs to me that there's another reason to cast a wide net in this sort of model fitting: it allows you the chance
to spot changes in the interplay among the variables of interest.  Sticking with the gender and smoking pairing, consider
the relationship between gender and smoking over the last 100 years:

100 years ago: very few women smoke, most men do
75 years ago: smoking incidence is more comparable among younger men and women (but lower for women),
but not among older women and men
50 years ago: smoking incidence is more equal across genders, although women are still less likely to smoke than men
25 years ago: smoking incidence is lower overall, and much more equal between the genders (it's dropped more for men than women)
today: overall incidence has dropped even more, this time more for women than men (that is, a random smoker was more
likely to be a women either today or 50 years ago than 25 years ago).

So, if you'd done a study back 100 years ago, when essentially only men smoked (thus completely confounding
gender and smoking), you could have produced a pruned model with gender but not smoking.  And you would have
missed the opportunity to switch to smoking (or adjust for the interaction between smoking and gender and age) as
time went on.

http://www.infoplease.com/ipa/A0762370.html
http://www.cdc.gov/tobacco/data_statistics/fact_sheets/adult_data/cig_smoking/
http://www.lung.org/finding-cures/our-research/trend-reports/Tobacco-Trend-Report.pdf (Table 4)

But that doesn't solve the problem of the confounding decreasing the reliability of the fitted estimates for
a variable in a set of confounded variables.

>>Kathy


-------------------------------------------
Katherine Godfrey
-------------------------------------------



Original Message:
Sent: 08-22-2014 09:50
From: Wayne Fischer
Subject: Interpretation and Use of Risk of Mortality Models

Thanks, Stephen.  

(1)  We are not told the method used to build the models.
(2)  What is "residual confounding?"
(3)  The amount of data should not be a concern since the models are based on using all members' data.
(4)  Given the well-documented issues due to overfitting, isn't a policy of including any variable that is biologically plausible as problematical as only including those with p less than 0.05?


-------------------------------------------
Wayne Fischer
Statistician
University of Texas Medical Branch
-------------------------------------------