Discussion: View Thread

Trade off between model sensitivity and specificity

  • 1.  Trade off between model sensitivity and specificity

    Posted 08-02-2012 16:53

    Dear All,
    I have a logistic regression model .The model specificity is 94.3% and sensitivity is 28.4%. Furthermore, the area under ROC curve is 0.75. Can we consider it a reasonable model. Does the low sensitivity of model indicates poor fit despite it's high specificity and a reasonable area under ROC curve?
    What are the possible reasons for low sensitivity of a model in general?
    Looking forward  to get enlightened from the groups expertise .
    Best Regards,
    Tasneem


    -------------------------------------------
    [Tasneem] [Zaihra]
    [Post Doctoral Fellow]
    [McGill University]
    -------------------------------------------


  • 2.  RE:Trade off between model sensitivity and specificity

    Posted 08-02-2012 17:03
    Hi Tasneem,

    You really aren't doing much better than calling everything negative.  Can you live with a false negative rate of 70%?  Despite the AUC, your combination of sensitivity and specificity indicates you are not able to separate positives from negatives.  I suspect your AUC reflects that you have a only a small proportion of your sample are positives.

    Bob
    -------------------------------------------
    Robert Gallavan
    Principal Statistician
    I3/Statprobe
    -------------------------------------------








  • 3.  RE:Trade off between model sensitivity and specificity

    Posted 08-02-2012 17:58
    The specificity and sensitivity for your fit would be okay for a criminal trial situation,
    where we want to avoid at all costs labeling an innocent person as guilty (a fase positive), where the cost might be letting some of the guilty slip through the cracks (false negatives).  Here specificity (if the jury
    finds you guilty, you almost certainly are guilty) trumps sensitivity (so a lot of guilty people
    win acquittal).

    But it's not so good for most screening conditions, where we would prefer to have
    some false positives than miss some true positives.  Low sensitivity means you're missing
    a lot of the positives, even if nearly everything you tag as positive is in fact positive.

    As Robert notes, this can happen if there aren't very many positives to begin with.

    >>Kathy

    -------------------------------------------
    Katherine Godfrey
    -------------------------------------------








  • 4.  RE:Trade off between model sensitivity and specificity

    Posted 08-02-2012 18:25

    My sample has 566 patients out of which 141 have certain condition [I am modeling probability of having that condition ==1 and not having it as 0] . The Hosmer Lemeshow goodness of fit test results (ChiSq = 3.82, DF = 8, Pr > ChiSq = 0.87), indicate a good fit, and the observed and expected frequencies show good agreement for all deciles of risk.  

    As per the suggestions can this be attributed mainly to the fact that I have only 141 patients with the condition?

    Thanks
    Tasneem

    -------------------------------------------
    [Tasneem] [Zaihra]
    [Post Doctoral Fellow]
    [McGill University]
    -------------------------------------------








  • 5.  RE:Trade off between model sensitivity and specificity

    Posted 08-03-2012 12:46
    Hello Tasneem,

    I do not believe your problem is because of the ratio of positives to negatives in your sample. 25% of teh data being 1s is not a bad proportion.  

    The sensitivity and specificity are functions of the cutoff that you choose for separating positives from negatives. I wonder if you might be using the wrong cutoff. How did you choose the cutoff value? usually, it is chosen to minimize both FPR and FNR simultaneously which doesn't seem to be the case here. 

    Mary 

    -------------------------------------------
    Mary Christman
    Statistical Consultant
    MCC Statistical Consulting
    -------------------------------------------








  • 6.  RE:Trade off between model sensitivity and specificity

    Posted 08-03-2012 14:31
    Mary in a personal response I have seen the whole ROC curve.  There is a point where sensitivity approaches 55% but the price is a drop in specificity to 77%.  I don't know if this is a satisfactory tradeoff.  But I guess in this case I would be inclined to see if with more data I could find more explanatory variables that could help the situation.

    -------------------------------------------
    Michael Chernick
    Director of Biostatistical Services
    Lankenau Institute for Medical Research
    -------------------------------------------








  • 7.  RE:Trade off between model sensitivity and specificity

    Posted 08-03-2012 19:28
    I'd like to suggest that one situation where Tazneem's numbers would look pretty good is the specific application of screening for prostate cancer.  In the July 6th, 2005 issue of the Journal of the American Medical Association (PubMed abstract at http://www.ncbi.nlm.nih.gov/pubmed/15998892), Thompson et al derive the Operating Characteristics of PSA among 8575 men enrolled on the placebo arm of the Prostate Cancer Prevention Trial.  Among these men, a PSA cutoff of 4.1 ng/mL yielded a sensitivity of 20.5% and specificity of 93.8%.  Given that the PSA cutoff used by our primary-care providers in the real world is 4.0 ng/mL, it becomes easy to see that Tazneem's numbers of 28.4% sensitivity and 94.3% specificity would represent a potentially important improvement in sensitivity over that of the current PSA test, if she were to obtain her numbers in the specific area of prostate-cancer screening.  


    -------------------------------------------
    Eric Siegel
    Biostatistician
    Univ of Arkansas for Medical Sciences
    -------------------------------------------





  • 8.  RE:Trade off between model sensitivity and specificity

    Posted 08-03-2012 10:35
    Whether this is acceptable depends on what the purpose of your model is.  

    If the key thing is getting an accurate picture of the probability of having the disease conditional on the predictor(s) in your model, then the Hosmer-Lemeshow statistics you got, with a sample size of 566, say that you are in good shape for that purpose.  The sensitivity and specificity and AUC don't really matter for that purpose.   The model will tell you, for example, that among people with a given set of predictor values, the prevalence of the disease is such and such, and it will be (close to) correct.  But it may not tell you much at all about which of those people are the ones with the disease.

    On the other hand, if you need accurate classifications of individuals for your purposes, you are in pretty bad shape with this sensitivity and specificity.  You might do better using a different prediction threshold--you can then get better sensitivity at the cost of losing specificity.  Whether that is better depends on a still more refined understanding of your purposes: what are the consequences of missing a case, and of wrongly diagnosing disease in a person who doesn't have it?   The balance between those will give you a sense of what achievable combination of sensitivity and specificity on your ROC curve is best for your purposes, and hence, at what cut-off to operate the model.

    -------------------------------------------
    Clyde Schechter
    -------------------------------------------








  • 9.  RE:Trade off between model sensitivity and specificity

    Posted 08-03-2012 11:09
    The sensitivity and specificity statistics reported in logistic regression are not really all that helpful. As I recall, programs like SPSS that report this statistic use a cut-off based on whether the predicted probability is above or below 0.5. If the probability of one of the two levels in your logistic regression model is much larger than the other, such a cut-off may not make a lot of sense. It also may not make a lot of sense if the costs of misclassification are much different (as another person has already noted). You should not trust a computer's choice here because the computer knows nothing about the context of the problem.

    You can manipulate the cutoff so that specificty is very high and sensitivity is very low. You can also manipulate the cutoff so that specificity is very low and sensitivity is very high. So the disparity that you see with the default cutoff of 0.5 is pretty much meaningless.

    Personally, I think it is very bad for programs to report sensitivity and specificity for a logistic regression model. It is almost never helpful and it is almost always confusing. Also, if you had to choose a value, 0.5 is probably one of the worst values for a cut-off. For highly skewed situations (situations where one level has a lot more data than another), it leads to really extreme values. A more logical choice would be to set the cutoff at the probability of the first level (usually the level coded as 0). That would be equivalent to a prior probability equal to the probabilities observed in your data and an assumption that the costs of misclassification are the same in either direction. But I wouldn't hold my breath waiting for this to happen. Every package develops a legacy of users over the years (and decades in the case of SPSS) who are so used to seeing things one way that they get upset when things change. Plus the developers of these programs would much rather roll out new features than to tweak old features.

    So just ignore sensitivity and specificity entirely.

    The area under the curve, however, is a more useful statistic. When it is close to 1, you have a good model and when it is close to 0.5, you have a bad model. Some people treat this statistic as a goodness of fit statistic, much like R-squared in linear regression. I think the analogy is a bit strained, but even so, it is a worthwhile statistic to report.

    With a single continuous variable, the area under the curve has an interesting physical interpretation. It represents P[X > Y] where X is a vector of the continuous variable associated with one of the levels of your dependent variable and Y is a vector of the continuous variable associated with the other level of your dependent variable. When this probability is large, it shows that small values of your independent variable are very likely to be associated with one level of your dependent variable and large values of your independent variable are very likely to be associated with the other level. This is a good thing.

    On the other hand if P[X >Y] is close to 0.5 it means that small and large values of your independent variable are just as likely to be associated with either level of your dependent variable.

    One minor point to keep things precise. The area under the curve is sometimes P[X < Y] rather than P[X > Y]. It depends on the sign of the coefficient in your model.

    So is 0.75 a good value? Well, it's larger than 0.5, but not very close to 1.0. Rather than compare this in a vacuum, consider this value to value observed in other models and for other data sets. If I had to describe what 0.75 means, I'd say "good, but not great." But without knowing more about your model and the results of competing models, I can't say a whole lot more.

    I hope this helps.

    -------------------------------------------
    Stephen Simon
    Independent Statistical Consultant
    P. Mean Consulting
    -------------------------------------------








  • 10.  RE:Trade off between model sensitivity and specificity

    Posted 08-03-2012 11:38
    Dear All,
    Thank you for your valuable inputs. It's very helpful.
    I really appreciate your time and interest in my issue.
    Best Regards,
    Tasneem

    -------------------------------------------
    [Tasneem] [Zaihra]
    [Post Doctoral Fellow]
    [McGill University]
    -------------------------------------------








  • 11.  RE:Trade off between model sensitivity and specificity

    Posted 08-03-2012 12:07
    I just want to echo the comment of Steve and Clyde regarding the ROC.  Most of the discussion centered on the low sensitivity.  But that is only a single point on the ROC curve.  If your software picked it as an optimal point then it is important to know what criterion was used.  I would much rather see the entire curve and make my own judgement as to whether or not I want to tradeoff a little specificity of increased sensitivity.  This tradeoff is very problem specific and depends on the importance of avoiding one error vs the other.  The fact that the AUC  is 0.75 is somewhat encouraging that you could probably bring sensitivity above 50% with a little loss in specificity.  But I don't know how much.  Kelly Zou coauthored a really good book on ROC and Lyle Broemeling's Bayesian diagnostic testing books could also be helpful.  But if you are not satisfied with the best tradeoff (perhaps because snesitivity is too low maybe you can find a way to improve the model to get the sensitivity up.

    -------------------------------------------
    Michael Chernick
    Director of Biostatistical Services
    Lankenau Institute for Medical Research
    -------------------------------------------








  • 12.  RE:Trade off between model sensitivity and specificity

    Posted 08-03-2012 18:05
    This is really a problem specific question. There is a comment paper by S. Greenland that talked about the trade-off between the sensiticvity and specificity and he suggested to quantify the cost of sensitivity and specificity in order to get an optimal cut-off point (of course, there are many other criteria and you can ref Pepe 2003).  Usually, you can set a minimum specificy you can tolerate (for example, 80%). That is, you can not tolerate the cost for having a specificity lower than that. Then, you can look at the sensitivity at this fixed specificity. 

    AUC is a good summary measure of the ROC curve. However, it include some range of specificity that are not rquite relevant. For example, the curve for lower specificities are not of interest even though you may have higher sensitivities there. Or, the region with extrememly low sensitivity  but high specificity (lower False Positive rate, FPR) is not of interest either (this is your case that have 0.943 Specificity and 0.294 sensitivity).  Partial AUC (restrict specificy to certain region) is usaully used in this situation when you do not want low specificities and sensitivities.  In short, I think if you have a scientific reason to set a fixed specificity, to look at the sensitivity at that point would be a good choice.  

    -------------------------------------------
    Nan Hu
    University of Utah
    -------------------------------------------








  • 13.  RE:Trade off between model sensitivity and specificity

    Posted 08-06-2012 14:44
    I basically agree with Stephen.  Consider a case-control study and a variable that you want to use to predict the status.  Sensitivity and specificity can be influenced by the distribution of cases to controls.  You usually spike it with cases, as controls are easy to get. The natural distribution can be very different. You need a Bayes type adjustment to any logistic regression.  You can have a prior distribution on the marginal probability that an individual coming to your clinic is a case.  You estimate the probability that an individual is a case in your spiked distibution given the covariates, andthen back adjust this for the natural distribution using your prior.  If you estimate the prior probability, you need to take the sampling error into account.  Using the delta method, you can get confidence intervals and point estimates for the probaility an individual is a case give the covariates. This could be useful for diagnosis.  But cutoffs are strongly discouraged, as if gives false impressions around the borders, and you are trying to make a continuous data problem into a discrete one--usually a bad idea.

    Jon

    -------------------------------------------
    Jon Shuster
    University of Florida
    -------------------------------------------