Discussion: View Thread

Really need to know some more information.  If you can mention it, it would help to know what the outcome of interest is.  How wide are the confidence intervals around the odds ratio you mention? Was any justification provided for not matching?  Is it likely that the outcomes occur in some particular sub population?  What characteristics of the population are included in the model? 

I am very suspicious.  I very rarely expect see that large an odds ratio.  It is of the magnitude one might expect looking at lung cancer and smoking.  

Bob 

Just to elaborate on the previous comments, what you have is a classic case-control design. Because the prevalence is artificially controlled, certain statistics like the relative risk are inappropriate for this setting. But Cornfield showed in 1950 that as long as the outcome is fairly rare, the odds ratio is a fine statistic to use. The case-control study was instrumental in first identifying a possible link between smoking and cancer. It also has helped identify that aspirin use was associated with Reye's Syndrome, and that HIV was a sexually transmitted disease. The CDC recently sent a team of epidemiologists to Brazil to conduct a case control study that more firmly established the link between the Zika virus and microcephaly.

I'm a bit surprised that the authors of the paper you are reviewing did not properly identify their study as a case-control study. Maybe it's because case-control studies have a (largely undeserved) bad reputation.

Anyway, I would be a bit nervous about an odds ratio of 80. As someone else has already noted, smoking and lung cancer has an odds ratio of around 10 (or maybe 20). The odds ratio for smoking and heart disease is around 2, and that is considered a fairly strong association.

There are several things that might cause such an abnormally large odds ratio. You might be seeing separation or quasi-separation of the data. If you have a large number of independent variables, you might be seeing overfitting. I like the suggestion of looking at the standard errors. It's hard to imagine that a poor goodness of fit could produce an artificially large odds ratio, though.

An Odds ratio of 80 (eighty?) ?   A long time ago (longer than I care to admit) I memorized the odds ratio that Mantel-Haenszel estimated for the risk of lung cancer from smoking in their classic paper - that was about -8-. That was ultimately enough for the Surgeon General to add a warning to cigarettes.  As you say, you are reviewing a manuscript. . You might consider recommending the  Firth correction and asking for the authors to check on convergence and for "separable data". 

Just to elaborate on the previous comments, what you have is a classic case-control design. Because the prevalence is artificially controlled, certain statistics like the relative risk are inappropriate for this setting. But Cornfield showed in 1950 that as long as the outcome is fairly rare, the odds ratio is a fine statistic to use. The case-control study was instrumental in first identifying a possible link between smoking and cancer. It also has helped identify that aspirin use was associated with Reye's Syndrome, and that HIV was a sexually transmitted disease. The CDC recently sent a team of epidemiologists to Brazil to conduct a case control study that more firmly established the link between the Zika virus and microcephaly.

I'm a bit surprised that the authors of the paper you are reviewing did not properly identify their study as a case-control study. Maybe it's because case-control studies have a (largely undeserved) bad reputation.

Anyway, I would be a bit nervous about an odds ratio of 80. As someone else has already noted, smoking and lung cancer has an odds ratio of around 10 (or maybe 20). The odds ratio for smoking and heart disease is around 2, and that is considered a fairly strong association.

There are several things that might cause such an abnormally large odds ratio. You might be seeing separation or quasi-separation of the data. If you have a large number of independent variables, you might be seeing overfitting. I like the suggestion of looking at the standard errors. It's hard to imagine that a poor goodness of fit could produce an artificially large odds ratio, though.

One technical comment: in properly designed / sampled population-based case-control studies, the logistic regression coefficients are consistently estimated by either the weighted model (that gives the controls their proper probability of selection weight; a very inefficient method when the sampling rates differ by a orders of magnitude, as in this case, 1:25) or by the quasi-ML that ignores the weights. The only problem for the latter, as Nancy pointed out, is the overall prevalence parameter which is controlled by the intercept, and that one is of course biased if you don't use weights (but you can figure the bias out if you know the sampling rate for the controls). So odds ratios of the factors should be fine. But 80 is indeed a bizarre number.

I would second the calls for the goodness of fit testing to be done properly.

Also, I would probably say that an OR of 40 with a CI of 25 to 65 is more convincing to me than the OR of 80 with a CI of 5 to 1000. In other words, you can just as well look at the lower confidence limit in order to decide which is a "better " model.

Do they described how exactly they subsampled the controls? If that was a random sample, it's probably OK. If that was a carefully matched sample, that's probably OK, too. If these were all last names starting with A (and I did have a sample like that in one of my projects... somebody underestimated the size of the list by a couple of orders of magnitude when they asked to pull each 100th name), that's problematic -- misses all those Russians whose last names tend to start with K ;).

Meant to give a reference on case-control stuff: http://www.statcan.gc.ca/pub/12-001-x/2006002/article/9546-eng.pdf

If your goal is to predict the outcome, I would use Random Forests on all the data. The authors can find a suitable cut off value then run several of them and see how well the models predict the outcome. 

Each Random Forest will show the authors which factors have an important impact on the outcome. It also shows the how well the model predicts each type of outcome. 

There are two big issues with logistic regression that we usually don't think about.

One is the odds ratio. Suppose the odds ratio shows a 5000% increase in the likely hood of disease. If there is a one in a billion odds of disease anyways, a 5000% increase, though dramatic, the odds of disease are still fairly rare. If you use your logistic regression model to predict the outcome, that person would still have little chance of getting the disease. That means a 5000% increase is pretty meaningless. 

The other issue is that logistic regressions try to minimize the error. I can predict everyone in the group will not get disease and be correct 99.96% of the time. If you use the logistic regression model to predict the outcome, most of the time it will fail to predict the correct outcome for those with the disease. 

With the random forests, you tune the model to correctly predict diseased have disease and healthy are healthy. You also end up breaking your data into 2-3 groups. So, you end up with internal validation of the model. By running multiple random forests, you improve the likelyhood of finding truly important factors. It's also like replicating the analysis with "new" data. So, you can get some very robust results.

Between Random Forests and Logistic regressions, I'll take RF all day every day. 

If your goal is to predict the outcome, I would use Random Forests on all the data. The authors can find a suitable cut off value then run several of them and see how well the models predict the outcome. 

Each Random Forest will show the authors which factors have an important impact on the outcome. It also shows the how well the model predicts each type of outcome. 

There are two big issues with logistic regression that we usually don't think about.

One is the odds ratio. Suppose the odds ratio shows a 5000% increase in the likely hood of disease. If there is a one in a billion odds of disease anyways, a 5000% increase, though dramatic, the odds of disease are still fairly rare. If you use your logistic regression model to predict the outcome, that person would still have little chance of getting the disease. That means a 5000% increase is pretty meaningless. 

The other issue is that logistic regressions try to minimize the error. I can predict everyone in the group will not get disease and be correct 99.96% of the time. If you use the logistic regression model to predict the outcome, most of the time it will fail to predict the correct outcome for those with the disease. 

With the random forests, you tune the model to correctly predict diseased have disease and healthy are healthy. You also end up breaking your data into 2-3 groups. So, you end up with internal validation of the model. By running multiple random forests, you improve the likelyhood of finding truly important factors. It's also like replicating the analysis with "new" data. So, you can get some very robust results.

Between Random Forests and Logistic regressions, I'll take RF all day every day. 

This is almost certainly a simple issue of over-fitting due to model selection that the final inference ignores.  The conditions are ripe:  your "positive" population is small, and more importantly model selection occurred.  Possibly a good deal of it.  I note your sentence:  "The purpose of the study is to model the odds of the outcome using a variety of factors."  So they went fishing.  And with a large data set with many factors, they don't have to fish much to get horribly biased results.

See Ambroise & McLachlan, 2002, "Selection Bias in Gene Extraction on the Basis of Microarray Gene-Expression Data."  Proceedings of the National Academy of Sciences, available for free download from pnas.org.  The problem is different but the principles are the same.

The final inference needs to encompass model selection.  As a rule of thumb for article reviews, I would suggest that any article reporting a statistical result in which modeling considered five or more variables should specifically describe how model-selection is addressed in inference, and make a convincing case that their methods prevent "selection bias", else they don't get published.  I'm so glad you wrote!  You're doing a wonderful job as reviewer!

There are multiple ways to make inference that includes model selection.  One possibility is to avoid model selection altogether:  use all variables, and (possibly) apply regularization methods (e.g., the Firth adjustment mentioned earlier) to make the problem computationally tractable.  Another possibility is to apply resampling methods (bootstrapping or cross-validation) to a fitting process that includes variable selection.  This is complicated because one doesn't get the same set of variables for every iteration...but that's exactly the point.  Still another possibility is to use Bayesian Model Averaging, such as with R's BMA package; there are now additional packages, and I couldn't comment on which is best.  Anyway, it's the authors who need to figure out the solution.

When I was in graduate school, in the days before data mining got big, we would discuss selecting a model and then interpreting the model.  There was always an acknowledgement that p-values and confidence intervals were biased due to model selection, but we sort of waved at the issue as we passed by, and comforted ourselves that with a modest degree of prior analysis, the effect would not be severe.  Fast forward many years, and people are routinely analyzing massive datasets with tens, hundreds, or thousands of variables, and selection bias can be enormous.  I wouldn't be entirely surprised if the true OR your authors find for their factor, once they control properly for model selection, is 1.0, i.e., completely null.  I can tell you I learned of this issue the hard way, by having a problem where I thought the data supported almost complete separation when in fact there was nothing reproducible there.

The classification and machine learning world has been badly burned by this (just as I was), and the community is now largely sensitized to the issue (I hope!).  The biostatistics community is not as broadly sensitized, presumably because it's less common there to see a large number of predictor variables.  But I think this example is exactly one of those occasions.

If anyone wants to discuss details, I would suggest spawning a new thread, because that discussion could utterly swamp this one.

Good luck!