ASA Connect

Dear Colleagues:

I am working on paper with a small sample, N = 115, with a low event rate (8/115), and suggested that the researcher cite "Firth, D. (1993). "Bias Reduction of Maximum Likelihood Estimates." Biometrika 80:27–38".   This is a common reference used for small samples that are analyzed with the Firth option in SAS Procs Logistic or PHReg. The researcher sent me a reply that the article couldn't be cited because it wasn't indexed in PubMed. 

Then, I did a google search and got this AI reply: "Bias Reduction of Maximum Likelihood Estimates" was published in Biometrika. While the original paper is not indexed on PubMed under its own entry, it is heavily cited and discussed across thousands of biomedical and epidemiological studies on PubMed."

Has anyone encountered problems with citing Firth (1993) or other articles that weren't indexed in PubMed?

------------------------------
Brandy Sinco, BS, MA, MS
Statistician Senior
Michigan Medicine
------------------------------

Presumably the reason Firth's paper is not in PubMed is because Biometrika is not a medical journal. Nevertheless, that's a very well known reference. It's featured in Julian Faraway's book "Extending the Linear Model with R" which I teach from in my GLM course, where Firth's method is explicitly cited as a solution to the well known paradox of logistic regression where there's a separating hyperplane between the positive and negative responses (in which case the regular GLM algorithm does not have a unique solution). There's also an R package called brglm that implements Firth's method. Good luck persuading the editor that this is all legitimate and well known statistical practice!

------------------------------
Richard Smith
University of North Carolina
------------------------------

Hi - I agree with Professor Smith, a peer reviewed, published paper from <any field!> is a legitimate resource/reference to cite. It is not correct to suggest that only papers in PubMed are "legitimate". PubMed is not "the scientific record". You do not have to make this argument, simply point out in your responses to reviewers that the manuscript is an established element in the field and you can even "respectfully disagree" with the idea that only papers in pub med are citable - that is patently false and even harmful, so correcting this reviewer's wrong thinking, and highlighting this for the editor who may not have noticed that error when reviewing reviewer comments prior to sending the input back to you, might be helpful in general. Good luck with the submission! If that's the most challenging comment to respond to, then I look forward to your paper being published soon! :)

------------------------------
--
Rochelle E. Tractenberg, PhD, MPH, PhD, PStat®, FASA, FAAAS
Director, Collaborative for Research on Outcomes and -Metrics
https://crom.georgetown.domains
Co-Director, National Capital Spinal Cord Injury Model System (NC-SCIMS)
Professor
Neurology
Biostatistics, Bioinformatics & Biomathematics
Rehabilitation Medicine
Georgetown University
Washington, D.C. 20057
------------------------------

Original Message:
Sent: 09-01-2026 09:10
From: Richard Smith
Subject: Cite Firth (1993), Even Though Not Indexed in PubMed?

Presumably the reason Firth's paper is not in PubMed is because Biometrika is not a medical journal. Nevertheless, that's a very well known reference. It's featured in Julian Faraway's book "Extending the Linear Model with R" which I teach from in my GLM course, where Firth's method is explicitly cited as a solution to the well known paradox of logistic regression where there's a separating hyperplane between the positive and negative responses (in which case the regular GLM algorithm does not have a unique solution). There's also an R package called brglm that implements Firth's method. Good luck persuading the editor that this is all legitimate and well known statistical practice!

------------------------------
Richard Smith
University of North Carolina
------------------------------

Hey Brandy,

I agree with Rochelle's reasoning: Pub Med isn't the record. That's why...

(1) different citation formats exist for different media (ex. newspaper articles, printed books, website, etc.)
(2) People in biosciences can make reference to methodological developments in astrophysics, engineering, top machine learning conferences, etc.

However, you might be able to circumvent a back-and-forth by reference to the goal journal's formal publication requirements: Just ask for / find the goal journal's rules as published on their website*. Do the journal's publication rules require that all cited material are indexed on PubMed? If not, then it's not a matter of preference or opinion. 

That might save you a round or two of arguing. Rochelle's argument is totally correct...but it relies on critical thinking. I'd be concerned that someone who carries around a "PubMed-only" Rule in their head, might not have critically evaluated that rule. ...So it might be more fruitful to supply them with an alternative rule.

Best,
Glen

*not citable of course, bc it's not on PubMed :^)

p.s. - Thanks Richard for the extra reference. I'll look into that.

------------------------------
Glen Wright Colopy
DPhil Oxon
Head of Data Science | Polaris Data Labs | https://polarisdatalabs.com/
Head of Data Science | IGOR - Your Personal Lab Assistant | https://www.igorlabapp.com/
------------------------------

Original Message:
Sent: 09-02-2026 08:17
From: Rochelle Tractenberg
Subject: Cite Firth (1993), Even Though Not Indexed in PubMed?

Hi - I agree with Professor Smith, a peer reviewed, published paper from <any field!> is a legitimate resource/reference to cite. It is not correct to suggest that only papers in PubMed are "legitimate". PubMed is not "the scientific record". You do not have to make this argument, simply point out in your responses to reviewers that the manuscript is an established element in the field and you can even "respectfully disagree" with the idea that only papers in pub med are citable - that is patently false and even harmful, so correcting this reviewer's wrong thinking, and highlighting this for the editor who may not have noticed that error when reviewing reviewer comments prior to sending the input back to you, might be helpful in general. Good luck with the submission! If that's the most challenging comment to respond to, then I look forward to your paper being published soon! :)

------------------------------
--
Rochelle E. Tractenberg, PhD, MPH, PhD, PStat®, FASA, FAAAS
Director, Collaborative for Research on Outcomes and -Metrics
https://crom.georgetown.domains
Co-Director, National Capital Spinal Cord Injury Model System (NC-SCIMS)
Professor
Neurology
Biostatistics, Bioinformatics & Biomathematics
Rehabilitation Medicine
Georgetown University
Washington, D.C. 20057
------------------------------


Original Message:
Sent: 09-01-2026 09:10
From: Richard Smith
Subject: Cite Firth (1993), Even Though Not Indexed in PubMed?

Presumably the reason Firth's paper is not in PubMed is because Biometrika is not a medical journal. Nevertheless, that's a very well known reference. It's featured in Julian Faraway's book "Extending the Linear Model with R" which I teach from in my GLM course, where Firth's method is explicitly cited as a solution to the well known paradox of logistic regression where there's a separating hyperplane between the positive and negative responses (in which case the regular GLM algorithm does not have a unique solution). There's also an R package called brglm that implements Firth's method. Good luck persuading the editor that this is all legitimate and well known statistical practice!

------------------------------
Richard Smith
University of North Carolina
------------------------------

Google AI states that no journal requires all references be on PubMed.   As statisticians, it is very common to cite methods papers when collaborating with non-statisticians.  I have never had a problem with such things.

Best,

Jon

------------------------------
Jonathan Shuster
------------------------------

Original Message:
Sent: 09-01-2026 09:10
From: Richard Smith
Subject: Cite Firth (1993), Even Though Not Indexed in PubMed?

Presumably the reason Firth's paper is not in PubMed is because Biometrika is not a medical journal. Nevertheless, that's a very well known reference. It's featured in Julian Faraway's book "Extending the Linear Model with R" which I teach from in my GLM course, where Firth's method is explicitly cited as a solution to the well known paradox of logistic regression where there's a separating hyperplane between the positive and negative responses (in which case the regular GLM algorithm does not have a unique solution). There's also an R package called brglm that implements Firth's method. Good luck persuading the editor that this is all legitimate and well known statistical practice!

------------------------------
Richard Smith
University of North Carolina
------------------------------

Thanks to everyone who replied and shared your opinions.  Many excellent points.

My reply to the researcher, who objected to the Firth article not having been indexed in PubMed, was that several other frequently cited statistics articles were not indexed in PubMed, yet were widely accepted in statistics methodology, such as:

The researcher withdrew objection to the Firth article not being indexed in Pub Med.  

Thanks again for taking the time to respond to my question.

------------------------------
Brandy Sinco, BS, MA, MS
Statistician Senior
Michigan Medicine
------------------------------

Authors are responsible for the accuracy of the entries in the reference list, regardless of the source. Relying on PubMed has risks. Some manuscripts I have reviewed contained badly garbled references that came from PubMed. The errors were obvious. The authors clearly did not proofread the manuscript.

Also, as a reviewer, I expect authors to have consulted each article that they cite. If they do not have access to the desired article, the entry for it in the reference list should report the article actually cited (for example, by saying "Cited by [actual source]"). Failure to do this is unprofessional and potentially embarrassing. For example, I have seen articles (in print) that judge the significance of a difference between two means by whether their ordinary confidence intervals overlap; and, worse, as the basis for that approach, they cite the 2001 article by Nat Schenker and Jane Gentleman (The American Statistician 55(3):182-186), which explains why that approach is incorrect!

------------------------------
David C. Hoaglin
------------------------------

Original Message:
Sent: 09-03-2026 14:28
From: Brandy Sinco
Subject: Cite Firth (1993), Even Though Not Indexed in PubMed?

Thanks to everyone who replied and shared your opinions.  Many excellent points.

My reply to the researcher, who objected to the Firth article not having been indexed in PubMed, was that several other frequently cited statistics articles were not indexed in PubMed, yet were widely accepted in statistics methodology, such as:

Kaplan Meier

1)_Kaplan EL.; Meier P. (1958). "Nonparametric estimation from incomplete observations". J. Amer. Statist. Assoc. 53 (282): 457–481. doi:10.2307/2281868. JSTOR 2281868.

 

Cox Proportional Hazards.

2)_Aalen, O. (1978). "Nonparametric Estimation of Partial Transition Probabilities in Multiple Decrement Models." Annals of Statistics 6:534–545.

 

3)_Cox, D. R. (1972). "Regression Models and Life-Tables." Journal of the Royal Statistical Society, Series B 34:187–220. With discussion.

 

4)_Cox, D. R. (1975). "Partial Likelihood." Biometrika 62:269–276.

The researcher withdrew objection to the Firth article not being indexed in Pub Med.  

Thanks again for taking the time to respond to my question.

------------------------------
Brandy Sinco, BS, MA, MS
Statistician Senior
Michigan Medicine
------------------------------