ASA Connect

06/28/2023

This Blog is presented out of concerns for future public health interests. My two papers (one in press and one accessible from the prior blog) represent two of the most important considerations for statisticians and biostatisticians so far in the 21^st century.  I blame no one for the misfortunes of the past.

Why this Blog is Must reading:  (1) In the past 3 years, there were over 41,000 articles listed in PubMed journals with "meta-analysis" in the title. If you do any consulting or education, you will almost certainly want to be informed; (2) Most "Evidence Pyramids" (Google this) place systematic reviews and meta-analysis at the top (most trusted) biomedical evidence; (3) There is no proof that the large sample distribution claims used by the mainstream have an evidence basis. See " An Important History Lesson" below. The Blog in fact proves the mainstream speculation on the asymptotic distribution of effect size uses contradictory assumptions; (4) as seen below, I have excellent credentials on this topic; and (5) reading this material will help you become part of the solution rather than part of the problem.

Knowing what you now know (or should know), I challenge readers to defend the ethics of using mainstream meta-analysis of randomized clinical trials where public health implications are at stake.

 

Of note (See last attachment), I invited the top biostatisticians/developers from the meta-analysis software companies that developed Comprehensive Meta Analysis and REVMAN for a pushback to this forum on the lack of scientific basis of the mainstream methods.  The deadline has passed with zero response.

 An Important History Lesson

Do not use mainstream methods (weights inversely proportional to the estimated variance).  If you read this material, and still insist on using the mainstream for an analysis of a collection of clinical trials of an intervention, your results could adversely impact public health.   The mainstream is not based on rigorous science.

Generally, statistics and biostatistics journals require mathematical proof that a new method is rigorous.  As this history lesson indicates, there have been major successes when these requirements were not met by the original authors, but later proven rigorously by others.  Large sample distribution theory for mainstream meta-analysis has never been proven to be correct, and now I have proven that the speculated distribution theory is scientifically unacceptable, in the sense that there is no evidence that its point estimates, confidence intervals, and p-values have an evidence basis.

The mainstream method for random effects meta-analysis, namely weights inversely proportional to the estimated variance, has three major references for use by statisticians and other scientists:  The Cochrane handbook (1); Introduction to Meta-Analysis (2); and Common Mistakes in Meta-Analysis (3). These references fail to recognize that in order to use the theory of linear combinations, the weights must be constant to a strong approximation (not seriously random variables as they are in truth).  The second enclosure provides the theorem they use with the a's the weights and the Y's as the estimates. The weights are in fact highly random variables. None of these references properly explain the assumptions underlying the methodology [assumption A4 in the first enclosure is needed to use the theory of linear (weighted) combinations of estimates].  None of these references offer any proof that the estimator of the main effect size has an asymptotic normal distribution or are approximately unbiased. The standard error formula is also incorrect.

This area of science was not nearly as fortunate as censored survival analysis. The speculated asymptotic distribution of the Kaplan-Meier estimator (4) was actually verified later by Efron. The speculated asymptotic distribution of Mantel's logrank test (5) for survival analysis was actually verified later by Breslow and Crowley. The speculated asymptotic distribution of Cox regression estimators (proportional hazards)  (6) was actually verified later by Tsiatis. Unfortunately, the statistical profession's good luck did not carry over to mainstream Meta-Analysis, where the science can no longer be viewed as sufficiently rigorous to trust in major public health decision making.  Fortunately, as described in my papers, we can use ratio estimation borrowed from survey sampling as a well-validated method. Documented Excel software to perform the calculations is available as a link in the first enclosure.

(1) Higgins J, Thompson J (2022) Cochrane Handbook for Systematic Reviews of Interventions. New York, 2022; NY: John Wiley and Sons, Publishers.

(2) Borenstein M, Hedges LV, Rothstein HR, Higgins JPT. Introduction to Meta-Analysis. New York, 2009; NY: John Wiley and Sons, Publishers.

(3) Borenstein M. Common Mistakes in Meta-Analysis and How to Avoid Them. 2019; Englewood, NJ: Biostat Inc, Publisher.

 (4) Kaplan, E. L.; Meier, Paul (1958). "Nonparametric Estimation from Incomplete Observations". Journal of the American Statistical Association. 53 (282): 457–481. doi:10.1080/01621459.1958.10501452. 

(5) Mantel, Nathan (1966). "Evaluation of survival data and two new rank order statistics arising in its consideration". Cancer Chemotherapy Reports. 50 (3): 163–70. PMID 5910392.

(6) Cox, David R (1972). "Regression Models and Life-Tables". Journal of the Royal Statistical Society, Series B. 34 (2): 187–220. JSTOR 2985181. MR 0341758.

My Relevant Credentials:

• Over 400 peer-reviewed articles with zero errata for statistics.  A fair number of these articles deal with meta-analysis
• Over $30 million in career NIH grants as Principal Investigator
• 15 years' service on NIH standing grant review committees
• Over 10 years as editorial board member of Research Synthesis Methods
• Invited by Ingram Olkin to review the Cochrane Handbook, second edition, which I published in Research Synthesis Methods.

Enclosures:

1.      Blog sent in early June

2.      Linear Combination theorem

3.      Reader feedback on the Blog

------------------------------
Jonathan Shuster
------------------------------

I understand Jonathan's frustration with substandard, even harmful, practice in an important area of application. Evidence of the deficiencies of the "mainstream" methods that he criticizes has been accumulating in the meta-analysis literature for many years.

Others have called for change. As examples, I mention two publications in 2014:

Cornell JE, Mulrow CD, Localio R, et al. (2014). Random-effects meta-analysis of inconsistent effects: a time for change.  Annals of Internal Medicine 160:267-270.

IntHout J, Ioannidis JPA, Borm GF (2014). The Hartung-Knapp-Sidik-Jonkman method for random effects meta-analysis is straightforward and considerably outperforms the standard DerSimonian-Laird method. BMC Medical Research Methodology 14:25.

Unfortunately, most meta-analyses appearing in the medical literature have continued to use deficient methods. Editors of major medical journals could push for improvement by instructing authors and, especially, reviewers to avoid such methods. Prominent editorials could emphasize this point. More dramatically, editors could choose a "nuclear option" by naming software (such as the Cochrane Collaboration's Review Manager) that provide (often as the default) deficient methods and announcing that results obtained by those methods are no longer acceptable for publication. This "nuclear option" may seem radical. It is. But when a decade passes with little or no improvement and potential harms are at stake, drastic remedies seem necessary.

I would not expect most authors of medical articles to follow the meta-analysis literature. As a reviewer, however, I have often seen little evidence that the authors involved in the meta-analysis have substantial experience in meta-analysis. The team may, understandably, expect that using well-known, seemingly authoritative software puts them on firm ground. Alas, they are usually mistaken.

I encourage others to contribute to this discussion and propose other remedies. 

------------------------------
David C. Hoaglin
------------------------------

Original Message:
Sent: 06-28-2023 13:49
From: Jonathan Shuster
Subject: Critically important follow-up to my June 2 post

06/28/2023

This Blog is presented out of concerns for future public health interests. My two papers (one in press and one accessible from the prior blog) represent two of the most important considerations for statisticians and biostatisticians so far in the 21^st century.  I blame no one for the misfortunes of the past.

Why this Blog is Must reading:  (1) In the past 3 years, there were over 41,000 articles listed in PubMed journals with "meta-analysis" in the title. If you do any consulting or education, you will almost certainly want to be informed; (2) Most "Evidence Pyramids" (Google this) place systematic reviews and meta-analysis at the top (most trusted) biomedical evidence; (3) There is no proof that the large sample distribution claims used by the mainstream have an evidence basis. See " An Important History Lesson" below. The Blog in fact proves the mainstream speculation on the asymptotic distribution of effect size uses contradictory assumptions; (4) as seen below, I have excellent credentials on this topic; and (5) reading this material will help you become part of the solution rather than part of the problem.

Knowing what you now know (or should know), I challenge readers to defend the ethics of using mainstream meta-analysis of randomized clinical trials where public health implications are at stake.

 

Of note (See last attachment), I invited the top biostatisticians/developers from the meta-analysis software companies that developed Comprehensive Meta Analysis and REVMAN for a pushback to this forum on the lack of scientific basis of the mainstream methods.  The deadline has passed with zero response.

 An Important History Lesson

Do not use mainstream methods (weights inversely proportional to the estimated variance).  If you read this material, and still insist on using the mainstream for an analysis of a collection of clinical trials of an intervention, your results could adversely impact public health.   The mainstream is not based on rigorous science.

Generally, statistics and biostatistics journals require mathematical proof that a new method is rigorous.  As this history lesson indicates, there have been major successes when these requirements were not met by the original authors, but later proven rigorously by others.  Large sample distribution theory for mainstream meta-analysis has never been proven to be correct, and now I have proven that the speculated distribution theory is scientifically unacceptable, in the sense that there is no evidence that its point estimates, confidence intervals, and p-values have an evidence basis.

The mainstream method for random effects meta-analysis, namely weights inversely proportional to the estimated variance, has three major references for use by statisticians and other scientists:  The Cochrane handbook (1); Introduction to Meta-Analysis (2); and Common Mistakes in Meta-Analysis (3). These references fail to recognize that in order to use the theory of linear combinations, the weights must be constant to a strong approximation (not seriously random variables as they are in truth).  The second enclosure provides the theorem they use with the a's the weights and the Y's as the estimates. The weights are in fact highly random variables. None of these references properly explain the assumptions underlying the methodology [assumption A4 in the first enclosure is needed to use the theory of linear (weighted) combinations of estimates].  None of these references offer any proof that the estimator of the main effect size has an asymptotic normal distribution or are approximately unbiased. The standard error formula is also incorrect.

This area of science was not nearly as fortunate as censored survival analysis. The speculated asymptotic distribution of the Kaplan-Meier estimator (4) was actually verified later by Efron. The speculated asymptotic distribution of Mantel's logrank test (5) for survival analysis was actually verified later by Breslow and Crowley. The speculated asymptotic distribution of Cox regression estimators (proportional hazards)  (6) was actually verified later by Tsiatis. Unfortunately, the statistical profession's good luck did not carry over to mainstream Meta-Analysis, where the science can no longer be viewed as sufficiently rigorous to trust in major public health decision making.  Fortunately, as described in my papers, we can use ratio estimation borrowed from survey sampling as a well-validated method. Documented Excel software to perform the calculations is available as a link in the first enclosure.

(1) Higgins J, Thompson J (2022) Cochrane Handbook for Systematic Reviews of Interventions. New York, 2022; NY: John Wiley and Sons, Publishers.

(2) Borenstein M, Hedges LV, Rothstein HR, Higgins JPT. Introduction to Meta-Analysis. New York, 2009; NY: John Wiley and Sons, Publishers.

(3) Borenstein M. Common Mistakes in Meta-Analysis and How to Avoid Them. 2019; Englewood, NJ: Biostat Inc, Publisher.

 (4) Kaplan, E. L.; Meier, Paul (1958). "Nonparametric Estimation from Incomplete Observations". Journal of the American Statistical Association. 53 (282): 457–481. doi:10.1080/01621459.1958.10501452. 

(5) Mantel, Nathan (1966). "Evaluation of survival data and two new rank order statistics arising in its consideration". Cancer Chemotherapy Reports. 50 (3): 163–70. PMID 5910392.

(6) Cox, David R (1972). "Regression Models and Life-Tables". Journal of the Royal Statistical Society, Series B. 34 (2): 187–220. JSTOR 2985181. MR 0341758.

My Relevant Credentials:

• Over 400 peer-reviewed articles with zero errata for statistics.  A fair number of these articles deal with meta-analysis
• Over $30 million in career NIH grants as Principal Investigator
• 15 years' service on NIH standing grant review committees
• Over 10 years as editorial board member of Research Synthesis Methods
• Invited by Ingram Olkin to review the Cochrane Handbook, second edition, which I published in Research Synthesis Methods.

Enclosures:

1.      Blog sent in early June

2.      Linear Combination theorem

3.      Reader feedback on the Blog

------------------------------
Jonathan Shuster
------------------------------