ASA Connect

Hello,

I am writing a couple articles for a chemistry journal on the Design and Analysis of Experiments. As we all know, these methods test different combinations of factors and levels. Meaning, we change multiple things, simultaneously. 

However, when you use your favorite web searching tool, and look for, "Designing a scientific experiment" or some other related searches, you'll find a lot of links that claim, "You CANNOT change more than one thing at a time during an experiment." Sometimes that is followed up with, "Statistics does NOT allow this." Others will say, "Because, you can't tell what made the change in the response." 

We all know this is non-sense. But, it is ingrained in scientific lore. Especially at the academic level. Where most of the journal article reviewers reside. 

I bring this up because I have written articles for other journals in the past where I used Plackett-Burman designs and Box-Behnken Designs. These articles were rejected by the reviewers for the reasons stated above. Sadly, in these articles, the first 8-15 references were stats textbooks. When I pointed this out to the reviewers. It angered them. Of the dozen or so reviewers I've dealt with, a pair of reviewers, one said something like, "This so called statistician should know that you can't change more than one thing at a time during the experiment." The other said something like, "Why does this author bore me with the basics of designed experiments. Everyone already knows this."  

When I was an undergrad in Physics and Biochemistry and a graduate student in Chemistry, all the articles we read used t-tests or multiple t-tests to compare groups, "control" vs "experimental group N". We used simple linear regression to find a relation between X and Y. In some articles, there was X1 to Xn and Y. The authors chose to use multiple, Simple Linear Regressions. Meaning X1 vs Y, X2 vs Y, Xn vs Y, etc.  

So, how do you handle a rejection based upon the editor's or the reviewer's lack of statistical knowledge?
How do you battle against bad statistics in science journal articles?  

Thanks,
Drew

------------------------------
Andrew Ekstrom

Statistician, Chemist, HPC Abuser;-)
------------------------------

Go to a different journal.

------------------------------
Douglas Landsittel
Professor of Biomedical Informatics
University of Pittsburgh
------------------------------

Original Message:
Sent: 01-11-2021 13:03
From: Andrew Ekstrom
Subject: What to do when journal reviewers don't believe some statistics methods exist

Hello,

I am writing a couple articles for a chemistry journal on the Design and Analysis of Experiments. As we all know, these methods test different combinations of factors and levels. Meaning, we change multiple things, simultaneously.

However, when you use your favorite web searching tool, and look for, "Designing a scientific experiment" or some other related searches, you'll find a lot of links that claim, "You CANNOT change more than one thing at a time during an experiment." Sometimes that is followed up with, "Statistics does NOT allow this." Others will say, "Because, you can't tell what made the change in the response."

We all know this is non-sense. But, it is ingrained in scientific lore. Especially at the academic level. Where most of the journal article reviewers reside.

I bring this up because I have written articles for other journals in the past where I used Plackett-Burman designs and Box-Behnken Designs. These articles were rejected by the reviewers for the reasons stated above. Sadly, in these articles, the first 8-15 references were stats textbooks. When I pointed this out to the reviewers. It angered them. Of the dozen or so reviewers I've dealt with, a pair of reviewers, one said something like, "This so called statistician should know that you can't change more than one thing at a time during the experiment." The other said something like, "Why does this author bore me with the basics of designed experiments. Everyone already knows this."

When I was an undergrad in Physics and Biochemistry and a graduate student in Chemistry, all the articles we read used t-tests or multiple t-tests to compare groups, "control" vs "experimental group N". We used simple linear regression to find a relation between X and Y. In some articles, there was X1 to Xn and Y. The authors chose to use multiple, Simple Linear Regressions. Meaning X1 vs Y, X2 vs Y, Xn vs Y, etc.

So, how do you handle a rejection based upon the editor's or the reviewer's lack of statistical knowledge?
How do you battle against bad statistics in science journal articles?

Thanks,
Drew

------------------------------
Andrew Ekstrom

Statistician, Chemist, HPC Abuser;-)
------------------------------

I tried that too. 

The first article I wanted to publish used fractional factorial designs and CCD response surfaces to optimize the extraction of chlorinated organic compounds. I used an EPA method as a basis for the experiment and data. The results were that following the EPA method gave significantly lower recoveries. So I had th edouble duty of using "non-existent" statistical methods AND claiming the EPA method was really bad. 

6 journals, 12 reviewers, one response, "You cannot change more than one thing at a time during an experiment. This so called statistician should know that!"

------------------------------
Andrew Ekstrom

Statistician, Chemist, HPC Abuser;-)
------------------------------

Original Message:
Sent: 01-12-2021 08:17
From: Douglas Landsittel
Subject: What to do when journal reviewers don't believe some statistics methods exist

Go to a different journal.

------------------------------
Douglas Landsittel
Professor of Biomedical Informatics
University of Pittsburgh

Original Message:
Sent: 01-11-2021 13:03
From: Andrew Ekstrom
Subject: What to do when journal reviewers don't believe some statistics methods exist

Hello,

I am writing a couple articles for a chemistry journal on the Design and Analysis of Experiments. As we all know, these methods test different combinations of factors and levels. Meaning, we change multiple things, simultaneously.

However, when you use your favorite web searching tool, and look for, "Designing a scientific experiment" or some other related searches, you'll find a lot of links that claim, "You CANNOT change more than one thing at a time during an experiment." Sometimes that is followed up with, "Statistics does NOT allow this." Others will say, "Because, you can't tell what made the change in the response."

We all know this is non-sense. But, it is ingrained in scientific lore. Especially at the academic level. Where most of the journal article reviewers reside.

I bring this up because I have written articles for other journals in the past where I used Plackett-Burman designs and Box-Behnken Designs. These articles were rejected by the reviewers for the reasons stated above. Sadly, in these articles, the first 8-15 references were stats textbooks. When I pointed this out to the reviewers. It angered them. Of the dozen or so reviewers I've dealt with, a pair of reviewers, one said something like, "This so called statistician should know that you can't change more than one thing at a time during the experiment." The other said something like, "Why does this author bore me with the basics of designed experiments. Everyone already knows this."

When I was an undergrad in Physics and Biochemistry and a graduate student in Chemistry, all the articles we read used t-tests or multiple t-tests to compare groups, "control" vs "experimental group N". We used simple linear regression to find a relation between X and Y. In some articles, there was X1 to Xn and Y. The authors chose to use multiple, Simple Linear Regressions. Meaning X1 vs Y, X2 vs Y, Xn vs Y, etc.

So, how do you handle a rejection based upon the editor's or the reviewer's lack of statistical knowledge?
How do you battle against bad statistics in science journal articles?

Thanks,
Drew

------------------------------
Andrew Ekstrom

Statistician, Chemist, HPC Abuser;-)
------------------------------

Hi Andrew,
I have encountered this thinking in the past and would appreciate knowing where/when you will publish your article.  It seems some academic degree programs hold to traditional experimental procedure vigorously and while they have proven themselves over the years, methods have certainly expanded beyond "one factor at a time" experimentation.

Thank you in advance,
Steve Reagan

------------------------------
Steven Reagan
Computational Modeling Manager
L&L Products, Inc.
------------------------------

Original Message:
Sent: 01-11-2021 13:03
From: Andrew Ekstrom
Subject: What to do when journal reviewers don't believe some statistics methods exist

Hello,

I am writing a couple articles for a chemistry journal on the Design and Analysis of Experiments. As we all know, these methods test different combinations of factors and levels. Meaning, we change multiple things, simultaneously.

However, when you use your favorite web searching tool, and look for, "Designing a scientific experiment" or some other related searches, you'll find a lot of links that claim, "You CANNOT change more than one thing at a time during an experiment." Sometimes that is followed up with, "Statistics does NOT allow this." Others will say, "Because, you can't tell what made the change in the response."

We all know this is non-sense. But, it is ingrained in scientific lore. Especially at the academic level. Where most of the journal article reviewers reside.

I bring this up because I have written articles for other journals in the past where I used Plackett-Burman designs and Box-Behnken Designs. These articles were rejected by the reviewers for the reasons stated above. Sadly, in these articles, the first 8-15 references were stats textbooks. When I pointed this out to the reviewers. It angered them. Of the dozen or so reviewers I've dealt with, a pair of reviewers, one said something like, "This so called statistician should know that you can't change more than one thing at a time during the experiment." The other said something like, "Why does this author bore me with the basics of designed experiments. Everyone already knows this."

When I was an undergrad in Physics and Biochemistry and a graduate student in Chemistry, all the articles we read used t-tests or multiple t-tests to compare groups, "control" vs "experimental group N". We used simple linear regression to find a relation between X and Y. In some articles, there was X1 to Xn and Y. The authors chose to use multiple, Simple Linear Regressions. Meaning X1 vs Y, X2 vs Y, Xn vs Y, etc.

So, how do you handle a rejection based upon the editor's or the reviewer's lack of statistical knowledge?
How do you battle against bad statistics in science journal articles?

Thanks,
Drew

------------------------------
Andrew Ekstrom

Statistician, Chemist, HPC Abuser;-)
------------------------------

You will encounter some version of this problem quite often if you are using any statistics beyond chi square, t-test, or linear regression. Having a little knowledge can be dangerous. If they ask for revisions, you first have to decide if your primary objetive is to get the paper published in that journal, or express your feelings about the reviewer.  You probably can't do both. I have found that the best approach, if you want to get published in that journal, is to a) educate the reviewers about the methods you are using and why they are best for the situation using citations, tables, and figures. b) explain why the method they are suggesting would not work or why it is not optimal. And most importantly, c) Try very hard to not to insult the reviewer.  Write as if you are engaging them in a dialogue about the best approach to use. Your attitude should be that their proposal is worthy of consideration, that in fact you considered it, but that you decided against it, and then explain why.

------------------------------
Richard Amdur
Clinical Professor and Lead Biostatistician
Medical Faculty Associates
------------------------------

Original Message:
Sent: 01-12-2021 08:48
From: Steven Reagan
Subject: What to do when journal reviewers don't believe some statistics methods exist

Hi Andrew,
I have encountered this thinking in the past and would appreciate knowing where/when you will publish your article.  It seems some academic degree programs hold to traditional experimental procedure vigorously and while they have proven themselves over the years, methods have certainly expanded beyond "one factor at a time" experimentation.

Thank you in advance,
Steve Reagan

------------------------------
Steven Reagan
Computational Modeling Manager
L&L Products, Inc.

Original Message:
Sent: 01-11-2021 13:03
From: Andrew Ekstrom
Subject: What to do when journal reviewers don't believe some statistics methods exist

Hello,

I am writing a couple articles for a chemistry journal on the Design and Analysis of Experiments. As we all know, these methods test different combinations of factors and levels. Meaning, we change multiple things, simultaneously.

However, when you use your favorite web searching tool, and look for, "Designing a scientific experiment" or some other related searches, you'll find a lot of links that claim, "You CANNOT change more than one thing at a time during an experiment." Sometimes that is followed up with, "Statistics does NOT allow this." Others will say, "Because, you can't tell what made the change in the response."

We all know this is non-sense. But, it is ingrained in scientific lore. Especially at the academic level. Where most of the journal article reviewers reside.

I bring this up because I have written articles for other journals in the past where I used Plackett-Burman designs and Box-Behnken Designs. These articles were rejected by the reviewers for the reasons stated above. Sadly, in these articles, the first 8-15 references were stats textbooks. When I pointed this out to the reviewers. It angered them. Of the dozen or so reviewers I've dealt with, a pair of reviewers, one said something like, "This so called statistician should know that you can't change more than one thing at a time during the experiment." The other said something like, "Why does this author bore me with the basics of designed experiments. Everyone already knows this."

When I was an undergrad in Physics and Biochemistry and a graduate student in Chemistry, all the articles we read used t-tests or multiple t-tests to compare groups, "control" vs "experimental group N". We used simple linear regression to find a relation between X and Y. In some articles, there was X1 to Xn and Y. The authors chose to use multiple, Simple Linear Regressions. Meaning X1 vs Y, X2 vs Y, Xn vs Y, etc.

So, how do you handle a rejection based upon the editor's or the reviewer's lack of statistical knowledge?
How do you battle against bad statistics in science journal articles?

Thanks,
Drew

------------------------------
Andrew Ekstrom

Statistician, Chemist, HPC Abuser;-)
------------------------------

The last time I dealt with journal reviewers, one reviewer basically said DOE methods do not exist. The other said everyone already knows about these methods. And wondered why I spent 2 pages describing how the methods work and why they actually do exist. 

When I asked the editor how to handle the reviewers comments. He said, "Good luck!" 

Since I knew I wouldn't be publishing in that journal, I decided to go off on the first reviewer. Unprofessional, YES. Do I care? NOPE!

------------------------------
Andrew Ekstrom

Statistician, Chemist, HPC Abuser;-)
------------------------------

Original Message:
Sent: 01-13-2021 07:18
From: Richard Amdur
Subject: What to do when journal reviewers don't believe some statistics methods exist

You will encounter some version of this problem quite often if you are using any statistics beyond chi square, t-test, or linear regression. Having a little knowledge can be dangerous. If they ask for revisions, you first have to decide if your primary objetive is to get the paper published in that journal, or express your feelings about the reviewer.  You probably can't do both. I have found that the best approach, if you want to get published in that journal, is to a) educate the reviewers about the methods you are using and why they are best for the situation using citations, tables, and figures. b) explain why the method they are suggesting would not work or why it is not optimal. And most importantly, c) Try very hard to not to insult the reviewer.  Write as if you are engaging them in a dialogue about the best approach to use. Your attitude should be that their proposal is worthy of consideration, that in fact you considered it, but that you decided against it, and then explain why.

------------------------------
Richard Amdur
Clinical Professor and Lead Biostatistician
Medical Faculty Associates

Original Message:
Sent: 01-12-2021 08:48
From: Steven Reagan
Subject: What to do when journal reviewers don't believe some statistics methods exist

Hi Andrew,
I have encountered this thinking in the past and would appreciate knowing where/when you will publish your article.  It seems some academic degree programs hold to traditional experimental procedure vigorously and while they have proven themselves over the years, methods have certainly expanded beyond "one factor at a time" experimentation.

Thank you in advance,
Steve Reagan

------------------------------
Steven Reagan
Computational Modeling Manager
L&L Products, Inc.

Original Message:
Sent: 01-11-2021 13:03
From: Andrew Ekstrom
Subject: What to do when journal reviewers don't believe some statistics methods exist

Hello,

I am writing a couple articles for a chemistry journal on the Design and Analysis of Experiments. As we all know, these methods test different combinations of factors and levels. Meaning, we change multiple things, simultaneously.

However, when you use your favorite web searching tool, and look for, "Designing a scientific experiment" or some other related searches, you'll find a lot of links that claim, "You CANNOT change more than one thing at a time during an experiment." Sometimes that is followed up with, "Statistics does NOT allow this." Others will say, "Because, you can't tell what made the change in the response."

We all know this is non-sense. But, it is ingrained in scientific lore. Especially at the academic level. Where most of the journal article reviewers reside.

I bring this up because I have written articles for other journals in the past where I used Plackett-Burman designs and Box-Behnken Designs. These articles were rejected by the reviewers for the reasons stated above. Sadly, in these articles, the first 8-15 references were stats textbooks. When I pointed this out to the reviewers. It angered them. Of the dozen or so reviewers I've dealt with, a pair of reviewers, one said something like, "This so called statistician should know that you can't change more than one thing at a time during the experiment." The other said something like, "Why does this author bore me with the basics of designed experiments. Everyone already knows this."

When I was an undergrad in Physics and Biochemistry and a graduate student in Chemistry, all the articles we read used t-tests or multiple t-tests to compare groups, "control" vs "experimental group N". We used simple linear regression to find a relation between X and Y. In some articles, there was X1 to Xn and Y. The authors chose to use multiple, Simple Linear Regressions. Meaning X1 vs Y, X2 vs Y, Xn vs Y, etc.

So, how do you handle a rejection based upon the editor's or the reviewer's lack of statistical knowledge?
How do you battle against bad statistics in science journal articles?

Thanks,
Drew

------------------------------
Andrew Ekstrom

Statistician, Chemist, HPC Abuser;-)
------------------------------

I tried to publish the articles in a bunch of different journals. It was such a long time ago that I don't remember everywhere. Sorry. 

But, if you look at some of the "top" research journals by the American Chemical Society (ACS) and some of the Springer journals in chemistry and related sciences, you'll find that OFAT is king. When Design of Experiments methods are used, the analysis is usually a bit off. 

IF you read through some of the Chemometrics books that are available, they do cover DoE methods. But, the few that I have, condense ALL of DOE into a chapter, if that. Again, then you analyze the textbook data properly, you find that the conclusions of the author(s) are usually not what a statistician would come to. Same thing for some of the Toxicology and Biology "Experimental Design" textbooks you can use.

------------------------------
Andrew Ekstrom

Statistician, Chemist, HPC Abuser;-)
------------------------------

Original Message:
Sent: 01-12-2021 08:48
From: Steven Reagan
Subject: What to do when journal reviewers don't believe some statistics methods exist

Hi Andrew,
I have encountered this thinking in the past and would appreciate knowing where/when you will publish your article.  It seems some academic degree programs hold to traditional experimental procedure vigorously and while they have proven themselves over the years, methods have certainly expanded beyond "one factor at a time" experimentation.

Thank you in advance,
Steve Reagan

------------------------------
Steven Reagan
Computational Modeling Manager
L&L Products, Inc.

Original Message:
Sent: 01-11-2021 13:03
From: Andrew Ekstrom
Subject: What to do when journal reviewers don't believe some statistics methods exist

Hello,

I am writing a couple articles for a chemistry journal on the Design and Analysis of Experiments. As we all know, these methods test different combinations of factors and levels. Meaning, we change multiple things, simultaneously.

However, when you use your favorite web searching tool, and look for, "Designing a scientific experiment" or some other related searches, you'll find a lot of links that claim, "You CANNOT change more than one thing at a time during an experiment." Sometimes that is followed up with, "Statistics does NOT allow this." Others will say, "Because, you can't tell what made the change in the response."

We all know this is non-sense. But, it is ingrained in scientific lore. Especially at the academic level. Where most of the journal article reviewers reside.

I bring this up because I have written articles for other journals in the past where I used Plackett-Burman designs and Box-Behnken Designs. These articles were rejected by the reviewers for the reasons stated above. Sadly, in these articles, the first 8-15 references were stats textbooks. When I pointed this out to the reviewers. It angered them. Of the dozen or so reviewers I've dealt with, a pair of reviewers, one said something like, "This so called statistician should know that you can't change more than one thing at a time during the experiment." The other said something like, "Why does this author bore me with the basics of designed experiments. Everyone already knows this."

When I was an undergrad in Physics and Biochemistry and a graduate student in Chemistry, all the articles we read used t-tests or multiple t-tests to compare groups, "control" vs "experimental group N". We used simple linear regression to find a relation between X and Y. In some articles, there was X1 to Xn and Y. The authors chose to use multiple, Simple Linear Regressions. Meaning X1 vs Y, X2 vs Y, Xn vs Y, etc.

So, how do you handle a rejection based upon the editor's or the reviewer's lack of statistical knowledge?
How do you battle against bad statistics in science journal articles?

Thanks,
Drew

------------------------------
Andrew Ekstrom

Statistician, Chemist, HPC Abuser;-)
------------------------------

If you want to take this on, I would recommend that you write an article with a deliberately provocative title like 'is it true that statistics says you can only change one factor at a time in design of experiments?'  you might also want to try to open a discussion with the journal editor on whether they would be supportive of an article on this topic appearing in their journal.  good luck!   -g


Glen Satten PhD
Emory University School of Medicine

------------------------------
Glen Satten
------------------------------

Original Message:
Sent: 01-11-2021 13:03
From: Andrew Ekstrom
Subject: What to do when journal reviewers don't believe some statistics methods exist

Hello,

I am writing a couple articles for a chemistry journal on the Design and Analysis of Experiments. As we all know, these methods test different combinations of factors and levels. Meaning, we change multiple things, simultaneously.

However, when you use your favorite web searching tool, and look for, "Designing a scientific experiment" or some other related searches, you'll find a lot of links that claim, "You CANNOT change more than one thing at a time during an experiment." Sometimes that is followed up with, "Statistics does NOT allow this." Others will say, "Because, you can't tell what made the change in the response."

We all know this is non-sense. But, it is ingrained in scientific lore. Especially at the academic level. Where most of the journal article reviewers reside.

I bring this up because I have written articles for other journals in the past where I used Plackett-Burman designs and Box-Behnken Designs. These articles were rejected by the reviewers for the reasons stated above. Sadly, in these articles, the first 8-15 references were stats textbooks. When I pointed this out to the reviewers. It angered them. Of the dozen or so reviewers I've dealt with, a pair of reviewers, one said something like, "This so called statistician should know that you can't change more than one thing at a time during the experiment." The other said something like, "Why does this author bore me with the basics of designed experiments. Everyone already knows this."

When I was an undergrad in Physics and Biochemistry and a graduate student in Chemistry, all the articles we read used t-tests or multiple t-tests to compare groups, "control" vs "experimental group N". We used simple linear regression to find a relation between X and Y. In some articles, there was X1 to Xn and Y. The authors chose to use multiple, Simple Linear Regressions. Meaning X1 vs Y, X2 vs Y, Xn vs Y, etc.

So, how do you handle a rejection based upon the editor's or the reviewer's lack of statistical knowledge?
How do you battle against bad statistics in science journal articles?

Thanks,
Drew

------------------------------
Andrew Ekstrom

Statistician, Chemist, HPC Abuser;-)
------------------------------

I plan on writing a few articles on the topic. 

My working title is, "The Misinterpretation of Statistics and It's Disastrous Consequences: Designing a Better Experiment Part 1" 

I plan on discussing "screening designs", P-values, T-tests and ANOVA. Then I plan on getting into more advanced topics. 

I have given a talk to chemists and other scientists titled, "Doing Science Better with Statistics". Few if any scientists want to science better.

------------------------------
Andrew Ekstrom

Statistician, Chemist, HPC Abuser;-)
------------------------------

Original Message:
Sent: 01-12-2021 09:10
From: Glen Satten
Subject: What to do when journal reviewers don't believe some statistics methods exist

If you want to take this on, I would recommend that you write an article with a deliberately provocative title like 'is it true that statistics says you can only change one factor at a time in design of experiments?'  you might also want to try to open a discussion with the journal editor on whether they would be supportive of an article on this topic appearing in their journal.  good luck!   -g


Glen Satten PhD
Emory University School of Medicine

------------------------------
Glen Satten

Original Message:
Sent: 01-11-2021 13:03
From: Andrew Ekstrom
Subject: What to do when journal reviewers don't believe some statistics methods exist

Hello,

I am writing a couple articles for a chemistry journal on the Design and Analysis of Experiments. As we all know, these methods test different combinations of factors and levels. Meaning, we change multiple things, simultaneously.

However, when you use your favorite web searching tool, and look for, "Designing a scientific experiment" or some other related searches, you'll find a lot of links that claim, "You CANNOT change more than one thing at a time during an experiment." Sometimes that is followed up with, "Statistics does NOT allow this." Others will say, "Because, you can't tell what made the change in the response."

We all know this is non-sense. But, it is ingrained in scientific lore. Especially at the academic level. Where most of the journal article reviewers reside.

I bring this up because I have written articles for other journals in the past where I used Plackett-Burman designs and Box-Behnken Designs. These articles were rejected by the reviewers for the reasons stated above. Sadly, in these articles, the first 8-15 references were stats textbooks. When I pointed this out to the reviewers. It angered them. Of the dozen or so reviewers I've dealt with, a pair of reviewers, one said something like, "This so called statistician should know that you can't change more than one thing at a time during the experiment." The other said something like, "Why does this author bore me with the basics of designed experiments. Everyone already knows this."

When I was an undergrad in Physics and Biochemistry and a graduate student in Chemistry, all the articles we read used t-tests or multiple t-tests to compare groups, "control" vs "experimental group N". We used simple linear regression to find a relation between X and Y. In some articles, there was X1 to Xn and Y. The authors chose to use multiple, Simple Linear Regressions. Meaning X1 vs Y, X2 vs Y, Xn vs Y, etc.

So, how do you handle a rejection based upon the editor's or the reviewer's lack of statistical knowledge?
How do you battle against bad statistics in science journal articles?

Thanks,
Drew

------------------------------
Andrew Ekstrom

Statistician, Chemist, HPC Abuser;-)
------------------------------

I have encountered this issue with healthcare journals.  I did some research on why this was happening and ended up collaborating on a paper on our discoveries.

It Is Time to Reconsider Factorial Designs: How Bradford Hill and R. A. Fisher Shaped the Standard of Clinical Evidence, has been published in Quality Management in Healthcare in Vol. 29, No. 2, April/June 2020.

We currated a number of articles using factorial designs that had been published in a variety of medical journals since the 1950's.  These can serve as good references to give to reviewers that have not had experience with muli-factor designs.

------------------------------
Lloyd Provost
Associates in Process Improvement
------------------------------

Original Message:
Sent: 01-11-2021 13:03
From: Andrew Ekstrom
Subject: What to do when journal reviewers don't believe some statistics methods exist

Hello,

I am writing a couple articles for a chemistry journal on the Design and Analysis of Experiments. As we all know, these methods test different combinations of factors and levels. Meaning, we change multiple things, simultaneously.

However, when you use your favorite web searching tool, and look for, "Designing a scientific experiment" or some other related searches, you'll find a lot of links that claim, "You CANNOT change more than one thing at a time during an experiment." Sometimes that is followed up with, "Statistics does NOT allow this." Others will say, "Because, you can't tell what made the change in the response."

We all know this is non-sense. But, it is ingrained in scientific lore. Especially at the academic level. Where most of the journal article reviewers reside.

I bring this up because I have written articles for other journals in the past where I used Plackett-Burman designs and Box-Behnken Designs. These articles were rejected by the reviewers for the reasons stated above. Sadly, in these articles, the first 8-15 references were stats textbooks. When I pointed this out to the reviewers. It angered them. Of the dozen or so reviewers I've dealt with, a pair of reviewers, one said something like, "This so called statistician should know that you can't change more than one thing at a time during the experiment." The other said something like, "Why does this author bore me with the basics of designed experiments. Everyone already knows this."

When I was an undergrad in Physics and Biochemistry and a graduate student in Chemistry, all the articles we read used t-tests or multiple t-tests to compare groups, "control" vs "experimental group N". We used simple linear regression to find a relation between X and Y. In some articles, there was X1 to Xn and Y. The authors chose to use multiple, Simple Linear Regressions. Meaning X1 vs Y, X2 vs Y, Xn vs Y, etc.

So, how do you handle a rejection based upon the editor's or the reviewer's lack of statistical knowledge?
How do you battle against bad statistics in science journal articles?

Thanks,
Drew

------------------------------
Andrew Ekstrom

Statistician, Chemist, HPC Abuser;-)
------------------------------

Drew -  

I have worked as a referee, and realize the effort that takes, but yes, it is possible to get a jaw dropping review.  Once, at a top survey journal, I'll tell you which one privately, a journal for which I have also refereed, I was told that I could not have gotten weighted least square regression results from SAS PROC REG, because SAS PROC REG would not do that.  What!?  You'd think I was getting a preview of the gaslighting of the last five years.  And the editor did not disagree with the referee!   

A friend had a paper rejected because it used a technique not in favor at a certain journal, so they published it with the ISR.  

So, your story sounds familiar.  You could try another journal, but you might get the same referees.  

Best wishes - Jim

------------------------------
James Knaub (Jim)
Retired Lead Mathematical Statistician
------------------------------

Original Message:
Sent: 01-11-2021 13:03
From: Andrew Ekstrom
Subject: What to do when journal reviewers don't believe some statistics methods exist

Hello,

I am writing a couple articles for a chemistry journal on the Design and Analysis of Experiments. As we all know, these methods test different combinations of factors and levels. Meaning, we change multiple things, simultaneously.

However, when you use your favorite web searching tool, and look for, "Designing a scientific experiment" or some other related searches, you'll find a lot of links that claim, "You CANNOT change more than one thing at a time during an experiment." Sometimes that is followed up with, "Statistics does NOT allow this." Others will say, "Because, you can't tell what made the change in the response."

We all know this is non-sense. But, it is ingrained in scientific lore. Especially at the academic level. Where most of the journal article reviewers reside.

I bring this up because I have written articles for other journals in the past where I used Plackett-Burman designs and Box-Behnken Designs. These articles were rejected by the reviewers for the reasons stated above. Sadly, in these articles, the first 8-15 references were stats textbooks. When I pointed this out to the reviewers. It angered them. Of the dozen or so reviewers I've dealt with, a pair of reviewers, one said something like, "This so called statistician should know that you can't change more than one thing at a time during the experiment." The other said something like, "Why does this author bore me with the basics of designed experiments. Everyone already knows this."

When I was an undergrad in Physics and Biochemistry and a graduate student in Chemistry, all the articles we read used t-tests or multiple t-tests to compare groups, "control" vs "experimental group N". We used simple linear regression to find a relation between X and Y. In some articles, there was X1 to Xn and Y. The authors chose to use multiple, Simple Linear Regressions. Meaning X1 vs Y, X2 vs Y, Xn vs Y, etc.

So, how do you handle a rejection based upon the editor's or the reviewer's lack of statistical knowledge?
How do you battle against bad statistics in science journal articles?

Thanks,
Drew

------------------------------
Andrew Ekstrom

Statistician, Chemist, HPC Abuser;-)
------------------------------

Dear Drew,

I'm sorry that you are having this problem. I know that it is frustrating. I think that replying to the reviewers and editors in the manner that Richard Amdur suggests is a good start, while being prepared to go to another journal if necessary. 

As you stated in your query, as a graduate student you learned t-tests, multiple t-tests, and linear regression. I suspect that your reviewers learned the same thing. When all you have is a hammer, even problem starts to look like a nail.

Therefore, part of the problem is on us as statisticians to (1) educate reviewers ever so gently and (2) advocate for changes in the graduate curriculum for other disciplines. Both of these items will take a long time. In the meantime, you or someone in the ASA should write to the websites and text authors, explain to them that their information is incorrect, and ask them to update it.

Cheers and Good Luck,
Monnie


------------------------------
Monnie McGee
Associate Professor
Southern Methodist University
Dallas,TX United States
------------------------------

Original Message:
Sent: 01-11-2021 13:03
From: Andrew Ekstrom
Subject: What to do when journal reviewers don't believe some statistics methods exist

Hello,

I am writing a couple articles for a chemistry journal on the Design and Analysis of Experiments. As we all know, these methods test different combinations of factors and levels. Meaning, we change multiple things, simultaneously.

However, when you use your favorite web searching tool, and look for, "Designing a scientific experiment" or some other related searches, you'll find a lot of links that claim, "You CANNOT change more than one thing at a time during an experiment." Sometimes that is followed up with, "Statistics does NOT allow this." Others will say, "Because, you can't tell what made the change in the response."

We all know this is non-sense. But, it is ingrained in scientific lore. Especially at the academic level. Where most of the journal article reviewers reside.

I bring this up because I have written articles for other journals in the past where I used Plackett-Burman designs and Box-Behnken Designs. These articles were rejected by the reviewers for the reasons stated above. Sadly, in these articles, the first 8-15 references were stats textbooks. When I pointed this out to the reviewers. It angered them. Of the dozen or so reviewers I've dealt with, a pair of reviewers, one said something like, "This so called statistician should know that you can't change more than one thing at a time during the experiment." The other said something like, "Why does this author bore me with the basics of designed experiments. Everyone already knows this."

When I was an undergrad in Physics and Biochemistry and a graduate student in Chemistry, all the articles we read used t-tests or multiple t-tests to compare groups, "control" vs "experimental group N". We used simple linear regression to find a relation between X and Y. In some articles, there was X1 to Xn and Y. The authors chose to use multiple, Simple Linear Regressions. Meaning X1 vs Y, X2 vs Y, Xn vs Y, etc.

So, how do you handle a rejection based upon the editor's or the reviewer's lack of statistical knowledge?
How do you battle against bad statistics in science journal articles?

Thanks,
Drew

------------------------------
Andrew Ekstrom

Statistician, Chemist, HPC Abuser;-)
------------------------------

When I was a grad student in Env Science, we could take stats classes at part of the degree. I asked if I could take Design of Experiments from the Industrial Engineering department. The program head said, "If you don't know how to design an experiment by now, you shouldn't be in the program!"

------------------------------
Andrew Ekstrom

Statistician, Chemist, HPC Abuser;-)
------------------------------

Original Message:
Sent: 01-13-2021 09:29
From: Monnie McGee
Subject: What to do when journal reviewers don't believe some statistics methods exist

Dear Drew,

I'm sorry that you are having this problem. I know that it is frustrating. I think that replying to the reviewers and editors in the manner that Richard Amdur suggests is a good start, while being prepared to go to another journal if necessary.

As you stated in your query, as a graduate student you learned t-tests, multiple t-tests, and linear regression. I suspect that your reviewers learned the same thing. When all you have is a hammer, even problem starts to look like a nail.

Therefore, part of the problem is on us as statisticians to (1) educate reviewers ever so gently and (2) advocate for changes in the graduate curriculum for other disciplines. Both of these items will take a long time. In the meantime, you or someone in the ASA should write to the websites and text authors, explain to them that their information is incorrect, and ask them to update it.

Cheers and Good Luck,
Monnie


------------------------------
Monnie McGee
Associate Professor
Southern Methodist University
Dallas,TX United States

Original Message:
Sent: 01-11-2021 13:03
From: Andrew Ekstrom
Subject: What to do when journal reviewers don't believe some statistics methods exist

Hello,

I am writing a couple articles for a chemistry journal on the Design and Analysis of Experiments. As we all know, these methods test different combinations of factors and levels. Meaning, we change multiple things, simultaneously.

However, when you use your favorite web searching tool, and look for, "Designing a scientific experiment" or some other related searches, you'll find a lot of links that claim, "You CANNOT change more than one thing at a time during an experiment." Sometimes that is followed up with, "Statistics does NOT allow this." Others will say, "Because, you can't tell what made the change in the response."

We all know this is non-sense. But, it is ingrained in scientific lore. Especially at the academic level. Where most of the journal article reviewers reside.

I bring this up because I have written articles for other journals in the past where I used Plackett-Burman designs and Box-Behnken Designs. These articles were rejected by the reviewers for the reasons stated above. Sadly, in these articles, the first 8-15 references were stats textbooks. When I pointed this out to the reviewers. It angered them. Of the dozen or so reviewers I've dealt with, a pair of reviewers, one said something like, "This so called statistician should know that you can't change more than one thing at a time during the experiment." The other said something like, "Why does this author bore me with the basics of designed experiments. Everyone already knows this."

When I was an undergrad in Physics and Biochemistry and a graduate student in Chemistry, all the articles we read used t-tests or multiple t-tests to compare groups, "control" vs "experimental group N". We used simple linear regression to find a relation between X and Y. In some articles, there was X1 to Xn and Y. The authors chose to use multiple, Simple Linear Regressions. Meaning X1 vs Y, X2 vs Y, Xn vs Y, etc.

So, how do you handle a rejection based upon the editor's or the reviewer's lack of statistical knowledge?
How do you battle against bad statistics in science journal articles?

Thanks,
Drew

------------------------------
Andrew Ekstrom

Statistician, Chemist, HPC Abuser;-)
------------------------------

Dear Andrew, 

I had similar issues a few times. What I would suggest is to write in the paper a few sentences (perhaps backed up with a Section in the Appendix) on why the methods you used are correct. At this point, you have quite a lot of information that reviewers think in some (perhaps not totally right) way. Just anticipate them! As you were mentioning, a paper on the topic could also help.

Happy Friday,


------------------------------
Michele Santacatterina
George Washington University
------------------------------

Original Message:
Sent: 01-13-2021 16:36
From: Andrew Ekstrom
Subject: What to do when journal reviewers don't believe some statistics methods exist

When I was a grad student in Env Science, we could take stats classes at part of the degree. I asked if I could take Design of Experiments from the Industrial Engineering department. The program head said, "If you don't know how to design an experiment by now, you shouldn't be in the program!"

------------------------------
Andrew Ekstrom

Statistician, Chemist, HPC Abuser;-)

Original Message:
Sent: 01-13-2021 09:29
From: Monnie McGee
Subject: What to do when journal reviewers don't believe some statistics methods exist

Dear Drew,

I'm sorry that you are having this problem. I know that it is frustrating. I think that replying to the reviewers and editors in the manner that Richard Amdur suggests is a good start, while being prepared to go to another journal if necessary.

As you stated in your query, as a graduate student you learned t-tests, multiple t-tests, and linear regression. I suspect that your reviewers learned the same thing. When all you have is a hammer, even problem starts to look like a nail.

Therefore, part of the problem is on us as statisticians to (1) educate reviewers ever so gently and (2) advocate for changes in the graduate curriculum for other disciplines. Both of these items will take a long time. In the meantime, you or someone in the ASA should write to the websites and text authors, explain to them that their information is incorrect, and ask them to update it.

Cheers and Good Luck,
Monnie


------------------------------
Monnie McGee
Associate Professor
Southern Methodist University
Dallas,TX United States

Original Message:
Sent: 01-11-2021 13:03
From: Andrew Ekstrom
Subject: What to do when journal reviewers don't believe some statistics methods exist

Hello,

I am writing a couple articles for a chemistry journal on the Design and Analysis of Experiments. As we all know, these methods test different combinations of factors and levels. Meaning, we change multiple things, simultaneously.

However, when you use your favorite web searching tool, and look for, "Designing a scientific experiment" or some other related searches, you'll find a lot of links that claim, "You CANNOT change more than one thing at a time during an experiment." Sometimes that is followed up with, "Statistics does NOT allow this." Others will say, "Because, you can't tell what made the change in the response."

We all know this is non-sense. But, it is ingrained in scientific lore. Especially at the academic level. Where most of the journal article reviewers reside.

I bring this up because I have written articles for other journals in the past where I used Plackett-Burman designs and Box-Behnken Designs. These articles were rejected by the reviewers for the reasons stated above. Sadly, in these articles, the first 8-15 references were stats textbooks. When I pointed this out to the reviewers. It angered them. Of the dozen or so reviewers I've dealt with, a pair of reviewers, one said something like, "This so called statistician should know that you can't change more than one thing at a time during the experiment." The other said something like, "Why does this author bore me with the basics of designed experiments. Everyone already knows this."

When I was an undergrad in Physics and Biochemistry and a graduate student in Chemistry, all the articles we read used t-tests or multiple t-tests to compare groups, "control" vs "experimental group N". We used simple linear regression to find a relation between X and Y. In some articles, there was X1 to Xn and Y. The authors chose to use multiple, Simple Linear Regressions. Meaning X1 vs Y, X2 vs Y, Xn vs Y, etc.

So, how do you handle a rejection based upon the editor's or the reviewer's lack of statistical knowledge?
How do you battle against bad statistics in science journal articles?

Thanks,
Drew

------------------------------
Andrew Ekstrom

Statistician, Chemist, HPC Abuser;-)
------------------------------

My suggestion, presuming there are no trade secrets involved,  is to get your own peer review to submit with your paper (or to rebut the review.  Even sending it to this forum and asking pointed questions to us can help.


------------------------------
Jon Shuster
------------------------------

Original Message:
Sent: 01-11-2021 13:03
From: Andrew Ekstrom
Subject: What to do when journal reviewers don't believe some statistics methods exist

Hello,

I am writing a couple articles for a chemistry journal on the Design and Analysis of Experiments. As we all know, these methods test different combinations of factors and levels. Meaning, we change multiple things, simultaneously.

However, when you use your favorite web searching tool, and look for, "Designing a scientific experiment" or some other related searches, you'll find a lot of links that claim, "You CANNOT change more than one thing at a time during an experiment." Sometimes that is followed up with, "Statistics does NOT allow this." Others will say, "Because, you can't tell what made the change in the response."

We all know this is non-sense. But, it is ingrained in scientific lore. Especially at the academic level. Where most of the journal article reviewers reside.

I bring this up because I have written articles for other journals in the past where I used Plackett-Burman designs and Box-Behnken Designs. These articles were rejected by the reviewers for the reasons stated above. Sadly, in these articles, the first 8-15 references were stats textbooks. When I pointed this out to the reviewers. It angered them. Of the dozen or so reviewers I've dealt with, a pair of reviewers, one said something like, "This so called statistician should know that you can't change more than one thing at a time during the experiment." The other said something like, "Why does this author bore me with the basics of designed experiments. Everyone already knows this."

When I was an undergrad in Physics and Biochemistry and a graduate student in Chemistry, all the articles we read used t-tests or multiple t-tests to compare groups, "control" vs "experimental group N". We used simple linear regression to find a relation between X and Y. In some articles, there was X1 to Xn and Y. The authors chose to use multiple, Simple Linear Regressions. Meaning X1 vs Y, X2 vs Y, Xn vs Y, etc.

So, how do you handle a rejection based upon the editor's or the reviewer's lack of statistical knowledge?
How do you battle against bad statistics in science journal articles?

Thanks,
Drew

------------------------------
Andrew Ekstrom

Statistician, Chemist, HPC Abuser;-)
------------------------------