As a Biochemistry and Physics undergrad, most of what I learned about statistical analysis was either very simple or very wrong, compared to what I learned in my Linear Regression and Design of Experiments classes. So, what constitutes as a flawed statistical analysis?
For example, If someone uses a One-way ANOVA to test for differences between multiple groups. Then they use 2-sample pooled T-tests to see if there are any differences between the groups AND they don't use any corrections for multiple comparisons, is that a flawed statistical analysis? If so, every "Intro to Stats" textbook I have ever used, does this. Many Intro to Stats teachers and profs do this too.
During the Flint Water Crisis, the scientists that analyzed the data used something called the Q-test, a statistical procedure to determine if a sample result is "too much" of an outlier. It assumes data comes from a "normal distribution". As taught in Analytical Chemistry, these outliers can be discarded. In that case, a couple of the readings should be thrown out. The remaining data shows there isn't much to worry about.... accept THERE WAS!!! The data should have been looked at as say Log-Normal. But, Log-Normal isn't taught in the sciences, nor many intro to stats classes. Even when it is, it gets maybe a section in a chapter and is treated as a side note or after thought. When, in reality, it IS more important than the normal distribution for most real world data.
Having worked in chemistry labs for 10+ years, most of the data we generated would be considered "repeated measures" not a true replicate, as we (statisticians) like to define it. At best, most of the data is NOT independent. But, they will use 2-sample T-tests on it anyways. When we (statisticians) analyze their data, we rarely think of it as repeated measures or highly correlated.
Having worked in chemistry labs for 10+ years, the version of "quality control" we had in our labs was poor, at best. One QC manager, I call him a QC mangler, believed that if our CCV (Calibration Curve Verification) and IS (Internal Standard) were between the lines, everything was fine. Had another QC mangler, at a different facility, that felt the same way. Our QC criteria were, "The reported concentration of the IS is between 70% and 130%, the sample passes QC protocols." When the reported concentration of the IS goes from 72% to 128% over the course say 15 samples, in a very linear way, it "meets" the QC criteria. So, even if hte samples were independently made and run on the instrument, because of the systematic bias induced by the instruments, they are no longer "independent" samples.
On those same instruments, if I have a sample with a reported "toxic level" of some chemical and say the IS reports 102%, then I rerun that sample twice in a row the next day and the IS reports say 74% and 77% and the reported level does not meet that "toxic" threshold, it is common to report all 3 results. But, claim that we did 2 "confirmation analyses" that showed the level wasn't "toxic" after all. Just close.
As a graduate student in the sciences, we had to read articles from difference scientists on different topics. In one article, an author used multiple simple liner regressions on their data instead of one Multiple Linear Regression. So, instead of Y = F(X1, X2, X3...) they had Y = F(X1), Y = F(X2), Y = F(X3), etc.
The director of that program denied my thesis proposal, one that used factorial designs to optimize the extraction efficiency of a chemical extraction, because, "You took all these stats classes and you STILL DON'T KNOW that you simply CANNOT CHANGE MORE THAN ONE THINGS AT A TIME DURING AN EXPERIMENT!!!!" And yes, he did yell that.
In all of these cases, scientists didn't use proper statistical methodologies, simply because they were never taught how. Or worse, taught they couldn't do something, even though they can.
Should scientists be held liable, or should those that looked the other way, never bothered to teach the right thing, never bothered to correct the scientists and blindly accept their data as "good quality" without even bothering to see if that is true, be held liable?
Just so we are clear, I was fired from 2 of those chemistry jobs, kicked out of the graduate program, and forced to give a highly flawed departmental final exam and had several classes taken away from me at that same university. I've stood up for good statistical analyses and statistical procedures. It's brought nothing but unemployment, pain and misery.
------------------------------
Andrew Ekstrom
Statistician, Chemist, HPC Abuser;-)
------------------------------