ASA Connect

Hello Everyone,

A few months ago I gave some assignments to my Data Science students. I gave them a data set. I told them to partition the data into a testing and training set. Then run various analyses on the data. Over the course of the class, they made Logistic Regression, CART and Random Forest models, among others. The data itself had 20-30 variables to pick from and about 10,000 rows. These were In Class assignments and activities. I had students write the results of their analyses on the board so we could all see. 

The end result was, most students didn't have the same answers as far as what variables in the data were "important". I had each student use a different random seed. Unlike a lot of the standard data sets out there for use, I made up these data points. So, I know, for a fact, what variables were used to create the response variables. I know how much randomness I added to each data set. 

No one ever got all the right variables from their analyses. However, by using all of there results, we were able to find 60% to 80% of the variables I choose for the models. We were also able to eliminate 90%+ of the spurious correlations. We were even able to "roughly" categorize the variables into "Most important", " Very Important", "Most likely random" and "Not important" based upon how often certain variables came up in models and those groupings most correlated to the size of the coefficients I used to create the models. 

For example, If X1, X3, X4 come up 100% - 90% of the time, X9, X13, X17 come up 90% to 70% of the time, and variables X7, X11, X12 and X28 never come up, we can classify these variables as "Most important", "Very Important" and "Not Important".   

I've been looking through the research a bit on reproducibility in statistical analyses. There is a lot of talk about making sure software is able to accurately reproduce results. But, I am not aware of any articles that discuss or "prove" that The nature of data itself might not lend itself to good reproducibility... at least not under certain conditions. 

Is anyone aware of studies that show data analyses techniques are not that reproducible? 

Dear Andrew,

Your result is not surprising to me. The first time I came across such issues was more than 15 years ago. One paper from that time is from the Erlangen group https://pubmed.ncbi.nlm.nih.gov/17019510/

Frank Harrell has been claiming problems with variable selection constantly. The best reference to this probably is section 4.3 in his book Regression Modeling Strategies (2nd ed; DOI 10.1007/978-3-319-19425-7).

In our own work, we use the following approach if we use feature selection: we run a cross-validation -- typically something like a 10-fold CV --, do the model building repeatedly within each fold. The more often a feature is selected, the more stable the final model is. However, to repeat Frank Harrell's observations and concerns in my own words: you are generally unable to identify the true underlying model. However, you are generally able to come up with a reasonably good prediction model.

I would like to hear what is the opinion of others.

Andreas

Hello Everyone,

A few months ago I gave some assignments to my Data Science students. I gave them a data set. I told them to partition the data into a testing and training set. Then run various analyses on the data. Over the course of the class, they made Logistic Regression, CART and Random Forest models, among others. The data itself had 20-30 variables to pick from and about 10,000 rows. These were In Class assignments and activities. I had students write the results of their analyses on the board so we could all see. 

The end result was, most students didn't have the same answers as far as what variables in the data were "important". I had each student use a different random seed. Unlike a lot of the standard data sets out there for use, I made up these data points. So, I know, for a fact, what variables were used to create the response variables. I know how much randomness I added to each data set. 

No one ever got all the right variables from their analyses. However, by using all of there results, we were able to find 60% to 80% of the variables I choose for the models. We were also able to eliminate 90%+ of the spurious correlations. We were even able to "roughly" categorize the variables into "Most important", " Very Important", "Most likely random" and "Not important" based upon how often certain variables came up in models and those groupings most correlated to the size of the coefficients I used to create the models. 

For example, If X1, X3, X4 come up 100% - 90% of the time, X9, X13, X17 come up 90% to 70% of the time, and variables X7, X11, X12 and X28 never come up, we can classify these variables as "Most important", "Very Important" and "Not Important".   

I've been looking through the research a bit on reproducibility in statistical analyses. There is a lot of talk about making sure software is able to accurately reproduce results. But, I am not aware of any articles that discuss or "prove" that The nature of data itself might not lend itself to good reproducibility... at least not under certain conditions. 

Is anyone aware of studies that show data analyses techniques are not that reproducible? 

Hello Everyone,

A few months ago I gave some assignments to my Data Science students. I gave them a data set. I told them to partition the data into a testing and training set. Then run various analyses on the data. Over the course of the class, they made Logistic Regression, CART and Random Forest models, among others. The data itself had 20-30 variables to pick from and about 10,000 rows. These were In Class assignments and activities. I had students write the results of their analyses on the board so we could all see. 

The end result was, most students didn't have the same answers as far as what variables in the data were "important". I had each student use a different random seed. Unlike a lot of the standard data sets out there for use, I made up these data points. So, I know, for a fact, what variables were used to create the response variables. I know how much randomness I added to each data set. 

No one ever got all the right variables from their analyses. However, by using all of there results, we were able to find 60% to 80% of the variables I choose for the models. We were also able to eliminate 90%+ of the spurious correlations. We were even able to "roughly" categorize the variables into "Most important", " Very Important", "Most likely random" and "Not important" based upon how often certain variables came up in models and those groupings most correlated to the size of the coefficients I used to create the models. 

For example, If X1, X3, X4 come up 100% - 90% of the time, X9, X13, X17 come up 90% to 70% of the time, and variables X7, X11, X12 and X28 never come up, we can classify these variables as "Most important", "Very Important" and "Not Important".   

I've been looking through the research a bit on reproducibility in statistical analyses. There is a lot of talk about making sure software is able to accurately reproduce results. But, I am not aware of any articles that discuss or "prove" that The nature of data itself might not lend itself to good reproducibility... at least not under certain conditions. 

Is anyone aware of studies that show data analyses techniques are not that reproducible? 

Hi, Andrew:

I wonder if you can identify the reasons for the failure to obtain the correct model.  One possibility is high multicolinearity, so that slight differences in correlations change which variables enter.

10,000 observations, even split, seems like enough to provide accurate estimates.  Were any of the analyses simulations that may have had too few runs?

What else?

Could you identify a "better" method that would more often be successful if you did it yourself as an expert?

Ed

Hello Everyone,

A few months ago I gave some assignments to my Data Science students. I gave them a data set. I told them to partition the data into a testing and training set. Then run various analyses on the data. Over the course of the class, they made Logistic Regression, CART and Random Forest models, among others. The data itself had 20-30 variables to pick from and about 10,000 rows. These were In Class assignments and activities. I had students write the results of their analyses on the board so we could all see. 

The end result was, most students didn't have the same answers as far as what variables in the data were "important". I had each student use a different random seed. Unlike a lot of the standard data sets out there for use, I made up these data points. So, I know, for a fact, what variables were used to create the response variables. I know how much randomness I added to each data set. 

No one ever got all the right variables from their analyses. However, by using all of there results, we were able to find 60% to 80% of the variables I choose for the models. We were also able to eliminate 90%+ of the spurious correlations. We were even able to "roughly" categorize the variables into "Most important", " Very Important", "Most likely random" and "Not important" based upon how often certain variables came up in models and those groupings most correlated to the size of the coefficients I used to create the models. 

For example, If X1, X3, X4 come up 100% - 90% of the time, X9, X13, X17 come up 90% to 70% of the time, and variables X7, X11, X12 and X28 never come up, we can classify these variables as "Most important", "Very Important" and "Not Important".   

I've been looking through the research a bit on reproducibility in statistical analyses. There is a lot of talk about making sure software is able to accurately reproduce results. But, I am not aware of any articles that discuss or "prove" that The nature of data itself might not lend itself to good reproducibility... at least not under certain conditions. 

Is anyone aware of studies that show data analyses techniques are not that reproducible? 

Hello Everyone,

A few months ago I gave some assignments to my Data Science students. I gave them a data set. I told them to partition the data into a testing and training set. Then run various analyses on the data. Over the course of the class, they made Logistic Regression, CART and Random Forest models, among others. The data itself had 20-30 variables to pick from and about 10,000 rows. These were In Class assignments and activities. I had students write the results of their analyses on the board so we could all see. 

The end result was, most students didn't have the same answers as far as what variables in the data were "important". I had each student use a different random seed. Unlike a lot of the standard data sets out there for use, I made up these data points. So, I know, for a fact, what variables were used to create the response variables. I know how much randomness I added to each data set. 

No one ever got all the right variables from their analyses. However, by using all of there results, we were able to find 60% to 80% of the variables I choose for the models. We were also able to eliminate 90%+ of the spurious correlations. We were even able to "roughly" categorize the variables into "Most important", " Very Important", "Most likely random" and "Not important" based upon how often certain variables came up in models and those groupings most correlated to the size of the coefficients I used to create the models. 

For example, If X1, X3, X4 come up 100% - 90% of the time, X9, X13, X17 come up 90% to 70% of the time, and variables X7, X11, X12 and X28 never come up, we can classify these variables as "Most important", "Very Important" and "Not Important".   

I've been looking through the research a bit on reproducibility in statistical analyses. There is a lot of talk about making sure software is able to accurately reproduce results. But, I am not aware of any articles that discuss or "prove" that The nature of data itself might not lend itself to good reproducibility... at least not under certain conditions. 

Is anyone aware of studies that show data analyses techniques are not that reproducible? 

Similar to what you describe, Silberzahn et al. (2018) demonstrated substantial variability across different researchers' models for the same dataset. They asked several teams to independently analyze the same dataset (real data, not simulated, IIRC) and the same overall research question: "whether soccer referees are more likely to give red cards to dark-skin-toned players than to light-skin-toned players." Each team came up with a different final model, and their conclusions ranged from strongly positive, to no-evidence-of-an-effect, to moderately negative.

"Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results"
Silberzahn et al. (2018)
https://doi.org/10.1177/2515245917747646

There have been other similar studies. The most recent one I've seen, Breznau et al. (2022), cites other examples in the paragraph just before their Methods section. I personally agree strongly with their own conclusion that scientists "should exercise humility and strive to better account for the uncertainty in their work."

"Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty"
Breznau et al. (2022)
https://doi.org/10.1073/pnas.2203150119

Hello Everyone,

A few months ago I gave some assignments to my Data Science students. I gave them a data set. I told them to partition the data into a testing and training set. Then run various analyses on the data. Over the course of the class, they made Logistic Regression, CART and Random Forest models, among others. The data itself had 20-30 variables to pick from and about 10,000 rows. These were In Class assignments and activities. I had students write the results of their analyses on the board so we could all see. 

The end result was, most students didn't have the same answers as far as what variables in the data were "important". I had each student use a different random seed. Unlike a lot of the standard data sets out there for use, I made up these data points. So, I know, for a fact, what variables were used to create the response variables. I know how much randomness I added to each data set. 

No one ever got all the right variables from their analyses. However, by using all of there results, we were able to find 60% to 80% of the variables I choose for the models. We were also able to eliminate 90%+ of the spurious correlations. We were even able to "roughly" categorize the variables into "Most important", " Very Important", "Most likely random" and "Not important" based upon how often certain variables came up in models and those groupings most correlated to the size of the coefficients I used to create the models. 

For example, If X1, X3, X4 come up 100% - 90% of the time, X9, X13, X17 come up 90% to 70% of the time, and variables X7, X11, X12 and X28 never come up, we can classify these variables as "Most important", "Very Important" and "Not Important".   

I've been looking through the research a bit on reproducibility in statistical analyses. There is a lot of talk about making sure software is able to accurately reproduce results. But, I am not aware of any articles that discuss or "prove" that The nature of data itself might not lend itself to good reproducibility... at least not under certain conditions. 

Is anyone aware of studies that show data analyses techniques are not that reproducible? 

My other thought is that you say "I told them to partition the data into a testing and training set." Even with large sample sizes, there is a lot of variability across different random train/test splits of the same dataset. Further, if your goal is to choose the correct model (not just a model that makes good predictions) and you're working with parametric models, asymptotically your split ratio n_train/n_test must go to 0 to ensure model-selection consistency (Shao 1997 https://www.jstor.org/stable/24306073 ; Wieczorek and Lei 2021 https://doi.org/10.1002/cjs.11635 ), though not necessarily for selection of nonparametric models (Yang 2007 https://doi.org/10.1214/009053607000000514 ). In other words, if you ask teams to form random splits where n_train > n_test as usual, you should expect each team to choose a *different* model based on test-set performance -- even if all the teams considered the same set of models (which it seems they did not).

Similar to what you describe, Silberzahn et al. (2018) demonstrated substantial variability across different researchers' models for the same dataset. They asked several teams to independently analyze the same dataset (real data, not simulated, IIRC) and the same overall research question: "whether soccer referees are more likely to give red cards to dark-skin-toned players than to light-skin-toned players." Each team came up with a different final model, and their conclusions ranged from strongly positive, to no-evidence-of-an-effect, to moderately negative.

"Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results"
Silberzahn et al. (2018)
https://doi.org/10.1177/2515245917747646

There have been other similar studies. The most recent one I've seen, Breznau et al. (2022), cites other examples in the paragraph just before their Methods section. I personally agree strongly with their own conclusion that scientists "should exercise humility and strive to better account for the uncertainty in their work."

"Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty"
Breznau et al. (2022)
https://doi.org/10.1073/pnas.2203150119

Hello Everyone,

A few months ago I gave some assignments to my Data Science students. I gave them a data set. I told them to partition the data into a testing and training set. Then run various analyses on the data. Over the course of the class, they made Logistic Regression, CART and Random Forest models, among others. The data itself had 20-30 variables to pick from and about 10,000 rows. These were In Class assignments and activities. I had students write the results of their analyses on the board so we could all see. 

The end result was, most students didn't have the same answers as far as what variables in the data were "important". I had each student use a different random seed. Unlike a lot of the standard data sets out there for use, I made up these data points. So, I know, for a fact, what variables were used to create the response variables. I know how much randomness I added to each data set. 

No one ever got all the right variables from their analyses. However, by using all of there results, we were able to find 60% to 80% of the variables I choose for the models. We were also able to eliminate 90%+ of the spurious correlations. We were even able to "roughly" categorize the variables into "Most important", " Very Important", "Most likely random" and "Not important" based upon how often certain variables came up in models and those groupings most correlated to the size of the coefficients I used to create the models. 

For example, If X1, X3, X4 come up 100% - 90% of the time, X9, X13, X17 come up 90% to 70% of the time, and variables X7, X11, X12 and X28 never come up, we can classify these variables as "Most important", "Very Important" and "Not Important".   

I've been looking through the research a bit on reproducibility in statistical analyses. There is a lot of talk about making sure software is able to accurately reproduce results. But, I am not aware of any articles that discuss or "prove" that The nature of data itself might not lend itself to good reproducibility... at least not under certain conditions. 

Is anyone aware of studies that show data analyses techniques are not that reproducible? 

I think the problem is that, in fact, there is no correct model. Assuming the variables were intercorrelated, there will be lots of models with trivially different "explanatory" power. One of your "important" variables could be irrelevant in the presence of some other set of variables. For data sets of this type, you can seek a parsimonious set of predictors for some outcome variable but thinking there is a correct set assumes you can tease causality out of correlational data. I remember decades ago reading in a statistics book in psychology that asserted that analysis of covariance can be used to detect true mean differences in non-experimental data if you include all important competing explanatory variables as covariates. One of the few times I've felt like burning a book (and I was still a grad student at the time). 

My other thought is that you say "I told them to partition the data into a testing and training set." Even with large sample sizes, there is a lot of variability across different random train/test splits of the same dataset. Further, if your goal is to choose the correct model (not just a model that makes good predictions) and you're working with parametric models, asymptotically your split ratio n_train/n_test must go to 0 to ensure model-selection consistency (Shao 1997 https://www.jstor.org/stable/24306073 ; Wieczorek and Lei 2021 https://doi.org/10.1002/cjs.11635 ), though not necessarily for selection of nonparametric models (Yang 2007 https://doi.org/10.1214/009053607000000514 ). In other words, if you ask teams to form random splits where n_train > n_test as usual, you should expect each team to choose a *different* model based on test-set performance -- even if all the teams considered the same set of models (which it seems they did not).

Similar to what you describe, Silberzahn et al. (2018) demonstrated substantial variability across different researchers' models for the same dataset. They asked several teams to independently analyze the same dataset (real data, not simulated, IIRC) and the same overall research question: "whether soccer referees are more likely to give red cards to dark-skin-toned players than to light-skin-toned players." Each team came up with a different final model, and their conclusions ranged from strongly positive, to no-evidence-of-an-effect, to moderately negative.

"Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results"
Silberzahn et al. (2018)
https://doi.org/10.1177/2515245917747646

There have been other similar studies. The most recent one I've seen, Breznau et al. (2022), cites other examples in the paragraph just before their Methods section. I personally agree strongly with their own conclusion that scientists "should exercise humility and strive to better account for the uncertainty in their work."

"Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty"
Breznau et al. (2022)
https://doi.org/10.1073/pnas.2203150119

Hello Everyone,

A few months ago I gave some assignments to my Data Science students. I gave them a data set. I told them to partition the data into a testing and training set. Then run various analyses on the data. Over the course of the class, they made Logistic Regression, CART and Random Forest models, among others. The data itself had 20-30 variables to pick from and about 10,000 rows. These were In Class assignments and activities. I had students write the results of their analyses on the board so we could all see. 

The end result was, most students didn't have the same answers as far as what variables in the data were "important". I had each student use a different random seed. Unlike a lot of the standard data sets out there for use, I made up these data points. So, I know, for a fact, what variables were used to create the response variables. I know how much randomness I added to each data set. 

No one ever got all the right variables from their analyses. However, by using all of there results, we were able to find 60% to 80% of the variables I choose for the models. We were also able to eliminate 90%+ of the spurious correlations. We were even able to "roughly" categorize the variables into "Most important", " Very Important", "Most likely random" and "Not important" based upon how often certain variables came up in models and those groupings most correlated to the size of the coefficients I used to create the models. 

For example, If X1, X3, X4 come up 100% - 90% of the time, X9, X13, X17 come up 90% to 70% of the time, and variables X7, X11, X12 and X28 never come up, we can classify these variables as "Most important", "Very Important" and "Not Important".   

I've been looking through the research a bit on reproducibility in statistical analyses. There is a lot of talk about making sure software is able to accurately reproduce results. But, I am not aware of any articles that discuss or "prove" that The nature of data itself might not lend itself to good reproducibility... at least not under certain conditions. 

Is anyone aware of studies that show data analyses techniques are not that reproducible? 

I think the problem is that, in fact, there is no correct model. Assuming the variables were intercorrelated, there will be lots of models with trivially different "explanatory" power. One of your "important" variables could be irrelevant in the presence of some other set of variables. For data sets of this type, you can seek a parsimonious set of predictors for some outcome variable but thinking there is a correct set assumes you can tease causality out of correlational data. I remember decades ago reading in a statistics book in psychology that asserted that analysis of covariance can be used to detect true mean differences in non-experimental data if you include all important competing explanatory variables as covariates. One of the few times I've felt like burning a book (and I was still a grad student at the time). 

My other thought is that you say "I told them to partition the data into a testing and training set." Even with large sample sizes, there is a lot of variability across different random train/test splits of the same dataset. Further, if your goal is to choose the correct model (not just a model that makes good predictions) and you're working with parametric models, asymptotically your split ratio n_train/n_test must go to 0 to ensure model-selection consistency (Shao 1997 https://www.jstor.org/stable/24306073 ; Wieczorek and Lei 2021 https://doi.org/10.1002/cjs.11635 ), though not necessarily for selection of nonparametric models (Yang 2007 https://doi.org/10.1214/009053607000000514 ). In other words, if you ask teams to form random splits where n_train > n_test as usual, you should expect each team to choose a *different* model based on test-set performance -- even if all the teams considered the same set of models (which it seems they did not).

Similar to what you describe, Silberzahn et al. (2018) demonstrated substantial variability across different researchers' models for the same dataset. They asked several teams to independently analyze the same dataset (real data, not simulated, IIRC) and the same overall research question: "whether soccer referees are more likely to give red cards to dark-skin-toned players than to light-skin-toned players." Each team came up with a different final model, and their conclusions ranged from strongly positive, to no-evidence-of-an-effect, to moderately negative.

"Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results"
Silberzahn et al. (2018)
https://doi.org/10.1177/2515245917747646

There have been other similar studies. The most recent one I've seen, Breznau et al. (2022), cites other examples in the paragraph just before their Methods section. I personally agree strongly with their own conclusion that scientists "should exercise humility and strive to better account for the uncertainty in their work."

"Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty"
Breznau et al. (2022)
https://doi.org/10.1073/pnas.2203150119

Hello Everyone,

A few months ago I gave some assignments to my Data Science students. I gave them a data set. I told them to partition the data into a testing and training set. Then run various analyses on the data. Over the course of the class, they made Logistic Regression, CART and Random Forest models, among others. The data itself had 20-30 variables to pick from and about 10,000 rows. These were In Class assignments and activities. I had students write the results of their analyses on the board so we could all see. 

The end result was, most students didn't have the same answers as far as what variables in the data were "important". I had each student use a different random seed. Unlike a lot of the standard data sets out there for use, I made up these data points. So, I know, for a fact, what variables were used to create the response variables. I know how much randomness I added to each data set. 

No one ever got all the right variables from their analyses. However, by using all of there results, we were able to find 60% to 80% of the variables I choose for the models. We were also able to eliminate 90%+ of the spurious correlations. We were even able to "roughly" categorize the variables into "Most important", " Very Important", "Most likely random" and "Not important" based upon how often certain variables came up in models and those groupings most correlated to the size of the coefficients I used to create the models. 

For example, If X1, X3, X4 come up 100% - 90% of the time, X9, X13, X17 come up 90% to 70% of the time, and variables X7, X11, X12 and X28 never come up, we can classify these variables as "Most important", "Very Important" and "Not Important".   

I've been looking through the research a bit on reproducibility in statistical analyses. There is a lot of talk about making sure software is able to accurately reproduce results. But, I am not aware of any articles that discuss or "prove" that The nature of data itself might not lend itself to good reproducibility... at least not under certain conditions. 

Is anyone aware of studies that show data analyses techniques are not that reproducible? 

This situation is an excellent opportunity to introduce the students to George Box: "Remember that all models are wrong; the practical question is how wrong do they have to be to not be useful."

I think the problem is that, in fact, there is no correct model. Assuming the variables were intercorrelated, there will be lots of models with trivially different "explanatory" power. One of your "important" variables could be irrelevant in the presence of some other set of variables. For data sets of this type, you can seek a parsimonious set of predictors for some outcome variable but thinking there is a correct set assumes you can tease causality out of correlational data. I remember decades ago reading in a statistics book in psychology that asserted that analysis of covariance can be used to detect true mean differences in non-experimental data if you include all important competing explanatory variables as covariates. One of the few times I've felt like burning a book (and I was still a grad student at the time). 

My other thought is that you say "I told them to partition the data into a testing and training set." Even with large sample sizes, there is a lot of variability across different random train/test splits of the same dataset. Further, if your goal is to choose the correct model (not just a model that makes good predictions) and you're working with parametric models, asymptotically your split ratio n_train/n_test must go to 0 to ensure model-selection consistency (Shao 1997 https://www.jstor.org/stable/24306073 ; Wieczorek and Lei 2021 https://doi.org/10.1002/cjs.11635 ), though not necessarily for selection of nonparametric models (Yang 2007 https://doi.org/10.1214/009053607000000514 ). In other words, if you ask teams to form random splits where n_train > n_test as usual, you should expect each team to choose a *different* model based on test-set performance -- even if all the teams considered the same set of models (which it seems they did not).

Similar to what you describe, Silberzahn et al. (2018) demonstrated substantial variability across different researchers' models for the same dataset. They asked several teams to independently analyze the same dataset (real data, not simulated, IIRC) and the same overall research question: "whether soccer referees are more likely to give red cards to dark-skin-toned players than to light-skin-toned players." Each team came up with a different final model, and their conclusions ranged from strongly positive, to no-evidence-of-an-effect, to moderately negative.

"Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results"
Silberzahn et al. (2018)
https://doi.org/10.1177/2515245917747646

There have been other similar studies. The most recent one I've seen, Breznau et al. (2022), cites other examples in the paragraph just before their Methods section. I personally agree strongly with their own conclusion that scientists "should exercise humility and strive to better account for the uncertainty in their work."

"Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty"
Breznau et al. (2022)
https://doi.org/10.1073/pnas.2203150119

Hello Everyone,

A few months ago I gave some assignments to my Data Science students. I gave them a data set. I told them to partition the data into a testing and training set. Then run various analyses on the data. Over the course of the class, they made Logistic Regression, CART and Random Forest models, among others. The data itself had 20-30 variables to pick from and about 10,000 rows. These were In Class assignments and activities. I had students write the results of their analyses on the board so we could all see. 

The end result was, most students didn't have the same answers as far as what variables in the data were "important". I had each student use a different random seed. Unlike a lot of the standard data sets out there for use, I made up these data points. So, I know, for a fact, what variables were used to create the response variables. I know how much randomness I added to each data set. 

No one ever got all the right variables from their analyses. However, by using all of there results, we were able to find 60% to 80% of the variables I choose for the models. We were also able to eliminate 90%+ of the spurious correlations. We were even able to "roughly" categorize the variables into "Most important", " Very Important", "Most likely random" and "Not important" based upon how often certain variables came up in models and those groupings most correlated to the size of the coefficients I used to create the models. 

For example, If X1, X3, X4 come up 100% - 90% of the time, X9, X13, X17 come up 90% to 70% of the time, and variables X7, X11, X12 and X28 never come up, we can classify these variables as "Most important", "Very Important" and "Not Important".   

I've been looking through the research a bit on reproducibility in statistical analyses. There is a lot of talk about making sure software is able to accurately reproduce results. But, I am not aware of any articles that discuss or "prove" that The nature of data itself might not lend itself to good reproducibility... at least not under certain conditions. 

Is anyone aware of studies that show data analyses techniques are not that reproducible? 

Hello Everyone,

A few months ago I gave some assignments to my Data Science students. I gave them a data set. I told them to partition the data into a testing and training set. Then run various analyses on the data. Over the course of the class, they made Logistic Regression, CART and Random Forest models, among others. The data itself had 20-30 variables to pick from and about 10,000 rows. These were In Class assignments and activities. I had students write the results of their analyses on the board so we could all see. 

The end result was, most students didn't have the same answers as far as what variables in the data were "important". I had each student use a different random seed. Unlike a lot of the standard data sets out there for use, I made up these data points. So, I know, for a fact, what variables were used to create the response variables. I know how much randomness I added to each data set. 

No one ever got all the right variables from their analyses. However, by using all of there results, we were able to find 60% to 80% of the variables I choose for the models. We were also able to eliminate 90%+ of the spurious correlations. We were even able to "roughly" categorize the variables into "Most important", " Very Important", "Most likely random" and "Not important" based upon how often certain variables came up in models and those groupings most correlated to the size of the coefficients I used to create the models. 

For example, If X1, X3, X4 come up 100% - 90% of the time, X9, X13, X17 come up 90% to 70% of the time, and variables X7, X11, X12 and X28 never come up, we can classify these variables as "Most important", "Very Important" and "Not Important".   

I've been looking through the research a bit on reproducibility in statistical analyses. There is a lot of talk about making sure software is able to accurately reproduce results. But, I am not aware of any articles that discuss or "prove" that The nature of data itself might not lend itself to good reproducibility... at least not under certain conditions. 

Is anyone aware of studies that show data analyses techniques are not that reproducible? 

The closest article I can think of is "Many Analysts, One Data Set" from https://journals.sagepub.com/doi/10.1177/2515245917747646 but that's a little different - in that article, each team is given the whole data set - but it sounds like you gave each of your students a random sample, which would introduce some extra variability.

I'd also mention Gelman's forking paths paper which is a bit more philosophical.

By the way, you mentioned your aggregated results screened out 90% of spurious variables - if you were using a 5% sig level I would have hoped it was closer to 95% (but maybe it was rounded coarsely or my mental math is off).

Best,

Neal

Hello Everyone,

A few months ago I gave some assignments to my Data Science students. I gave them a data set. I told them to partition the data into a testing and training set. Then run various analyses on the data. Over the course of the class, they made Logistic Regression, CART and Random Forest models, among others. The data itself had 20-30 variables to pick from and about 10,000 rows. These were In Class assignments and activities. I had students write the results of their analyses on the board so we could all see. 

The end result was, most students didn't have the same answers as far as what variables in the data were "important". I had each student use a different random seed. Unlike a lot of the standard data sets out there for use, I made up these data points. So, I know, for a fact, what variables were used to create the response variables. I know how much randomness I added to each data set. 

No one ever got all the right variables from their analyses. However, by using all of there results, we were able to find 60% to 80% of the variables I choose for the models. We were also able to eliminate 90%+ of the spurious correlations. We were even able to "roughly" categorize the variables into "Most important", " Very Important", "Most likely random" and "Not important" based upon how often certain variables came up in models and those groupings most correlated to the size of the coefficients I used to create the models. 

For example, If X1, X3, X4 come up 100% - 90% of the time, X9, X13, X17 come up 90% to 70% of the time, and variables X7, X11, X12 and X28 never come up, we can classify these variables as "Most important", "Very Important" and "Not Important".   

I've been looking through the research a bit on reproducibility in statistical analyses. There is a lot of talk about making sure software is able to accurately reproduce results. But, I am not aware of any articles that discuss or "prove" that The nature of data itself might not lend itself to good reproducibility... at least not under certain conditions. 

Is anyone aware of studies that show data analyses techniques are not that reproducible? 

Hello Everyone,

A few months ago I gave some assignments to my Data Science students. I gave them a data set. I told them to partition the data into a testing and training set. Then run various analyses on the data. Over the course of the class, they made Logistic Regression, CART and Random Forest models, among others. The data itself had 20-30 variables to pick from and about 10,000 rows. These were In Class assignments and activities. I had students write the results of their analyses on the board so we could all see. 

The end result was, most students didn't have the same answers as far as what variables in the data were "important". I had each student use a different random seed. Unlike a lot of the standard data sets out there for use, I made up these data points. So, I know, for a fact, what variables were used to create the response variables. I know how much randomness I added to each data set. 

No one ever got all the right variables from their analyses. However, by using all of there results, we were able to find 60% to 80% of the variables I choose for the models. We were also able to eliminate 90%+ of the spurious correlations. We were even able to "roughly" categorize the variables into "Most important", " Very Important", "Most likely random" and "Not important" based upon how often certain variables came up in models and those groupings most correlated to the size of the coefficients I used to create the models. 

For example, If X1, X3, X4 come up 100% - 90% of the time, X9, X13, X17 come up 90% to 70% of the time, and variables X7, X11, X12 and X28 never come up, we can classify these variables as "Most important", "Very Important" and "Not Important".   

I've been looking through the research a bit on reproducibility in statistical analyses. There is a lot of talk about making sure software is able to accurately reproduce results. But, I am not aware of any articles that discuss or "prove" that The nature of data itself might not lend itself to good reproducibility... at least not under certain conditions. 

Is anyone aware of studies that show data analyses techniques are not that reproducible? 

To me, Ekstrom's post illustrates the need to add a new paradigm to our statistical repertoire.  For many of the data sets I have worked with, there is no true model.  For those data sets, finding the true model is not a sensible goal.  A more sensible goal is to find many disparate models that all do a good job of describing the data.  When we find many good models then we can answer questions like "Do all of these models have a substantially positive beta_1?" or "Is it possible to construct good models that don't use X_1?"

Presently most of our algorithms are geared toward producing one model.  It would be useful to create new algorithms that are geared toward finding many good models, and it would be even better to find good models that differ substantially from each other.  In my opinion, such model searches would be useful even in applied problems that have a true model.

I know of only a very few papers that make a start on constructing such useful algorithms.  They are by Rohan Joseph, Cynthia Rudin, myself, and a few others.

Hello Everyone,

A few months ago I gave some assignments to my Data Science students. I gave them a data set. I told them to partition the data into a testing and training set. Then run various analyses on the data. Over the course of the class, they made Logistic Regression, CART and Random Forest models, among others. The data itself had 20-30 variables to pick from and about 10,000 rows. These were In Class assignments and activities. I had students write the results of their analyses on the board so we could all see. 

The end result was, most students didn't have the same answers as far as what variables in the data were "important". I had each student use a different random seed. Unlike a lot of the standard data sets out there for use, I made up these data points. So, I know, for a fact, what variables were used to create the response variables. I know how much randomness I added to each data set. 

No one ever got all the right variables from their analyses. However, by using all of there results, we were able to find 60% to 80% of the variables I choose for the models. We were also able to eliminate 90%+ of the spurious correlations. We were even able to "roughly" categorize the variables into "Most important", " Very Important", "Most likely random" and "Not important" based upon how often certain variables came up in models and those groupings most correlated to the size of the coefficients I used to create the models. 

For example, If X1, X3, X4 come up 100% - 90% of the time, X9, X13, X17 come up 90% to 70% of the time, and variables X7, X11, X12 and X28 never come up, we can classify these variables as "Most important", "Very Important" and "Not Important".   

I've been looking through the research a bit on reproducibility in statistical analyses. There is a lot of talk about making sure software is able to accurately reproduce results. But, I am not aware of any articles that discuss or "prove" that The nature of data itself might not lend itself to good reproducibility... at least not under certain conditions. 

Is anyone aware of studies that show data analyses techniques are not that reproducible? 

Hello Everyone,

A few months ago I gave some assignments to my Data Science students. I gave them a data set. I told them to partition the data into a testing and training set. Then run various analyses on the data. Over the course of the class, they made Logistic Regression, CART and Random Forest models, among others. The data itself had 20-30 variables to pick from and about 10,000 rows. These were In Class assignments and activities. I had students write the results of their analyses on the board so we could all see. 

The end result was, most students didn't have the same answers as far as what variables in the data were "important". I had each student use a different random seed. Unlike a lot of the standard data sets out there for use, I made up these data points. So, I know, for a fact, what variables were used to create the response variables. I know how much randomness I added to each data set. 

No one ever got all the right variables from their analyses. However, by using all of there results, we were able to find 60% to 80% of the variables I choose for the models. We were also able to eliminate 90%+ of the spurious correlations. We were even able to "roughly" categorize the variables into "Most important", " Very Important", "Most likely random" and "Not important" based upon how often certain variables came up in models and those groupings most correlated to the size of the coefficients I used to create the models. 

For example, If X1, X3, X4 come up 100% - 90% of the time, X9, X13, X17 come up 90% to 70% of the time, and variables X7, X11, X12 and X28 never come up, we can classify these variables as "Most important", "Very Important" and "Not Important".   

I've been looking through the research a bit on reproducibility in statistical analyses. There is a lot of talk about making sure software is able to accurately reproduce results. But, I am not aware of any articles that discuss or "prove" that The nature of data itself might not lend itself to good reproducibility... at least not under certain conditions. 

Is anyone aware of studies that show data analyses techniques are not that reproducible? 

Not published anywhere, but I once read a paper that used Lasso for confounder selection. The results were nonsensical (e.g., lagged variables from N-2, N-3,N-8, were "important") and obviously methodologically unsound if you are trying to find an unbiased estimator for whatever effect they were interested. I was going to write a response with a simulation study, but alas never got to it. Here are some simulation results (lasso selection on github). I looked at the Lasso for a set number of N, varied number of predictors, true value of one parameter, the overall measurement error, and correlation. The results are as expected...low signal to noise, less likely to recover the true parameters from the data generating process. Code is four years old so take with a grain of salt...

Hello Everyone,

A few months ago I gave some assignments to my Data Science students. I gave them a data set. I told them to partition the data into a testing and training set. Then run various analyses on the data. Over the course of the class, they made Logistic Regression, CART and Random Forest models, among others. The data itself had 20-30 variables to pick from and about 10,000 rows. These were In Class assignments and activities. I had students write the results of their analyses on the board so we could all see. 

The end result was, most students didn't have the same answers as far as what variables in the data were "important". I had each student use a different random seed. Unlike a lot of the standard data sets out there for use, I made up these data points. So, I know, for a fact, what variables were used to create the response variables. I know how much randomness I added to each data set. 

No one ever got all the right variables from their analyses. However, by using all of there results, we were able to find 60% to 80% of the variables I choose for the models. We were also able to eliminate 90%+ of the spurious correlations. We were even able to "roughly" categorize the variables into "Most important", " Very Important", "Most likely random" and "Not important" based upon how often certain variables came up in models and those groupings most correlated to the size of the coefficients I used to create the models. 

For example, If X1, X3, X4 come up 100% - 90% of the time, X9, X13, X17 come up 90% to 70% of the time, and variables X7, X11, X12 and X28 never come up, we can classify these variables as "Most important", "Very Important" and "Not Important".   

I've been looking through the research a bit on reproducibility in statistical analyses. There is a lot of talk about making sure software is able to accurately reproduce results. But, I am not aware of any articles that discuss or "prove" that The nature of data itself might not lend itself to good reproducibility... at least not under certain conditions. 

Is anyone aware of studies that show data analyses techniques are not that reproducible? 

Hello Everyone,

A few months ago I gave some assignments to my Data Science students. I gave them a data set. I told them to partition the data into a testing and training set. Then run various analyses on the data. Over the course of the class, they made Logistic Regression, CART and Random Forest models, among others. The data itself had 20-30 variables to pick from and about 10,000 rows. These were In Class assignments and activities. I had students write the results of their analyses on the board so we could all see. 

The end result was, most students didn't have the same answers as far as what variables in the data were "important". I had each student use a different random seed. Unlike a lot of the standard data sets out there for use, I made up these data points. So, I know, for a fact, what variables were used to create the response variables. I know how much randomness I added to each data set. 

No one ever got all the right variables from their analyses. However, by using all of there results, we were able to find 60% to 80% of the variables I choose for the models. We were also able to eliminate 90%+ of the spurious correlations. We were even able to "roughly" categorize the variables into "Most important", " Very Important", "Most likely random" and "Not important" based upon how often certain variables came up in models and those groupings most correlated to the size of the coefficients I used to create the models. 

For example, If X1, X3, X4 come up 100% - 90% of the time, X9, X13, X17 come up 90% to 70% of the time, and variables X7, X11, X12 and X28 never come up, we can classify these variables as "Most important", "Very Important" and "Not Important".   

I've been looking through the research a bit on reproducibility in statistical analyses. There is a lot of talk about making sure software is able to accurately reproduce results. But, I am not aware of any articles that discuss or "prove" that The nature of data itself might not lend itself to good reproducibility... at least not under certain conditions. 

Is anyone aware of studies that show data analyses techniques are not that reproducible? 

Thank you all for your comments. Please keep them coming. I am thankful for all of you who have and will participate. 

Hello Everyone,

A few months ago I gave some assignments to my Data Science students. I gave them a data set. I told them to partition the data into a testing and training set. Then run various analyses on the data. Over the course of the class, they made Logistic Regression, CART and Random Forest models, among others. The data itself had 20-30 variables to pick from and about 10,000 rows. These were In Class assignments and activities. I had students write the results of their analyses on the board so we could all see. 

The end result was, most students didn't have the same answers as far as what variables in the data were "important". I had each student use a different random seed. Unlike a lot of the standard data sets out there for use, I made up these data points. So, I know, for a fact, what variables were used to create the response variables. I know how much randomness I added to each data set. 

No one ever got all the right variables from their analyses. However, by using all of there results, we were able to find 60% to 80% of the variables I choose for the models. We were also able to eliminate 90%+ of the spurious correlations. We were even able to "roughly" categorize the variables into "Most important", " Very Important", "Most likely random" and "Not important" based upon how often certain variables came up in models and those groupings most correlated to the size of the coefficients I used to create the models. 

For example, If X1, X3, X4 come up 100% - 90% of the time, X9, X13, X17 come up 90% to 70% of the time, and variables X7, X11, X12 and X28 never come up, we can classify these variables as "Most important", "Very Important" and "Not Important".   

I've been looking through the research a bit on reproducibility in statistical analyses. There is a lot of talk about making sure software is able to accurately reproduce results. But, I am not aware of any articles that discuss or "prove" that The nature of data itself might not lend itself to good reproducibility... at least not under certain conditions. 

Is anyone aware of studies that show data analyses techniques are not that reproducible? 

Hi Andrew,

I do a series of similar exercises with my Design of Experiments students but my exercises are much more basic than yours. The data for these exercises are generated with macros so the underlying model is the same but there is superimposed random noise. When we use standard experiment designs everyone gets similar answers. Later in class, to emphasize the value of using a designed experiment, I ask them to analyze data sets with more complicated structure and of course their results diverge. We talk about some of the special methods available to analyze such data but those are beyond the scope of the class. If I am successful, the students learn to prefer designed experiments and distrust models built from happenstance data. I am lucky that most of my students come from the engineering and science community where they have some control over how their data are collected and they can choose to used designed experiments.

Long ago Angela Dean from OSU or Susan Collins from Lubrizol (I think they were working together) told us about a data set from a supersaturated experiment design that was published with a request for people to analyze the data and submit their results. A prize was offered for the best analysis. The purpose of the exercise was to assess analysis reproducibility. My recollection was that a small subset of the analyses submitted were on target but most missed by a wide margin - similar results to yours. 

Kaggle's competitions provide a similar opportunity to assess analysis reproducibility. I haven't invested any time there so I don't know what kinds of problems have analyses that are shared. Maybe someone has already written something up? If would be fun/malicious to assign Kaggle projects with shared analyses for students to do their own analysis reproducibility evaluation.

I'm going to share this thread with my students in the future when we reach this topic.

Paul Mathews
Mathews Malnar and Baily, Inc.

Thank you all for your comments. Please keep them coming. I am thankful for all of you who have and will participate. 

Hello Everyone,

A few months ago I gave some assignments to my Data Science students. I gave them a data set. I told them to partition the data into a testing and training set. Then run various analyses on the data. Over the course of the class, they made Logistic Regression, CART and Random Forest models, among others. The data itself had 20-30 variables to pick from and about 10,000 rows. These were In Class assignments and activities. I had students write the results of their analyses on the board so we could all see. 

The end result was, most students didn't have the same answers as far as what variables in the data were "important". I had each student use a different random seed. Unlike a lot of the standard data sets out there for use, I made up these data points. So, I know, for a fact, what variables were used to create the response variables. I know how much randomness I added to each data set. 

No one ever got all the right variables from their analyses. However, by using all of there results, we were able to find 60% to 80% of the variables I choose for the models. We were also able to eliminate 90%+ of the spurious correlations. We were even able to "roughly" categorize the variables into "Most important", " Very Important", "Most likely random" and "Not important" based upon how often certain variables came up in models and those groupings most correlated to the size of the coefficients I used to create the models. 

For example, If X1, X3, X4 come up 100% - 90% of the time, X9, X13, X17 come up 90% to 70% of the time, and variables X7, X11, X12 and X28 never come up, we can classify these variables as "Most important", "Very Important" and "Not Important".   

I've been looking through the research a bit on reproducibility in statistical analyses. There is a lot of talk about making sure software is able to accurately reproduce results. But, I am not aware of any articles that discuss or "prove" that The nature of data itself might not lend itself to good reproducibility... at least not under certain conditions. 

Is anyone aware of studies that show data analyses techniques are not that reproducible? 

Hi Andrew,

I do a series of similar exercises with my Design of Experiments students but my exercises are much more basic than yours. The data for these exercises are generated with macros so the underlying model is the same but there is superimposed random noise. When we use standard experiment designs everyone gets similar answers. Later in class, to emphasize the value of using a designed experiment, I ask them to analyze data sets with more complicated structure and of course their results diverge. We talk about some of the special methods available to analyze such data but those are beyond the scope of the class. If I am successful, the students learn to prefer designed experiments and distrust models built from happenstance data. I am lucky that most of my students come from the engineering and science community where they have some control over how their data are collected and they can choose to used designed experiments.

Long ago Angela Dean from OSU or Susan Collins from Lubrizol (I think they were working together) told us about a data set from a supersaturated experiment design that was published with a request for people to analyze the data and submit their results. A prize was offered for the best analysis. The purpose of the exercise was to assess analysis reproducibility. My recollection was that a small subset of the analyses submitted were on target but most missed by a wide margin - similar results to yours. 

Kaggle's competitions provide a similar opportunity to assess analysis reproducibility. I haven't invested any time there so I don't know what kinds of problems have analyses that are shared. Maybe someone has already written something up? If would be fun/malicious to assign Kaggle projects with shared analyses for students to do their own analysis reproducibility evaluation.

I'm going to share this thread with my students in the future when we reach this topic.

Paul Mathews
Mathews Malnar and Baily, Inc.

Thank you all for your comments. Please keep them coming. I am thankful for all of you who have and will participate. 

Hello Everyone,

A few months ago I gave some assignments to my Data Science students. I gave them a data set. I told them to partition the data into a testing and training set. Then run various analyses on the data. Over the course of the class, they made Logistic Regression, CART and Random Forest models, among others. The data itself had 20-30 variables to pick from and about 10,000 rows. These were In Class assignments and activities. I had students write the results of their analyses on the board so we could all see. 

The end result was, most students didn't have the same answers as far as what variables in the data were "important". I had each student use a different random seed. Unlike a lot of the standard data sets out there for use, I made up these data points. So, I know, for a fact, what variables were used to create the response variables. I know how much randomness I added to each data set. 

No one ever got all the right variables from their analyses. However, by using all of there results, we were able to find 60% to 80% of the variables I choose for the models. We were also able to eliminate 90%+ of the spurious correlations. We were even able to "roughly" categorize the variables into "Most important", " Very Important", "Most likely random" and "Not important" based upon how often certain variables came up in models and those groupings most correlated to the size of the coefficients I used to create the models. 

For example, If X1, X3, X4 come up 100% - 90% of the time, X9, X13, X17 come up 90% to 70% of the time, and variables X7, X11, X12 and X28 never come up, we can classify these variables as "Most important", "Very Important" and "Not Important".   

I've been looking through the research a bit on reproducibility in statistical analyses. There is a lot of talk about making sure software is able to accurately reproduce results. But, I am not aware of any articles that discuss or "prove" that The nature of data itself might not lend itself to good reproducibility... at least not under certain conditions. 

Is anyone aware of studies that show data analyses techniques are not that reproducible? 

I am hosting a a symposium for the American Chemical Society this summer. I've been looking for a topic to discuss during my "Data Science in Chemical Research"  talk. I haven't been able to come up with anything new this year.... Until about 20 mins ago. 

Do you mind if I borrow your idea about the analysis of "designed experiments" vs "observational" data analysis? 

I think a title like, "Is scientific reproducibility a fallacy, a misunderstanding or sign of poorly designed experiments?"   who knows, I might actually get more than 5 people watching.... 

I teach my students about the reproducibility of results vs conclusions. Adding in my data analysis as well as your idea for generating data with a macro, I should be able to show the power of designed experiments.... which brings up the another challenge. How do you "prove" to scientists, that have been doing research for decades, that their OFAT methods are bad AND that you can and should change more than one thing at a time?

Hi Andrew,

I do a series of similar exercises with my Design of Experiments students but my exercises are much more basic than yours. The data for these exercises are generated with macros so the underlying model is the same but there is superimposed random noise. When we use standard experiment designs everyone gets similar answers. Later in class, to emphasize the value of using a designed experiment, I ask them to analyze data sets with more complicated structure and of course their results diverge. We talk about some of the special methods available to analyze such data but those are beyond the scope of the class. If I am successful, the students learn to prefer designed experiments and distrust models built from happenstance data. I am lucky that most of my students come from the engineering and science community where they have some control over how their data are collected and they can choose to used designed experiments.

Long ago Angela Dean from OSU or Susan Collins from Lubrizol (I think they were working together) told us about a data set from a supersaturated experiment design that was published with a request for people to analyze the data and submit their results. A prize was offered for the best analysis. The purpose of the exercise was to assess analysis reproducibility. My recollection was that a small subset of the analyses submitted were on target but most missed by a wide margin - similar results to yours. 

Kaggle's competitions provide a similar opportunity to assess analysis reproducibility. I haven't invested any time there so I don't know what kinds of problems have analyses that are shared. Maybe someone has already written something up? If would be fun/malicious to assign Kaggle projects with shared analyses for students to do their own analysis reproducibility evaluation.

I'm going to share this thread with my students in the future when we reach this topic.

Paul Mathews
Mathews Malnar and Baily, Inc.

Thank you all for your comments. Please keep them coming. I am thankful for all of you who have and will participate. 

Hello Everyone,

A few months ago I gave some assignments to my Data Science students. I gave them a data set. I told them to partition the data into a testing and training set. Then run various analyses on the data. Over the course of the class, they made Logistic Regression, CART and Random Forest models, among others. The data itself had 20-30 variables to pick from and about 10,000 rows. These were In Class assignments and activities. I had students write the results of their analyses on the board so we could all see. 

The end result was, most students didn't have the same answers as far as what variables in the data were "important". I had each student use a different random seed. Unlike a lot of the standard data sets out there for use, I made up these data points. So, I know, for a fact, what variables were used to create the response variables. I know how much randomness I added to each data set. 

No one ever got all the right variables from their analyses. However, by using all of there results, we were able to find 60% to 80% of the variables I choose for the models. We were also able to eliminate 90%+ of the spurious correlations. We were even able to "roughly" categorize the variables into "Most important", " Very Important", "Most likely random" and "Not important" based upon how often certain variables came up in models and those groupings most correlated to the size of the coefficients I used to create the models. 

For example, If X1, X3, X4 come up 100% - 90% of the time, X9, X13, X17 come up 90% to 70% of the time, and variables X7, X11, X12 and X28 never come up, we can classify these variables as "Most important", "Very Important" and "Not Important".   

I've been looking through the research a bit on reproducibility in statistical analyses. There is a lot of talk about making sure software is able to accurately reproduce results. But, I am not aware of any articles that discuss or "prove" that The nature of data itself might not lend itself to good reproducibility... at least not under certain conditions. 

Is anyone aware of studies that show data analyses techniques are not that reproducible? 

Andrew, 

In response to your question:  How do you "prove" to scientists, that have been doing research for decades, that their OFAT methods are bad AND that you can and should change more than one thing at a time?

My experience has been that when they see/experience an important interaction of multiple factors that gives insight (or even solves) a problem they are interested in, they become open to learn about mult-factor designs.  I have found this situation much worse with health scientists (clinicians and researchers) than in other industries.  I put together a research team during Covid to try and understand why and found some interesting history that made this happen. 

We wrote about this in attached paper: It Is Time to Reconsider Factorial Designs: How Bradford Hill and R. A. Fisher Shaped the Standard of Clinical Evidence, Quality Management in Healthcare in Vol. 29, No. 2, April/June 2020.

I am hosting a a symposium for the American Chemical Society this summer. I've been looking for a topic to discuss during my "Data Science in Chemical Research"  talk. I haven't been able to come up with anything new this year.... Until about 20 mins ago. 

Do you mind if I borrow your idea about the analysis of "designed experiments" vs "observational" data analysis? 

I think a title like, "Is scientific reproducibility a fallacy, a misunderstanding or sign of poorly designed experiments?"   who knows, I might actually get more than 5 people watching.... 

I teach my students about the reproducibility of results vs conclusions. Adding in my data analysis as well as your idea for generating data with a macro, I should be able to show the power of designed experiments.... which brings up the another challenge. How do you "prove" to scientists, that have been doing research for decades, that their OFAT methods are bad AND that you can and should change more than one thing at a time?

Hi Andrew,

I do a series of similar exercises with my Design of Experiments students but my exercises are much more basic than yours. The data for these exercises are generated with macros so the underlying model is the same but there is superimposed random noise. When we use standard experiment designs everyone gets similar answers. Later in class, to emphasize the value of using a designed experiment, I ask them to analyze data sets with more complicated structure and of course their results diverge. We talk about some of the special methods available to analyze such data but those are beyond the scope of the class. If I am successful, the students learn to prefer designed experiments and distrust models built from happenstance data. I am lucky that most of my students come from the engineering and science community where they have some control over how their data are collected and they can choose to used designed experiments.

Long ago Angela Dean from OSU or Susan Collins from Lubrizol (I think they were working together) told us about a data set from a supersaturated experiment design that was published with a request for people to analyze the data and submit their results. A prize was offered for the best analysis. The purpose of the exercise was to assess analysis reproducibility. My recollection was that a small subset of the analyses submitted were on target but most missed by a wide margin - similar results to yours. 

Kaggle's competitions provide a similar opportunity to assess analysis reproducibility. I haven't invested any time there so I don't know what kinds of problems have analyses that are shared. Maybe someone has already written something up? If would be fun/malicious to assign Kaggle projects with shared analyses for students to do their own analysis reproducibility evaluation.

I'm going to share this thread with my students in the future when we reach this topic.

Paul Mathews
Mathews Malnar and Baily, Inc.

Thank you all for your comments. Please keep them coming. I am thankful for all of you who have and will participate. 

Hello Everyone,

A few months ago I gave some assignments to my Data Science students. I gave them a data set. I told them to partition the data into a testing and training set. Then run various analyses on the data. Over the course of the class, they made Logistic Regression, CART and Random Forest models, among others. The data itself had 20-30 variables to pick from and about 10,000 rows. These were In Class assignments and activities. I had students write the results of their analyses on the board so we could all see. 

The end result was, most students didn't have the same answers as far as what variables in the data were "important". I had each student use a different random seed. Unlike a lot of the standard data sets out there for use, I made up these data points. So, I know, for a fact, what variables were used to create the response variables. I know how much randomness I added to each data set. 

No one ever got all the right variables from their analyses. However, by using all of there results, we were able to find 60% to 80% of the variables I choose for the models. We were also able to eliminate 90%+ of the spurious correlations. We were even able to "roughly" categorize the variables into "Most important", " Very Important", "Most likely random" and "Not important" based upon how often certain variables came up in models and those groupings most correlated to the size of the coefficients I used to create the models. 

For example, If X1, X3, X4 come up 100% - 90% of the time, X9, X13, X17 come up 90% to 70% of the time, and variables X7, X11, X12 and X28 never come up, we can classify these variables as "Most important", "Very Important" and "Not Important".   

I've been looking through the research a bit on reproducibility in statistical analyses. There is a lot of talk about making sure software is able to accurately reproduce results. But, I am not aware of any articles that discuss or "prove" that The nature of data itself might not lend itself to good reproducibility... at least not under certain conditions. 

Is anyone aware of studies that show data analyses techniques are not that reproducible? 

I am hosting a a symposium for the American Chemical Society this summer. I've been looking for a topic to discuss during my "Data Science in Chemical Research"  talk. I haven't been able to come up with anything new this year.... Until about 20 mins ago. 

Do you mind if I borrow your idea about the analysis of "designed experiments" vs "observational" data analysis? 

I think a title like, "Is scientific reproducibility a fallacy, a misunderstanding or sign of poorly designed experiments?"   who knows, I might actually get more than 5 people watching.... 

I teach my students about the reproducibility of results vs conclusions. Adding in my data analysis as well as your idea for generating data with a macro, I should be able to show the power of designed experiments.... which brings up the another challenge. How do you "prove" to scientists, that have been doing research for decades, that their OFAT methods are bad AND that you can and should change more than one thing at a time?

Hi Andrew,

I do a series of similar exercises with my Design of Experiments students but my exercises are much more basic than yours. The data for these exercises are generated with macros so the underlying model is the same but there is superimposed random noise. When we use standard experiment designs everyone gets similar answers. Later in class, to emphasize the value of using a designed experiment, I ask them to analyze data sets with more complicated structure and of course their results diverge. We talk about some of the special methods available to analyze such data but those are beyond the scope of the class. If I am successful, the students learn to prefer designed experiments and distrust models built from happenstance data. I am lucky that most of my students come from the engineering and science community where they have some control over how their data are collected and they can choose to used designed experiments.

Long ago Angela Dean from OSU or Susan Collins from Lubrizol (I think they were working together) told us about a data set from a supersaturated experiment design that was published with a request for people to analyze the data and submit their results. A prize was offered for the best analysis. The purpose of the exercise was to assess analysis reproducibility. My recollection was that a small subset of the analyses submitted were on target but most missed by a wide margin - similar results to yours. 

Kaggle's competitions provide a similar opportunity to assess analysis reproducibility. I haven't invested any time there so I don't know what kinds of problems have analyses that are shared. Maybe someone has already written something up? If would be fun/malicious to assign Kaggle projects with shared analyses for students to do their own analysis reproducibility evaluation.

I'm going to share this thread with my students in the future when we reach this topic.

Paul Mathews
Mathews Malnar and Baily, Inc.

Thank you all for your comments. Please keep them coming. I am thankful for all of you who have and will participate. 

Hello Everyone,

A few months ago I gave some assignments to my Data Science students. I gave them a data set. I told them to partition the data into a testing and training set. Then run various analyses on the data. Over the course of the class, they made Logistic Regression, CART and Random Forest models, among others. The data itself had 20-30 variables to pick from and about 10,000 rows. These were In Class assignments and activities. I had students write the results of their analyses on the board so we could all see. 

The end result was, most students didn't have the same answers as far as what variables in the data were "important". I had each student use a different random seed. Unlike a lot of the standard data sets out there for use, I made up these data points. So, I know, for a fact, what variables were used to create the response variables. I know how much randomness I added to each data set. 

No one ever got all the right variables from their analyses. However, by using all of there results, we were able to find 60% to 80% of the variables I choose for the models. We were also able to eliminate 90%+ of the spurious correlations. We were even able to "roughly" categorize the variables into "Most important", " Very Important", "Most likely random" and "Not important" based upon how often certain variables came up in models and those groupings most correlated to the size of the coefficients I used to create the models. 

For example, If X1, X3, X4 come up 100% - 90% of the time, X9, X13, X17 come up 90% to 70% of the time, and variables X7, X11, X12 and X28 never come up, we can classify these variables as "Most important", "Very Important" and "Not Important".   

I've been looking through the research a bit on reproducibility in statistical analyses. There is a lot of talk about making sure software is able to accurately reproduce results. But, I am not aware of any articles that discuss or "prove" that The nature of data itself might not lend itself to good reproducibility... at least not under certain conditions. 

Is anyone aware of studies that show data analyses techniques are not that reproducible?