Discussion: View Thread

Hi

Would anyone be willing to share a demonstration of how multiple looks at data inflates the type 1 error rate or alpha level?  Many people don't get this concept of how looking early inflates your error. 

I recall a classmate from graduate school who TA'd an undergraduate level course had an excel spreadsheet that simulated data (I think it was getting a head or tail from a coin toss many times) and tested the hypothesis of P(getting a head)=0.50.  I think she had different testing scenarios to show how the type I error rate differed under mutplie testing vs.single testing conditions.

Thanks!!
-------------------------------------------
Shelley-Ann Walters
3M
-------------------------------------------

-------------------------------------------
Joe Massaro
-------------------------------------------

I coincidentally received an e-mail below from a colleague this morning.  I have not read the article yet, but it looks like it might help you address the issue:

"In case you missed it, the Neuroscience Ig Nobel prize was given for  "Neural correlates of interspecies perspective taking in the post-mortem Atlantic Salmon: An argument for multiple comparisons correction," Craig M. Bennett, Abigail A. Baird, Michael B. Miller, and George L. Wolford, 2009 (http://www.jsur.org/ar/jsur_ben102010.pdf)

It's basically an example of a massive multiple-testing problem that found significant fMRI responses to stimuli in a dead salmon.  Very readable and a great example."

Thanks.

I have several papers listed at
--> http://www.pmean.com/category/MultipleComparisons.html
though I must confess that it is a rather haphazard list. But there is one gem worth mentioning on that page:

Jerry Dallal. There must be something buried in here somewhere!. Description: This webpage uses a simulation to illustrate what happens with one hundred simultaneous independent tests of significance. [Accessed February 23, 2011]. Available at: http://www.jerrydallal.com/LHSP/multtest.htm.

-------------------------------------------
Stephen Simon
Independent Statistical Consultant
P. Mean Consulting
-------------------------------------------

While these examples are good for multiple comparisons (I would refer
interested parties to the xkcd jelly bean strip, which has been mentioned
of late here as well), it seems to me that Dr. Walters is looking for a 'peeking'
example rather than instances of multiple comparisons; this would be
more in line with interim analyses in a RCT.

Attached is a simulated dataset along the lines of what your classmate had.
A coin is flipped 50 times and we want to know if it is fair.
The first column is the flip number, the second column is the result,
the third column is the sum of heads to that point.

Instead of waiting for all 50 flips to come in, our impatient investigator
checks after 10, 20, 30 and 40 flips as well. The fourth and fifth columns
have high and low critical values for two-sided alpha=0.05 based on binomial
probabilities under the fair coin assumption; actually alpha is smaller than 0.05
in most of the checks but I didn't want to put random decision processes into
this simple example.

In this data set, the fourth interim check would have (incorrectly) rejected the fair coin null.

-------------------------------------------
John Dawson
Postdoctoral Scholar
University of Alabama at Birmingham
-------------------------------------------







Original Message:
Sent: 10-11-2012 16:18
From: Stephen Simon
Subject: Demo of how muliple analytical looks at data inflate the type I error/alpha?

I have several papers listed at
--> http://www.pmean.com/category/MultipleComparisons.html
though I must confess that it is a rather haphazard list. But there is one gem worth mentioning on that page:

Jerry Dallal. There must be something buried in here somewhere!. Description: This webpage uses a simulation to illustrate what happens with one hundred simultaneous independent tests of significance. [Accessed February 23, 2011]. Available at: http://www.jerrydallal.com/LHSP/multtest.htm.

-------------------------------------------
Stephen Simon
Independent Statistical Consultant
P. Mean Consulting
-------------------------------------------

Dr. Dawson has a fine idea.  That could be implimented is R, for example, easily and then simulated to get the empirical type I error rate for that scenario.  I claim the real type I error rate is between 0.05 and 1 - (1 - 0.05)^5 = 0.2262191.

-------------------------------------------
Brian Johnson
Biostatistician
Essentia Institute of Rural Health
-------------------------------------------







Original Message:
Sent: 10-11-2012 16:47
From: John Dawson
Subject: Demo of how muliple analytical looks at data inflate the type I error/alpha?

While these examples are good for multiple comparisons (I would refer
interested parties to the xkcd jelly bean strip, which has been mentioned
of late here as well), it seems to me that Dr. Walters is looking for a 'peeking'
example rather than instances of multiple comparisons; this would be
more in line with interim analyses in a RCT.

Attached is a simulated dataset along the lines of what your classmate had.
A coin is flipped 50 times and we want to know if it is fair.
The first column is the flip number, the second column is the result,
the third column is the sum of heads to that point.

Instead of waiting for all 50 flips to come in, our impatient investigator
checks after 10, 20, 30 and 40 flips as well. The fourth and fifth columns
have high and low critical values for two-sided alpha=0.05 based on binomial
probabilities under the fair coin assumption; actually alpha is smaller than 0.05
in most of the checks but I didn't want to put random decision processes into
this simple example.

In this data set, the fourth interim check would have (incorrectly) rejected the fair coin null.

-------------------------------------------
John Dawson
Postdoctoral Scholar
University of Alabama at Birmingham
-------------------------------------------

Since the tests aren't independent the Sidak is a wide upper bound here.
Empirically I get 0.094 based on 10K sims.

-------------------------------------------
John Dawson
Postdoctoral Scholar
University of Alabama at Birmingham
-------------------------------------------




Original Message:
Sent: 10-11-2012 17:03
From: Brian Johnson
Subject: Demo of how muliple analytical looks at data inflate the type I error/alpha?

Dr. Dawson has a fine idea.  That could be implimented is R, for example, easily and then simulated to get the empirical type I error rate for that scenario.  I claim the real type I error rate is between 0.05 and 1 - (1 - 0.05)^5 = 0.2262191.

-------------------------------------------
Brian Johnson
Biostatistician
Essentia Institute of Rural Health
-------------------------------------------







Original Message:
Sent: 10-11-2012 16:47
From: John Dawson
Subject: Demo of how muliple analytical looks at data inflate the type I error/alpha?

While these examples are good for multiple comparisons (I would refer
interested parties to the xkcd jelly bean strip, which has been mentioned
of late here as well), it seems to me that Dr. Walters is looking for a 'peeking'
example rather than instances of multiple comparisons; this would be
more in line with interim analyses in a RCT.

Attached is a simulated dataset along the lines of what your classmate had.
A coin is flipped 50 times and we want to know if it is fair.
The first column is the flip number, the second column is the result,
the third column is the sum of heads to that point.

Instead of waiting for all 50 flips to come in, our impatient investigator
checks after 10, 20, 30 and 40 flips as well. The fourth and fifth columns
have high and low critical values for two-sided alpha=0.05 based on binomial
probabilities under the fair coin assumption; actually alpha is smaller than 0.05
in most of the checks but I didn't want to put random decision processes into
this simple example.

In this data set, the fourth interim check would have (incorrectly) rejected the fair coin null.

-------------------------------------------
John Dawson
Postdoctoral Scholar
University of Alabama at Birmingham
-------------------------------------------

Whoops! Coding error. Make that 0.176

-------------------------------------------
John Dawson
Postdoctoral Scholar
University of Alabama at Birmingham
-------------------------------------------



Original Message:
Sent: 10-11-2012 17:03
From: Brian Johnson
Subject: Demo of how muliple analytical looks at data inflate the type I error/alpha?

Dr. Dawson has a fine idea.  That could be implimented is R, for example, easily and then simulated to get the empirical type I error rate for that scenario.  I claim the real type I error rate is between 0.05 and 1 - (1 - 0.05)^5 = 0.2262191.

-------------------------------------------
Brian Johnson
Biostatistician
Essentia Institute of Rural Health
-------------------------------------------







Original Message:
Sent: 10-11-2012 16:47
From: John Dawson
Subject: Demo of how muliple analytical looks at data inflate the type I error/alpha?

While these examples are good for multiple comparisons (I would refer
interested parties to the xkcd jelly bean strip, which has been mentioned
of late here as well), it seems to me that Dr. Walters is looking for a 'peeking'
example rather than instances of multiple comparisons; this would be
more in line with interim analyses in a RCT.

Attached is a simulated dataset along the lines of what your classmate had.
A coin is flipped 50 times and we want to know if it is fair.
The first column is the flip number, the second column is the result,
the third column is the sum of heads to that point.

Instead of waiting for all 50 flips to come in, our impatient investigator
checks after 10, 20, 30 and 40 flips as well. The fourth and fifth columns
have high and low critical values for two-sided alpha=0.05 based on binomial
probabilities under the fair coin assumption; actually alpha is smaller than 0.05
in most of the checks but I didn't want to put random decision processes into
this simple example.

In this data set, the fourth interim check would have (incorrectly) rejected the fair coin null.

-------------------------------------------
John Dawson
Postdoctoral Scholar
University of Alabama at Birmingham
-------------------------------------------

Thank you all for your response.  I did find exactly what I was looking for so I am motivated somewhat to create a spreadsheet to help me demostrate the inflation of type 1 error with early analysis of data. 

-------------------------------------------
Shelley-Ann Walters
3M
-------------------------------------------







Original Message:
Sent: 10-11-2012 17:34
From: John Dawson
Subject: Demo of how muliple analytical looks at data inflate the type I error/alpha?

Whoops! Coding error. Make that 0.176

-------------------------------------------
John Dawson
Postdoctoral Scholar
University of Alabama at Birmingham
-------------------------------------------



Original Message:
Sent: 10-11-2012 17:03
From: Brian Johnson
Subject: Demo of how muliple analytical looks at data inflate the type I error/alpha?

Dr. Dawson has a fine idea.  That could be implimented is R, for example, easily and then simulated to get the empirical type I error rate for that scenario.  I claim the real type I error rate is between 0.05 and 1 - (1 - 0.05)^5 = 0.2262191.

-------------------------------------------
Brian Johnson
Biostatistician
Essentia Institute of Rural Health
-------------------------------------------







Original Message:
Sent: 10-11-2012 16:47
From: John Dawson
Subject: Demo of how muliple analytical looks at data inflate the type I error/alpha?

While these examples are good for multiple comparisons (I would refer
interested parties to the xkcd jelly bean strip, which has been mentioned
of late here as well), it seems to me that Dr. Walters is looking for a 'peeking'
example rather than instances of multiple comparisons; this would be
more in line with interim analyses in a RCT.

Attached is a simulated dataset along the lines of what your classmate had.
A coin is flipped 50 times and we want to know if it is fair.
The first column is the flip number, the second column is the result,
the third column is the sum of heads to that point.

Instead of waiting for all 50 flips to come in, our impatient investigator
checks after 10, 20, 30 and 40 flips as well. The fourth and fifth columns
have high and low critical values for two-sided alpha=0.05 based on binomial
probabilities under the fair coin assumption; actually alpha is smaller than 0.05
in most of the checks but I didn't want to put random decision processes into
this simple example.

In this data set, the fourth interim check would have (incorrectly) rejected the fair coin null.

-------------------------------------------
John Dawson
Postdoctoral Scholar
University of Alabama at Birmingham
-------------------------------------------