ASA Connect

What common statistical misconceptions do you come across?

-------------------------------------------
Elaine Eisenbeisz
Owner and Principal Statistician
Omega Statistics
-------------------------------------------

"It's the Law of Averages."  This is a popular misinterpretation of the Laws of Large Numbers.  It turns independent events into dependent ones to create mean reversion.

-------------------------------------------
Charles Coleman
-------------------------------------------


Original Message:
Sent: 11-10-2014 09:53
From: Elaine Eisenbeisz
Subject: Statistical Misconceptions

What common statistical misconceptions do you come across?

-------------------------------------------
Elaine Eisenbeisz
Owner and Principal Statistician
Omega Statistics
-------------------------------------------

Original Message------

What common statistical misconceptions do you come across?

-------------------------------------------
Elaine Eisenbeisz
Owner and Principal Statistician
Omega Statistics
-------------------------------------------

Eric,

Some authors have concluded that the misconception you describe is unavoidable unless students develop a firm footing in both bayesian & frequentist reasoning. Bayesian statements are statements about hypotheses given data, but frequentist statements (such as p itself) are ones about data given hypotheses.

Applied science aims to make statements about generalities (hypotheses) based on observations (data), which bayesian statements are equipped to supply; however, the results most statisticians offer are frequentist; hence, we are inviting & perhaps even encouraging the misconception.

I sometimes illustrate the difference between what applied science is looking for, vs. what frequentist results indicate, using a "measles-spots" analogy.

"Measles" correlates to a hypothesis.

"Spots" correlates to a datum.

Plainly, P(spots | measles) = nearly 100%; almost everyone with measles displays spots. This is the frequentist type of statement.

In contrast, P(measles | spots) = much less than 100%, since someone could display spots for many reasons other than having measles. This is the applied science, bayesian type of statement.

Beyond that, the discussion & references I gave in my Post #8 of this thread might trigger some ideas for you as well.

-------------------------------------------
Andrew Hartley
Associate Statistical Science Director
PPD, Inc.
-------------------------------------------


Original Message:
Sent: 11-11-2014 09:16
From: Eric Vance
Subject: Statistical Misconceptions

Common misconception: p<0.05 means the probability that the null hypothesis is true is less than 0.05.

Does anyone have a good analogy for what a p-value is and what it isn't? The idea behind an analogy is that it describes the intuition behind the technical concept and doesn't have to be 100% technically correct itself.

I don't have a great analogy to explain this concept. The sort-of analogy I use is from Freedman, Pisani, Purves "Statistics" and goes like this in my re-telling of it:

In many court drama movies the defense lawyer advises his client who is up for cross-examination, "Only answer the question the prosecuting lawyer asks you. Don't answer the question you think he should be asking or the question you want him to ask. Answer only the question he actually asks."

In an opposite vein, it doesn't matter what question you ask about the null hypothesis, the p-value only gives an answer to one specific question, namely, "If we assume the null hypothesis to be true, what is the probability we would observe data as extreme or more extreme than the data we actually observed?" You can go ahead and ask the p-value any question you want, but it only answers the above question.

I'm not satisfied with this "analogy" for p-values. Does anyone have a better one?

--
Eric Vance, PhD
Director of LISA (Laboratory for Interdisciplinary Statistical Analysis),
http://www.lisa.stat.vt.edu
Assistant Research Professor, Virginia Tech Department of Statistics
403G Hutcheson Hall (0439), 250 Drillfield Drive, Blacksburg, VA 24061,
540-231-4597, http://www.stat.vt.edu/people/faculty/Vance-Eric.html
ervance@vt.edu
----------------------

Hi:  I use the "chance cause" concept, following Walter Shewhart (1930's).  The p-value is the probability of what we obseve (at or more extremne then what we have observed) if operatuing under a system of chance causes.  I then give some sime examples, for example from games of chance.  I also tie this in with the concept of significance.  Thus, when the p-value is small, there is greater "significance" to the event we observe - meaning that it is likely that something other than a chance cause that was at play in bringing about what we have observed,  and this moves us to look for causs other rhan chance. 

-------------------------------------------
Stephen Luko
Statistician, fellow, UTC Aerospace Systems
stephen.luko@hs.utc.com
-------------------------------------------


Original Message:
Sent: 11-11-2014 09:16
From: Eric Vance
Subject: Statistical Misconceptions

Common misconception: p<0.05 means the probability that the null hypothesis is true is less than 0.05.

Does anyone have a good analogy for what a p-value is and what it isn't? The idea behind an analogy is that it describes the intuition behind the technical concept and doesn't have to be 100% technically correct itself.

I don't have a great analogy to explain this concept. The sort-of analogy I use is from Freedman, Pisani, Purves "Statistics" and goes like this in my re-telling of it:

In many court drama movies the defense lawyer advises his client who is up for cross-examination, "Only answer the question the prosecuting lawyer asks you. Don't answer the question you think he should be asking or the question you want him to ask. Answer only the question he actually asks."

In an opposite vein, it doesn't matter what question you ask about the null hypothesis, the p-value only gives an answer to one specific question, namely, "If we assume the null hypothesis to be true, what is the probability we would observe data as extreme or more extreme than the data we actually observed?" You can go ahead and ask the p-value any question you want, but it only answers the above question.

I'm not satisfied with this "analogy" for p-values. Does anyone have a better one?

--
Eric Vance, PhD
Director of LISA (Laboratory for Interdisciplinary Statistical Analysis),
http://www.lisa.stat.vt.edu
Assistant Research Professor, Virginia Tech Department of Statistics
403G Hutcheson Hall (0439), 250 Drillfield Drive, Blacksburg, VA 24061,
540-231-4597, http://www.stat.vt.edu/people/faculty/Vance-Eric.html
ervance@vt.edu
----------------------

Regarding p-values:  Two thoughts, first some background, then a brief class demonstration  I've used to illustrate p-values.

Background:  I find it useful to think about p-values relying on Arthur Dempster's distinction between "predictive" and "postdictive" uses of probability.  Bayesian intervals are "predictive" in the sense that they describe uncertainty about an outcome not yet revealed, whereas p-values are postdictive, in that they assign a number to an outcome that is known to have occurred.  Dempster regards p-values as a quantitative measure of surprise at an outcome, or as Nick Maxwell puts it so well (I'm paraphrasing) "How embarrassed should the null hypothesis be in the face of the data?"  (References at the end)

Class demonstration:  I show the class a two-headed coin, tell them I'm going to toss it several times, and ask a student to guess the outcome of each toss.  After each toss and guess, I say, "You're right" regardless of the truth.  The first time the guess is "right" it's no big deal.  Ditto the second time.  On we go ...  By the fifth toss and "right" guess, almost everyone is highly suspicious, and we can have a good discussion:  At what point did you start to get suspicious?  At what point were you persuaded that there was something fishy going on?  What was your reasoning?  I use this demo to bring out various points, e.g., intuitively, everyone is starting from a null model and using that model to evaluate a p-value, at least informally, and that the p-value asks "If I trust my model, how likely is the observed outcome?  The p-value is a postdictive measure of surprise or suspicion.  For most students, 3 "right" guesses is still meh, 4 in a row (p=1/16) is enough to raise questions, and 5 in a row (p=1/32) is pretty convincing.  Moreover, in this context, no one thinks the p-value is the chance that the null is true.  We can discuss various alternatives to the fair coin model, but it is clear that there is no easy way to assign probabilities to alternatives based solely on the data.

George

A.P. Dempster (1971).  "Model Searching and Estimation in the Logic of Inference" in Foundations of Statistical Inference: A Symposium, edited by Godambe and Sprott.  Holt Rinehart and Winston.

Nick Maxwell (2004).  Data Matters.  Wiley.

-------------------------------------------
George Cobb
Mount Holyoke College
-------------------------------------------


Original Message:
Sent: 11-11-2014 09:16
From: Eric Vance
Subject: Statistical Misconceptions

Common misconception: p<0.05 means the probability that the null hypothesis is true is less than 0.05.

Does anyone have a good analogy for what a p-value is and what it isn't? The idea behind an analogy is that it describes the intuition behind the technical concept and doesn't have to be 100% technically correct itself.

I don't have a great analogy to explain this concept. The sort-of analogy I use is from Freedman, Pisani, Purves "Statistics" and goes like this in my re-telling of it:

In many court drama movies the defense lawyer advises his client who is up for cross-examination, "Only answer the question the prosecuting lawyer asks you. Don't answer the question you think he should be asking or the question you want him to ask. Answer only the question he actually asks."

In an opposite vein, it doesn't matter what question you ask about the null hypothesis, the p-value only gives an answer to one specific question, namely, "If we assume the null hypothesis to be true, what is the probability we would observe data as extreme or more extreme than the data we actually observed?" You can go ahead and ask the p-value any question you want, but it only answers the above question.

I'm not satisfied with this "analogy" for p-values. Does anyone have a better one?

--
Eric Vance, PhD
Director of LISA (Laboratory for Interdisciplinary Statistical Analysis),
http://www.lisa.stat.vt.edu
Assistant Research Professor, Virginia Tech Department of Statistics
403G Hutcheson Hall (0439), 250 Drillfield Drive, Blacksburg, VA 24061,
540-231-4597, http://www.stat.vt.edu/people/faculty/Vance-Eric.html
ervance@vt.edu
----------------------

Hello, George,
much of your description follows the arguments of Richard Royall in "Statistical Evidence: A Likelihood Paradigm."

If we want to substantiate our reasoning mathematically as much as possible, though, we need to base what is "suspicious" or "surprising" not only on the evidence given by the coin tosses, but also on the costs of being wrong or right (the loss function) & our prior beliefs about the coin & the tossing mechanism.

If wrongly concluding the coin is unfair (say) is very costly, then we should use a higher standard for concluding that. We would also need stronger evidence from the tosses if, before the tosses, the coin had been examined very carefully from a physical point of view & found to follow design specifications very closely.

-------------------------------------------
Andrew Hartley
Associate Statistical Science Director
PPD, Inc.
-------------------------------------------


Original Message:
Sent: 11-12-2014 10:08
From: George Cobb
Subject: Statistical Misconceptions

Regarding p-values:  Two thoughts, first some background, then a brief class demonstration  I've used to illustrate p-values.

Background:  I find it useful to think about p-values relying on Arthur Dempster's distinction between "predictive" and "postdictive" uses of probability.  Bayesian intervals are "predictive" in the sense that they describe uncertainty about an outcome not yet revealed, whereas p-values are postdictive, in that they assign a number to an outcome that is known to have occurred.  Dempster regards p-values as a quantitative measure of surprise at an outcome, or as Nick Maxwell puts it so well (I'm paraphrasing) "How embarrassed should the null hypothesis be in the face of the data?"  (References at the end)

Class demonstration:  I show the class a two-headed coin, tell them I'm going to toss it several times, and ask a student to guess the outcome of each toss.  After each toss and guess, I say, "You're right" regardless of the truth.  The first time the guess is "right" it's no big deal.  Ditto the second time.  On we go ...  By the fifth toss and "right" guess, almost everyone is highly suspicious, and we can have a good discussion:  At what point did you start to get suspicious?  At what point were you persuaded that there was something fishy going on?  What was your reasoning?  I use this demo to bring out various points, e.g., intuitively, everyone is starting from a null model and using that model to evaluate a p-value, at least informally, and that the p-value asks "If I trust my model, how likely is the observed outcome?  The p-value is a postdictive measure of surprise or suspicion.  For most students, 3 "right" guesses is still meh, 4 in a row (p=1/16) is enough to raise questions, and 5 in a row (p=1/32) is pretty convincing.  Moreover, in this context, no one thinks the p-value is the chance that the null is true.  We can discuss various alternatives to the fair coin model, but it is clear that there is no easy way to assign probabilities to alternatives based solely on the data.

George

A.P. Dempster (1971).  "Model Searching and Estimation in the Logic of Inference" in Foundations of Statistical Inference: A Symposium, edited by Godambe and Sprott.  Holt Rinehart and Winston.

Nick Maxwell (2004).  Data Matters.  Wiley.

-------------------------------------------
George Cobb
Mount Holyoke College
-------------------------------------------


Original Message:
Sent: 11-11-2014 09:16
From: Eric Vance
Subject: Statistical Misconceptions

Common misconception: p<0.05 means the probability that the null hypothesis is true is less than 0.05.

Does anyone have a good analogy for what a p-value is and what it isn't? The idea behind an analogy is that it describes the intuition behind the technical concept and doesn't have to be 100% technically correct itself.

I don't have a great analogy to explain this concept. The sort-of analogy I use is from Freedman, Pisani, Purves "Statistics" and goes like this in my re-telling of it:

In many court drama movies the defense lawyer advises his client who is up for cross-examination, "Only answer the question the prosecuting lawyer asks you. Don't answer the question you think he should be asking or the question you want him to ask. Answer only the question he actually asks."

In an opposite vein, it doesn't matter what question you ask about the null hypothesis, the p-value only gives an answer to one specific question, namely, "If we assume the null hypothesis to be true, what is the probability we would observe data as extreme or more extreme than the data we actually observed?" You can go ahead and ask the p-value any question you want, but it only answers the above question.

I'm not satisfied with this "analogy" for p-values. Does anyone have a better one?

--
Eric Vance, PhD
Director of LISA (Laboratory for Interdisciplinary Statistical Analysis),
http://www.lisa.stat.vt.edu
Assistant Research Professor, Virginia Tech Department of Statistics
403G Hutcheson Hall (0439), 250 Drillfield Drive, Blacksburg, VA 24061,
540-231-4597, http://www.stat.vt.edu/people/faculty/Vance-Eric.html
ervance@vt.edu
----------------------

I have found that a Svengali deck of cards is useful for conveying some of these ideas. With such a deck, anyone picking a card at random from the deck ends up picking the same card (specific to the deck). It is available from any magic store or online (at Amazon, for example). When I start discussing probability, I ask the class what they think the probability is that a card picked at random is, say, the king of spades. Everyone responds "one out of 52," indicating that they understand the basics of probability. I describe that as theoretical probability, and contrast it with experimental probability, where we carry out an experiment and see how often something occurs. I then pull out my Svengali deck and ask a student to pick a card and show it to the class. Everyone laughs when they see that the card is the king of spades. I say that perhaps this is just a coincidence, cut the deck a few times, and ask a second student to pick a card. Students are again amazed, but I repeat for a third time. I point out that at this point they are probably suspicious that this is no ordinary deck, because with this deck, P(K of spades) is not 1/52 but 3/3 = 1. I then describe how this is essentially what we're doing when we do hypothesis testing: we make a hypothesis, and when the data makes the initial hypothesis implausible, we reject it. I also point out that people complain that you can't tell anything with a sample size of 1000, yet the students were willing to make a conclusion based on a sample size of 3. Clearly small samples are sometimes sufficient.

-------------------------------------------
Raymond Greenwell
Professor of Mathematics
Department of Mathematics
-------------------------------------------


Original Message:
Sent: 11-12-2014 10:08
From: George Cobb
Subject: Statistical Misconceptions

Regarding p-values:  Two thoughts, first some background, then a brief class demonstration  I've used to illustrate p-values.

Background:  I find it useful to think about p-values relying on Arthur Dempster's distinction between "predictive" and "postdictive" uses of probability.  Bayesian intervals are "predictive" in the sense that they describe uncertainty about an outcome not yet revealed, whereas p-values are postdictive, in that they assign a number to an outcome that is known to have occurred.  Dempster regards p-values as a quantitative measure of surprise at an outcome, or as Nick Maxwell puts it so well (I'm paraphrasing) "How embarrassed should the null hypothesis be in the face of the data?"  (References at the end)

Class demonstration:  I show the class a two-headed coin, tell them I'm going to toss it several times, and ask a student to guess the outcome of each toss.  After each toss and guess, I say, "You're right" regardless of the truth.  The first time the guess is "right" it's no big deal.  Ditto the second time.  On we go ...  By the fifth toss and "right" guess, almost everyone is highly suspicious, and we can have a good discussion:  At what point did you start to get suspicious?  At what point were you persuaded that there was something fishy going on?  What was your reasoning?  I use this demo to bring out various points, e.g., intuitively, everyone is starting from a null model and using that model to evaluate a p-value, at least informally, and that the p-value asks "If I trust my model, how likely is the observed outcome?  The p-value is a postdictive measure of surprise or suspicion.  For most students, 3 "right" guesses is still meh, 4 in a row (p=1/16) is enough to raise questions, and 5 in a row (p=1/32) is pretty convincing.  Moreover, in this context, no one thinks the p-value is the chance that the null is true.  We can discuss various alternatives to the fair coin model, but it is clear that there is no easy way to assign probabilities to alternatives based solely on the data.

George

A.P. Dempster (1971).  "Model Searching and Estimation in the Logic of Inference" in Foundations of Statistical Inference: A Symposium, edited by Godambe and Sprott.  Holt Rinehart and Winston.

Nick Maxwell (2004).  Data Matters.  Wiley.

-------------------------------------------
George Cobb
Mount Holyoke College
-------------------------------------------


Original Message:
Sent: 11-11-2014 09:16
From: Eric Vance
Subject: Statistical Misconceptions

Common misconception: p<0.05 means the probability that the null hypothesis is true is less than 0.05.

Does anyone have a good analogy for what a p-value is and what it isn't? The idea behind an analogy is that it describes the intuition behind the technical concept and doesn't have to be 100% technically correct itself.

I don't have a great analogy to explain this concept. The sort-of analogy I use is from Freedman, Pisani, Purves "Statistics" and goes like this in my re-telling of it:

In many court drama movies the defense lawyer advises his client who is up for cross-examination, "Only answer the question the prosecuting lawyer asks you. Don't answer the question you think he should be asking or the question you want him to ask. Answer only the question he actually asks."

In an opposite vein, it doesn't matter what question you ask about the null hypothesis, the p-value only gives an answer to one specific question, namely, "If we assume the null hypothesis to be true, what is the probability we would observe data as extreme or more extreme than the data we actually observed?" You can go ahead and ask the p-value any question you want, but it only answers the above question.

I'm not satisfied with this "analogy" for p-values. Does anyone have a better one?

--
Eric Vance, PhD
Director of LISA (Laboratory for Interdisciplinary Statistical Analysis),
http://www.lisa.stat.vt.edu
Assistant Research Professor, Virginia Tech Department of Statistics
403G Hutcheson Hall (0439), 250 Drillfield Drive, Blacksburg, VA 24061,
540-231-4597, http://www.stat.vt.edu/people/faculty/Vance-Eric.html
ervance@vt.edu
----------------------

Regarding what Dr. Vance writes about what a p-value answers, namely "the p-value only gives an answer to one specific question, namely, "If we assume the null hypothesis to be true, what is the probability we would observe data as extreme or more extreme than the data we actually observed?" You can go ahead and ask the p-value any question you want, but it only answers the above question.":

This is a good point. To extend it to a "P-Value Meaning", try this:

The probability that a value of the statistic (or corresponding test stat) in the direction(s) of H1 and as extreme as the one that actually did occur would occur if Ho were true.

As far as Stat 101 is concerned, I would still go with what Dr. Vance writes above whereas my "Meaning" could perhaps be used in a step-by-step explanation when introducing the test-of-significance concept by example.

Related to this for those not familiar with it, there is the Stat 101-like monograph titled "What Is A P-Value Anyway? 34 [Short] Stories To Help You Actually Understand Statistics?" by Andrew Vickers, which would probably be good for the significant others of statisticians who are not such and prefer reading short stories...

Hi all,

Christina Kendziorski gives a funny analogy in her class on hypothesis testing, which seems to amuse the students.  Referring to a loved one (e.g. husband...[not that it's based on fact!!]...

``If you loved me (null), you would not have been such a jerk about **[fill in the blank...birthday, anniversary,...]**
  Since you were behaving like a jerk (i.e. data),  it must be you don't love me! ''

-Michael N.

-------------------------------------------
Michael Newton
University of Wisconsin-Madison
-------------------------------------------


Original Message:
Sent: 11-13-2014 00:28
From: David Bernklau
Subject: Statistical Misconceptions

Regarding what Dr. Vance writes about what a p-value answers, namely "the p-value only gives an answer to one specific question, namely, "If we assume the null hypothesis to be true, what is the probability we would observe data as extreme or more extreme than the data we actually observed?" You can go ahead and ask the p-value any question you want, but it only answers the above question.":

This is a good point. To extend it to a "P-Value Meaning", try this:

The probability that a value of the statistic (or corresponding test stat) in the direction(s) of H1 and as extreme as the one that actually did occur would occur if Ho were true.

As far as Stat 101 is concerned, I would still go with what Dr. Vance writes above whereas my "Meaning" could perhaps be used in a step-by-step explanation when introducing the test-of-significance concept by example.

Related to this for those not familiar with it, there is the Stat 101-like monograph titled "What Is A P-Value Anyway? 34 [Short] Stories To Help You Actually Understand Statistics?" by Andrew Vickers, which would probably be good for the significant others of statisticians who are not such and prefer reading short stories...

-------------------------------------------
David Bernklau
(David Bee on Internet)
--------------------


Original Message:
Sent: 11-11-2014 09:16
From: Eric Vance
Subject: Statistical Misconceptions

Common misconception: p<0.05 means the probability that the null hypothesis is true is less than 0.05.

Does anyone have a good analogy for what a p-value is and what it isn't? The idea behind an analogy is that it describes the intuition behind the technical concept and doesn't have to be 100% technically correct itself.

I don't have a great analogy to explain this concept. The sort-of analogy I use is from Freedman, Pisani, Purves "Statistics" and goes like this in my re-telling of it:

In many court drama movies the defense lawyer advises his client who is up for cross-examination, "Only answer the question the prosecuting lawyer asks you. Don't answer the question you think he should be asking or the question you want him to ask. Answer only the question he actually asks."

In an opposite vein, it doesn't matter what question you ask about the null hypothesis, the p-value only gives an answer to one specific question, namely, "If we assume the null hypothesis to be true, what is the probability we would observe data as extreme or more extreme than the data we actually observed?" You can go ahead and ask the p-value any question you want, but it only answers the above question.

I'm not satisfied with this "analogy" for p-values. Does anyone have a better one?

--
Eric Vance, PhD
Director of LISA (Laboratory for Interdisciplinary Statistical Analysis),
http://www.lisa.stat.vt.edu
Assistant Research Professor, Virginia Tech Department of Statistics
403G Hutcheson Hall (0439), 250 Drillfield Drive, Blacksburg, VA 24061,
540-231-4597, http://www.stat.vt.edu/people/faculty/Vance-Eric.html
ervance@vt.edu
----------------------

Eric Vance asked for an analogy to explain P-value. I believe it was Oscar Kempthorne who called the P-value a measure of the distance [or proximity] of the data from [or to] the null hypothesis. If, for example, the null hypothesis is about mean temperature in degrees C, we start out with the distance between the hypothetical population mean and the empirical sample mean in deg C, then standardize it to the units of a Z-score or T-score, then, rescale again to the probability scale, a universal measure, regardless of the original units and regardless of the particular test of significance (Z, T, Chi-squared, ...). Nonetheless, we must always consider the power or sensitivity or severeness of the test.
-------------------------------------------
Golde Holtzman
Associate Professor Emeritus of Statistics
Virginia Tech (VPI)
-------------------------------------------



Original Message:
Sent: 11/11/2014 9:16:41 AM
From: ervance@vt.edu
Subject: RE: Statistical Misconceptions

Common misconception: p<0.05 means the probability that the null hypothesis is true is less than 0.05.

Does anyone have a good analogy for what a p-value is and what it isn't? The idea behind an analogy is that it describes the intuition behind the technical concept and doesn't have to be 100% technically correct itself.

I don't have a great analogy to explain this concept. The sort-of analogy I use is from Freedman, Pisani, Purves "Statistics" and goes like this in my re-telling of it:

In many court drama movies the defense lawyer advises his client who is up for cross-examination, "Only answer the question the prosecuting lawyer asks you. Don't answer the question you think he should be asking or the question you want him to ask. Answer only the question he actually asks."

In an opposite vein, it doesn't matter what question you ask about the null hypothesis, the p-value only gives an answer to one specific question, namely, "If we assume the null hypothesis to be true, what is the probability we would observe data as extreme or more extreme than the data we actually observed?" You can go ahead and ask the p-value any question you want, but it only answers the above question.

I'm not satisfied with this "analogy" for p-values. Does anyone have a better one?

-- Eric Vance, PhD Director of LISA (Laboratory for Interdisciplinary Statistical Analysis), http://www.lisa.stat.vt.edu Assistant Research Professor, Virginia Tech Department of Statistics 403G Hutcheson Hall (0439), 250 Drillfield Drive, Blacksburg, VA 24061, 540-231-4597, http://www.stat.vt.edu/people/faculty/Vance-Eric.html ervance@vt.edu ----------------------

The most common statistical misconception is about probability itself. If the weatherman states tomorrow there is a 10% chance of rain, and it rains tomorrow, we can expect people to say: "well, what do these forecasters know anyway". Similarly, if you tell a student that the chance of winning the lottery is, say 1/150 million (of course, this varies by state), they might say: "But the cousin of the brother of the neighbor of the son of a friend of mine, won the lottery. Therefore, I could win the lottery too". There is no realization there is a difference between YOU winning the lottery and SOMEONE winning the lottery (ignoring the fact that anecdotal evidence is being used to support the argument to play the lottery).

Put in another way, if I start a discussion with students about probability with a case of the infinite monkey theorem such that the probability of writing a phrase of Shakespeare is 1/150 million, and ask them if they would hire monkeys for a newspaper; there would be an overwhelming "No" answer. Asking students why, they reply that it would simply be silly given the low chances. However, if next in the discussion, I ask them if they would play a lottery to win $1,000,000 where the probability of winning is 1/150 million, most would answer yes. People's use of probability in making decisions depends on whether the event is seen as a positive thing or a negative thing (a well known fact related to expected values in behavioral finance).

-------------------------------------------
Roberto Rivera
Associate professor
University of Puerto Rico Mayaguez
-------------------------------------------


Original Message:
Sent: 11-10-2014 09:53
From: Elaine Eisenbeisz
Subject: Statistical Misconceptions

What common statistical misconceptions do you come across?

-------------------------------------------
Elaine Eisenbeisz
Owner and Principal Statistician
Omega Statistics
-------------------------------------------

Another statistical misconception related to lottery is independence. When people choose their lottery numbers, people often try to use the most (or least) frequently observed numbers even though each draw is independent of previous ones. 

-------------------------------------------
Seong-Tae "Ty" Kim
Assistant Professor of Statistics
North Carolina A&T State University
-------------------------------------------


Original Message:
Sent: 11-11-2014 10:00
From: Roberto Rivera
Subject: Statistical Misconceptions

The most common statistical misconception is about probability itself. If the weatherman states tomorrow there is a 10% chance of rain, and it rains tomorrow, we can expect people to say: "well, what do these forecasters know anyway". Similarly, if you tell a student that the chance of winning the lottery is, say 1/150 million (of course, this varies by state), they might say: "But the cousin of the brother of the neighbor of the son of a friend of mine, won the lottery. Therefore, I could win the lottery too". There is no realization there is a difference between YOU winning the lottery and SOMEONE winning the lottery (ignoring the fact that anecdotal evidence is being used to support the argument to play the lottery).

Put in another way, if I start a discussion with students about probability with a case of the infinite monkey theorem such that the probability of writing a phrase of Shakespeare is 1/150 million, and ask them if they would hire monkeys for a newspaper; there would be an overwhelming "No" answer. Asking students why, they reply that it would simply be silly given the low chances. However, if next in the discussion, I ask them if they would play a lottery to win $1,000,000 where the probability of winning is 1/150 million, most would answer yes. People's use of probability in making decisions depends on whether the event is seen as a positive thing or a negative thing (a well known fact related to expected values in behavioral finance).

-------------------------------------------
Roberto Rivera
Associate professor
University of Puerto Rico Mayaguez
-------------------------------------------


Original Message:
Sent: 11-10-2014 09:53
From: Elaine Eisenbeisz
Subject: Statistical Misconceptions

What common statistical misconceptions do you come across?

-------------------------------------------
Elaine Eisenbeisz
Owner and Principal Statistician
Omega Statistics
-------------------------------------------

The lottery player below might actually be making a wise decision.  His expected gain is only negative if we assume the value of money to be proportional to the amount of money.  Many people may consider losing a dollar to be totally inconsequential but winning a million is life changing.  

-------------------------------------------
Emil M Friedman, PhD
emil.friedman@alum.mit.edu (forwards to day job)
emilfrie@alumni.princeton.edu (home)
http://www.statisticalconsulting.org
-------------------------------------------


Original Message:
Sent: 11-11-2014 10:00
From: Roberto Rivera
Subject: Statistical Misconceptions

The most common statistical misconception is about probability itself. If the weatherman states tomorrow there is a 10% chance of rain, and it rains tomorrow, we can expect people to say: "well, what do these forecasters know anyway". Similarly, if you tell a student that the chance of winning the lottery is, say 1/150 million (of course, this varies by state), they might say: "But the cousin of the brother of the neighbor of the son of a friend of mine, won the lottery. Therefore, I could win the lottery too". There is no realization there is a difference between YOU winning the lottery and SOMEONE winning the lottery (ignoring the fact that anecdotal evidence is being used to support the argument to play the lottery).

Put in another way, if I start a discussion with students about probability with a case of the infinite monkey theorem such that the probability of writing a phrase of Shakespeare is 1/150 million, and ask them if they would hire monkeys for a newspaper; there would be an overwhelming "No" answer. Asking students why, they reply that it would simply be silly given the low chances. However, if next in the discussion, I ask them if they would play a lottery to win $1,000,000 where the probability of winning is 1/150 million, most would answer yes. People's use of probability in making decisions depends on whether the event is seen as a positive thing or a negative thing (a well known fact related to expected values in behavioral finance).

-------------------------------------------
Roberto Rivera
Associate professor
University of Puerto Rico Mayaguez
-------------------------------------------


Original Message:
Sent: 11-10-2014 09:53
From: Elaine Eisenbeisz
Subject: Statistical Misconceptions

What common statistical misconceptions do you come across?

-------------------------------------------
Elaine Eisenbeisz
Owner and Principal Statistician
Omega Statistics
-------------------------------------------

The common ones - but they are so pervasive:
(1) the meaning of significance and p-values
(2) confidence intervals (when prediction interval is appropriate)

-------------------------------------------
James McCurley
Software Engineering Institute
-------------------------------------------


Original Message:
Sent: 11-10-2014 09:53
From: Elaine Eisenbeisz
Subject: Statistical Misconceptions

What common statistical misconceptions do you come across?

-------------------------------------------
Elaine Eisenbeisz
Owner and Principal Statistician
Omega Statistics
-------------------------------------------

One thing that I come across more often that I'd like is the notion some of my clients have that a p-Value < 0.05 in their favor means it is certain that the result true and they "win". It doesn't matter if the p-Value is 0.049999999 or <0.000000001, the conclusion is the same. Trying to explain to them it isn't necessarily so falls on deaf ears. Ugh! 

-------------------------------------------
Nestor Rohowsky
President and Principal Consultant
Integrated Data Consultation Svcs, Inc.
-------------------------------------------


Original Message:
Sent: 11-11-2014 10:06
From: James McCurley
Subject: Statistical Misconceptions

The common ones - but they are so pervasive:
(1) the meaning of significance and p-values
(2) confidence intervals (when prediction interval is appropriate)

-------------------------------------------
James McCurley
Software Engineering Institute
-------------------------------------------


Original Message:
Sent: 11-10-2014 09:53
From: Elaine Eisenbeisz
Subject: Statistical Misconceptions

What common statistical misconceptions do you come across?

-------------------------------------------
Elaine Eisenbeisz
Owner and Principal Statistician
Omega Statistics
-------------------------------------------

Trying again. 

Complete disbelief that a survey sample of 1200 from the USA has the same precision as a survey sample from an Arkansas county.

-------------------------------------------
John Senner
-------------------------------------------


Original Message:
Sent: 11-11-2014 10:12
From: John Senner
Subject: Statistical Misconceptions



-------------------------------------------
John Senner
-------------------------------------------

In the same area, I hear very often the confusion between power and precision. The sample size for a small proportion is smaller than a proportion of 0.5. But we need a big sample size to find a small difference and can do with a smaller sample to find a big difference.

Any good analogies for these?

Sharon

-------------------------------------------
Sharon Kühlmann-Berenzon, PhD
Sr Biostatistician
Public Health Agency of Sweden
Stockholm
-------------------------------------------



Original Message:
Sent: 11/11/2014 10:15:54 AM
From: j_senner@yahoo.com
Subject: RE: Statistical Misconceptions

Trying again.  Complete disbelief that a survey sample of 1200 from the USA has the same precision as a survey sample from an Arkansas county. ------------------------------------------- John Senner -------------------------------------------
Original Message: Sent: 11-11-2014 10:12 From: John Senner Subject: Statistical Misconceptions ------------------------------------------- John Senner -------------------------------------------

When speaking to non-statisticians, the two most common misconceptions are:

1) It's easy to lie with statistics.  I.E., you can prove anything by manipulating statistics.
When I hear that, I point out that statistical models have important assumptions and we can not
report that an effect is significant when the p-value indicates that it's non-significant.

2) Statistics are done by computers and statisticians are no longer needed.
When I hear this one, I tell the person that software is a computation aid and that statistics is a branch of mathematics,
with heavy reliance on calculus and matrix algebra.  I have even offered to show the person some books in my statistics library.

-------------------------------------------
Brandy Sinco
Research Associate
-------------------------------------------


Original Message:
Sent: 11-10-2014 09:53
From: Elaine Eisenbeisz
Subject: Statistical Misconceptions

What common statistical misconceptions do you come across?

-------------------------------------------
Elaine Eisenbeisz
Owner and Principal Statistician
Omega Statistics
-------------------------------------------

I think the most common misconception by far is regarding statements about data (whether observed or hypothetical) as if they were statements about the population.  

This is the basis of the common mistake known as the "transposition of conditioning," in which P(D | H) is taken to be, or to imply something about, P(H | D), where

H = some hypothesis &
D = data or datum. 

For instance, once an experiment has been run, a p-value (p) is the probability of results (Y), in an imaginary repetition of the experiment, at least as extreme as those (x) actually observed, assuming the tested hypothesis H:

P(Y>x | H).

As you can see, this is a probability about data, given a hypothesis.

However, numerous psychological experiments, sampling from both college students & college faculty, have shown that the most common inference made about H, based on p, is that p = P(H | x), which is a probability about a hypothesis, given data. See

Oakes, M. (1986). Statistical inference: A commentary for the social and behavioral sciences. Chichester, UK: Wiley.

Davidoff (1999). Standing statistics right side up. Annals of Internal Medicine (130), 12, 1019-1021.

Another common misconception of the same kind is known as the "confidence trick," in which the estimand, theta, is taken to lie within a given X% confidence interval with X% probability (a statement about a hypothesis given data). This is despite the facts that

1. the "X%" portion of the term "X% confidence interval" refers only to the pre-experimental probability of containing theta (again, a statement about the data)
2. the frequentist paradigm has no conception of the probability that a given CI contains an estimand

Foster (2014).  Confidence Trick: The Interpretation of Confidence Intervals. Canadian Journal of Science, Mathematics and Technology Education, Volume 14, Issue 1.

 

I could say much more about this, as I have strong feelings concerning the reasons it's so pervasive, but I'd better stop there & assess your interest.

-------------------------------------------
Andrew Hartley
Associate Statistical Science Director
PPD, Inc.
-------------------------------------------


Original Message:
Sent: 11-10-2014 09:53
From: Elaine Eisenbeisz
Subject: Statistical Misconceptions

What common statistical misconceptions do you come across?

-------------------------------------------
Elaine Eisenbeisz
Owner and Principal Statistician
Omega Statistics
-------------------------------------------

I did sample designs, usually for audit work or cost allocation.  The most common misconceptions I encountered were:
1.  Doubling the sample cuts the margin of error in half
2.  The binomial sample size formula is all you need to pick a sample plan
3.  Failure to use optimal allocation across strata (or any of several other choices a practitioner makes) invalidates the sample.  A corollary is that using equal-sized strata samples requires its own special estimation formula.  Still can't figure that one out.
4.  A biased estimator reflects the statistician's preconceptions.
5.  Sample plans are good, or at least neutral, while sample schemes are somehow nefarious.
6.  There is a magic sampling rate or sample size you need for the sample to be valid
7.  Concern about mathematical assumptions is academic pettifoggery; anybody who passed an intro stats course can design and carry out a valid sample.

-------------------------------------------
Robert Lovell
-------------------------------------------


Original Message:
Sent: 11-10-2014 09:53
From: Elaine Eisenbeisz
Subject: Statistical Misconceptions

What common statistical misconceptions do you come across?

-------------------------------------------
Elaine Eisenbeisz
Owner and Principal Statistician
Omega Statistics
-------------------------------------------

As I wrote the other day in the thread about conveying statistical concepts, I  have definitely encountered  #6 (believe it or not, even among students of statistics!). However, in my profession (economics), by far the most common and consequential misconception is that specifications of econometric models should be determined by goodness of fit, independent of what theory, common sense, or other sources of evidence would suggest
.   

-------------------------------------------
David Luskin
USDOT-FHWA
-------------------------------------------


Original Message:
Sent: 11-11-2014 14:29
From: Robert Lovell
Subject: Statistical Misconceptions

I did sample designs, usually for audit work or cost allocation.  The most common misconceptions I encountered were:
1.  Doubling the sample cuts the margin of error in half
2.  The binomial sample size formula is all you need to pick a sample plan
3.  Failure to use optimal allocation across strata (or any of several other choices a practitioner makes) invalidates the sample.  A corollary is that using equal-sized strata samples requires its own special estimation formula.  Still can't figure that one out.
4.  A biased estimator reflects the statistician's preconceptions.
5.  Sample plans are good, or at least neutral, while sample schemes are somehow nefarious.
6.  There is a magic sampling rate or sample size you need for the sample to be valid
7.  Concern about mathematical assumptions is academic pettifoggery; anybody who passed an intro stats course can design and carry out a valid sample.

-------------------------------------------
Robert Lovell
-------------------------------------------


Original Message:
Sent: 11-10-2014 09:53
From: Elaine Eisenbeisz
Subject: Statistical Misconceptions

What common statistical misconceptions do you come across?

-------------------------------------------
Elaine Eisenbeisz
Owner and Principal Statistician
Omega Statistics
-------------------------------------------

A statement I often hear from students at the start of their first course: "I have been told that if it is statistics, it must be hard" ------------------------------------------- Elizabeth Murff Eastern Washington University -------------------------------------------
Original Message: Sent: 11-10-2014 09:53 From: Elaine Eisenbeisz Subject: Statistical Misconceptions What common statistical misconceptions do you come across? ------------------------------------------- Elaine Eisenbeisz Owner and Principal Statistician Omega Statistics -------------------------------------------

I've come across many, many statistical misconceptions. See discussions at http://www.ma.utexas.edu/users/mks/CommonMistakes2014/commonmistakeshome2014.html,  http://www.ma.utexas.edu/users/mks/statmistakes/TOC.html, http://www.ma.utexas.edu/users/mks/Cautions2014/cautions2014.html, and http://www.ma.utexas.edu/blogs/mks/

-------------------------------------------
Martha Smith
University of Texas
-------------------------------------------


Original Message:
Sent: 11-10-2014 09:53
From: Elaine Eisenbeisz
Subject: Statistical Misconceptions

What common statistical misconceptions do you come across?

-------------------------------------------
Elaine Eisenbeisz
Owner and Principal Statistician
Omega Statistics
-------------------------------------------

Original Message------

What common statistical misconceptions do you come across?

-------------------------------------------
Elaine Eisenbeisz
Owner and Principal Statistician
Omega Statistics
-------------------------------------------

One that should also be high on the list is the misconception that a larger value of the Chi-square statistic means a stronger association, e.g., in the test for independence between two categorical variables. Such as in "The large value of the Chi-squared statistic (X^2=123, P<0.0001) indicates a very strong association between ...". The Chi-squared statistic is proportional to the sample size n, so the larger n, the larger X^2.  

-------------------------------------------
Bernhard Klingenberg
Williams College
-------------------------------------------


Original Message:
Sent: 11-10-2014 09:53
From: Elaine Eisenbeisz
Subject: Statistical Misconceptions

What common statistical misconceptions do you come across?

-------------------------------------------
Elaine Eisenbeisz
Owner and Principal Statistician
Omega Statistics
-------------------------------------------