ASA Connect

 View Only
Back to eGroups
Expand all | Collapse all

Statistical shenanigans - examples requested

  • 1.  Statistical shenanigans - examples requested

    Posted 07-01-2015 13:53

    Dear colleagues,

    I'm writing a book for a popular audience about the use and misuse of statistics, to be published by Penguin Random House next year. It will be a successor to the excellent books by Joel Best and Leon Huff. If you have examples of the misuse of statistics or graphs, particularly as reported in reputable media (NY Times, WSJ, TV networks, Corporate Reports, etc. —— sites like TMZ and The Star  just make too easy targets) I'd be grateful if you'd forward them to me. Contributions will of course be credited.

    ------------------------------
    Daniel Levitin, Ph.D.
    Dean of Arts & Humanities
    Minerva Schools at Keck Graduate Institute
    ------------------------------



  • 2.  RE: Statistical shenanigans - examples requested

    Posted 07-02-2015 13:17

    hi!  were you also interested in anecdotes, or only hard-print verifiable?

    ------------------------------
    Dean Barron
    Twobluecats.com
    ------------------------------


    Original Message


  • 3.  RE: Statistical shenanigans - examples requested

    Posted 07-03-2015 14:01

    Hi, Dean. Only hard-print verifiable from respected (or somewhat respected) sources. Thanks!

    ------------------------------
    Daniel Levitin
    Dean
    Minerva Schools at Keck Graduate Institute
    ------------------------------


    Original Message


  • 4.  RE: Statistical shenanigans - examples requested

    Posted 07-02-2015 13:38

    You might work through the Climate change threads on this forum - not all of them can be right.

    ------------------------------
    Steve Schwarzbek
    HPC Analyst
    Northrop Grumman
    ------------------------------


    Original Message


  • 5.  RE: Statistical shenanigans - examples requested

    Posted 09-03-2015 08:23

    Excellent suggestion!  I was shocked to see several ASA professionals show graphs and statistics that they say showed that global warming has not stopped or even slowed down since 2000.  The NOAA data, graphed as a 5 year moving average, is here.  Readers, draw your own conclusions.  My conclusion is when even statisticians can't agree that black is black and white is white, what hope is there to educate the general public?



    ------------------------------
    Morris Olitsky
    Statistician
    USDA
    ------------------------------


    Original Message


  • 6.  RE: Statistical shenanigans - examples requested

    Posted 07-02-2015 15:14

    You might look up reports of sexual abuse in the military, in the spring of 2013. On a Tuesday night the Dept of Defense "estimated" that there had been 26,000 cases in 2012. Wednesday morning they announced that 89.5 % of the 26,000 had not been reported. a) 26,000 moved from estimate to something stronger. b) 89.5 made it sound like they had serious calculations instead of speculations.

    ------------------------------
    Nancy Hall
    University of Delaware
    ------------------------------


    Original Message


  • 7.  RE: Statistical shenanigans - examples requested

    Posted 07-03-2015 08:27

    Daniel, for some time the RSS News had a column called Forsooth! collecting examples of misuse of statistics in the media. It can be now found on the StatsLife website StatsLife - The Royal Statistical Society forsooth | StatsLife. I believe you need to register first with their website.

    Over the years, I kept the hard copy of several of these with the idea of showing them to students. In case you have trouble accessing the website, feel free to drop me an email and I would be glad to send you a scanned copy.


    ------------------------------
    Marco Geraci
    Associate Professor, PhD, CStat
    University of South Carolina-Department of Epidemiology & Biostatistics
    ------------------------------


    Original Message


  • 8.  RE: Statistical shenanigans - examples requested

    Posted 07-06-2015 11:12

    There's also a collection of Forsooth's from Chance News at https://www.causeweb.org/wiki/chance/index.php/Collected_Forsooths

    ------------------------------
    Emil M Friedman, PhD
    emil.friedman@alum.mit.edu (forwards to day job)
    emilfrie@alumni.princeton.edu (home)
    http://www.statisticalconsulting.org



  • 9.  RE: Statistical shenanigans - examples requested

    Posted 07-07-2015 12:17


    Personally I have found the following graph to be an abomination.


    ------------------------------
    David Mangen
    Owner
    Mangen Research Associates, Inc.
    ------------------------------


    Original Message


  • 10.  RE: Statistical shenanigans - examples requested

    Posted 07-03-2015 09:46

    Daniel,

    You are welcome to review my 'Weird Statistics' blog, link below.   There I find publications that usually contain some statistical comment, and comment on them.   I specialize in unusual ones and many of them are good examples of bad statistics that I make fun of.   I focus on articles that are likely to make people smile or groan depending on their profession so you are unlikely to see academic articles there.    If you use anything from my blog, I only ask that you acknowledge it!   I've been doing this for a few years so there are a hundred or two things to look at.   You should find something useful there that fits your purpose.   

    Statistics in the News

    Scoop.it remove preview
    Statistics in the News
    Interesting or unusual and useful statistical data analysis articles.
    View this on Scoop.it >



    ------------------------------

    Bill Bentley
    Owner/President, Value-Train
    678-921-5238

    http://www.value-train.com

    billb@value-train.com
    Author of "Lean Six Sigma Secrets for the CIO"

    www.linkedin.com/in/billbentley/

    http://www.scoop.it/t/weird-statistics


    ------------------------------


    Original Message


  • 11.  RE: Statistical shenanigans - examples requested

    Posted 07-06-2015 14:55

    Cool site Bill! I'm going to share it on my facebook page.

    ------------------------------
    Elaine Eisenbeisz
    Owner and Principal Statistician
    Omega Statistics
    ------------------------------


    Original Message


  • 12.  RE: Statistical shenanigans - examples requested

    Posted 07-07-2015 11:29

    Are you interested in serious, apparently intentional misuse of statistics or just "fun" examples?  

    ------------------------------
    Charles Mann
    Charles R Mann Associates, Inc.
    ------------------------------


    Original Message


  • 13.  RE: Statistical shenanigans - examples requested

    Posted 07-08-2015 11:17

    Here's one that's not simple, but consequential.

    In many applications of statistical methods, the expected distribution of p-values under the null hypothesis is uniform - but in genome-wide association studies (GWAS), the smallest possible p-value depends on the minor allele frequency and, thus, the cumulative expected distribution of p-values in the frequently used QQ plot is a convex curve below the diagonal (http://www.nature.com/articles/tp2013124). For almost 10 years people have dismissed relevant findings from GWAS because the QQ plots were misinterpreted (among a couple of other issues).

    Knut

    ------------------------------
    Knut M. Wittkowski
    Head, Dept. Biostatistics, Epidemiology, and Research Design
    The Rockefeller University
    ------------------------------



  • 14.  RE: Statistical shenanigans - examples requested

    Posted 07-09-2015 07:34

    This is my favorite, which I have used with my students discussing how the media can skew studies: "Study: smelling farts may good to your health"

    http://theweek.com/speedreads/450160/study-smelling-farts-may-good-health

    Also reported in dailymail, Time and The Guardian though several websites later made corrections about the study conclusions (the researchers never made such a claim about farts. For a summary of the study see here:http://www.nbcnews.com/health/health-news/no-farts-dont-prevent-cancer-claims-dont-pass-smell-test-n156136)
    ------------------------------
    Roberto Rivera
    Associate professor
    University of Puerto Rico Mayaguez
    ------------------------------


    Original Message


  • 15.  RE: Statistical shenanigans - examples requested

    Posted 07-12-2015 17:02

    Here is my most recent favorite:

    Your Hardcore Run May Be as Dangerous as Sitting Around Doing Nothing

    Bloomberg.com remove preview
    Your Hardcore Run May Be as Dangerous as Sitting Around Doing Nothing
    (Bloomberg) -- Vigorous running -- faster than 7 miles an hour, more than 2.4 hours a week, more than three times a week - - could be almost as harmful as sitting around doing nothing, according to a new cardiology study that's likely to stoke the debate over how much exercise is too much.
    View this on Bloomberg.com >

    Below is a refutation piblished in the NY Times.

    http://www.nytimes.com/2015/02/06/upshot/no-more-running-probably-isnt-bad-for-you.html?_r=1&abt=0002&abg=1



    ------------------------------
    Brydon Grant
    ------------------------------


    Original Message


  • 16.  RE: Statistical shenanigans - examples requested

    Posted 09-04-2015 10:28

    There's the common mistake of muddling predictive-ness with measurement. Specifically, VAM for teacher evaluations have employed very "bad statistics". The ASA has written about this (quoted in the linked article from the WP).

    In addition to the predictive-ness/measurement mistake, another glaring error is the omission of a factor to adequately account for starting scores. It seems whoever constructed these models did it in such a way that teachers with already high-performing students would be penalized (since those students could not achieve much "improvement").

    http://www.washingtonpost.com/blogs/answer-sheet/wp/2015/04/01/teacher-how-my-highest-scoring-students-actually-hurt-my-evaluation/



  • 17.  RE: Statistical shenanigans - examples requested

    Posted 09-07-2015 06:03

    A related problem, that I "discovered" when creating my own VAM back in 2009 is that many of the tests are designed to measure students in the middle with precision but not students at the ends.  So those good students who are getting great to perfect scores on the test fluctuate in huge bounds from year to year.  A difference of one question can be worth 100 points or more at the top of the scaled score compared to 15 or so in the middle.  Unless the model addresses this, teachers of even one excellent student are being judged on large differences in test scores that are actually meaningless.

    ------------------------------
    Ruth Wunderlich
    Linden Hall School for Girls
    ------------------------------


    Original Message


  • 18.  RE: Statistical shenanigans - examples requested

    Posted 09-10-2015 22:04

    Another interesting phenomenon about standardized tests arises because they are devised so that scores from the target population have a normal distribution. See part b of Problem 4 (at the very end) in the handout at  http://www.ma.utexas.edu/users/mks/ProbStatGradTeach/NormalAndCLT.pdf for a hypothetical example.

    ------------------------------
    Martha Smith
    University of Texas
    ------------------------------


    Original Message


  • 19.  RE: Statistical shenanigans - examples requested

    Posted 09-04-2015 11:38

    Not a shenanigan, but an instance worthy of noting: recent news program noted a drop in (average) SAT scores. I assume this refers to the average score of all who took the SAT. But the is not a random/representative sample. As a higher percent of the relevant population take the SAT, the (on average) "sample" is dipping lower into the achievement percentiles, so we can expect a lower average. This confounder is never mentioned: percent of "population" taking the SAT. When there is/was a push for all to go to college, a result was declining actual average SAT results.  

    ------------------------------
    Kenneth Burnham
    Colorado State University
    ------------------------------


    Original Message


  • 20.  RE: Statistical shenanigans - examples requested

    Posted 09-07-2015 11:22

    I believe that the SAT was "re-normed" back in the mid 90's and perhaps for that reason.

    ------------------------------
    Walter Dyar
    Retired
    ------------------------------


    Original Message


  • 21.  RE: Statistical shenanigans - examples requested

    Posted 09-08-2015 13:30

    Our nation's Unemployment rate is a major misuse of statistics.  It only counts people currently collecting unemployment; Once unemployment is maxed out, the "unemployed" go on welfare, disability, public assistance, social security, work "under the table", or go out on the street.   The Unemployment Rate needs to be modified to take into account the aforementioned variables, so that the American public is truly informed about this

    Edward Kosiarski


    ------------------------------



  • 22.  RE: Statistical shenanigans - examples requested

    Posted 09-09-2015 11:15

    This us a political issue, not a mathematical one. All these data are collected and are available on the website of the Bureau of Labor Statistics. The data are available to economists and researchers, so I'm not sure "shenanigans" is the right word to describe it.

    ------------------------------
    David Corliss
    Rapid Response Team / Predictive Analytics
    Ford Motor Company
    ------------------------------


    Original Message


  • 23.  RE: Statistical shenanigans - examples requested

    Posted 09-10-2015 11:38

    As noted by another poster, the unemployment rate as typically reported in monthly news items is not the only one that the BLS calculates or reports. However, none of the BLS figures is based on tabulating who's collecting unemployment insurance. You can see the BLS' definitions of being unemployed (the numerator of the U3 ratio) and being in the labor force (the denominator) here:

    How the Government Measures Unemployment

    <quote>

    Some people think that to get these figures on unemployment, the government uses the number of people collecting unemployment insurance (UI) benefits under state or federal government programs. But some people are still jobless when their benefits run out, and many more are not eligible at all or delay or never apply for benefits. So, quite clearly, UI information cannot be used as a source for complete information on the number of unemployed.

    </quote>

    You may not agree with how the BLS defines unemployment or U3, but it's unfair to accuse the Bureau of shenanigans.

    Although they do not figure in the BLS's own unemployment ratios, reports of initial claims of unemployment insurance and the number collecting UI are also available from the Department of Labor:

    Unemployment Insurance Weekly Claims Data, Employment & Training Administration (ETA) - U.S. Department of Labor

    http://workforcesecurity.doleta.gov/unemploy/claims_arch.asp

    The latest press release from the Office of Unemployment Insurance is here:

    http://workforcesecurity.doleta.gov/unemploy/claims_arch.asp

    >>Kathy


    ------------------------------
    Katherine Godfrey
    ------------------------------


    Original Message