ASA Connect

 View Only
Expand all | Collapse all

About Deficient Statistics in Political Polls in the Media

  • 1.  About Deficient Statistics in Political Polls in the Media

    Posted 5 days ago

    Many political polls currently discussed in the media are deficient or incorrect from a statistical standpoint. This is not good for the country, nor for our profession (they give statistics a bad name). We, as professional statisticians, both individually and collectively, should counter and explain such deficiencies. For example: 

    (25) Post | Feed | LinkedIn

    (https://www.linkedin.com/feed/update/urn:li:activity:7243766222251528192/)

    Keep safe/jorge.



    ------------------------------
    Jorge L. Romeu
    Emeritus SUNY Faculty
    Adjunct Professor, Syracuse U.
    https://web.cortland.edu/romeu/
    https://www.researchgate.net/profile/Jorge_Romeu
    ------------------------------


  • 2.  RE: About Deficient Statistics in Political Polls in the Media

    Posted 4 days ago

    Most, if not all, of the political polls have a more serious problem: extremely high rates of nonresponse. In recent years, response rates have been substantially below 10 percent. The polling organizations produce margins of error, though media organizations do not always report them. The polling organizations seldom disclose their response rates.

    The problem is that the nonresponse is unlikely to be random. Thus, the estimates calculated from the completed sample are likely to suffer from biases, whose size and direction are unknown. I am deeply skeptical that, even with careful design and sophisticated methods of analysis, the polling organizations can satisfactorily adjust for the biases. I would welcome convincing evidence to the contrary. Meanwhile, I have for some time advised friends and acquaintances to ignore the results of political polls.

    It is discouraging that serious journalists, who supposely base their work on facts, have not carefully examined the workings of political polls and reported the findings.



    ------------------------------
    David C. Hoaglin
    ------------------------------



  • 3.  RE: About Deficient Statistics in Political Polls in the Media

    Posted 4 days ago

    You are absolutely right! Drop in survey sample size is very large. But in many cases, there is a good reason for it. 

    Sometimes, political opponents, the Media and/or celebrities portray very negatively contenders, candidates or political parties, using unfavorable comments or misrepresentations. Such negative portrayal may cause voters who support said candidates or parties to suppress their support, or refuse to participate in surveys, to avoid being "socially marginalized ", or suffer other types of hindrances. These actions produce a distorting effect on survey results, which are based on the total freedom of expression of sample subjects. When absent, this creates a type of bias that is not always accounted for by pollsters, and that we may call the "Violeta Chamorro Effect".

    Let's present a real-life example of the statistical effects of said Violeta Chamorro Effect.

    In the Nicaraguan presidential campaign of 1990, doña Violeta Chamorro, the widow of a well-known journalist murdered during the Somoza regime, was running against the incumbent Sandinista president Daniel Ortega, who headed the leftist, revolutionary government. The opposition party UNO, a coalition of fourteen groups across all political persuasions, campaigned on improving Nicaragua's economic downfall by supporting free enterprise, and ending the civil war between the government and the Contra Guerrillas. Sandinistas accused doña Violeta of trying to reverse the revolution, restoring capitalism and dependance on the United States.

    A poll taken at the time showed three times as many people supporting the Sandinistas (28%) than all of the opposition parties (9%); 59% did not identify with any political party. So, many, including the Ortega government, thought that the Sandinistas would win the election. To most everyone's surprise, however, the election was won by Mrs. Chamorro, who took 55% of the vote, to 41% for Ortega. How and why were the surveys ever so wrong?

    Perhaps the social pressure that existed at the time can explain this. Nicaraguan voters were concerned about possible social and political implications of stating that they were supporting opposition UNO, and the Chamorro candidate. So, they lied to the polls thus biasing their results.

    Under social pressure, some "undecided" voters may not want to disclose their real preference. But, once inside the voting booth, they may vote differently than expressed in polls. In a very close election such as this one, even if such voters represent 1% of the electorate, they may well decide the election. And surveys will have had very little, if any, real value.

    Thanks again for your valuable comment. Keep safe/jorge.



    ------------------------------
    Jorge L. Romeu
    Emeritus SUNY Faculty
    Adjunct Professor, Syracuse U.
    https://web.cortland.edu/romeu/
    https://www.researchgate.net/profile/Jorge_Romeu
    ------------------------------



  • 4.  RE: About Deficient Statistics in Political Polls in the Media

    Posted 3 days ago

    In addition to all the important issues with polling previously mentioned -- I have also experienced a surge of fundraising spam disguised as polls. In the past couple of years, I have responded to a few phone calls or texts which seemed like they might plausibly be from legitimate polling organizations, but which quickly turned into partisan demands to donate to a specific political party.

    Is this just my experience, or are others seeing this too?

    Even though I am a statistician who works on survey sampling myself, I have stopped responding to nearly all survey and poll invitations. Unless the invitation clearly explains who is running the poll, and I can verify for myself that this is a legitimate organization, it's a waste of my time to participate.

    And unfortunately this seems to be out of our hands. No matter how well we statisticians can design our surveys, many people won't trust us if they are used to receiving fake spam "surveys" much more often than real ones.



    ------------------------------
    Jerzy Wieczorek
    Associate Professor
    Colby College
    ------------------------------



  • 5.  RE: About Deficient Statistics in Political Polls in the Media

    Posted 3 days ago

    Excellent comment! The same has happened to me. I have also shied from replying, as it could well be another physhing expedition.

    I do not agree, however, that we cannot do anything about it. Individually, as professional statisticians, we can write to the media and provide some Tech information on the correct way to carry out surveys, and about how to interpret their results.

    Collectively, the ASA could put out some notice about these issues, for the benefit of the public at large, and of the political process.

    Thanks again for your comment. Keep safe./jorge.



    ------------------------------
    Jorge L. Romeu
    Emeritus SUNY Faculty
    Adjunct Professor, Syracuse U.
    https://web.cortland.edu/romeu/
    https://www.researchgate.net/profile/Jorge_Romeu
    ------------------------------



  • 6.  RE: About Deficient Statistics in Political Polls in the Media

    Posted 3 days ago

    There was an article about polling methods possibly in Atlantic that I am unable to find. And as I -very vaguely- recall, the author (a pollster) wrote about the difficulty of actually obtaining random samples in their polls and that particular author seemed to suggest they used --convenience - type samples rather than random samples. 

    The only relevant link I could find was this one from Pew Research which mentions using panels

    https://www.pewresearch.org/methods/2023/04/19/how-public-polling-has-changed-in-the-21st-century/. 

    and AAPOR appears to have completed very thorough research on the topic of errors in polls

    https://aapor.org/wp-content/uploads/2022/11/AAPOR-Task-Force-on-2020-Pre-Election-Polling_Report-FNL.pdf

    I'm unclear on whether or how the pollsters may (or may not) have revised their polls in light of AAPOR' s research

    I am unable to find the article that seemed to suggest that a pollster might be using a convenience sample



    ------------------------------
    Chris Barker, Ph.D.
    Past Chair
    Statistical Consulting Section
    Consultant and
    Adjunct Associate Professor of Biostatistics
    www.barkerstats.com


    ---
    "In composition you have all the time you want to decide what to say in 15 seconds, in improvisation you have 15 seconds."
    -Steve Lacy
    ------------------------------



  • 7.  RE: About Deficient Statistics in Political Polls in the Media

    Posted 3 days ago
      |   view attached

    Thank you, Chris, for your interesting sources. Prof. Ron Kennet just sent me the attached review paper, on surveys, that may be of interest to our conversation. Keep safe/jorge.



    ------------------------------
    Jorge L. Romeu
    Emeritus SUNY Faculty
    Adjunct Professor, Syracuse U.
    https://web.cortland.edu/romeu/
    https://www.researchgate.net/profile/Jorge_Romeu
    ------------------------------

    Attachment(s)

    pdf
    KenettEtAlARSA2017.pdf   302 KB 1 version


  • 8.  RE: About Deficient Statistics in Political Polls in the Media

    Posted 2 days ago

    Jorge: most large media outlets (NYTimes, WashPost, CBS, etc.) have polling units with very sophisticated data analysts who are perfectly aware of the problems encountered nowadays with polls. And to add to Chris's comment, the American Association for Public Opinion Research (AAPOR), of which I am a member, has done a lot of outreach with the press to "educate" them abut polling and all the difficulties involved.

    That said, there has almost never been a "random" sample in the history of polling, going back to the 1930s (not to mention the Literary Digest polls). Perhaps, at one point, when random digit sampling came out, samples came close to being "random". But since the mid-70s response rates have been in free fall. As a result, a lot of complex statistical tools have been used to correct these essentially self-selected samples to make them look like (one hopes) random samples.

    Cheers - Dominic



    ------------------------------
    Dominic Lusinchi
    Independent researcher & consultant (retired)
    San Francisco, Calif.
    ------------------------------



  • 9.  RE: About Deficient Statistics in Political Polls in the Media

    Posted 2 days ago

    Thanks to Dominick, Michael and Keith for their useful contributions to the discussion. Yes, it is true that we are "over-surveyed", that to some level (as in the case of Harris replacing Biden) polls show a trend, and that pollsters know of these problems and have tried to fix or counter them. The real problem is not the pollsters work, or even the lack of sample randomness, or the drastic drop of the sample sizes, but the way politicians, media and pundits mis-interpret these poll results. We, as statisticians and the ASA, as our professional guild, should take an active stand and educate the people about them.

    For example, during the Covd-19 pandemic, several political groups took the vaccination procedures to task, to support or oppose their political agendas. The Medical societies, and some of its members, worked hard to clarify the issues in dispute. That is what the institutions composing "civil society" are supposed to do (I think).

    Imagine that some groups take data, selected indiscriminately, develop regression models without doing residual analyses or check of model assumptions, and use its significant and non-significant untested coefficients to establish or propose government programs.

    Should we, statisticians, and our professional association clarify all the inconsistencies and model violations used, and question their results, for the benefit of the citizenry and the dignity of our profession? I would say this constitutes the core of our discussion.

    Thank you all. Keep safe/jorge.



    ------------------------------
    Jorge L. Romeu
    Emeritus SUNY Faculty
    Adjunct Professor, Syracuse U.
    https://web.cortland.edu/romeu/
    https://www.researchgate.net/profile/Jorge_Romeu
    ------------------------------



  • 10.  RE: About Deficient Statistics in Political Polls in the Media

    Posted 3 days ago

    Two other factors to consider. First we are simply over surveyed. You cannot make a purchase on line,  go to a medical appointment, eat at a restaurant without being surveyed about how we did. This over surveying has to depress response rates.

    Second, political campaigns use what is termed push polls. These appear to be legitimate polls for the first few questions and then you will get a question like "do you approve of the fact that Senator XYZ has unnatural relations with barn animals?" So even if you are willing to  answer a legitimate poll you really are discouraged from doing so.

    I simply think the survey tool in many, many contexts has been ruined by these widespread malpractices.



    ------------------------------
    Michael Sack Elmaleh
    Principal
    Michael Sack Elmaleh CPA, CVA
    ------------------------------



  • 11.  RE: About Deficient Statistics in Political Polls in the Media

    Posted 2 days ago

    The many criticisms made of these polls are well-founded. However, there is one sense in which they are useful: discerning trends. The same poll questions are used repeatedly from week to week, with the same biases and inconsistencies built in each time.  While these problems preclude accurate assessment of absolute levels of support, changes (e.g. Harris replacing Biden) can be assessed at least to some degree.



    ------------------------------
    Keith Ord
    Professor Emeritus
    Georgetown University
    ------------------------------