ASA Connect

 View Only
Expand all | Collapse all

About Deficient Statistics in Political Polls in the Media

  • 1.  About Deficient Statistics in Political Polls in the Media

    Posted 09-30-2024 01:04

    Many political polls currently discussed in the media are deficient or incorrect from a statistical standpoint. This is not good for the country, nor for our profession (they give statistics a bad name). We, as professional statisticians, both individually and collectively, should counter and explain such deficiencies. For example: 

    (25) Post | Feed | LinkedIn

    (https://www.linkedin.com/feed/update/urn:li:activity:7243766222251528192/)

    Keep safe/jorge.



    ------------------------------
    Jorge L. Romeu
    Emeritus SUNY Faculty
    Adjunct Professor, Syracuse U.
    https://web.cortland.edu/romeu/
    https://www.researchgate.net/profile/Jorge_Romeu
    ------------------------------


  • 2.  RE: About Deficient Statistics in Political Polls in the Media

    Posted 10-01-2024 09:04

    Most, if not all, of the political polls have a more serious problem: extremely high rates of nonresponse. In recent years, response rates have been substantially below 10 percent. The polling organizations produce margins of error, though media organizations do not always report them. The polling organizations seldom disclose their response rates.

    The problem is that the nonresponse is unlikely to be random. Thus, the estimates calculated from the completed sample are likely to suffer from biases, whose size and direction are unknown. I am deeply skeptical that, even with careful design and sophisticated methods of analysis, the polling organizations can satisfactorily adjust for the biases. I would welcome convincing evidence to the contrary. Meanwhile, I have for some time advised friends and acquaintances to ignore the results of political polls.

    It is discouraging that serious journalists, who supposely base their work on facts, have not carefully examined the workings of political polls and reported the findings.



    ------------------------------
    David C. Hoaglin
    ------------------------------



  • 3.  RE: About Deficient Statistics in Political Polls in the Media

    Posted 10-01-2024 14:15

    You are absolutely right! Drop in survey sample size is very large. But in many cases, there is a good reason for it. 

    Sometimes, political opponents, the Media and/or celebrities portray very negatively contenders, candidates or political parties, using unfavorable comments or misrepresentations. Such negative portrayal may cause voters who support said candidates or parties to suppress their support, or refuse to participate in surveys, to avoid being "socially marginalized ", or suffer other types of hindrances. These actions produce a distorting effect on survey results, which are based on the total freedom of expression of sample subjects. When absent, this creates a type of bias that is not always accounted for by pollsters, and that we may call the "Violeta Chamorro Effect".

    Let's present a real-life example of the statistical effects of said Violeta Chamorro Effect.

    In the Nicaraguan presidential campaign of 1990, doña Violeta Chamorro, the widow of a well-known journalist murdered during the Somoza regime, was running against the incumbent Sandinista president Daniel Ortega, who headed the leftist, revolutionary government. The opposition party UNO, a coalition of fourteen groups across all political persuasions, campaigned on improving Nicaragua's economic downfall by supporting free enterprise, and ending the civil war between the government and the Contra Guerrillas. Sandinistas accused doña Violeta of trying to reverse the revolution, restoring capitalism and dependance on the United States.

    A poll taken at the time showed three times as many people supporting the Sandinistas (28%) than all of the opposition parties (9%); 59% did not identify with any political party. So, many, including the Ortega government, thought that the Sandinistas would win the election. To most everyone's surprise, however, the election was won by Mrs. Chamorro, who took 55% of the vote, to 41% for Ortega. How and why were the surveys ever so wrong?

    Perhaps the social pressure that existed at the time can explain this. Nicaraguan voters were concerned about possible social and political implications of stating that they were supporting opposition UNO, and the Chamorro candidate. So, they lied to the polls thus biasing their results.

    Under social pressure, some "undecided" voters may not want to disclose their real preference. But, once inside the voting booth, they may vote differently than expressed in polls. In a very close election such as this one, even if such voters represent 1% of the electorate, they may well decide the election. And surveys will have had very little, if any, real value.

    Thanks again for your valuable comment. Keep safe/jorge.



    ------------------------------
    Jorge L. Romeu
    Emeritus SUNY Faculty
    Adjunct Professor, Syracuse U.
    https://web.cortland.edu/romeu/
    https://www.researchgate.net/profile/Jorge_Romeu
    ------------------------------



  • 4.  RE: About Deficient Statistics in Political Polls in the Media

    Posted 10-02-2024 08:34

    In addition to all the important issues with polling previously mentioned -- I have also experienced a surge of fundraising spam disguised as polls. In the past couple of years, I have responded to a few phone calls or texts which seemed like they might plausibly be from legitimate polling organizations, but which quickly turned into partisan demands to donate to a specific political party.

    Is this just my experience, or are others seeing this too?

    Even though I am a statistician who works on survey sampling myself, I have stopped responding to nearly all survey and poll invitations. Unless the invitation clearly explains who is running the poll, and I can verify for myself that this is a legitimate organization, it's a waste of my time to participate.

    And unfortunately this seems to be out of our hands. No matter how well we statisticians can design our surveys, many people won't trust us if they are used to receiving fake spam "surveys" much more often than real ones.



    ------------------------------
    Jerzy Wieczorek
    Associate Professor
    Colby College
    ------------------------------



  • 5.  RE: About Deficient Statistics in Political Polls in the Media

    Posted 10-02-2024 11:14

    Excellent comment! The same has happened to me. I have also shied from replying, as it could well be another physhing expedition.

    I do not agree, however, that we cannot do anything about it. Individually, as professional statisticians, we can write to the media and provide some Tech information on the correct way to carry out surveys, and about how to interpret their results.

    Collectively, the ASA could put out some notice about these issues, for the benefit of the public at large, and of the political process.

    Thanks again for your comment. Keep safe./jorge.



    ------------------------------
    Jorge L. Romeu
    Emeritus SUNY Faculty
    Adjunct Professor, Syracuse U.
    https://web.cortland.edu/romeu/
    https://www.researchgate.net/profile/Jorge_Romeu
    ------------------------------



  • 6.  RE: About Deficient Statistics in Political Polls in the Media

    Posted 10-02-2024 14:39

    There was an article about polling methods possibly in Atlantic that I am unable to find. And as I -very vaguely- recall, the author (a pollster) wrote about the difficulty of actually obtaining random samples in their polls and that particular author seemed to suggest they used --convenience - type samples rather than random samples. 

    The only relevant link I could find was this one from Pew Research which mentions using panels

    https://www.pewresearch.org/methods/2023/04/19/how-public-polling-has-changed-in-the-21st-century/. 

    and AAPOR appears to have completed very thorough research on the topic of errors in polls

    https://aapor.org/wp-content/uploads/2022/11/AAPOR-Task-Force-on-2020-Pre-Election-Polling_Report-FNL.pdf

    I'm unclear on whether or how the pollsters may (or may not) have revised their polls in light of AAPOR' s research

    I am unable to find the article that seemed to suggest that a pollster might be using a convenience sample



    ------------------------------
    Chris Barker, Ph.D.
    Past Chair
    Statistical Consulting Section
    Consultant and
    Adjunct Associate Professor of Biostatistics
    www.barkerstats.com


    ---
    "In composition you have all the time you want to decide what to say in 15 seconds, in improvisation you have 15 seconds."
    -Steve Lacy
    ------------------------------



  • 7.  RE: About Deficient Statistics in Political Polls in the Media

    Posted 10-02-2024 14:55
      |   view attached

    Thank you, Chris, for your interesting sources. Prof. Ron Kennet just sent me the attached review paper, on surveys, that may be of interest to our conversation. Keep safe/jorge.



    ------------------------------
    Jorge L. Romeu
    Emeritus SUNY Faculty
    Adjunct Professor, Syracuse U.
    https://web.cortland.edu/romeu/
    https://www.researchgate.net/profile/Jorge_Romeu
    ------------------------------

    Attachment(s)

    pdf
    KenettEtAlARSA2017.pdf   302 KB 1 version


  • 8.  RE: About Deficient Statistics in Political Polls in the Media

    Posted 10-03-2024 09:44

    Jorge: most large media outlets (NYTimes, WashPost, CBS, etc.) have polling units with very sophisticated data analysts who are perfectly aware of the problems encountered nowadays with polls. And to add to Chris's comment, the American Association for Public Opinion Research (AAPOR), of which I am a member, has done a lot of outreach with the press to "educate" them abut polling and all the difficulties involved.

    That said, there has almost never been a "random" sample in the history of polling, going back to the 1930s (not to mention the Literary Digest polls). Perhaps, at one point, when random digit sampling came out, samples came close to being "random". But since the mid-70s response rates have been in free fall. As a result, a lot of complex statistical tools have been used to correct these essentially self-selected samples to make them look like (one hopes) random samples.

    Cheers - Dominic



    ------------------------------
    Dominic Lusinchi
    Independent researcher & consultant (retired)
    San Francisco, Calif.
    ------------------------------



  • 9.  RE: About Deficient Statistics in Political Polls in the Media

    Posted 10-03-2024 11:59

    Thanks to Dominick, Michael and Keith for their useful contributions to the discussion. Yes, it is true that we are "over-surveyed", that to some level (as in the case of Harris replacing Biden) polls show a trend, and that pollsters know of these problems and have tried to fix or counter them. The real problem is not the pollsters work, or even the lack of sample randomness, or the drastic drop of the sample sizes, but the way politicians, media and pundits mis-interpret these poll results. We, as statisticians and the ASA, as our professional guild, should take an active stand and educate the people about them.

    For example, during the Covd-19 pandemic, several political groups took the vaccination procedures to task, to support or oppose their political agendas. The Medical societies, and some of its members, worked hard to clarify the issues in dispute. That is what the institutions composing "civil society" are supposed to do (I think).

    Imagine that some groups take data, selected indiscriminately, develop regression models without doing residual analyses or check of model assumptions, and use its significant and non-significant untested coefficients to establish or propose government programs.

    Should we, statisticians, and our professional association clarify all the inconsistencies and model violations used, and question their results, for the benefit of the citizenry and the dignity of our profession? I would say this constitutes the core of our discussion.

    Thank you all. Keep safe/jorge.



    ------------------------------
    Jorge L. Romeu
    Emeritus SUNY Faculty
    Adjunct Professor, Syracuse U.
    https://web.cortland.edu/romeu/
    https://www.researchgate.net/profile/Jorge_Romeu
    ------------------------------



  • 10.  RE: About Deficient Statistics in Political Polls in the Media

    Posted 10-07-2024 07:46

    hi all:

    This was submitted to significance (this is the second submission). After an original revise and resubmit decision, I compromised with this submission. This was as far as I could possibly go without compromising my own statistical philosophies.  This resulted in an impasse, so it never was published.

    Best,

    Jon

    Is the Kansas Abortion Amendment Survey a Wake-up Call on Election Survey Science?

    In this essay, Jonathan Shuster asks statisticians to weigh in on the titled question.   Rather than advocating an answer one way or the other, it is being proposed as a debate topic in which the statistical community needs to become engaged. The 2022 Kansas state constitutional amendment will be used as a case study.

    The main reasons for keeping the statistical community involved include the following: (1) The public has a substantial interest in voter preference for a wide variety of political issues. If statisticians put themselves on the sidelines, less quantitative analysts are likely to increase the likelihood of adverse impact on the data analysis; (2) Statistician involvement can contribute to optimal design of surveys, getting the most from limited resources; (3) Statisticians are well suited to interpret the data and to make informed decisions about the scope of bias in the survey; (4) Many election surveys are conducted by groups with a vested interest in the outcome.  When results go against these interests, they can easily go unreported. Statisticians have long championed reporting non-significant results, and are well poised to act against such reporting bias.

    But there are also important reasons for staying on the sidelines, including: (1) Political polls have notoriously low participation rates (often below 20%).  See this link: Phone survey response rates decline again | Pew Research Center . Getting a true sampling frame is difficult. For example, even with voter registration rolls, you never know if the registered voter will actually cast a ballot; (2) Can a survey in of itself influence election results? As Cernat and Keusch 1 note, results of surveys can have an impact on behavior. For example, if a landslide prediction is publicly reported, will that in of itself adversely affect voter turnout as compared to a too close to call prediction? (3) We have all seen numerous situations where the survey result was completely inconsistent with the actual result.  This link Election polls are 95% confident but only 60% accurate, Berkeley Haas study finds | Haas News | Berkeley Haas

    gives us insight into how close to pure guesswork election surveys may be. (4) As a consulting statistician, if you believe the science behind a request for you to participate in any particular project  is faulty, you are within your rights to respectfully opt out.

    The 2022 Kansas Amendment Vote on Abortion Rights

    As an important case study, we shall compare results of a survey (held two weeks before the official vote)  and actual results of the vote on an August 2, 2022 state constitutional amendment in the US state of Kansas. A Yes vote would essentially remove abortion rights in the state.

    After the results were announced, this author was asked why the results seemed so peculiar in light of the "Co/Efficient" poll of 1500 likely voters seemed so divergent from the actual vote.  Of the 922,321 actual voters 378,466 voted Yes (favored removal of right to choose language from the state constitution), and 543,885 (59.0%) voted no (favored keeping the language in the constitution), while in the random poll of likely voters, 47% favored Yes, 43% favored No, while 10% voiced no opinion.  

    The retrospective question is: Were the poll results consistent with a true random sample of Kansas voters on this issue?   By  pessimistically projecting all unknown elements in the survey to favor survey responses of "Yes" , we shall demonstrate with very high certainty that if the survey was truly a random sample from the actual amendment vote, the survey results would have been implausible.

    We make two necessarily biased assumptions favoring a Yes answer to this question: (1) All individuals who would have voiced no opinion in the survey actually voted No in the actual election; (2) Because the Poll results were rounded to whole percentages, we presume the actual percentage results were as close as possible to a No outcome: 46.50% Yes, 43.50% No, and 10.0% No opinion.  Any other configuration of these two issues make the actual survey results even more implausible.

    Here are the data from the actual vote, cross-tabulated by conservative assumption (1) as to how voters would have responded to the survey: 

    Table 1: Conservative tabulation of actual voting percentages

    Voted Yes

    Favor Eliminate right to Choose

    Voted No

    Favor Retain right to Choose

    Would respond to Question

    41.0%

    49.0%

    Refuse to answer

    0.0%

    10.0%

    Total

    41.0%

    59.0%

    Of the population who would have responded to the poll, 49.0%/90.0%=54.44% would have reported a No vote. In the Co/Efficient poll of 1500 voters of the estimated 1350 who gave an opinion, 43.5/90=48.33% voted no, a discrepancy of 6.11%.  The standard error of this estimate is conservatively 0.5/sqrt(1350)=1.36%. The discrepancy is 6.11/1.36=4.49 standard errors.  The probability that a standard normal random variable exceeds 4.49 in absolute value is 7.1 in a million (0.0000071).

    Beyond a reasonable doubt, the survey results were not representative of the actual results.

    Discussion

    One question is whether the survey results influenced the final vote. Could it be that apparently being behind but within striking distance made the No leaning side more motivated to vote than the Yes leaning side?  Did the closeness of the survey results stimulate greater campaigning to try to bridge the deficit for the No leaning side, whereas the Yes leaning leadership became complacent?

    Survey results should be reported to a reasonable number of significant digits in reports made to the public. In a binomial survey of 1500, the standard error is at most 1.29%, and hence results should have had at least 3 to 4 significant digits not two.  It was fortunate that the discrepancy between the poll and election were robust enough to overcome the uncertainty caused by roundoff.

    In non-controversial settings, survey sampling can be very effective, provided that the investigators have a sampling frame and can retain identifiers to detect non-responders for second or even third contacts.  This essay's scope is restricted to pre-election polling including exit polling.

     

    Disclosure statement

    The author is self-funded and has no competing interest

    Reference

    1.       Cernat A. and Keusch F. (2022) Do surveys change behaviours? Significance 19(4), 10-11.

    Jonathan Shuster is Professor Emeritus, College of Medicine, University of Florida.  Homepage: https://hobi.med.ufl.edu/profile/shuster-jonathan/



    ------------------------------
    Jonathan Shuster
    ------------------------------



  • 11.  RE: About Deficient Statistics in Political Polls in the Media

    Posted 10-02-2024 15:14

    Two other factors to consider. First we are simply over surveyed. You cannot make a purchase on line,  go to a medical appointment, eat at a restaurant without being surveyed about how we did. This over surveying has to depress response rates.

    Second, political campaigns use what is termed push polls. These appear to be legitimate polls for the first few questions and then you will get a question like "do you approve of the fact that Senator XYZ has unnatural relations with barn animals?" So even if you are willing to  answer a legitimate poll you really are discouraged from doing so.

    I simply think the survey tool in many, many contexts has been ruined by these widespread malpractices.



    ------------------------------
    Michael Sack Elmaleh
    Principal
    Michael Sack Elmaleh CPA, CVA
    ------------------------------



  • 12.  RE: About Deficient Statistics in Political Polls in the Media

    Posted 10-03-2024 10:12

    The many criticisms made of these polls are well-founded. However, there is one sense in which they are useful: discerning trends. The same poll questions are used repeatedly from week to week, with the same biases and inconsistencies built in each time.  While these problems preclude accurate assessment of absolute levels of support, changes (e.g. Harris replacing Biden) can be assessed at least to some degree.



    ------------------------------
    Keith Ord
    Professor Emeritus
    Georgetown University
    ------------------------------