hi all:
This was submitted to significance (this is the second submission). After an original revise and resubmit decision, I compromised with this submission. This was as far as I could possibly go without compromising my own statistical philosophies. This resulted in an impasse, so it never was published.
Best,
Jon
Is the Kansas Abortion Amendment Survey a Wake-up Call on Election Survey Science?
In this essay, Jonathan Shuster asks statisticians to weigh in on the titled question. Rather than advocating an answer one way or the other, it is being proposed as a debate topic in which the statistical community needs to become engaged. The 2022 Kansas state constitutional amendment will be used as a case study.
The main reasons for keeping the statistical community involved include the following: (1) The public has a substantial interest in voter preference for a wide variety of political issues. If statisticians put themselves on the sidelines, less quantitative analysts are likely to increase the likelihood of adverse impact on the data analysis; (2) Statistician involvement can contribute to optimal design of surveys, getting the most from limited resources; (3) Statisticians are well suited to interpret the data and to make informed decisions about the scope of bias in the survey; (4) Many election surveys are conducted by groups with a vested interest in the outcome. When results go against these interests, they can easily go unreported. Statisticians have long championed reporting non-significant results, and are well poised to act against such reporting bias.
But there are also important reasons for staying on the sidelines, including: (1) Political polls have notoriously low participation rates (often below 20%). See this link: Phone survey response rates decline again | Pew Research Center . Getting a true sampling frame is difficult. For example, even with voter registration rolls, you never know if the registered voter will actually cast a ballot; (2) Can a survey in of itself influence election results? As Cernat and Keusch 1 note, results of surveys can have an impact on behavior. For example, if a landslide prediction is publicly reported, will that in of itself adversely affect voter turnout as compared to a too close to call prediction? (3) We have all seen numerous situations where the survey result was completely inconsistent with the actual result. This link Election polls are 95% confident but only 60% accurate, Berkeley Haas study finds | Haas News | Berkeley Haas
gives us insight into how close to pure guesswork election surveys may be. (4) As a consulting statistician, if you believe the science behind a request for you to participate in any particular project is faulty, you are within your rights to respectfully opt out.
The 2022 Kansas Amendment Vote on Abortion Rights
As an important case study, we shall compare results of a survey (held two weeks before the official vote) and actual results of the vote on an August 2, 2022 state constitutional amendment in the US state of Kansas. A Yes vote would essentially remove abortion rights in the state.
After the results were announced, this author was asked why the results seemed so peculiar in light of the "Co/Efficient" poll of 1500 likely voters seemed so divergent from the actual vote. Of the 922,321 actual voters 378,466 voted Yes (favored removal of right to choose language from the state constitution), and 543,885 (59.0%) voted no (favored keeping the language in the constitution), while in the random poll of likely voters, 47% favored Yes, 43% favored No, while 10% voiced no opinion.
The retrospective question is: Were the poll results consistent with a true random sample of Kansas voters on this issue? By pessimistically projecting all unknown elements in the survey to favor survey responses of "Yes" , we shall demonstrate with very high certainty that if the survey was truly a random sample from the actual amendment vote, the survey results would have been implausible.
We make two necessarily biased assumptions favoring a Yes answer to this question: (1) All individuals who would have voiced no opinion in the survey actually voted No in the actual election; (2) Because the Poll results were rounded to whole percentages, we presume the actual percentage results were as close as possible to a No outcome: 46.50% Yes, 43.50% No, and 10.0% No opinion. Any other configuration of these two issues make the actual survey results even more implausible.
Here are the data from the actual vote, cross-tabulated by conservative assumption (1) as to how voters would have responded to the survey:
Table 1: Conservative tabulation of actual voting percentages
|
Voted Yes
Favor Eliminate right to Choose
|
Voted No
Favor Retain right to Choose
|
Would respond to Question
|
41.0%
|
49.0%
|
Refuse to answer
|
0.0%
|
10.0%
|
Total
|
41.0%
|
59.0%
|
Of the population who would have responded to the poll, 49.0%/90.0%=54.44% would have reported a No vote. In the Co/Efficient poll of 1500 voters of the estimated 1350 who gave an opinion, 43.5/90=48.33% voted no, a discrepancy of 6.11%. The standard error of this estimate is conservatively 0.5/sqrt(1350)=1.36%. The discrepancy is 6.11/1.36=4.49 standard errors. The probability that a standard normal random variable exceeds 4.49 in absolute value is 7.1 in a million (0.0000071).
Beyond a reasonable doubt, the survey results were not representative of the actual results.
Discussion
One question is whether the survey results influenced the final vote. Could it be that apparently being behind but within striking distance made the No leaning side more motivated to vote than the Yes leaning side? Did the closeness of the survey results stimulate greater campaigning to try to bridge the deficit for the No leaning side, whereas the Yes leaning leadership became complacent?
Survey results should be reported to a reasonable number of significant digits in reports made to the public. In a binomial survey of 1500, the standard error is at most 1.29%, and hence results should have had at least 3 to 4 significant digits not two. It was fortunate that the discrepancy between the poll and election were robust enough to overcome the uncertainty caused by roundoff.
In non-controversial settings, survey sampling can be very effective, provided that the investigators have a sampling frame and can retain identifiers to detect non-responders for second or even third contacts. This essay's scope is restricted to pre-election polling including exit polling.
Disclosure statement
The author is self-funded and has no competing interest
Reference
1. Cernat A. and Keusch F. (2022) Do surveys change behaviours? Significance 19(4), 10-11.
Jonathan Shuster is Professor Emeritus, College of Medicine, University of Florida. Homepage: https://hobi.med.ufl.edu/profile/shuster-jonathan/
------------------------------
Jonathan Shuster
------------------------------
Original Message:
Sent: 10-03-2024 09:44
From: Dominic Lusinchi
Subject: About Deficient Statistics in Political Polls in the Media
Jorge: most large media outlets (NYTimes, WashPost, CBS, etc.) have polling units with very sophisticated data analysts who are perfectly aware of the problems encountered nowadays with polls. And to add to Chris's comment, the American Association for Public Opinion Research (AAPOR), of which I am a member, has done a lot of outreach with the press to "educate" them abut polling and all the difficulties involved.
That said, there has almost never been a "random" sample in the history of polling, going back to the 1930s (not to mention the Literary Digest polls). Perhaps, at one point, when random digit sampling came out, samples came close to being "random". But since the mid-70s response rates have been in free fall. As a result, a lot of complex statistical tools have been used to correct these essentially self-selected samples to make them look like (one hopes) random samples.
Cheers - Dominic
------------------------------
Dominic Lusinchi
Independent researcher & consultant (retired)
San Francisco, Calif.
Original Message:
Sent: 10-02-2024 14:55
From: Jorge Romeu
Subject: About Deficient Statistics in Political Polls in the Media
Thank you, Chris, for your interesting sources. Prof. Ron Kennet just sent me the attached review paper, on surveys, that may be of interest to our conversation. Keep safe/jorge.
------------------------------
Jorge L. Romeu
Emeritus SUNY Faculty
Adjunct Professor, Syracuse U.
https://web.cortland.edu/romeu/
https://www.researchgate.net/profile/Jorge_Romeu
Original Message:
Sent: 10-02-2024 14:38
From: Chris Barker
Subject: About Deficient Statistics in Political Polls in the Media
There was an article about polling methods possibly in Atlantic that I am unable to find. And as I -very vaguely- recall, the author (a pollster) wrote about the difficulty of actually obtaining random samples in their polls and that particular author seemed to suggest they used --convenience - type samples rather than random samples.
The only relevant link I could find was this one from Pew Research which mentions using panels
https://www.pewresearch.org/methods/2023/04/19/how-public-polling-has-changed-in-the-21st-century/.
and AAPOR appears to have completed very thorough research on the topic of errors in polls
https://aapor.org/wp-content/uploads/2022/11/AAPOR-Task-Force-on-2020-Pre-Election-Polling_Report-FNL.pdf
I'm unclear on whether or how the pollsters may (or may not) have revised their polls in light of AAPOR' s research
I am unable to find the article that seemed to suggest that a pollster might be using a convenience sample
------------------------------
Chris Barker, Ph.D.
Past Chair
Statistical Consulting Section
Consultant and
Adjunct Associate Professor of Biostatistics
www.barkerstats.com
---
"In composition you have all the time you want to decide what to say in 15 seconds, in improvisation you have 15 seconds."
-Steve Lacy
Original Message:
Sent: 09-30-2024 01:03
From: Jorge Romeu
Subject: About Deficient Statistics in Political Polls in the Media
Many political polls currently discussed in the media are deficient or incorrect from a statistical standpoint. This is not good for the country, nor for our profession (they give statistics a bad name). We, as professional statisticians, both individually and collectively, should counter and explain such deficiencies. For example:
(25) Post | Feed | LinkedIn
(https://www.linkedin.com/feed/update/urn:li:activity:7243766222251528192/)
Keep safe/jorge.
------------------------------
Jorge L. Romeu
Emeritus SUNY Faculty
Adjunct Professor, Syracuse U.
https://web.cortland.edu/romeu/
https://www.researchgate.net/profile/Jorge_Romeu
------------------------------