ASA Connect

Many political polls currently discussed in the media are deficient or incorrect from a statistical standpoint. This is not good for the country, nor for our profession (they give statistics a bad name). We, as professional statisticians, both individually and collectively, should counter and explain such deficiencies. For example: 

(25) Post | Feed | LinkedIn

(https://www.linkedin.com/feed/update/urn:li:activity:7243766222251528192/)

Keep safe/jorge.

Most, if not all, of the political polls have a more serious problem: extremely high rates of nonresponse. In recent years, response rates have been substantially below 10 percent. The polling organizations produce margins of error, though media organizations do not always report them. The polling organizations seldom disclose their response rates.

The problem is that the nonresponse is unlikely to be random. Thus, the estimates calculated from the completed sample are likely to suffer from biases, whose size and direction are unknown. I am deeply skeptical that, even with careful design and sophisticated methods of analysis, the polling organizations can satisfactorily adjust for the biases. I would welcome convincing evidence to the contrary. Meanwhile, I have for some time advised friends and acquaintances to ignore the results of political polls.

It is discouraging that serious journalists, who supposely base their work on facts, have not carefully examined the workings of political polls and reported the findings.

Many political polls currently discussed in the media are deficient or incorrect from a statistical standpoint. This is not good for the country, nor for our profession (they give statistics a bad name). We, as professional statisticians, both individually and collectively, should counter and explain such deficiencies. For example: 

(25) Post | Feed | LinkedIn

(https://www.linkedin.com/feed/update/urn:li:activity:7243766222251528192/)

Keep safe/jorge.

You are absolutely right! Drop in survey sample size is very large. But in many cases, there is a good reason for it. 

Sometimes, political opponents, the Media and/or celebrities portray very negatively contenders, candidates or political parties, using unfavorable comments or misrepresentations. Such negative portrayal may cause voters who support said candidates or parties to suppress their support, or refuse to participate in surveys, to avoid being "socially marginalized ", or suffer other types of hindrances. These actions produce a distorting effect on survey results, which are based on the total freedom of expression of sample subjects. When absent, this creates a type of bias that is not always accounted for by pollsters, and that we may call the "Violeta Chamorro Effect".

Let's present a real-life example of the statistical effects of said Violeta Chamorro Effect.

In the Nicaraguan presidential campaign of 1990, doña Violeta Chamorro, the widow of a well-known journalist murdered during the Somoza regime, was running against the incumbent Sandinista president Daniel Ortega, who headed the leftist, revolutionary government. The opposition party UNO, a coalition of fourteen groups across all political persuasions, campaigned on improving Nicaragua's economic downfall by supporting free enterprise, and ending the civil war between the government and the Contra Guerrillas. Sandinistas accused doña Violeta of trying to reverse the revolution, restoring capitalism and dependance on the United States.

A poll taken at the time showed three times as many people supporting the Sandinistas (28%) than all of the opposition parties (9%); 59% did not identify with any political party. So, many, including the Ortega government, thought that the Sandinistas would win the election. To most everyone's surprise, however, the election was won by Mrs. Chamorro, who took 55% of the vote, to 41% for Ortega. How and why were the surveys ever so wrong?

Perhaps the social pressure that existed at the time can explain this. Nicaraguan voters were concerned about possible social and political implications of stating that they were supporting opposition UNO, and the Chamorro candidate. So, they lied to the polls thus biasing their results.

Under social pressure, some "undecided" voters may not want to disclose their real preference. But, once inside the voting booth, they may vote differently than expressed in polls. In a very close election such as this one, even if such voters represent 1% of the electorate, they may well decide the election. And surveys will have had very little, if any, real value.

Thanks again for your valuable comment. Keep safe/jorge.

Most, if not all, of the political polls have a more serious problem: extremely high rates of nonresponse. In recent years, response rates have been substantially below 10 percent. The polling organizations produce margins of error, though media organizations do not always report them. The polling organizations seldom disclose their response rates.

The problem is that the nonresponse is unlikely to be random. Thus, the estimates calculated from the completed sample are likely to suffer from biases, whose size and direction are unknown. I am deeply skeptical that, even with careful design and sophisticated methods of analysis, the polling organizations can satisfactorily adjust for the biases. I would welcome convincing evidence to the contrary. Meanwhile, I have for some time advised friends and acquaintances to ignore the results of political polls.

It is discouraging that serious journalists, who supposely base their work on facts, have not carefully examined the workings of political polls and reported the findings.

Many political polls currently discussed in the media are deficient or incorrect from a statistical standpoint. This is not good for the country, nor for our profession (they give statistics a bad name). We, as professional statisticians, both individually and collectively, should counter and explain such deficiencies. For example: 

(25) Post | Feed | LinkedIn

(https://www.linkedin.com/feed/update/urn:li:activity:7243766222251528192/)

Keep safe/jorge.

In addition to all the important issues with polling previously mentioned -- I have also experienced a surge of fundraising spam disguised as polls. In the past couple of years, I have responded to a few phone calls or texts which seemed like they might plausibly be from legitimate polling organizations, but which quickly turned into partisan demands to donate to a specific political party.

Is this just my experience, or are others seeing this too?

Even though I am a statistician who works on survey sampling myself, I have stopped responding to nearly all survey and poll invitations. Unless the invitation clearly explains who is running the poll, and I can verify for myself that this is a legitimate organization, it's a waste of my time to participate.

And unfortunately this seems to be out of our hands. No matter how well we statisticians can design our surveys, many people won't trust us if they are used to receiving fake spam "surveys" much more often than real ones.

You are absolutely right! Drop in survey sample size is very large. But in many cases, there is a good reason for it. 

Sometimes, political opponents, the Media and/or celebrities portray very negatively contenders, candidates or political parties, using unfavorable comments or misrepresentations. Such negative portrayal may cause voters who support said candidates or parties to suppress their support, or refuse to participate in surveys, to avoid being "socially marginalized ", or suffer other types of hindrances. These actions produce a distorting effect on survey results, which are based on the total freedom of expression of sample subjects. When absent, this creates a type of bias that is not always accounted for by pollsters, and that we may call the "Violeta Chamorro Effect".

Let's present a real-life example of the statistical effects of said Violeta Chamorro Effect.

In the Nicaraguan presidential campaign of 1990, doña Violeta Chamorro, the widow of a well-known journalist murdered during the Somoza regime, was running against the incumbent Sandinista president Daniel Ortega, who headed the leftist, revolutionary government. The opposition party UNO, a coalition of fourteen groups across all political persuasions, campaigned on improving Nicaragua's economic downfall by supporting free enterprise, and ending the civil war between the government and the Contra Guerrillas. Sandinistas accused doña Violeta of trying to reverse the revolution, restoring capitalism and dependance on the United States.

A poll taken at the time showed three times as many people supporting the Sandinistas (28%) than all of the opposition parties (9%); 59% did not identify with any political party. So, many, including the Ortega government, thought that the Sandinistas would win the election. To most everyone's surprise, however, the election was won by Mrs. Chamorro, who took 55% of the vote, to 41% for Ortega. How and why were the surveys ever so wrong?

Perhaps the social pressure that existed at the time can explain this. Nicaraguan voters were concerned about possible social and political implications of stating that they were supporting opposition UNO, and the Chamorro candidate. So, they lied to the polls thus biasing their results.

Under social pressure, some "undecided" voters may not want to disclose their real preference. But, once inside the voting booth, they may vote differently than expressed in polls. In a very close election such as this one, even if such voters represent 1% of the electorate, they may well decide the election. And surveys will have had very little, if any, real value.

Thanks again for your valuable comment. Keep safe/jorge.

Most, if not all, of the political polls have a more serious problem: extremely high rates of nonresponse. In recent years, response rates have been substantially below 10 percent. The polling organizations produce margins of error, though media organizations do not always report them. The polling organizations seldom disclose their response rates.

The problem is that the nonresponse is unlikely to be random. Thus, the estimates calculated from the completed sample are likely to suffer from biases, whose size and direction are unknown. I am deeply skeptical that, even with careful design and sophisticated methods of analysis, the polling organizations can satisfactorily adjust for the biases. I would welcome convincing evidence to the contrary. Meanwhile, I have for some time advised friends and acquaintances to ignore the results of political polls.

It is discouraging that serious journalists, who supposely base their work on facts, have not carefully examined the workings of political polls and reported the findings.

Many political polls currently discussed in the media are deficient or incorrect from a statistical standpoint. This is not good for the country, nor for our profession (they give statistics a bad name). We, as professional statisticians, both individually and collectively, should counter and explain such deficiencies. For example: 

(25) Post | Feed | LinkedIn

(https://www.linkedin.com/feed/update/urn:li:activity:7243766222251528192/)

Keep safe/jorge.

Many political polls currently discussed in the media are deficient or incorrect from a statistical standpoint. This is not good for the country, nor for our profession (they give statistics a bad name). We, as professional statisticians, both individually and collectively, should counter and explain such deficiencies. For example: 

(25) Post | Feed | LinkedIn

(https://www.linkedin.com/feed/update/urn:li:activity:7243766222251528192/)

Keep safe/jorge.

There was an article about polling methods possibly in Atlantic that I am unable to find. And as I -very vaguely- recall, the author (a pollster) wrote about the difficulty of actually obtaining random samples in their polls and that particular author seemed to suggest they used --convenience - type samples rather than random samples. 

The only relevant link I could find was this one from Pew Research which mentions using panels

https://www.pewresearch.org/methods/2023/04/19/how-public-polling-has-changed-in-the-21st-century/. 

and AAPOR appears to have completed very thorough research on the topic of errors in polls

https://aapor.org/wp-content/uploads/2022/11/AAPOR-Task-Force-on-2020-Pre-Election-Polling_Report-FNL.pdf

I'm unclear on whether or how the pollsters may (or may not) have revised their polls in light of AAPOR' s research

I am unable to find the article that seemed to suggest that a pollster might be using a convenience sample

Many political polls currently discussed in the media are deficient or incorrect from a statistical standpoint. This is not good for the country, nor for our profession (they give statistics a bad name). We, as professional statisticians, both individually and collectively, should counter and explain such deficiencies. For example: 

(25) Post | Feed | LinkedIn

(https://www.linkedin.com/feed/update/urn:li:activity:7243766222251528192/)

Keep safe/jorge.

Thank you, Chris, for your interesting sources. Prof. Ron Kennet just sent me the attached review paper, on surveys, that may be of interest to our conversation. Keep safe/jorge.

There was an article about polling methods possibly in Atlantic that I am unable to find. And as I -very vaguely- recall, the author (a pollster) wrote about the difficulty of actually obtaining random samples in their polls and that particular author seemed to suggest they used --convenience - type samples rather than random samples. 

The only relevant link I could find was this one from Pew Research which mentions using panels

https://www.pewresearch.org/methods/2023/04/19/how-public-polling-has-changed-in-the-21st-century/. 

and AAPOR appears to have completed very thorough research on the topic of errors in polls

https://aapor.org/wp-content/uploads/2022/11/AAPOR-Task-Force-on-2020-Pre-Election-Polling_Report-FNL.pdf

I'm unclear on whether or how the pollsters may (or may not) have revised their polls in light of AAPOR' s research

I am unable to find the article that seemed to suggest that a pollster might be using a convenience sample

Many political polls currently discussed in the media are deficient or incorrect from a statistical standpoint. This is not good for the country, nor for our profession (they give statistics a bad name). We, as professional statisticians, both individually and collectively, should counter and explain such deficiencies. For example: 

(25) Post | Feed | LinkedIn

(https://www.linkedin.com/feed/update/urn:li:activity:7243766222251528192/)

Keep safe/jorge.

Jorge: most large media outlets (NYTimes, WashPost, CBS, etc.) have polling units with very sophisticated data analysts who are perfectly aware of the problems encountered nowadays with polls. And to add to Chris's comment, the American Association for Public Opinion Research (AAPOR), of which I am a member, has done a lot of outreach with the press to "educate" them abut polling and all the difficulties involved.

That said, there has almost never been a "random" sample in the history of polling, going back to the 1930s (not to mention the Literary Digest polls). Perhaps, at one point, when random digit sampling came out, samples came close to being "random". But since the mid-70s response rates have been in free fall. As a result, a lot of complex statistical tools have been used to correct these essentially self-selected samples to make them look like (one hopes) random samples.

Cheers - Dominic

Thank you, Chris, for your interesting sources. Prof. Ron Kennet just sent me the attached review paper, on surveys, that may be of interest to our conversation. Keep safe/jorge.

There was an article about polling methods possibly in Atlantic that I am unable to find. And as I -very vaguely- recall, the author (a pollster) wrote about the difficulty of actually obtaining random samples in their polls and that particular author seemed to suggest they used --convenience - type samples rather than random samples. 

The only relevant link I could find was this one from Pew Research which mentions using panels

https://www.pewresearch.org/methods/2023/04/19/how-public-polling-has-changed-in-the-21st-century/. 

and AAPOR appears to have completed very thorough research on the topic of errors in polls

https://aapor.org/wp-content/uploads/2022/11/AAPOR-Task-Force-on-2020-Pre-Election-Polling_Report-FNL.pdf

I'm unclear on whether or how the pollsters may (or may not) have revised their polls in light of AAPOR' s research

I am unable to find the article that seemed to suggest that a pollster might be using a convenience sample

Many political polls currently discussed in the media are deficient or incorrect from a statistical standpoint. This is not good for the country, nor for our profession (they give statistics a bad name). We, as professional statisticians, both individually and collectively, should counter and explain such deficiencies. For example: 

(25) Post | Feed | LinkedIn

(https://www.linkedin.com/feed/update/urn:li:activity:7243766222251528192/)

Keep safe/jorge.

Many political polls currently discussed in the media are deficient or incorrect from a statistical standpoint. This is not good for the country, nor for our profession (they give statistics a bad name). We, as professional statisticians, both individually and collectively, should counter and explain such deficiencies. For example: 

(25) Post | Feed | LinkedIn

(https://www.linkedin.com/feed/update/urn:li:activity:7243766222251528192/)

Keep safe/jorge.

Two other factors to consider. First we are simply over surveyed. You cannot make a purchase on line,  go to a medical appointment, eat at a restaurant without being surveyed about how we did. This over surveying has to depress response rates.

Second, political campaigns use what is termed push polls. These appear to be legitimate polls for the first few questions and then you will get a question like "do you approve of the fact that Senator XYZ has unnatural relations with barn animals?" So even if you are willing to  answer a legitimate poll you really are discouraged from doing so.

I simply think the survey tool in many, many contexts has been ruined by these widespread malpractices.

Many political polls currently discussed in the media are deficient or incorrect from a statistical standpoint. This is not good for the country, nor for our profession (they give statistics a bad name). We, as professional statisticians, both individually and collectively, should counter and explain such deficiencies. For example: 

(25) Post | Feed | LinkedIn

(https://www.linkedin.com/feed/update/urn:li:activity:7243766222251528192/)

Keep safe/jorge.