Statisticians should refuse compensation and never participate in election polls. We have known the truth about them for decades. There is no such thing as a "margin of error" since it deals with sampling error. Bias in these polls which cannot be effectively eliminated are a much larger factor. Today, political polls are lucky to get better than a 10% response rate. These polls may be used for reasons other than calling an outcome or estimating preferences. If 70% of the respondents say they oppose a new tax, you can claim there is dissention, but cannot claim a true majority. The apathetic do not talk to the pollsters, can be very different creatures for the responders.
My recent Google poll of members of the ASA Consulting Section had 52/1558 responding to one of my 4 waves of requests. This was on a topic of vital interest to our group, namely about inappropriate requests for analysis from our clients. This was not intended to be a scientific survey, but it illustrates the problem in our own house. Nonetheless, I am reporting the survey with the caveat that the response rate was exceedingly low. It does illustrate diversity on the responders, as well as apathy in many to the question of our responsibility when we face such requests.
Survey Sampling has a role, but not in predicting elections, rating performance of those in office, or in straw votes on candidates prior to elections.
Original Message:
Sent: 12-25-2018 22:18
From: Jonathan Siegel
Subject: Why electoral polls fail
I think a key problem is one in which the statistics community has repeatedly failed, failure to take into account correlated errors.
Most election polls assume individuals, localities, states, etc. are independent, so that if the chance of being wrong in calling any one election is being independent of the chance of being wrong in calling any other. Assuming this makes the estimated probability of calling a large number of elections incorrectly very low.
We have seen models based on this assumption crash and burn before. An example is the models used to estimate the rate of mortgage defaults preceding the financial collapse of 2008. These models also assumed that the chance of every mortgage defaulting was independent of every other.
independence of model errors is rarely a good assumption in observational social science models. People are interconnected and subject to unmeasured effects. When our model is wrong, it tends to be wrong systematically. A recession or downturn in housing increases everyone's chance of default simultaneously. A new political movement makes prior election assumptions simultaneously less reliable in every election.
This means that assurances of reliability based on simplistic assumptions are rarely valid. And it is astonishing to see the sorts of simplistic assumptions taught in school - assumptions made only to make hand-calculation easier for students being taught how to calculate by hand - persisting into professional work. Independence of one of these. Correlated error problems are difficult and make calculations hard.
I can't help but think that one of the reasons we have so often focused on the easy-to-calculate problems over what the real world has to offer us has been the discipline's insistence on mastery of calculation as the essential skill, with the mechanics of the mathematical calculations deemed the essential skill that distinguishes deep from shallow knowledge.
This doesn't just mean that there isn't time to teach what sorts of assumptions are appropriate in real-world settings. It means a deep understanding of what sorts of assumptions are and aren't appropriate, an ability to look at a real-world situation often simply isn't valued. The student who can hand-calculate more quickly has, all too often, been preferred to the student who can look at a real-world situation, compare it to a model, and tell you where the model's statistical assumptions are and are not likely to break down. We teach in a way that values the student who can start with known problems and solve them faster or more elegantly, and we don't value the student who can look at a complex reality and assess what the problem is.
This in turn has resulted in the continuation of sandbox assumptions designed to support hand calculation into professional work in the real world. No wonder the statistics profession keeps failing society in cases like this.
------------------------------
Jonathan Siegel
Deputy Director Clinical Statistics
Original Message:
Sent: 12-19-2018 07:02
From: Edmundo Pimentel
Subject: Why electoral polls fail
The design of the investigations by sampling includes the type of sampling to be used, the selection of the sample and the design of the measurement instrument. The main objective of the design of the sample is the representativeness of the selected individuals with respect to the population to be investigated.
The problem is that following the design methodologies of the sample only a high probability of it being representative is obtained, but you never get to know how representative it turned out to be, therefore it could not be representative. When several samples are taken in the follow-up of the electoral contest, this problem is minimized, but it does not disappear.
In the case of the design of the measurement instrument, it is intended to measure the psychological construct of the attitude of favoring a candidate with the vote. This psychological construct has become more complex and in some cases is no longer one-dimensional, so the instruments that are traditionally used to measure it are not efficient.
When the psychological construct is not one-dimensional, the fundamental precepts of psychometrics must be applied to obtain an accurate measurement, which is why it is highly recommended to use the Factorial Analysis technique.
------------------------------
Edmundo Pimentel
Teacher
Calle Los Laboratorios
Original Message:
Sent: 12-05-2018 14:34
From: Edmundo Pimentel
Subject: Why electoral polls fail
Dear colleagues:
Thank you for your contributions, I know it is a controversial issue; the idea of bringing it to the forum is not to evidence of errors but seek solutions. It is obvious that the problem exists and its statistical regularity is sufficient evidence to take care of him, which is necessary because it affects the credibility of the guild.
I hope that in this exchange of opinions, a proposal will emerge that will help mitigate the flaws in the electoral surveys.
Edmundo
------------------------------
Edmundo Pimentel
Teacher
Calle Los Laboratorios
Original Message:
Sent: 12-04-2018 12:46
From: Herman Autore
Subject: Why electoral polls fail
Professor Pimentel,
Thank you for such an interesting description of current events in international poll-taking. I had heard in US media that the result of introspection by polling firms after the 2016 US elections concluded that the bad prediction was the result of underfunded local polling. From what they said, the presidential election prediction is based on amalgamating all local predictions nation-wide. The problem then stemmed from poor funding, they claim, in rural polling. For example, more urban areas had enough funding to poll enough people. Rural areas lacked the funding to get accurate measurements. The evidence they point to is that after the election local urban predictions were more accurate than local rural predictions. The argument is convincing, but begs the question as to how so many rural polls will get the resources they need in future elections. See How polling has changed since the 2016 election, from PBS News Hour.
Nate Silver from FiveThirtyEight.com has a more nuanced explanation, however, one that doesn't even mention funding:
They also suggest there are real shortcomings in how American politics are covered, including pervasive groupthink among media elites, an unhealthy obsession with the insider's view of politics, a lack of analytical rigor, a failure to appreciate uncertainty, a sluggishness to self-correct when new evidence contradicts pre-existing beliefs, and a narrow viewpoint that lacks perspective from the longer arc of American history.
Indeed, he says that the issue is so complicated that it has required him to write a series of articles to address it. See The Real Story Of 2016 from FiveThirtyEight.
------------------------------
Herman Autore
Florida State University
Original Message:
Sent: 12-03-2018 13:45
From: Edmundo Pimentel
Subject: Why electoral polls fail
Lo que hemos visto en los últimos años es que las compañías de demoscopia han fracasado por completo en algunos de sus pronósticos. Los errores van de lo que sucedió en Inglaterra con David Cameron al Brexit. También se equivocaron en Argentina, Colombia, España, Panamá y Perú, entre otros casos. Finalmente, el error se hizo explícito en las elecciones de los Estados Unidos, donde se demostró un rotundo fracaso.
En Panamá encuesta tras encuesta Varela apareció en tercer lugar. Pero en la votación el vicepresidente Juan Carlos Varela alcanzó el triunfo. ¿Por qué las encuestas de opinión fallaron constantemente? El resultado de la elección presidencial panameña, asociado con los recientes resultados electorales en Honduras, Costa Rica y El Salvador, requiere un trabajo exhaustivo e introspectivo por parte de las firmas encuestadoras.
Las últimas elecciones presidenciales en Perú estuvieron marcadas por el mismo fracaso previsto. A lo largo de la campaña electoral, la candidata Keiko Fujimori ganó en la encuesta, pero finalmente, en la segunda ronda de las elecciones del 5 de junio, el actual presidente, Pedro Pablo Kuczynski, ganó.
En Ecuador, los encuestadores fallaron en casi todos los resultados que publicaron: tanto en las encuestas de salida (ninguno tuvo éxito en la primera y la segunda ronda al mismo tiempo), como en encuestas anteriores, los resultados no fueron convergentes.
Las encuestas en Costa Rica predijeron dos cosas: que las elecciones presidenciales serían ganadas por el evangélico Fabricio Alvarado y que habría una gran abstención. Finalmente, el partido gobernante Carlos Alvarado ganó y más personas votaron que en la primera ronda.
Las encuestas en Chile no pudieron predecir el resultado de las elecciones presidenciales. Aunque cada encuesta tuvo un énfasis diferente, las fallas principales se encuentran en la sobreestimación de los votos de Sebastián Piñera y en la subestimación de los votos de Beatriz Sánchez. "Reconocemos humildemente que no fuimos precisos al estimar el voto en particular de Sebastián Piñera y Beatriz Sánchez", admitió el Centro de Estudios Públicos (CPE), que anunció que trabajará en la búsqueda del instrumento "más preciso" posible.
En el caso de Argentina, las encuestas anteriores indicaron que habrá diferencias de hasta 10 puntos de ventaja para Daniel Scioli sobre Mauricio Macri, "pero esta diferencia terminó siendo de dos puntos, que supera el margen de error de las encuestas", explica Guillermo Cumsille, profesor de opinión pública de la Facultad de Ciencias Sociales de la U. de Chile y socio de la consultora Demoscópica.
En España, la noche electoral del 26-J dejó varios titulares y uno de los más destacados fue el rotundo fracaso de las urnas. Todas las encuestas, incluida la del Centro de Investigación Sociológica, habían reflejado durante semanas que Podemos podría superar con más o menos flexibilidad el PSOE. Pero la realidad dio una fuerte bofetada a las compañías demoscópicas: los socialistas mantuvieron el segundo lugar, superaron a los que tenían a Pablo Iglesias por 14 escaños y el PP obtuvo una victoria más amplia de lo esperado.
Según lo expresado por la Asociación Española de Comunicación Científica (AECC), es común que las encuestas pronostiquen un resultado electoral que no se reúna más adelante. ¿Hay alguna razón científica para explicarlo? Ellos piensan que sí y consideran que el problema podría ser la aplicación incorrecta de las teorías matemáticas detrás de las encuestas.
Arie Kapteyn, an old professor of economics at the University of Southern California, was the only one who, with his surveys for the Los Angeles Times, was able to predict Trump's victory. No other major US media was right in their predictions; all skated loudly predicting Hillary Clinton as the winner.
------------------------------
Edmundo Pimentel
Profesor
Calle Los Laboratorios
-------------- ----------------