Discussion: View Thread

Hi all,

I was reading an article recently and was curious what everyone thought of something:
http://fivethirtyeight.blogs.nytimes.com/2012/06/10/where-electoral-forecasts-agree-and-disagree/

Specifically, the author calculates the standard deviation of the ratings from 6 different sources, where the rating is on a scale of -2, -1, 0, 1, 2.

For instance, if the ratings from the 6 sources are {0, 1, 1, 2, 0, 1} such as in the second row of his table (for Pennsylvania), then the mean is 0.83333 and the SD is 0.687.

He uses this SD as a measure of the "disagreement" between the 6 ratings. I hadn't ever seen a SD of a discrete scale used in this way before, and it didn't necessarily make sense to me as a valid method. However, I'm curious what others on this mailing list would have to say about it?


Thanks,
Gabe



-------------------------------------------
Gabriel Farkas
-------------------------------------------

I'm a very applied market researcher, and we routinely treat 1-to-4 ratings scales (or 1-5, 1-6, 1-7, 1-10, 0-10 or anthing else) as continuous.  No one even thinks twice about it.

Even by those standards, this seems a bit of a stretch.  The key implied assumption is that solid differs from leaning as much as leaning does from tossup.  Maybe, but who knows?  It would depend a lot on how those terms were defined in the first place.  And "differs as much" is defined on what scale (probability of winning?  percentage lead in polls?  etc?)?

But, if you look at how the article is actually using it, I don't see a real problem.  He's not using the SDs to draw conclusions, or directly interpreting them, but only using them to order the table in a helpful way.  Had he omitted the SDs from the table and not mentioned them, the table would still be in a helpful order and we might not even be wondering how that order was arrived at.

-------------------------------------------
David Lyon
Aurora Market Modeling
-------------------------------------------

I've not seen SD used that way before, either.  But if one believes it is reasonable to equate dispersion with disagreement, and if SD is a well-accepted descriptive measure of dispersion, then using SD in the article's context would be reasonable. 

I did happen to notice that the article's author calculates population SDs, not sample SDs.  That may also be reasonable depending on how the article's author views the six sources. 

-------------------------------------------
Eric Siegel
Biostatistician
Univ of Arkansas for Medical Sciences
-------------------------------------------

Hello all, I've spent a good deal of my career time working with rater reliability aka agreement, disagreement using measures corrected for chance such as kappa.  There is a multinomial, multiple rater kappa that could be applicable to the situation described herein.  the usual caveats apply, what was the purpose of the study, what questions was the study designed or intended for etc.

All the best.  This site provides good thinking exercise.  John   

-------------------------------------------
John Bartko
Consulting Biostatistician
-------------------------------------------







Original Message:
Sent: 06-14-2012 18:27
From: Eric Siegel
Subject: Calculating standard deviation for a 5-point scale

I've not seen SD used that way before, either.  But if one believes it is reasonable to equate dispersion with disagreement, and if SD is a well-accepted descriptive measure of dispersion, then using SD in the article's context would be reasonable. 

I did happen to notice that the article's author calculates population SDs, not sample SDs.  That may also be reasonable depending on how the article's author views the six sources. 

-------------------------------------------
Eric Siegel
Biostatistician
Univ of Arkansas for Medical Sciences
-------------------------------------------