Lies, damned lies and polls?
Anyone who reads the In Box knows that I’m a poll junky. I study them obsessively.
I’m fascinated by what they do (and don’t) tell us about the American people, and I think there’s good evidence that they’re a valuable part of our political system.
I know, I know — a lot of In Boxers disagree.
And there’s an increasingly feisty debate in the wider political blogosphere, about the roll of public opinion surveys in our civic culture, and the influence of poll-watchers like Nate Silver, who is currently experiencing something of a backlash for the strong Obama lean in his prognostications.
This is all very small beer in the wider scheme of thing, but I will say this: 2012 will provide a very good test caste for, if nothing else, how accurate these polls are. Can they really find and measure the sentiment of citizens?
Are there hidden pockets of Americans — a “silent majority” that could send the 2012 election in an unexpected direction.
It’s a good test case because the polls are essentially universal in their estimation that Barack Obama has a whopping advantage ahead of Tuesday’s vote.
Scroll through the last two weeks of surveys and you will find that the vast majority of surveys in battleground states show the race either tied or (in most instances) leaning toward the Democrat by 1 to 5 points.
(The exceptions are Florida and North Carolina, where surveys show a tie or a modest lean toward Romney of 1 to 5 points.)
Similarly, the national polls have slowly gravitated back toward a modest Obama tilt. Of five surveys released on Sunday, three showed Obama leading by 1 to 3 points. Three showed a dead tie.
Which means that we have a near-perfect test model for assessing the accuracy of these political tools.
If the pollsters have it right, Obama will almost certainly win (by narrow margins) at least enough states to capture a second term.
If the pollsters are wrong, and Romney finds pockets of passion, momentum and support that the survey models couldn’t identify and measure, then he wins a major upset and the pollsters go back to the drawing board.
Some statisticians will argue against this assessment. They will point out that even Nate Silver gives Romney a not inconsiderable chance, roughly 20%, of winning the election.
I don’t think that washes. If Romney pulls off a big sweep — especially if he pushes successfully into states like Michigan, Minnesota, and Pennsylvania — I think the polling industry will need to do some serious soul searching.
And so, too, will political reporters like myself, who have come to rely on polls to help us understand the state of American politics.
Tags: election12, polls
Brian – Nate Silvers 20percent is the probability that the polls are wrong. (And wrong in Obama’s favor).
To clarify, as of today Silver gives Romney roughly a 14% chance of winning the White House. That is still a 1-in-8 chance, which most statisticians would argue is a reasonably high chance when compared with other long-shot phenomena — winning the lottery, pitching a no-hitter, etc.
–Brian, NCPR
Right, Silver puts Romney’s chances of winning at 13.7%.
And presumably that’s how often historically the polls are off enough a day before the election for the other guy to win.
Without the polls Mitt wouldn’t have known what to say to all those different groups. As the polls go, so goes Mitt.
Brian,
If I say there’s a 1 in 5 chance of something happening, then I’m wrong if it never happens. If Romney wins, it can just as easily be interpreted as Nate Silver and other pollsters being right, and this is the 1 in 5 time. In this post, you are buying into the polls as magic theme. It’s not about projecting your gut feelings, but about finding scientific ways to measure the electorate’s preferences. Polling is quite accurate. But again, a prediction is wrong if the 1 in 5 never happens. And Peter, if the 1 in 5 happens, that doesn’t mean the polling is “off.” It means it’s on. It’s really a simple concept, but one that many people resist, I don’t know why. It’s like the tradition in journalism of quoting prices from decades ago — “When coffee costs a dime and gas was 60 cents a gallon” — without any cost of living context. A dime for a cup of coffee was expensive in 1910.
Will –
If hundreds of polls over the last two months all suggest an Obama tilt and Obama loses, I think that’s more interesting and problematic than one guy getting a 1-in-5 prediction wrong.
(You’re right – taking Silver to task alone would be silly. It’s the wider methodology I’m talking about.)
Remember, these polls aren’t supposed to satisfy statisticians. They’re supposed to reflect reality.
So if a vast amount of surveying data — from dozens of different polling firms — suggests one thing and another thing happens, I think it raises questions.
–Brian, NCPR
Well, the good news is that 48 hours from now we should (hopefully) know the results of the only poll that matters. All prognostication aside, anything can, and often does, happen. As they say in baseball, “that’s why we play the games.”
Brian: Well, I don’t. If the polls were all way off, then yes. But this race is predicted to be close by all polls. If Romney wins, then that is simply something that happens a certain percentage of time, when a race is close. Again, if this race were run 100 times in these exact conditions, the polls are predicting that Romney would win a certain number of times. The polls choose favorites, they do not predict absolute winners and losers.
Will – This from Pollster.com is saying it better than I’m able to:
“The only real remaining question is whether the the final polling averages will prove to be accurate or whether some systematic error in the swing state surveys is concealing a hidden Romney advantage that will reveal itself when all the votes are counted.”
http://www.huffingtonpost.com/2012/11/05/polls-2012_n_2038645.html
–Brian, NCPR
An observation.
No matter who wins, the USA is fast becoming No Country for Old White Men.
If Obama wins, it’s going that way faster than I think.
I think polls can be insightful to a certain degree on issues. Is there popular support for marriage equality or deporting immigrants? If there’s not a referenda on certain issues, polls are the only indicator we really have of popular sentiment.
That isn’t true of electoral races. I don’t think they are of little use to the public on election races. They remind me a bit of the polls in college basketball. Polls don’t really matter because the championship is decided on the court. And electoral polls don’t matter (to anyone other than campaign flacks and donors who use it to target resources) because the election is decided in the voting booth. Polls are really a tail wagging the dog situation.
I love numbers. I have a math degree with a concentration in statistics. As such, I understand their limitations better than most.
Correction: I obviously meant to say, “I don’t think they [polls] are of much use to the public on election races…”
It feels like there is a fundamental misunderstanding of probabilities here.
As Will tried to point out, poll analysis – like Nate Silver is doing – is not a prediction on who will win or lose… anymore than betting lines are. It is a statement of the probabilities that either will win or lose.
Yes, that can then be used to inform predictions, but just like you wouldn’t (or, shouldn’t) say a betting line was wrong if the favorite didn’t win (especially if the favorite was a favorite by only a modest margin), you also wouldn’t say a poll analysis was wrong if its favorite doesn’t win.
All you can say is that the outcome with the lower probability occurred. That happens, no soul searching required.
Btw, for those interested, the betting lines right now favor Obama anywhere from -260 to -400. He’s the odds on favorite… in case you trust Vegas bookies more than stat geeks like Silver.
Polls have actually been very accurate in predicting and calling elections given the complexity involved, particularly exit polls.
Brian Mann,
I don’t mean to be impolite, but I think you’re showing some innumeracy here, especially when you say polls showing Obama with “a whopping advantage” and in your discussion of Silver. As is pointed out upthread, Silver is not predicting victory. He’s giving odds on it. He would bet on it. But he’s not guaranteeing it. He’s hedging, which is what Mitt Romney does, and does well, for a living. Smart money goes where it think it will win. Sometimes smart money is wrong. It’s pointed out well in this post from last week:
“…Mr. Obama is not a sure thing, by any means. It is a close race. His chances of holding onto his Electoral College lead and converting it into another term are equivalent to the chances of an N.F.L. team winning when it leads by a field goal with three minutes left to play in the fourth quarter. There are plenty of things that could go wrong, and sometimes they will.
But it turns out that an N.F.L. team that leads by a field goal with three minutes left to go winds up winning the game 79 percent of the time. Those were Mr. Obama’s chances in the FiveThirtyEight forecast as of Wednesday: 79 percent….”
And what Larry said. There’s an election Tuesday, and against most odds, our man Mitt will win, and the country will be the greater for it!
Redskins lost yesterday. I think that means Obama loses tomorrow.
What Mr. said!
OA is exactly right. The question is not whether the polls are of use to the public, whatever that means. The question is, are they accurate? I think they are, within an admitted margin of error. The analogy to college basketball, or any other sport, Brian (MOFYC) is perfect. Of course, the games, like the elections, are decided on the court, and unexpected things can happen. But statisticians who concentrate on sports are good at predicting the favorites (Nate Silver started as a baseball statistician) and those who concentrate on politics are also. If you’re good at statistics, Brian, then you must understand the notion of one team, or candidate, being a 3:1 or a 10:1 or a 100:1 favorite. It doesn’t predict victory, it predicts the likelihood of victory, taking various factors into account and giving them statistical weight. It’s pretty accurate, which can be confirmed, in the sports world, by checking whether bookmakers who follow professional odds make money.
“The analogy to college basketball, or any other sport, Brian (MOFYC) is perfect. Of course, the games, like the elections, are decided on the court, and unexpected things can happen.”
The analogy to college basketball is far from perfect. Polls of voters are asking people what they are planning to do. They just have to not be lying– it takes no particular skill or luck for them to accomplish what they intend. That, and the poll sample has to be representative of all of the voters it is attempting to model.
The odds-making on sports is totally different. In theory, if you sampled enough voters, you could be virtually certain of the outcome. Nothing you can do could make sports odds-making equally accurate.
Walker, there are two things at play here. Polls, and poll analysis. I agree with you that the former, polls themselves, are not at all like odds-making. They are just data collection.
The later, however, poll analysis – which is what Nate Silver and a lot of others are now doing – is exactly like odds-making in that an analysis of the data is done and probabilities are determined.
If that analysis gives Obama an 80% chance of winning… and Romney wins instead… that does not mean the analysis was wrong, anymore than the fact that long shots occasionally upset favorites at the track somehow means the odds system there is flawed. It happens, less often, but it does happen. That is probability.
Now, to Brian M’s point, the quality of any analysis does rely on the accuracy of the data being used. So, if all (or a good majority) of the polls are fundamentally flawed, then the analysis would not be very reliable. Crap in, crap out. However, a good analysis of polls will account for this as best as it can. Using historical accuracy, compensating for outliers, etc etc. This is what Nate Silver does, and he is exceptional at it. For his analysis to be unreliable, in my opinion, there would have to be a total collapse of the polling process, which would produce an overwhelming amount of bad data in his analysis.
Such a total collapse of these polling systems would be, sticking to the subject, highly improbable (but not impossible!)
Will: I understand the use of such odds and statistics for gamblers. And I understand their use to the campaigns and big donors, who use them to target their resources.
But my point is that for us voters, most of whom belong to neither category, there is no use.
Is anyone going to vote differently because of these polls? I sure as hell hope not.
To the rest of us, these poll analyses are just a diversion from debate about and reporting of real issues.
Is Obama 4:1 to win tomorrow? Maybe. But so what?
The odds don’t matter. The result does.
Sorry… hit return submit prematurely.
Basically, I have no beef with the accuracy of the numbers. My beef is with the overemphasis on them for an audience for whom they really don’t serve any purpose other than entertainment.
Perhaps we should just hold this election one hundred times in a row—see how many of them Obama wins, how many go for Romney, and settle this once and for all.
I may be a voice in the wilderness, but I hate modern campaign coverage in that so much of it focuses on the horse race aspect and so little on the actual issues.
Oh Brother! We need more math literacy in this country. Turns out that we’ve been talking about the horse race all this time and hardly anyone understands how the odds are determined.
How about next election NPR doesn’t send a reporter with the candidates on the road to tell us what the candidate said every day and the reporter can spend the time (and money) that is saved in analyzing the candidates policy positions.
Sorry scratchy, I grew so frustrated with being in the remedial math class that I skipped over your comment.
Here is a helpful graphic which illustrates a look at probabilities. It was already on NCPR but apparently nobody looked at it.
http://www.nytimes.com/interactive/2012/11/02/us/politics/paths-to-the-white-house.html
In all of these discussions and all of these polls, I’m troubled by the huge number of folks who don’t vote at all. Who are they? What do the think? What are their lives like? Do they watch and listen to the same election news as the rest of us? And the biggest question of all…what would it take to bring them over to the other side…to take part in this democracy??