Has Nate Silver lost his mind on NY-19?
Okay, I admit it, the blog title is deliberately provocative. A fairer question would be:
Are the statistical pundits — people who claim to be predicting the 2010 election results based on facts and statistical models, rather than raw opini0n — full of beans?
The reason that this matters is that more and more of the political narrative is being shaped by supposedly impartial statistical models that push well beyond simple polling.
This has been a persistent thread here on the In Box the past few days, as I’ve continued to stumble across instances where supposedly stat-based analyses seem, frankly, improbable.
The case in point today is New York’s 19th congressional district, currently represented by Democratic incumbent John Hall.
This is, by any estimation, a close race, which the most recent poll describes as “a virtual tie.”
The Monmouth University Poll finds the incumbent Democrat with 49% of the vote and the Republican with 48% among likely voters in this district.
In fact, every poll conducted during this election season in NY-19 has landed well within the margin of error, with most finding the race to be perfectly deadlocked.
Nate Silver’s big picture analysis, as he acknowledges, includes a fairly subjective range of factors, including “polling, expert forecasts, fundraising data, past election returns and other indicators.”
Throw in all those kitchen-sink variables — “other indicators”? — and Silver predicts the Republican will pull off a 50.4-47.2% win.
That’s still pretty reasonable, right?
Especially because Silver acknowledges in the fine print that this prognostication includes a huge +-6.4% margin of error.
Which means that he, too, is basically saying that the race is too close for his measuring stick to give us a clear picture.
Now here’s where things get funky.
Using all those variables, Silver then runs “100,000 simulations” and comes up with the prediction that the Republican has nearly a 72% chance of winning.
That’s right. When Silver’s tea leaves settle at the bottom of the cup, his model shows that the Republican is approaching a lock on this race.
Now, I’m not a statistician.
And I know that there are some general rules in politics that pundits use to frame these predictions.
There is a conventional wisdom, for example, that incumbents polling under 50% are in serious trouble, and that the vote often breaks a bit toward the challenger.
But this race also has some significant factors that break the other way, including a deeply unpopular Republican, Carl Paladino, at the top of the ticket.
Also, the Democrats have a powerful GOTV machine in New York, and they have the fact that this is, bluntly, a downstate New York district.
Does this mean I think Hall is a lock? Heck, no. I have no idea what’s going to happen in this race.
But do I believe that the Republican has a 72% chance of winning? Also, no.
As much as I respect Silver’s track record over the years, I just don’t buy it.
I would love to see him show his work on a prediction like this, so that we can peak inside the statistical models that are framing this year’s mid-term.
Tags: election10
Brian, I think the problem is in not understanding what it means to run 100,000 simulations. This is exactly what the climate modelers do; it helps to tease out the cumulative effects of various small changes.
It will be sooo interesting to see how this election plays out. I’m guessing: many surprises!
Brian,
I think you and many misunderstand the Republican lean, Democrat Lean and Toss Up methods. To your layman, all three are toss ups. It’s different for the prognosticators too, though Nate is putting down specific chips, at the end of the day he isn’t going to hold up the exact races he bet on, but the total number of races he projects. He and others are looking for roughly 50 seats to flip. They don’t really care where and the percentages are just putting math to the paper. If Nate hits 50, that is what he’ll hold up, not which races actually swung one way or another. Thus he’ll put NY19 at 70% because there is a chance that NY25 at 88% for Maffei, might swing the opposite direction. In actuality, he’s hedging bets this way.
That aside, I think that you have most certainly missed many, many problems that John Hall has. Himself being to biggest obsticle:
He is not running an effective campaign like Maffei, Murphy and Owens. In fact, he has been disengaged with his district. For example, OFA is doing his canvassing as opposed to grassroots, he sent a mayor to debate for him instead of showing up, and he had a NJ congressman come do the talking at one of his few town halls. He has name recognition from being a musician, but not a resume like Nan Hayworth and in this election, I think the resume wins.
Finally Paladino may be dragging the top of the ticket, but Cuomo has a problem and it is that he isn’t lifting the ticket. Cuomo isn’t campaigning and he is an unaspiring candidate. People are going to pull the lever for him, but not because they want to. Heck, I wouldn’t be surprised if the Rent is Too Damn High party doesn’t get a good showing come election day.
Sam has a point, I may write for Patterson!
Lies, damn lies and statistics. Polls are pretty much the same, eh?
Ticket splitting may be the key to NY’s close congressional races next Tuesday. Sam’s point that many Independents (and even Democrats) have not warmed to Cuomo is true enough. Though it seems safe to say Cuomo hasn’t done quite the demolition job on the affections of loyal partisans that Paladino has.
Too, there is the question of viable alternative candidates at the top of the ticket. If you eliminate the novelty candidacies of Davis and McMillen (wiht sincerest apologies to potheads and martial artists), of the three remaining non-major party candidates who articulated policy positions, two were on the liberal side of the spectrum, and one on the conservative/libertarian side. Point being that fewer options on the right may result in a lower conservative turn out. Frankly, the gulf is so wide these days between the two major parties that full-spectrum (R-to-D or vice versa) ticket splitting seems unlikely. Most of whatever ticket splitting occurs will be done on either side of the divide.
So I’m inclined to give NY’s Democratic congressional candidates the edge in the close races (which is precisely the same logical analysis I used to predict Mondale’s Presidential victory in 1984).
Sam –
I’m interested in your take on John Hall’s inadequacies as a candidate; you may well be right that that will sway the election beyond what we’re actually seeing in the polls.
But I don’t think your argument builds much confidence in Silver’s method for telling me what’s happening in particular races around the country.
If his purpose is to reflect reality — and not to win at a series of gambler’s bets — then his system should include the capacity to say, “I don’t know.”
When I look at his own data, his own margin-of-error admissions, I think this is a “too close to call” situation.
It CERTAINLY isn’t a 72% lean situation…
Brian, NCPR
Let me say again what I have said before.
With the advent of caller ID, polling is losing its ability to forecast.
Let me add a feeling. I am a bit suspicious of the objectivity of polling and worry they are a form of political ads.
I say this because so many people are sheep and will jump on any bandwagon to be part of the maddening crowd.
If most people think this or that is true, it must be true is the logic of following the crowd.
Look at this way. Suppose the World Series matches a team that won 99 games against one which won 94. A stat geek running 100,000 simulated series may conclude that the 99-win team will win say 62.3% of the time which sounds like a pretty solid favorite. And yet, in the world on the field, common sense indicates going in, with two teams that just won 2 playoff series, the match-up is just about even.
To me some of these forecasts that because of a couple statistical factors candidate A is a 70% favorite seem similarly off base. Another question is the mental leap..the forecaster is suggesting a 70% chance of victory for A, not that A is going to get 70% of the vote.
Brian,
On NY-19, sure thing, shoot me an email sometime.
Here is Nate explaining Nate.
http://fivethirtyeight.blogs.nytimes.com/2010/09/30/the-uncanny-accuracy-of-polling-averages-part-2-what-the-numbers-say/#more-1607
I agree with your point but if you think about it, more than a 1 in 4 chance is not really a throw-away race, that’s probably the fringe of toss up if you ask me.
If you are interested in some other models…although not a race by race projection as 538 does, Dr. James Campbell at UB recently devised a new model based in part on Charlie Cook’s. I’m publishing an article with an interview I conducted at Daily Caller. Not out yet though.
Brian,
I think you’re just not getting it. A 1 in 4 chance of winning does not mean the other guy has a lock. It means 1 guy has a 1 in 4 chance of winning and the other has a 3 in 4 chance of winning. It’s not a tossup and it’s not a lock. Certainly, the 1 in 4 chance guy can win — out of 10 or 20 such races, the 1 in 4 candidate will probably win, probably 1 in 4 times. Silver is very good at what he does. For reference, look back at his analysis of Murphy-Tedisco, when that race was waiting for absentees. He (and I, by the way) looked at where those absentees were coming from and correctly predicted Murphy would win, to the great annoyance of some local commenters, who claimed it was a guess. It wasn’t. Also, I’m guessing (this is a guess) the recent Siena poll (Gibson 9 points up) has it wrong. Maybe things look good for Gibson right now, but I bet the election will be close, whoever wins.
I have been struggling with my own conflict over respecting Nate and my growing skepticism over his models. If he’s basing his zillion simulations on flawed data (and I believe he is), then his conclusions are equally flawed.
I took a deep dive into the toplines of the polls he’s using, and they’re demographically flawed, they’re heavily weighted toward traditional midterm models and don’t adjust for the aberrant political climate we are in right now. I have the sense that November 3rd will leave many scratching their heads.
You should know that the translation of the vote percentages (say 50.4 to 47.2), which you find at least plausible, into probabilities of victory is rather straighforward. It is based on lookin at previous elections, one week prior to election day, and seeing how often a candidate with a 3.2 point margin at that time actually won on election day. There’s no fudging or guessing, it’s all based on past history. That 3.2 point margin in a general election (not a primary, which is likely to be more volatile) wins 72% of the time on election day.
The input into determining the vote percentages (50.4-47.2) involves a pretty complicated set of calculations, with which you might disagree. But the input into translating that spread into a probability of victory is very straightforward. Nate discusses this here: http://fivethirtyeight.blogs.nytimes.com/2010/09/30/the-uncanny-accuracy-of-polling-averages-part-2-what-the-numbers-say/
What they said – the split between the polls may be very close, even within margin of error, but running simulations with varying inputs – high voter turnout, low voter turnout, age differentials, etc. – gives you the odds. There’s a reason they call it a horserace: even the favorite loses sometimes. Hall has a slightly better than 1-in-4 chance of winning. I don’t know anything about betting, but I can’t imagine that would be a very risky bet to take.