Nate Silver runs the numbers with Playbook
January, 10, 2013
By Joe DeMartino | ESPN.com
Nate Silver got exactly two things wrong about the 2012 election: the outcome of Senate elections in North Dakota and Minnesota. Everything else about the election fell before the might of Silver’s analysis on his FiveThirtyEight blog -- he correctly forecast the results of 31 other Senate elections and, most crucially, which way every single state would swing in the presidential race. This isn’t a new thing for Silver -- he got 49 out of 50 states right in the 2008 elections, and was a pioneer in baseball forecasting with Baseball Prospectus. Playbook caught up with him during his visit to Bristol to ask him about statistical uncertainty, the Baseball Hall of Fame, and whether or not he’s a witch.
You received a lot of dismissal from pundits during the election -- saying "this guy doesn't know what he's doing" or "we're not familiar with his methods." The same thing happened to some extent in baseball with PECOTA. Where did you receive more pushback?
Probably in politics. Bill James pioneered a lot of sabermetric stuff back in the 1980s, so it was a fight that was already in the midst of occurring by the time I had stepped into it -- “Moneyball” was written in 2002, and I started releasing the first PECOTAs in 2003. So the ground had been trodden on a little bit more, whereas in politics, it's really as if you're speaking a foreign language when you bring up numbers and statistics. You also don't have the same incentives. The good thing in sports is, for better or for worse, we have a way to evaluate success and failure. You test yourself every day based on the results of a baseball game. In politics, you can be wrong for an awful long time -- for four years -- before you get a dose of reality sometimes.
There's no "spin room" in sports.
Yeah, I mean, hey, some of the shows here, people are spinning things, but if you were saying that, say, "Notre Dame is underrated and will beat Alabama" --
You'd be wrong...
--that spin isn't very persuasive for people when you have the reality check the very next day.
Who's worse about predicting outcomes -- pundits, or sports analysts?
Pundits are worse. There have been studies where pundits have been analyzed over a long period of time. “The McLaughlin Group” is a show that's been on syndicated TV for a very long time. I spent a couple of weeks going through five years worth of predictions they had made and scoring them as right or wrong -- this is all a labor of love -- and they got exactly half right. You'd do just as well flipping a coin. At least in sports, you get more practice in seeing how things play out on the field. I love stats and building models and so forth, but you stay a little bit more grounded.
There are only so many factors that go into sports.
It can be complicated. PECOTA was a complicated projection system -- we were looking at something like 13 factors that predict how well a player might do -- but you have a lot of data. You've got about 750 players in the majors at any given time, with 140 seasons of professional baseball. Multiply that together and you have thousands of data points, whereas we've only had 17 presidential elections since World War II. It's much harder to design your model in politics, but it also means that people may overrate unimportant things because they don't get to test their ideas as much.
Do you think punditry is obsolete?
Yes, I guess is the short answer. The long answer is, I'm not sure it'll ever be. It's entertainment, basically. I think it adds very little actual value to what people understand about what happens in an election or what the democratic system is like. It's purely entertainment value for people who like to watch "Meet the Press" and so forth.
Are there any that you trust, in particular? Pundits or reporters?
I make a distinction there. I think reporting is very valuable, potentially. Some people have a better sense for it than others. I like John Heilemann from New York Magazine, the co-author of "Game Change". He does some reporting and has read history and political science, so he can put his stuff in more context with more of a filter for what's BS and what isn't.
Whenever I see people address the predictions you made in 2008 and 2012, it seems like they think you're making a 1 or 0 prediction -- either this will happen or it won't, but it seems like there's not a lot of understanding about the level of uncertainty that goes into it.
This is one of the things sports fans are better about understanding. If you watch the NCAA tournament, you know that sometimes a 15-seed will beat a 2-seed in the first round, but you wouldn’t want to bet on any one 15-seed against a 2-seed in any one game. Those long-term probabilities, we understand them more in sports. We understand the role luck can play in any one given day -- will a free throw go in when it has to, what will the refereeing be like -- whereas in politics, what might come down to luck, people might attribute more to “divine intervention.” You can always have some of that in sports too -- say, if the weather is bad enough to mess up a team’s game plan. We might attribute that to the coaching plan, where maybe there are other factors at play as well.
What do you think was the biggest factor of uncertainty in the 2012 elections?
It was a pretty close election. Obama’s margin was fairly persuasive in the electoral college, he still only won by about 4 percent. It’s almost 50/50. I think that period after the first debate in Denver, where Romney’s numbers clearly improved quite a bit, the question was, when would his rise stop? We noticed after about seven or 10 days, he stopped rising in the polls. He hadn’t gotten quite close enough in states like Ohio and Virginia and so forth. That was the moment when I probably got the most heat from people. The press thought that Romney still had momentum, but I define momentum in more of a mathematical way -- are you going in the same direction, or has there been a stall?
Will his momentum tail off, or is there a high-water mark...
A high-water mark, exactly. We saw his numbers recede a bit, which meant that he had some chance but still needed to see some other event that would help his numbers.
Reading certain pundits, it seemed like they saw that first debate as a real turning point. I followed Andrew Sullivan at that time, and he was panicking -- it had been a golden opportunity for the president that he’d totally blown.
This is another thing in both sports and politics: the latest news development, people overrate how important it’ll be in the end. You have one debate where Romney did well and Obama did badly -- no one is in denial about that -- but you had two more debates where Obama did better. Hurricane Sandy intervened. You had the president’s ground operation. You had a lot of things in the campaign that got lost when a big news story hit.
Say Romney had emerged the winner -- obviously that was factored into your prediction, because you gave him a nine percent chance to win and things that have a nine percent chance of happening, happen all the time -- how would you have reacted to it?
I’d probably go hide out in Mexico or something. It would have been tough. That’s the other thing about the NCAA tournament -- we make probabilistic predictions every year for the tournament, and you get some right, and some wrong. I knew I would get too much credit if Obama won and too much blame if Romney won. I’m trying to bask in the “too much credit” part of it right now. If you have a prediction that goes badly, then you have to ask yourself, “Was I unlucky, or did I design my model badly? Did I make stupid assumptions?” or something. If Romney had won, it was an unlikely enough event where I’d have looked under the hood and asked if I did something wrong, or if it was out of my control. In sports, that’s easier to do, because of the structure of the game. The rules are always the same, so you can say more definitively, say, a team down 3-2 in the ninth inning will win 12 percent of the time, or whatever. You have possibilities based on thousands and thousands of games played, whereas in politics you have a certain structural uncertainty as to why people vote and what will happen in the long run.
The New York Times, your employer, put out a blank cover page on their sports section, saying “And the Inductees are...”. There were three guys who did get in (by the Veterans Committee), but they’re all dead. Do you think the voting process needs to be reformed at all?
There are little tweaks -- when guys get kicked off the ballot, for instance. Bernie Williams and Kenny Lofton got knocked off the ballot -- I thought they were guys that should marinate a little bit longer. That was unfortunate. People blame the voting process when they should be blaming the voters themselves. I would have voted for seven or eight guys yesterday. Some of them would have been accused of using steroids. I don’t understand, for example, why Curt Schilling got something like 38 percent of the vote. All these sportswriters talk about the importance of postseason play. He’s one of the most successful postseason pitchers in the history of the game.
That’s the Jack Morris argument.
That’s what doesn’t make sense, is the head-to-head comparisons. If you compare Jack Morris and Curt Schilling, they’re similar except Schilling had a lower ERA even though he played in an offensive era where it was easier to score runs. He was a more dominant pitcher in terms of strikeout totals. Obviously they both had postseason success, but Schilling did it over more distinct playoff seasons. If you look at that comparison, I don’t understand the rationale for voting for Morris but not Schilling. The stats guys have him as a fair bit better in the regular season and to some extent in the postseason. People are going more for a gut feel test. Jack Morris has been a weird cause, almost, for some old-school guys, in the way that Bert Blyleven became a cause for new-school guys for a long time. The number of All-Star teams that Jack Morris made was five or six [note: five] , I think, but he wasn’t a guy who was a superstar player. If I had a vote -- and of course I don’t -- I’d like the guys who were dominant, who you could clearly say were the best at some phase of the game at their position for a long period of time. Schilling was dominant in a way that Morris probably wasn’t as much. [Morris] would not be the worst pitcher in the Hall of Fame by any means.
That’s not exactly an endorsement, though.
Right, I mean, if you had a team made up of the worst players in the Hall of Fame, it’d be a good baseball team, I guess, but not a great one. I don’t think you’d win the World Series, necessarily. There have been some mistakes made over time. I’m a Tigers fan, so I would not feel horrible if Jack Morris was in the Hall of Fame. It is pretty random. He has a stat of having the most wins of any pitcher in the 1980s. People kind of back-cast arguments and reasons to vote for people.
Would you want Alan Trammell in the Hall of Fame, as a Tigers fan?
Yeah, and probably Lou Whitaker as well.
He’s fallen off the ballot, though.
Yeah, I mean, if you have [Joe] Tinker, [Johnny] Evers, and [Frank] Chance, why not Trammell and Whitaker? Being an elite shortstop for a decade or so is a tough thing to do. You have to account for positional value. That’s why I don’t understand why [Craig] Biggio didn’t get in. [Mike] Piazza’s a case where people are suspecting him despite no hard evidence linking him to steroids. Those guys seem like the easy cases to me. A guy makes 10 All-Star teams, is the best at his position for a decade, he should probably get in. Jack Morris wasn’t really the best pitcher in his league for any period of time.
You’re something of a celebrity now. You mentioned people giving you too much credit -- there’s a #drunknatesilver hashtag on Twitter, you have people asking if you’re a witch. Is this enjoyable for you, or is it horrifying?
It can be both at once. It’s weird when you’re rushing to catch your plane or something. I got spotted three times at O’Hare airport the other week. Things are different, but it’s good. I think you have to be careful not to be too caught up in your own press clippings. You have to maintain your hunger and your edge a little bit. You don’t want people to throw rose petals around you and say “whatever you say is brilliant.” If you look at my book, “The Signal and the Noise”, it’s all about stories where people made predictions that looked good on paper but failed miserably later on. I used to play poker, and you can be running good for a long time where you feel like it’s the easiest game ever. Then you hit a downstreak, and suddenly...
... you’re the idiot.
Yeah. I’m aware of some of the good fortune I think I’ve had in life. You combine that with hard work and some talent, and you hope to do as well as you can. I don’t want to get soft and lose my edge. If you get opportunities to do too many interviews and people are saying nice things to you all the time, then I worry about that a little bit.
What’s next for FiveThirtyEight?
My goal is to expand FiveThirtyEight to cover a wider range of domains. Sports would probably be one of those. I’ve been writing more sports pieces on FiveThirtyEight because I have all this pent-up energy from the election. It’s terrible that the Summer Olympics were in the same year. You don’t get to follow sports as much. In any field where I think a lot of value can be added with statistics and advanced analysis, that’s where we’d go. There are some political stories where we can’t add much -- for example, Barack Obama’s cabinet nominations. You can’t do anything data-driven on that. I wouldn’t want to do something where we’re not adding insight and value. If there’s something that FiveThirtyEight does, I’d want us to do it well, in our own style. Too many people producing web content just want to expand and do things, instead of thinking about “where can I best apply my energy”?