US Elections: a probability is not a prediction

Next weeks US presidential election is fascinating to watch because the clash between old school journalism and more data driven approaches looks to be very much part of the story itself.

The difference between probability and prediction is a key to this but not so easy to explain. The best data based analysis of the US elections comes from Nate Silver over at fivethirtyeight where Nate reviews a series of polls, accounts for bias and then runs a number of scenarios and tests against incoming events and polls to try and get the best possible data on the election.

Mark Coddington gets to the heart of the matter when he describes the different approaches as

“fundamentally an issue of epistemology — how journalists know what they know…”

Where political journalists’ information is privileged, his (Nates) is public, coming from poll results that all the rest of us see, too.

Where political journalists’ information is evaluated through a subjective and nebulous professional/cultural sense of judgment, his evaluation is systematic and scientifically based. It involves judgment, too, but because it’s based in a scientific process, we can trace how he applied that judgment to reach his conclusions.

Both of those different ways of knowing inevitably result in different types of conclusions. Silver’s conclusions are at once much more specific and much less certain than those of the political punditry. The process of journalistic objectivity can’t possibly produce that kind of specificity; that’s outside of its epistemological capabilities.

I have been reading Nates book “The Signal and the Noise: Why So Many Predictions Fail-but Some Don’t”

The book is a great read and it covers off the background reasons to why forecasting works better when the process is open ended and not driven by an idealogue.

Part of what makes Nate Silvers predictions much better is that he constanting reravaluating the data and not trying to filter it by being distracted at some level about what has happened in the past or what we think should happen.

So often the journalists are being guided by all kinds of inside knowledge that helps them to feel they have a much clearer understanding of the politics than they do.

Looking at the image to your left you should see that Obama has a very big lead on the electoral college and an 83.7% chance of winning there which is where it counts. The popular vote is much closer at 50% Obama and 48% Romney – only a 2% difference there.

Most journalists are saying it is “too close to call” but with less than a week to go it is much more likely that Obama is going to win and quite easily. In 2000 Bush won 271 electoral college seats but lost the popular vote by something like 600,000 votes. Obama is on track to win something like 300 electoral college seats.

In Nates book the section about foxes and hedgehogs in the famous essay by Isaiah Berlin

“The fox knows many things, but the hedgehog knows one big thing.” This fragment of verse by the Greek poet Archilochus describes the central thesis of Isaiah Berlin’s masterly essay on Tolstoy, in which he underlines a fundamental distinction between those people (foxes) who are fascinated by the infinite variety of things and those (hedgehogs) who relate everything to a central, all embracing system.

It has also been suggested by a few that a close race is more exciting for the media. I’d guess that overall voter turnout will be higher, especially in the swing states of Ohio others like that.

I would recommend reading the full Mark Coddington post and the Nate Silver book also.

Here is a recent clip of Nate talking about how the numbers work.