…especially if you think it won’t make any difference

This Thursday will see one of the most important UK elections in which the general public will be looking to the polls more than ever before. However, can they be trusted? 

In 2008 Nate Silver did something extraordinary. He predicted the winners of all 100 U.S. Senate seats. Polling companies had correctly predicted the result of a presidential race before, but no-one had ever so accurately predicted a Senate race. Overnight Nate Silver became a Data Science superstar with interviews on CNN and a partnership with the New York Times to support his blog.  

Full of confidence, Silver turned his attention and his methodology to the 2010 UK general election. And he got it completely wrong. Undeterred he had a crack at the 2015 and 2017 UK general elections, and he struck out again. After his reputation remained intact by correctly predicted the US elections during the intervening years, he eventually gave up on UK elections citing poor quality data.   

In spite of his high-profile fails, Nate’s UK election experience does point to an interesting insight into the unique difficulty with UK general election predictions rooted within building a representative sample of voters.  

As most already know UK voters are grouped into 650 constituencies of between 60k to 80k voters. The party with the most votes in each constituency sends an MP to parliament. The party with the most MPs in parliament forms a government (usually). So to predict an election result requires a robust sample in all 650 constituencies. The YouGov/Sunday Times poll – one of the most comprehensive and respected – is currently running at 1,680 respondents. That equates to 2.8 voters in each constituency. Hardly statistically significant. 

This does beg the question as to what is the value of polls then? The simple answer is storytelling. They give news outlets something ‘concrete’ to talk about whether it’s Boris winning a TV debate or the Lib Dems’ Remain story having traction. And in the absence of any more robust evidence to the contrary, then it’s worth reporting and acting upon.  

The best of the bunch we found was https://www.electoralcalculus.co.uk/ which is based on a reasonable 10,827 voters (18 voters/constituency). Despite having a sample size 10x larger than Yougov, it doesn’t tend to get too much coverage. While they didn’t get 2015 or 2017, they did at least get 2010 right so they’re one-up on poor old Nate. Above all else, what we like about Electoral Calculus is that they are open-sourced with their methodology – remember what your primary school maths teacher once told you… 

This all points to several watch-outs for our industry when it comes to insight derivation and analysis. The first is always question the source of an insight and the underlying data. The proliferation of new data and techniques is exciting but if these new approaches are black box and you can’t unpick the methodology then handle the insights with care. Secondly and more broadly, beware of the insatiable desire for ‘the answer.’ This can put pressure on whoever’s doing your analytics to deliver any answer. Create an environment whereby people can challenge the robustness of data with confidence.  

And in case you’re curious and despite our misgivings about data quality you’re still keen to see a prediction for the election, we have reweighted Electoral Calculus predictions (as they were still predicting 650 MP’s) and had a stab at the results…  

So remember on Thursday – whether you’re in a ‘critical swing’ seat being targeted by an opposing party or if your local MP is nailed on to win, question the source of that wisdom and the robustness of the sample when you’re queuing to vote at your local polling station.  

James Londal, Chief Data Officer, Hearts & Science UK

Garrett O’Reilly, Managing Director, Hearts & Science UK