Hung Parliament - appraising the polls

So, the votes are in and uncertainty reigns in the immediate aftermath of the election. But for the polling companies, weeks of estimates, margins of error and questioning have been replaced with cold, hard certainty. We now know that the Conservative Party won 42% of the vote, followed by Labour at 40%. With one seat left to declare, the Conservatives lead the way on 318 seats.

In an election with few clear winners, two stand out. The first is YouGov, and specifically their new modelling methodology. This combines 50,000 (non-representative) interviews each week with machine learning to model the outcome in terms of both vote share and seats. Though YouGov claimed that this model successfully bucked the trend and correctly predicted the Brexit result, its forecasts were widely dismissed in the week before the vote.

If we look back at those forecasts though, the YouGov model predicted a hung parliament and a Conservative vote share of 39-44%. Labour were forecast to achieve 36-41%. This forecast also relied on a high turnout of young voters, something that was dismissed as unrealistic by many observers.

The success of this forecasting method, in contrast to the general consensus of a Conservative majority of 50+, should be a landmark moment for the polling industry. If machine learning can overcome the potential discrepancies in a large, unrepresentative sample and provide highly accurate results, it may well represent the future in opinion polling, and even general mass research. Getting this election and the EU referendum right is a startling achievement, given the struggles of traditional polling methods.

The other big winner of the night was the broadcasters’ exit poll. In the 60 minute result vacuum that followed it’s fanfare reveal, experts and insiders from all parties expressed doubt on its accuracy. It was left to John Curtice, Professor of Politics at the University of Strathclyde to explain the numbers on the BBC, and he gradually became social media’s hero of the night as reality began to reflect the forecast and his mild-mannered, accessible analysis shone through.

When the dust settles on an uncertain electoral outcome, the polling and research industry have some food for thought. But unlike recent years, there is cause for optimism in the shape of a big data, machine learning methodology that might just represent a brave new world of research capability.