As X Factor winner Matt Cardle storms the charts to take this year’s Christmas number one, YouGov’s Director of Political and Social Research Joe Twyman looks back at predicting the result for this year’s show.
Polling for elections is hardly new – in fact it has been done in this country for every election since 1945. Though they are hard work, they are also usually exciting. Crucially they also provide researchers with a very public opportunity to test the accuracy of their predictions against a measurable outcome.
The downside of this is that you need to get things right – and if you get your prediction wrong then a lot of people are watching. Put simply, you have to be accurate because a company’s reputation, or even an entire industry’s reputation, can suffer the effects of a bad call. In 1992 when the General Election result surprised many in the research business the ripples were felt far and wide.
Outside of election time there are actually relatively few occasions when such a comparison of a prediction against a measurable outcome is possible, at least in public. However, reality television competitions provide just such an opportunity and the explosion of this type of programming over the ten years since YouGov’s formation has provided a brave new world for those who are sufficiently fearless.
Throughout YouGov’s ten years we have regularly dived headfirst into such realms where angels (and other research agencies) fear to tread. Back at the beginning of 2002 YouGov was the only company to test their accuracy with a public prediction for the first series of Pop Idol. The timing of the contest itself provided the perfect opportunity for what was, at the time, still a fledgling methodology.
YouGov’s ground-breaking online approach had been very accurate in predicting the result of the 2001 General Election and then spot on with the ensuing Conservative leadership election. However, Pop Idol represented a significant departure and some outside the company questioned whether moving beyond the realm of the political was even possible, let alone wise.
Fuel was added to the flames of doubt when our final prediction was published. ‘Conventional wisdom’ overwhelmingly stated that contestant Gareth Gates would triumph over Will Young. YouGov’s published prediction of 53% for Will Young and only 47% for Gareth raised eyebrows to such an extent that none of the newspapers we talked to were willing to publish our results, fearing they were wrong.
Of course 53% is exactly the percentage of votes that Will Young did win. We had accurately predicted results from politics and reality television, which was great news to us and anyone who had used the results published the Friday before the final to place a bet at the bookmakers.
What many outside the company had not understood was that the methodological principles we had used for Pop Idol were, at their heart, essentially the same as we had used for politics. Namely we made sure we asked the right questions, to the right people, in the right proportion.
This guiding principle was replicated successfully through two series of Pop Idol, various incarnations of Big Brother and most recently X Factor.
With audiences of a size usually reserved for disappointing performances by the England football team, this year’s season of X Factor was the most popular ever - and having predicted the final three in the correct order last year the pressure was on once again.
However, what some people do not realise is that accurately predicting the result of the average reality television contest is actually much more difficult than predicting the average general election.
There are a variety of reasons for this difference. Firstly, a great deal fewer people vote on reality TV than at general elections. Though it is sometimes easy to forget, only approximately five million votes were cast for the final of X Factor across the whole weekend. This compares to around 30 million in the 2010 general election. The result of this is that the pool of potential respondents is smaller.
Secondly, at general elections people can only vote once. In most reality TV contests people have the option to vote as many times as they wish and each vote is counted equally, though they do have to pay for each one. The total number of votes needs to be considered in any final prediction.
Thirdly, only registered voters aged 18 or over are eligible to vote in general elections. With reality TV anybody can, in theory at least, dial the premium rate phone number regardless of their age, nationality or even location. Standard polling of adult respondents will not capture these people.
Lastly, with X Factor particularly, the performance of contestants on the night can potentially have a massive effect on the final result. A spluttering, wheezing rendition or a barn-storming crowd-pleaser can cause shifts in support just one day after the publishing of a prediction.
Given all these considerations it is perhaps little surprise so few agencies stick their head above the parapet at all, and almost certainly explains why YouGov are the only ones to accurately predict the order of the top four.
How do we do it? The key to our accuracy, and the secret to our success, lies with preparation. The X Factor experience shows that the overwhelming majority of respondents will not decide on a favourite until after the live shows begin, and only after a few weeks will their choice begin to solidify. Therefore, because an accurate prediction is virtually impossible so early on we instead spent weeks on internal polling designed to build up a database of X Factor viewers, their profile, their behaviours and their preferences.
As the live shows progressed 27 year-old Matt Cardle quickly established himself as both the bookies’ favourite and the leader in our internal polling, but the position of the other contestants fluctuated week to week and only stabilised later in the contest.
[Full disclosure: Like X Factor host Dermot O’Leary, I grew up in Colchester, my family still live in Colchester, with my brother and his family living down the road from Matt Cardle]
In the last few weeks the situation was further cemented and by the time our first poll for The Sun was commissioned we had data from over 18,000 X Factor viewers. For our eve-of-final prediction around 1,500 of these people were re-contacted to take part in our survey. First, second and third preferences, voting patterns and strength of support were all measured along with additional variables to aid the prediction. This largely followed the approach we took to the survey in 2009 when our prediction of the final three was correct.
This time, although many agreed with our prediction for the winner, some commentators felt we had underestimated the support for the boy band One Direction. However, our data showed that while One Direction generated a lot of buzz it was not actually translating to votes. Though it is easy to forget, fan noise does not always correlate strongly with popularity.
Our final results placed Matt first, Rebecca in second, One Direction in third and Cher fourth. Anyone betting on that order just before the final show began would have been given odds of around 4 to 1.
Although many agreed that Matt’s continued illness meant he underperformed on the night it did not make too much difference to the final outcome. When Cher was eliminated our confidence grew and when One Direction followed her I knew we had it right again.
All that remains now is to wait for next year and the whole thing to start all over again.
It’s not easy, but it is exciting.
