Anthony Wells provides an explains why voting intention surveys work the way they do
Accuracy is the most important thing to any polling company, and drives the decisions we make about how to frame questions.
Our final polling figures before the European elections came very close to the actual result, being within 1 or 2 points for most political parties. It also accurately predicted the Labour and Conservative collapse, the Liberal Democrats being in second place and the Greens beating the Conservatives.
The one exception was the Brexit Party. While we successfully predicted their victory, we overstated their actual level of support by five points.
After each election we conduct a review of polling performance, looking carefully at the data to identify any errors and determine potential causes. If necessary we make changes to our methodology. As part of a review following the recent European elections we have also been looking at how we ask Westminster voting intention.
What is prompting and why it is a challenge
Prompting for party names may seem like a very simple decision – just list all the parties standing, just as on the ballot paper, and who people will vote for. The reality is a little more complicated.
The Electoral Commission has hundreds of parties registered, most of whom will put up hardly any candidates, so it would be absurd to list them all. More to the point, including smaller parties in the polling question may risk overstating their support. Some of the biggest problems for smaller parties are getting the public to notice them at all, and then being seen as a serious legitimate option on a par with the bigger parties rather than a wasted vote. If a polling company includes them in their question it is, intrinsically, sending a message to respondents that they are a big party, and our job is to measure public opinion, not to lead it.
But we do still want to record support for smaller parties. And so, for many years, we have used a “two stage” approach. We ask people if they will vote Conservative, Labour, Liberal Democrat, SNP, Plaid or “other”. For people who pick “other”, we then show them a second list of smaller parties.
However, there comes a point when a political party breaks through and is big enough, well-known enough and popular enough that it should be included in the main prompt: a point when not including them would risk understating their support. Exactly when to make that shift is more difficult than you might imagine. There isn’t a magic level of support that qualifies for “promotion”, as we have to ensure that the move won’t artificially boost their support. Like any change in approach, it is one we only make after thoroughly testing the impact of the change.
Testing for changes to our methodology
Over the last week, we have been testing how we ask voting intention, including running a survey without any prompts at all, just an empty text box for people to type their answer in. This is not a practical solution in itself as it can lead to subjective responses and more “don’t know” answers due to respondents simply not wanting to type out the party name. However, it allows us to see what the results would be if we stripped out any prompting impact.
We also tested a one-stage prompted question, with the Green party and Brexit Party included in a list, along with Conservative, Labour, Liberal Democrat, SNP, Plaid and Other.
Below are the results of these tests compared to the results from last week’s poll (asked as our standard two-stage question).
When comparing these results, it is important to remember that all polls are subject to a margin of error of a few percentage points, and that there may have been some movement over time since last week’s poll. Bearing both these things in mind, the level of support for the Brexit Party was much the same in the completely unprompted question, as it was when they were included in the “other” section. It was also only slightly higher in the “one stage question”, and certainly within the margin of error.
Including them in the “other” section therefore does not appear to have been suppressing their support. Likewise, including them in the main prompt does not appear to be resulting in any undue leap in support.
For the Green party, there was also little difference between the level of support we found in the two-stage question and in the unprompted version. Given the one-stage version produced a level of support in between the unprompted version and two-stage question, it seems including the Green party in the main prompt no longer carries any obvious risk of overstating them.
As such, we’ll be including both the Brexit Party and the Green party in our main survey prompt in future surveys, the first of which has also been released today and finds the following: Brexit 26%, Lab 20%, Lib Dem 20%, Con 18%, Green 9%.
Photo: Getty
