Designing a questionnaire is not always straightforward. In fact, it can at times be a long and difficult process. There are options to consider, approaches to assess and rules to follow – not to mention the various operational considerations that inevitably arise.
The upcoming referendum on whether or not to adopt the Alternative Vote (AV) for Westminster elections has proved particularly interesting in this respect.
As with the better known example of voting intention, many people have different theories about the best way to get the most accurate result, but nobody can ever be 100% sure that their method is absolutely right.
With AV there are many issues to consider: Do you provide respondents with information about both systems or not? If you are providing information, what information do you provide and how much? Do you precisely replicate the question on the ballot paper or provide an adapted version? Do you weight by turnout or just use standard weighting?
There is no collective agreement on what is the ‘right question’ to ask – and with so few national referendums having taken place in Britain, it is unlikely a definitive solution will be found any time soon.
Therefore, what we have done at YouGov is to look at the data derived from a variety of different approaches, both the individual results and the long-term trends.
Just this week, we have run the question exactly as it appears on the ballot paper, and then weighted the results by likelihood to vote. Taking that approach, the result was as follows:
- Yes 37%
- No 44%
- Don't know 19%
We also asked a longer question that provided respondents with a relatively brief explanation of the two electoral systems. In that instance, the result was not weighted by likelihood to vote and was as follows:
- Yes 33%
- No 45%
- Don't know 15%
- Wouldn't vote 6%
You can view the most recent results from those two different approaches, the full question wording and the associated tracking data here, by clicking on the 'Alternative Vote Referendum' link on this page [link].
Finally, we asked the exact ballot question at the end of a detailed survey exploring arguments for and against different procedural aspects of reform of the electoral system.
This approach produced the following results, again NOT weighted to likelihood to vote:
- Yes 45%
- No 33%
- Don’t know 17%
- Wouldn’t vote 6%
So then - three different answers, but which is ‘right’? As mentioned, nobody can know for sure. All three are legitimate approaches to take, but what is clear is that through the separate approaches, you are clearly measuring three slightly different things.
A simplistic summary analysis of the data would suggest that the more detailed exploration provided in the third survey produces only a slight reduction in ‘Don’t knows’ compared to the no explanation version, but an increase in support for ‘Yes’ at the expense of ‘No’.
In contrast, providing no explanation of the different systems leads to higher levels of respondents saying ‘Don't know’, with ‘No’ not as far ahead when compared to an approach that just provides a brief explanation.
With such differences, it is perhaps, therefore, more sensible to consider the long-term trends when predicting what the result might actually be on May 5th. The long-term trends show an increase in support for First Past the Post, while support for AV has remained relatively constant.
If (and it is a relatively big ‘if’) these trends continue the No campaign will triumph. However, there is clearly a large number of people who do not know how they will vote, and there are still three weeks of the campaign to come.
The variation between the different approaches illustrates that there is potential for respondents to change their mind in light of the campaign to come, and so therefore, there is still everything to play for.
As every A-level Politics student knows, ‘a week is a long time in politics’- and we have three ‘long times’ remaining.
