YouGov experiments dig deeper into the importance of question framing for the measurement of public opinion
Question wording, and its effect on the results about public opinion that pollsters measure, has again been subject to some recent debate among practitioners and watchers of the polling industry. Two weeks ago, YouGov survey experiments reminded us of the importance of question wording — and, in particular, not using leading statements — in the context of the UK’s obligations to the European Convention on Human Rights.
This week, a survey experiment ran by another polling firm concluded that using the word “illegally” in a question in reference to migrants arriving on small boats from across the English Channel made little to no difference in the measured overall levels of support for the government’s new policy to send the people involved back to France and ban them from returning to the UK in the future.
While this was a welcome contribution to our knowledge about how and when particular wording matters, we can not and should not conclude from that evidence alone that question wording and issue framing never affects people’s views or policy preferences on migrants arriving by small boats.
Rather, fresh YouGov survey experiments now show that in fact the public will sometimes react quite differently to different ways of framing the policy and the debate itself. Namely, that support for the government’s position drops both when (a) using different framing of the issue, and (b) putting the government policy up against an alternative measure which offers a different resolution.
This week, we asked 2,155 respondents one of three random questions. The first potential question replicated the question fielded by other polling firms in their assessment of support for the government’s new policy proposal:
Do you support or oppose the following?
Migrants arriving in the UK illegally on small boats across the Channel should be removed from the UK and blocked from returning in future
A second question took exactly the same proposal but changed the framing; for instance, instead of ‘migrants’, we described the people as ‘refugees’. We also changed the language around ‘small boats’ to instead simply describe their entry as ‘by sea’:
Do you support or oppose the following?
Refugees arriving in the UK from France by sea should be removed from the UK and blocked from returning in future
Both wordings accurately represent the government’s policy, but use very different language to describe what it entails. The second question moves the framing of the policy away from the government position on the issue — that individuals arriving on boats across the channel are entering the UK illegally, and ought to be treated as law breakers — and instead toward arguments and definitions made by, for instance, human rights associations such as Amnesty International — that the individuals involved are asylum seekers who are making their way to the UK, by the only route available to them, to claim their rights under international law.
We noticed a meaningful difference in support for the government’s position based on which of these two questions were shown. In the first instance, replicating the wording from other polling firms, we found 53% in support of the idea, while 30% were opposed.
However, when we used the refugee framing, we found that support dropped by 5 points to 48%. Opposition to the proposal registered at 32%. This significant change in levels of support between the two treatments — binary logit coefficient: -0.23, p < 0.05 — highlights well how important it is to consider how question framing, not just the individual words or terms, can shape responses.
Though there was a difference, we still found that, overall, the public were more likely to support the policy than oppose it, regardless of which of the two wordings were used. In this sense, changing the wording does not change the overall finding of the poll (that the public are broadly in support of the policy), but it does change our understanding of the extent of that support.
The final question respondents could have seen put the government policy directly up against an alternative policy suggestion in a “which comes closest to your view” framework. References to ‘migrants’ or ‘refugees’ were removed and instead replaced with ‘people’.
This question design allows us to move away from testing policy proposals in isolation (which tends to inflate support, as respondents are not necessarily thinking about tradeoffs, priorities, or alternatives while answering), and also moves us past potential acquiescence bias problems where people have more tendency to agree with a given suggestion than to disagree with it (which can also come into play when using the support v oppose wording).
The alternative policy proposal offered in the question presented a different, but nonetheless specific, process for dealing with people arriving in the UK by boats across the Channel:
Which of the following comes closest to your view?
People arriving in the UK on small boats across the Channel should be removed from the UK and blocked from returning in future
People arriving in the UK on small boats across the Channel should be processed immediately and have their asylum applications assessed as quickly as possible
Compared to asking the question in a straightforward support/oppose framework about removal and blocking, we found a significant decrease in overall backing for the government’s policy proposal. Just 37% said that the government’s position came closest to their view. Meanwhile, support for the alternative suggestion – processing migrants' asylum claims as fast as possible – was roughly the same.
Put another way, support for the government policy drops by 16 points against the original framing, and by 11 points versus the alternative framing, when the government policy is put directly up against an alternative policy suggestion.
This does not mean that we should conclude that the government’s policy proposal is unpopular — but it serves as evidence to suggest that the British public are supportive of action being taken on the issue in general, and there are a range of potential policies they would support (not just the government’s).
So, what can we conclude from these latest experiments?
It seems clear that support for the government’s policy proposal to send people arriving by small boats across the Channel back to France, and ban them from entering the UK again, depends very much on how exactly the policy is framed.
If we replicate the government’s own language on the issue, we tend to see somewhere around half of the public (53%) in support of the idea.
However, if we use an alternative framing – namely, one which describes the people involved as ‘refugees’ rather than ‘illegal migrants’, we see that support for the proposal drops (to 48%).
Furthermore, we can also conclude that the policy itself is by no means the most popular (or only) potential route for the government to take on the issue. When tested directly against another proposal, the government’s position was effectively tied with the hypothetical alternative.
All of which should serve as a reminder that question wording and issue framing is always important to consider in public opinion research. As is the type of question set up we use when assessing support for policies.
Image: Adobe Stock
