Interpreting results: Anthony Wells explains margin of error and highlights why some results can't always be taken at face value
In the small print of opinion polls you'll often find a ‘margin of error’ quoted, normally of plus or minus 3%. This means that 19 times out of 20, the figures in the opinion poll will be within 3% of the ‘true’ answer you'd get if you interviewed the entire population.
A poll of 1,000 people has a margin of error of +/- 3%, a poll of 2,000 people a margin of error of +/- 2%. The smaller the sample, the less precise it is and the wider the margin of error. Strictly speaking, these calculations are based on the assumption that polls are genuine random samples, with every member of the population having an equal chance of being selected. In many cases this isn't true ‒ polls are carried out by quota sampling, or from panels of volunteers. Even polls done by randomly dialling phone numbers aren't truly random, as the majority of people decline to take part. Even so, the margin of error is still a good rough guide to how precise a poll in, and indeed, when measured against real events like general elections most polls are indeed within the margin of error of the real result.
However, it is important to note that a margin of error applies to the whole sample. All pollsters who are members of the British Polling Council, like YouGov, will publish computer tables showing the detailed results of the poll, which will include crossbreaks breaking down respondents by age, gender, social class, region and other demographics. While these offer great insight into patterns of public opinion, they do, naturally, have smaller sample sizes. For example, a poll of 1000 people will normally have around 500 men and 500 women, and the margins of error on those figures will be around +/- 4%
For smaller demographic groups, sample sizes are even smaller and these bring with them much larger margins of error. For example, a poll of 1000 people would have a margin of error of +/- 3%, but if there were only 100 Scottish respondents within that poll the Scottish figures would have a margin of error of +/- 10%. This means unless the difference between what Scottish respondents said was different to what the rest of the sample said by more than 10 percentage points, it would not be statistically significant. It could just be random error.
The error is particularly common when looking at responses of ethnic minorities or religious minorities in national polls. Britain is an overwhelmingly Christian or secular country, meaning that in any properly representative poll of the British population, only a small percentage of respondents will be Muslim, Hindu or Jewish, and any crossbreaks by religion or ethnicity will be based on very small numbers with very large margins of error.
Remember, it isn't just the sample size of the overall poll that counts, but the sample sizes of the crossbreaks too. It is very rare that crossbreaks of fewer than 50 or 100 respondents will tell you anything reliable or useful.
On Monday the 21st of November 2011, think tank Demos published A Place for Pride - a report about patriotism which built on a series of focus groups and a YouGov poll of 2,086 British people. One of the press releases accompanying the launch of that report made reference to the fact that 83% of Muslims covered by the poll responded that they were 'proud to be a British citizen', comparing that figure to a baseline - drawn from the whole sample - of 79%. This aspect of the Demos report was widely reported in the press.
It is true that this finding, accompanied in the research report by findings from focus groups in which British Muslims participated, is broadly in line with previous polling evidence on the matter.
It should be noted, however, that the sample size for British Muslims was relatively small, just 48 people, and it is questionable whether confident statements can be made on that basis about one group being more proud of their British identity than another (as we saw in some of the press coverage). The more relevant point, stressed in the Demos report, is that people who declare a faith are more likely to be proud of their British identity than the population at large which argues that we should not perceive minority faiths as potential threats to either cohesion or patriotism.
