For eight years YouGov conducted regular surveys for the Daily Telegraph. These helped to establish YouGov’s reputation as Britain’s most accurate polling company – not only in the 2005 general election, but in the Conservative leadership elections of 2001 and 2005, and in last June’s Euro-elections.
YouGov parted company with the Daily Telegraph a few weeks ago, when we agreed a deal with News International to supply daily polls to The Sun and Sunday Times.
This week, Robert Winnett, a member of the Telegraph’s political staff, emailed me saying that he planned to write an article about YouGov in tomorrow’s newspaper. Below are Winnett’s comments and questions in full, and my replies.
Robert Winnett: The Daily Telegraph has been analysing the methodology employed by YouGov in producing its opinion polls over the past few weeks.
As you will be aware, the opinion polls produced by your company are currently more favourable to Labour than your competitors. According to our calculations, during March, your polls have showed an average Conservative lead of 4.9 percent compared to 8 percent for your competitors.
Peter Kellner: Comparing the average of our March results with those of our established rivals (ICM, Ipsos-MORI, ComRes, TNS), I calculate that the figures are:
YouGov: Con 37%, Lab 32%, Lib Dem 18%
Other companies: Con 38%, Lab 31%, Lib Dem 20%
The remarkable thing, given the variety of methods employed, is how close we are, not how far apart.
The "other companies" figures change if you add in Angus Reid, Opinium and Harris. All the double-digit leads reported in March were from polls conducted by these three companies. They are more out of line with the rest of the polling fraternity than we are. In particular, Opinium and Harris have reported “other” parties averaging 17% in March, and Angus Reid 15%, compared with 12% for the rest of us, including YouGov. If you are looking for outliers, that is where you will find them.
I have not included Populus, which IS an established company, but which published no GB voting intention polls in March. However, Populus DID conduct a poll in Labour marginals for The Times, around the same time as we conducted one for Channel 4 News. The two polls produced virtually identical results – the Tories gaining enough seats to be the largest party but not quite enough for an overall majority. A separate, slightly later, Ipsos-MORI poll in labour marginals recorded a slightly smaller swing. So no evidence there of any "Labour bias" in our data relative to other companies.
RW: You may also be aware that senior Conservatives are becoming increasingly vocal in their criticism of your polling techniques accusing YouGov of having a “pro-Labour bias”.
PK: Their views cannot be very widespread, for the party has commissioned a considerable amount of private polling from us. If such "senior Conservatives" in fact exist, I suspect they are making ill-informed comments about us as part of some internal party friction and using gullible journalists to advance their cause. It wouldn't be the first time this has happened, as you well know.
RW: Against this background, I would be grateful if you would respond to the following points:
You recently conducted a joint analysis of polling data with Experian. It is our understanding that Experian’s initial analysis showed the Conservatives had a lead of 12.5 percent over Labour. We also understand that Experian were satisfied that this analysis was statistically sound. However, you changed the figures to show only a six percent lead. It has been alleged to us that the re-weighting was unnecessary considering the large sample size. Please could you explain your justification for doing this and what statistical formula you applied?
PK: The raw data we supplied to Experian was unweighted, and from a number of polls, some of which were for market research purposes and not specifically designed to be politically representative. It is common practice to weight raw data. All reputable polling companies do this. You say Experian regards the data as "statistically sound". So do we. That is not the same thing as saying it did not need weighting. The person trying to persuade you this is a serious issue is either ignorant, or mischievous, or both.
RW: Labour responders are given an increased weighting in your polls. Can you explain why this is the case?
PK: This is not the case at present to any significant degree. I have looked at the data from our last five surveys. In total, the figures for Labour are: unweighted, 2011; weighted: 2087. We therefore upweighted Labour by 76 people. In total samples of c. 7500, this makes a difference of just one percentage point. If you look at figures for different parties produced by different polling companies, you will find that this is by no means exceptional - in ICM's latest poll, they downweighted Labour from 235 to 202: a change of more than three percentage points, in a total sample of 1003. I say this not to criticise ICM, which is a well-run company and was right to do what it did; but if your point is that variations between weighted and unweighted data are evidence of ‘sin’ (which I would dispute), then there are far worse ‘sinners’ than YouGov! Again, I think someone who should know better is taking you for a ride.
RW: Has You Gov changed its weighting policy in the run-up to this election? If so, please could you give details?
PK: See the following commentary on our website, from two weeks ago:
All good polling companies keep their sampling and weighting systems under review, change them when necessary, and are transparent about it. We are no exception. If you seek to imply otherwise, you are simply wrong. (For example, MORI, to their credit, made significant changes to their methods after the London Mayoral elections two years ago.)
RW: We understand that you have privately conceded that the Sunday Times poll showing a two percent lead during the Conservative spring conference was a “rogue poll”. Why is this the case and do you regret releasing it?
PK: I have made no such comment, either in public or private. Normally, polls are subject to a margin of error of 2-3 percentage points (depending on method and sample size) at the 95% confidence level. That is, 19 times out of 20, they should be correct to within 2-3 points. But one time in 20, the laws of probability dictate that a perfectly well-conducted poll will fall outside this range. This is the normal definition of a "rogue poll". Our Sunday Times poll, showing Con 37%-Lab 35%, was NOT a rogue, in my judgement. I believe that the figures for Conservative and Labour were both well within the margin of error at the 95% confidence level.
RW: We also understand that, last month, The Sun rejected a YouGov poll showing a one percent Tory lead. Do you believe they were right to do so?
PK: Untrue. Our daily voting intention polls started appearing in the Sun on February 18. To test our systems, we started asking about voting intention, never intended for publication, for some weeks preceding that. Our poll showing a one-point Conservative lead was one of these. (It was conducted immediately after Piers Morgan's interview with Gordon Brown which, I believe, caused a real but short-lived movement in voting intention.) But this finding was never destined for The Sun and therefore never rejected by it. The Sun has published every voting intention result we have supplied.
One final point. This is not the first time our critics have attacked our methods. The last time was in April 2008, when Ken Livingstone at least had the decency to put his name to the criticism. He threatened to complain to the Market Research Society, saying our mayoral polls (which showed Boris Johnson consistently ahead) were wrong, because all other polls put him ahead or, at worse, neck-and-neck. In the event, our final poll, for the Evening Standard, was exactly right (Boris 53%, Ken 47%). Ken never pursued his complaint.
This time it seems our critics are sheltering behind the cowardly screen of anonymity.
You might also recall that our final poll in last year's Euro-elections for the Telegraph was almost exactly right, and closer to the result than any other company's polls.
Given the Telegraph's proud history of impartial and rigorous news reporting, I look forward to you providing a fair and balanced story!
