Proper surveys should avoid leading questions

 

The Market Research Society defines a “Leading question” as one that is “…badly constructed [and] tends to steer respondents toward a particular answer.” It is for obvious reasons never a good idea to have such a question in any poll whose findings you want to use seriously.

The leading pollster YouGov has recently published poll results which purport to show British attitudes to immigration. The answers to two of the questions have generated headlines. This is how YouGov themselves presented the results:

“Moving onto the issue of immigration, 76% of people support David Cameron’s stated aim of reducing immigration to the “tens of thousands”, but the overwhelmingly majority (83%) of people think it is unlikely he will achieve it, only 9% think it is likely. When YouGov asked the same question two years ago 15% thought it was likely Cameron would hit his target, so while net immigration has fallen somewhat over recent years, its not registering with the public.”

But when we actually look at the survey results and in particular the questions this looks like very sloppy research and the comment is tendentious. Here are the questions:

 

Both of these questions are “leading”. In the first one the respondent is lead by the “David Cameron has pledged to reduce net immigration” introduction into believing that such a policy must be a good thing. There is no attempt at balance in the question. Obviously if reducing immigration is a good thing then the bigger the reduction the better. So 76% agree that it’s a good thing. Hardly surprisingly.

In the second question the very nature of the question suggests that for Cameron to be able to “deliver the pledge” is questionable. And given inbuilt attitudes to politics and politicians at the moment the respondent is likely to disbelief the pledge – as 83% of the respondents do.

You could ask the same questions in a different way and get completely different results. For example if you said in place of the Cameron pledge intro “A recent study by University College London has shown that immigrants have a net positive effect on the UK economy…” then respondents would answer very differently. This would, of course, be equally wrong! Leading questions are always wrong.

A fellow twitterer, @baggins_dil , has drawn my attention to this from “Yes Minister” which makes the point very well and humorously. Essential viewing !