Sunday, 5 January 2014

What Do You Think?

Sutton Council are asking people to fill out a survey giving their thoughts on the recent changes to the Central Road area in Worcester Park. You may have had one of these (shown right) through your door. There is also an electronic version which can be found here. However more than one person has contacted me to say they feel the survey itself is a bit biased. One person used the phrase:
“…don't get me started there on how biased that survey is.”
So I thought it was worth having a closer look…

Answer Options

I note the four possible answers for the questions are: ‘agree’, ‘agree somewhat’, ‘disagree’ and ‘not sure’. Does something appear to be missing?

Imagine an agreement scale with five positions, with Agree at one end and Disagree at the other end (just ignore the ‘not sure’ option for the moment).

Agree Disagree

If the scale is unbiased then the central position should be the neutral point, let’s call it ‘neither agree nor disagree’.

Agree Neither agree nor disagree  Disagree

‘Agree somewhat’ should then slot into place between 'Agree' and the neutral point:

Agree Agree somewhat Neither agree nor disagree  Disagree

Leaving the fourth spot for ‘Disagree somewhat’

Agree Agree somewhat Neither agree nor disagree      Disagree somewhat      Disagree

That would be a balanced unbiased survey that allows the full range of options for what people want to say.

In a three option unbiased survey (again ignoring ‘not sure’) then these options should be:

Agree  Neither agree nor disagree  Disagree

What looks to have happened here is that a five option survey has been cut down to a different three option survey by having a different two options removed leaving this:

Agree Agree somewhat Disagree

This means that anyone who wants to take a neutral position will have to choose ‘Agree somewhat’ as the closest available option.

Reporting the results

Now when these survey results are collated and published, what will they say?

One way statistics are often reported is to lump groups together so if equal numbers had ticked each box on the full five category survey question (shown above), you could say that 2 out of 5 (40%) of people ‘agree or agree somewhat’ with the statement. That is 1 in 5 or 20% who chose ‘agree’ added to the 1 in 5 (or 20%) who chose ‘agree somewhat’ making 40% in total. And in the same way 40% ‘disagreed or disagreed somewhat’ with it. You can then change the language a bit and say that 40% of people generally agreed with whatever is being asked.

However with this survey, if an equal amount of people tick each of the three boxes, it could be reported, quite honestly, using the same technique that two thirds (one third choosing 'Agree' plus another third choosing 'Agree somewhat') of people generally agreed that all these changes have ‘improved Worcester Park’ (perhaps rounded to 67%). Even though many of those who ‘generally agreed’ may have actually wanted to indicate a more neutral position that would have been better described as ‘Neither agree nor disagree’. Even those who might otherwise be inclined to ‘Disagree somewhat’ may choose the ‘Agree somewhat’ option rather than the ‘Disagree’ option, feeling that the ‘middle option’ better reflects that they don’t hold a particularly strong opinion even though it would fall on the disagreeing side.

So in fact we may end up with all the ‘neutral’ and some slightly negative responses being added to the ‘positive’ responses when the results are announced.

This begs the question, “Do they want to know what people actually think or are they looking to get a specific result for some reason?”

Choice Of Questions

Also I can’t help but feel that most of their questions focus only on the likely positive aspects of the works without asking about aspects that may not be so popular. Additionally the works are called improvements all the way through the survey – so if you so happen to think these changes have made the area worse – well you’re just wrong aren’t you! That’s not to say that they have made it worse or that the changes can’t be considered improvements, just that an impartial survey shouldn’t make value judgements up front about the very things it is asking people to comment on.

In order to rebalance what some have considered an inherent bias in the official council survey, I have devised my own survey. I have merely added the two missing response options and the questions which I think should have been on the official one but have been missed off for some reason. I have not included the questions from the main survey, only the missed out ones so you can do this one in conjunction with the official one.

Please let me know if you would like other questions added to this as I can still add some!

The missing questions: