On survey bias

How do you bias a survey to get just the result you want?

Bias the device

Here’s how SNCF does it. They put some kind of device in the new renovated Saint-Lazare station. The display invites passers-by to tell how much they love the new station (of course, it is beautiful, it’s brand new, full of natural light, and though we suffered a lot with years of works, it was worth the pain). You have two choices : either click on a pink button with a big heart shape (saying, I love it), or take a picture of a QR code “to explain why you don’t love it”.

The surest way to get more than 90% approval.

Bias the questionnaire

But it gets better. After clicking a few times on the button (ok, maybe they handle that kind of childish behavior), I scanned the QR code. There, after waiting for long seconds, I was retargeted to a page saying “3221 people love the new station, and you?” with… a button to say “I love it”, and a smaller one to go to a new page to enter a message. All in a positive tone (help us improve!) So cute.

I left a message saying something about statistics and fiability. For now, I have just received the standard reply, with a thank you.

What is it good for?

From the url linked by the QR code, I found their campaign was using MyFeelBack solutions. So there are some customer feedback professionals making this kind of survey, and there are people making money. Well, I can only hope they take into account the bias introduced in their procedure when they analyze the result. How can they do this?

  • compare the number of clicks with the number of people coming to the station everyday (or, if they have the figure, the number that can pass near the device),
  • add a new device with a random question and estimate how much people want to click on it,
  • count the number of customers from which you can expect feedback with a button vs. a QR code and a form,
  • only take the written feedback into account.

Maybe the device is not entirely useless, after all? What do you think?