Detecting AI Bots in Surveys - The New Challenge for Researchers

Like most researchers, surveys are an important part of the toolkit. However, in recent years obtaining valid survey completes have become ever more challenging. There is general fatigue towards surveys and many would be aware of the falling completion rates. As a researcher you have to be quite creative to ensure that the right people are targeted, they give truthful answers and complete the survey to the end. This challenge was bad enough but now we have the rise of AI bots. Sometimes their answers are as good as human and very difficult to discern.

An excellent webinar by Cloud Research explained how the rise in bot responses is fuelled by 'Click Farms' based in countries where relatively small fees for completing surveys are well above the minimum wage in their countries. Using VPN and other tricks, those behind these scams pretend that they reside in the relevant country to qualify to take the survey. Some of the standard due diligence research techniques can be used to spot them but increasingly more sophistication is required to weed them out of research results. A couple of giveaways if you're looking to hunt them down:-

1. They tend not to speak the language and therefore any verbatim responses are usually unintelligible or not strictly relevant

2. They so much want to qualify for the survey that they will click fictitious products or places in your list questions

3. A simple survey is completed very quickly but a more complex one takes the bot longer than a human being to finish, because it has not been trained to answer many of the more specialised questions.

   

4. Acquiescence bias is common - so answers lean towards the positive.

5. Using User Research techniques when you view how a survey was completed you tend to see a lot of straight lines as it moves from question to question, rather than an arc shape which is typical of human beings. Additionally, they will be going off screen quite a bit to another tab with a translator to ensure that they understand the questions.

To ensure valid results we have to use a range of techniques to prevent these AI bots from spoiling results. For understandable reasons, these bots tend to be shy of face-to-face interview research. So maybe by adding a statement to your intro saying that the last two questions will be answered via a quick face-to-face interview may deter them. Read more about this in the whitepaper here.