Impact of Lying on Surveys

Have you ever wondered what the impact of respondents lying on your survey might be? Repeated lying on surveys is known as farming and refers specifically to repeatedly completing a survey (correctly or otherwise). Thomas Chesney of Nottingham University Business School in Nottingham, UK and Kay Penny of Edinburgh Napier University in Edinburgh, UK have recently completed an interesting study to examine this possibility.

Web based businesses such as SurveyMonkey allow researchers or businesses to create and distribute online questionnaires quickly and at low costs. One of the main detractors to this method is the lack of certainty about who responded and how many times. As a secondary concern, how accurate are the demographic responses. Are the respondents who they claim to be? Some of the more sophisticated survey processes will not allow the same code to be used more than once; however, there may still be ways of providing multiple respondents. For example, it has been suggested that some respondents voted more than once for Sarah Palin's daughter when she was competing on the TV program "Dancing with the Stars."

There have been studies that show that an incentivize survey often leads to an increased response rate and the greater the incentive,the greater the increase in response. In the same way others have found that one big prize increases the response more than many small prizes despite the lower odds of winning. Another way to increase participation to use a prize draw rather than prepaid or promised monetary incentives.

A particular study by Morabia and Zheng in 2009 investigated the influence of entry into a raffle as an incentive for participation to an urban transportation survey in New York and found that approximately 4.7% of the participants were thought to have responded twice since they gave the same email address (are we talking dumb here).

A more subtle concern about lying on a survey is the respondent may be asked to answer questions that my impact their job. Something that business people often overlook is the possibility that one or more competitors may be responding to their surveys in ways that would distort the results and increase the possibility that decisions made as a result of the responses would be in error.

In reality there appears to be three basic types of farming; namely, the first is to repeatedly tell the truth, the second is to provide a random answer to each question and the third is to give perceived average responses to every response.

The finding of these researchers is that farming can lead to real and potential problems which cannot be predicted. The errors can be either a type I or type II error with no simple way to reduce or eliminate the errors.

The bottom line is that surveys on the web present real opportunities for lying and distorting the truth The best way to remove this possibility is good management. Of course, it can be virtually eliminated with phone or other direct survey techniques which are both more timely and expensive. Just another concern for those of us who attempt to extract accurate information from surveys.