Thursday, December 30, 2010

The Halo Effect on Satisfaction Metrics

There has been some very interesting research on the Halo effect on satisfaction metrics. The research was performed by Joachim Buschken, Thomas Otter and Greg M. Allenby. Their work is entitled "Do We Halo or Form? A Bayesian Mixture Model for Customer Satisfaction Data".

When we look at customer satisfaction data we often look at what are the drivers of overall satisfaction. We naively assume that each question is answered independently so that we can use techniques such as regression analysis to identify those factors that most strongly relate to overall satisfaction. When this assumption is satisfied the "drivers of satisfaction" are valid and management actions taken as a result should have a positive impact on the organization.

When the assumption of independence is not met, the identification of drivers loses its validity. One of the most serious deviations comes from "Halos". Halos occur when a customer responds to a survey such that all the scores are identical and usually all top box (or bottom box). The customer is saying that the company scores top box with respect to every question. In other words, the company is perfect and all the areas requested for evaluation have outstanding performance. When all the scores are identical the term often used to describe this response is Halo. There can be a negative Halo as well as a positive Halo. The same logic works for the negative Halo, the customer is saying that all areas of the company being evaluated are equally dramatically under performing.

The Halo effect occurs most often when the respondents recall their overall satisfaction and then assign scores for each question consistent with their assessment of the overall score. When this occurs the components are uninformative about the drivers of overall satisfaction. In this case regression of overall satisfaction or dissatisfaction is not possible because the components are co-linear.

When conducting statistical analysis to determine the influence of drivers on overall satisfaction the halo responses must be removed from the response data set.
While many survey analysts try to find ways to remove outliers, the Halos are responses that should be eliminated from any statistical analysis.

The researchers point out that respondents who are familiar with the product or service are more capable of providing insights are found to be more likely to Halo their responses. On the other hand those respondents who are less aware of the products or services and are less capable of providing informed are more likely to make their responses more independent and less likely to Halo their responses. The problem is that neither of these scenarios are what would be appropriate for driver analysis.

The bottom line is that Halos are bad news for satisfaction and loyalty surveys. It is clear that analysts who are doing driver analysis need to be aware of the impact of Halos. This is one exception to the rule that outliers must be included. As the researchers note "the analysis of customer satisfaction data for the purpose of understanding drivers of overall satisfaction requires removal of the Haloed responses. From a managerial standpoint, Haloed responses are uninformative about the drivers of overall satisfaction because the component-specific information is suppressed."

Monday, December 27, 2010

Impact of Lying on Surveys

Have you ever wondered what the impact of respondents lying on your survey might be? Repeated lying on surveys is known as farming and refers specifically to repeatedly completing a survey (correctly or otherwise). Thomas Chesney of Nottingham University Business School in Nottingham, UK and Kay Penny of Edinburgh Napier University in Edinburgh, UK have recently completed an interesting study to examine this possibility.

Web based businesses such as SurveyMonkey allow researchers or businesses to create and distribute online questionnaires quickly and at low costs. One of the main detractors to this method is the lack of certainty about who responded and how many times. As a secondary concern, how accurate are the demographic responses. Are the respondents who they claim to be? Some of the more sophisticated survey processes will not allow the same code to be used more than once; however, there may still be ways of providing multiple respondents. For example, it has been suggested that some respondents voted more than once for Sarah Palin's daughter when she was competing on the TV program "Dancing with the Stars."

There have been studies that show that an incentivize survey often leads to an increased response rate and the greater the incentive,the greater the increase in response. In the same way others have found that one big prize increases the response more than many small prizes despite the lower odds of winning. Another way to increase participation to use a prize draw rather than prepaid or promised monetary incentives.

A particular study by Morabia and Zheng in 2009 investigated the influence of entry into a raffle as an incentive for participation to an urban transportation survey in New York and found that approximately 4.7% of the participants were thought to have responded twice since they gave the same email address (are we talking dumb here).

A more subtle concern about lying on a survey is the respondent may be asked to answer questions that my impact their job. Something that business people often overlook is the possibility that one or more competitors may be responding to their surveys in ways that would distort the results and increase the possibility that decisions made as a result of the responses would be in error.

In reality there appears to be three basic types of farming; namely, the first is to repeatedly tell the truth, the second is to provide a random answer to each question and the third is to give perceived average responses to every response.

The finding of these researchers is that farming can lead to real and potential problems which cannot be predicted. The errors can be either a type I or type II error with no simple way to reduce or eliminate the errors.

The bottom line is that surveys on the web present real opportunities for lying and distorting the truth The best way to remove this possibility is good management. Of course, it can be virtually eliminated with phone or other direct survey techniques which are both more timely and expensive. Just another concern for those of us who attempt to extract accurate information from surveys.

Thursday, December 16, 2010

Customer Advocacy

With the increasing use of social media, the concept of customer advocacy is starting to have legs. The Gartner research group has reported that more than $1.2 billion is spent on loyalty programs in the US. They also suggest that more than 75% of consumers have at least one loyalty card and more than 1/3 of the shopping population has two or more loyalty cards.

The idea of customer advocacy is based on the notion that someone who espouses a product, service or company is an advocate. Some loyalty scales used in the market describe people who score high on satisfaction surveys as advocates. The bad news is creating an advocate may not necessarily follow a score on a satisfaction survey. We have not seen enough evidence to validate the fact that a high satisfaction score creates an advocate; however, it is fairly certain that an advocate should be very satisfied before that person starts advocating a product, service or company.

The good news is that WOM is relatively inexpensive. There is some interesting research that bears prominently on this advocacy topic. The research was published in the September, 2009 issue of the Journal of Marketing. The study was titled "Effects of Word-of-Mouth Versus Traditional Marketing: Findings from an Internet Social Networking Site." The major finding was that WOM has a much longer carryover period. This suggests that WOM will persist long after normal activities designed to increase customer loyalty. The technical description of the result is that WOM is about 2.5 times higher than the average advertising elasticity.

The bottom line is that companies should start to focus their programs toward WOM on social networking sites and reward customer advocacy behavior. The challenge is going to be how best to get customers to become advocates. We will be searching the market for successes in finding and tracking customer advocates on the social networking sites.

Tuesday, December 7, 2010

Endowed Progress

Have you ever heard of "endowed progress"? If you never have, you might want to read a research paper by authors Nunes and Dreze entitled "The Endowed Progress Effect: How Artificial Advancement Increases Effort." These researchers uncovered another people trait that improves sales, profit and (probably) loyalty. Endowed progress means that people who are given an artificial advancement toward a goal (such as a free trip) show greater persistence toward reaching the goal than they otherwise would. Artificial advancement is a way of appearing to give a customer a head start toward a goal even though it still takes the same effort to reach the goal. The example they use is an offer of a free airline ticket after 8 trips will not have as many people reaching the goal of 8 trips as when the offer is for a free trip after 10 trips but with a two trip head start. While both offers are the same, these researchers proved that customers were more likely to complete the 8 trips in a 10 trip program with a two trip head start than just doing 8 trips.

What is happening is that the task is reframed as a program that is already started. The customer tends to increase the chance that 8 trips will be taken and in a shorter period of time. They found that as people progress toward a goal, their effort will increase to get completion and as a result completion time will decrease given the head start.

Another finding from the research is that customer persistence depends on the relative progress made by the customer but NOT on the amount of the reward points or miles or other goal. thus, we might expect the customer will work harder as the goal comes into sight.

Their third finding was that endowed progress appears to be more effective when the customers are provided with a reason for the endowment. The customers want to know that there is a reason you are giving them a head start. The reason may be as simple as "we are starting a new loyalty program and are offering you this incentive to join our rewards program".

While I have never been a proponent of rewards programs, if you must use them, you might want to take advantage of this research. Although it suggests we are like sheep, their research is compelling. Try it, it just might provide a way of increasing sales, profit and loyalty.

web visitor stats
OptiPlex 755 Desktops