The Halo Effect on Satisfaction Metrics

There has been some very interesting research on the Halo effect on satisfaction metrics. The research was performed by Joachim Buschken, Thomas Otter and Greg M. Allenby. Their work is entitled "Do We Halo or Form? A Bayesian Mixture Model for Customer Satisfaction Data".

When we look at customer satisfaction data we often look at what are the drivers of overall satisfaction. We naively assume that each question is answered independently so that we can use techniques such as regression analysis to identify those factors that most strongly relate to overall satisfaction. When this assumption is satisfied the "drivers of satisfaction" are valid and management actions taken as a result should have a positive impact on the organization.

When the assumption of independence is not met, the identification of drivers loses its validity. One of the most serious deviations comes from "Halos". Halos occur when a customer responds to a survey such that all the scores are identical and usually all top box (or bottom box). The customer is saying that the company scores top box with respect to every question. In other words, the company is perfect and all the areas requested for evaluation have outstanding performance. When all the scores are identical the term often used to describe this response is Halo. There can be a negative Halo as well as a positive Halo. The same logic works for the negative Halo, the customer is saying that all areas of the company being evaluated are equally dramatically under performing.

The Halo effect occurs most often when the respondents recall their overall satisfaction and then assign scores for each question consistent with their assessment of the overall score. When this occurs the components are uninformative about the drivers of overall satisfaction. In this case regression of overall satisfaction or dissatisfaction is not possible because the components are co-linear.

When conducting statistical analysis to determine the influence of drivers on overall satisfaction the halo responses must be removed from the response data set.
While many survey analysts try to find ways to remove outliers, the Halos are responses that should be eliminated from any statistical analysis.

The researchers point out that respondents who are familiar with the product or service are more capable of providing insights are found to be more likely to Halo their responses. On the other hand those respondents who are less aware of the products or services and are less capable of providing informed are more likely to make their responses more independent and less likely to Halo their responses. The problem is that neither of these scenarios are what would be appropriate for driver analysis.

The bottom line is that Halos are bad news for satisfaction and loyalty surveys. It is clear that analysts who are doing driver analysis need to be aware of the impact of Halos. This is one exception to the rule that outliers must be included. As the researchers note "the analysis of customer satisfaction data for the purpose of understanding drivers of overall satisfaction requires removal of the Haloed responses. From a managerial standpoint, Haloed responses are uninformative about the drivers of overall satisfaction because the component-specific information is suppressed."