The Halo Effect

I was recently reading an article where the author noted, quite correctly I might add, that if your agents are rated highly and all your performance metrics are good, especially if they hold up when bench marked, then you know that it is the product or another area of business that is at fault. The other side of the coin is that sometimes a customer will rate the customer service agent low even though the agent performed adequately. When this happens the scores for all areas tend to have the same value. If the customer was extremely unhappy, the scores in all areas might be very low. On the other hand if all the scores are high the customer might have been extremely happy. When a survey response indicates a consistently high or low score, it is often noted as a halo effect.

I have often seen survey survey questionnaires that showed that the company scored top box in every area. While it is nice to imagine your company is perfect, it is probably not true. Similarly, I have also seen many surveys that showed the company scored bottom box in every area. Just as having all top box scores is unlikely, having all scores in the bottom box is equally unlikely. These responses indicate customers who are either very happy or very unhappy. The phrase halo effect was initially used to identify customers who were very happy and were inclined to report that everything was perfect. These customers saw a halo over everything the company did. It quickly expanded to include those customers who were very unhappy and were sending a message that that the company could do nothing correctly.

There are even variations on the halo effect. I saw a survey which had a scale of 1 to 10 and the customer had scored every question 9. The comment from the customer was that only God could receive a 10. Once again, there is a halo reflecting an overall perception rather than a thoughtful consideration of the products/services received.

The question is what to do with responses that indicate a possible halo. The first problem is which question triggered the halo. Without knowing which question instigated the halo, the choices are to either accept all the scores or none of the scores. If the comments suggest which question triggered the halo, that question score would be reasonable to include any analysis. Once again the problem is whether or not to include the other data or ignore them. If the sample is small, I suggest that the others be ignored since the variation caused by the small sample may distort the results and lead to erroneous conclusions.

The bottom line is that halos can be found in most survey data and if they are ignored the results and conclusions may be erroneous. Many people who offer themselves as survey experts know little about the many ways results can be distorted and what to do with them. When people talk about scrubbing the data, it is not a simple process. Remember the adage "garbage in gives you garbage out" is no longer valid. The real adage is "garbage in gives you gospel out" since no one ever questions the computer.