Bad Metrics

There was an article in the March 3, 2008 issue of BusinessWeek that announced the 2008 winners of customer service. They list the top 25 companies that are, according to BusinessWeek, the Customer Service Champs. Sounds good. The top company is USAA with a score of 1030.66 points The last company on the list of the top 25 is True Value with a score of 857.21

My guess there are a lot of happy customer service executives that were happy to see their company on the top 25 list. In general, I think the companies listed do provide excellent customer service and may, in fact, be the customer service champs. When I look further into the way the companies were selected and then how the scoring was accomplished I cringe. This is a bad metric - it has too many flaws to be accurate.

The first problem is that there is no indication that the more than 1000 readers who responded to the survey represent a probability sample. Without a probability sample the results are questionable at best. Then J. D. Power created a web-based survey for the brands and surveyed at least 100 customers (for each brand?) to get a comparable score (whatever that means).

Then J. D. Power ranked all brands using scores from both their database and the supplemental surveys (ranking based on what?). Next the folks at BusinessWeek combined the "people" and the "process" scores from J. D. Power's data to create the service index with people weighted at 60% and process at 40%. There is no description of how the scores were combined (probably added). Further, the question is why choose a 60/40 split. No rationale is given for the ratio. Why not 50/50 or maybe 90/10?

That is not the end of this nightmare. Brands that were ranked first in their category received 100 bonus points (why not 82 or any other number - no rationale given). Those that ranked second received an additional 50 points (why not - this makes no more sense than the 100 points for first place). Hold on, those companies that fell below third place had 50 points subtracted (I can't believe a publisher would allow this to be published when company reputations are on the line).

Finally, those companies that did particularly well in the reader poll were awarded an extra 25 points if their vote-to-complaint ratios were in the top 10%. I guess 25points makes as much sense as the 50 and 100 points that are being thrown around.

The bottom line is that this is a bad metric because it is unlikely to be a good probability sample and then the scoring is absolutely arbitrary. Company reputations are being helped or hindered based on a metric that is not well founded.