Measurements Raise More Questions Than Answers

As we read some of the publications that are found on the web some of the information raises interesting questions. This may be one of the most boring blogs written for The Customer Institute. I am basing my concern on a chart that had some very interesting and, in my opinion, some worthwhile information. The chart was titled "The Impact of Problem Resolution on Loyalty." The chart showed information taken from five different industries and noted how the "loyalty" changed depending whether problems left the customer satisfied, mollified or dissatisfied. It also added the score if there was no problem. The group of customers that had no problem recorded the highest level of loyalty.

I actually liked this chart and the reason I am using it as an example is that it is a great example of making a very important point but then gets lost when the scales on the axes don't follow the content of the chart. The concern I have with this particular chart is that the scale used on the y-axis of the chart is labeled "Repurchase Intention." WOW!

How do you get from measures of loyalty to repurchase intention? While it is easy to say there is some relationship between loyalty and retention, it requires a GIANT leap of faith to then put it into a chart that does not indicate loyalty on the y-axis. Some of the questions that come to mind are:

1. Is the relationship between loyalty and repurchase intention synonymous? If yes, then it is ok.
2. If they are not synonymous is there a linear relationship between the two terms?
3. If there is a known linear relationship between these two variables, I have yet to see the data and research to support it.

The bottom line is that we must always pay attention to the charts that always seem so compelling. There is no guarantee that the charts represent the information correctly. I have witnessed past transgressions that included showing a linear relationship between customer satisfaction and loyalty. While the scatter plot of the data implied a linear relationship, the r-square for the linear relationship was so low that the assumption that a linear relationship existed took a lot of courage to swallow.

We need some carefully designed experiments to demonstrate the relationships between these terms that we live with every day. What is the relationship between satisfaction, loyalty, repurchase, etc.?

Measurement is Important

I recently responded to a blog that was critiquing the NPS measurement. My point was that measurement accuracy is important. Some of the comments in the blog suggested that even though NPS might not be accurate, it is simple to compute and easy to present to "the boardroom." Thus, the measure of NPS should be used. I disagreed with this approach and made the point that we must strive for measurement accuracy. Anything less has two major problems; namely,
1. The measurement may provide misleading results that would encourage a company to invest in areas that may have little or no impact on the company performance. This is especially true with NPS since its major selling point is that its use will lead to improved financial performance.
2. Perhaps even more important than providing misleading information is the notion that those of us who are in the customer measurement business are responsible for the accuracy of the measurements we use. If there is any doubt about the validity of the measurements, it will reflect on us.

An interesting measurement occurred while I was employed by Xerox Corporation. At the time I was responsible for measuring service parameters for all copiers and service operations in the United States. The measurements were important since they were used to establish the field service budet (which at that time was large enough to rank the service business as a Fortune 500 company). We introduced a new desktop copier and projected the installation time to be about 1 hour. We thought we would check this out by tracking the first several thousand installations and compare the actual installation times with our estimate so that we could provide the best estimate for the field budget.

After several thousand installations we examined the data and found that the average install time was 1.00 hours with ZERO standard deviation. This is easily translated into the following statement "EVERY IINSTALLATION TOOK EXACTLY 1.00 HOURS." There is no way that this could be true! So, what did we learn about this measurement?
1. We learned that this measurement was not accurate since there was no chance of tha many copiers to take exactly 1 hour of installation time.
2. We learned that the field service organization had inside information that we had projected a 1 hour installation time.
3. The field organization believed that by giving us what we projected they were helping us.
4. We now had no idea of how long it took to install that copier and hence could not provide an install time for the field budget.

The good news is that this process was easily fixed by pointing out the impact of their actions and to my knowledge it has never happened again.

So, the moral of this episode at Xerox has several dimensions which can easily be extrapolated to the current situation with the NPS measurement.
1. Be careful what you measure.
2. Make sure the measurement makes sense.
3. Know who is providing the data,
4. Make sure you know the agenda of those providing the data.

I think it is necessary for the industry to make sure that the measurements we support are valid; otherwise, we will reap the consequences.