Measurement is Important

I recently responded to a blog that was critiquing the NPS measurement. My point was that measurement accuracy is important. Some of the comments in the blog suggested that even though NPS might not be accurate, it is simple to compute and easy to present to "the boardroom." Thus, the measure of NPS should be used. I disagreed with this approach and made the point that we must strive for measurement accuracy. Anything less has two major problems; namely,
1. The measurement may provide misleading results that would encourage a company to invest in areas that may have little or no impact on the company performance. This is especially true with NPS since its major selling point is that its use will lead to improved financial performance.
2. Perhaps even more important than providing misleading information is the notion that those of us who are in the customer measurement business are responsible for the accuracy of the measurements we use. If there is any doubt about the validity of the measurements, it will reflect on us.

An interesting measurement occurred while I was employed by Xerox Corporation. At the time I was responsible for measuring service parameters for all copiers and service operations in the United States. The measurements were important since they were used to establish the field service budet (which at that time was large enough to rank the service business as a Fortune 500 company). We introduced a new desktop copier and projected the installation time to be about 1 hour. We thought we would check this out by tracking the first several thousand installations and compare the actual installation times with our estimate so that we could provide the best estimate for the field budget.

After several thousand installations we examined the data and found that the average install time was 1.00 hours with ZERO standard deviation. This is easily translated into the following statement "EVERY IINSTALLATION TOOK EXACTLY 1.00 HOURS." There is no way that this could be true! So, what did we learn about this measurement?
1. We learned that this measurement was not accurate since there was no chance of tha many copiers to take exactly 1 hour of installation time.
2. We learned that the field service organization had inside information that we had projected a 1 hour installation time.
3. The field organization believed that by giving us what we projected they were helping us.
4. We now had no idea of how long it took to install that copier and hence could not provide an install time for the field budget.

The good news is that this process was easily fixed by pointing out the impact of their actions and to my knowledge it has never happened again.

So, the moral of this episode at Xerox has several dimensions which can easily be extrapolated to the current situation with the NPS measurement.
1. Be careful what you measure.
2. Make sure the measurement makes sense.
3. Know who is providing the data,
4. Make sure you know the agenda of those providing the data.

I think it is necessary for the industry to make sure that the measurements we support are valid; otherwise, we will reap the consequences.