Benchmarks - How Good Are They?

In the latest issue of Quirk's Marketing Research Review (April 2010), there is a survey of researchers regarding benchmarking. The survey was performed in the Spring of 2009 with 97 responses from people involved in market research in their organizations (83 have a designated market research function). The survey was performed by Jennifer Van de Meulebroecke and Michele Sims both members of the staff at TRC Market Research in Fort Washington, PA. The researchers represented many markets including insurance, utilities, high-tech, health care and financial services.

The researchers noted that:
1. 75% of the researchers collected benchmark data as part of an internal study,
2. 56% of the researchers collected benchmark data separately or at a different time than other studies. Some also use an outside vendor for assistance.
3. 56% of the researchers used a syndicated source for their benchmark data.

The research went farther by asking the researchers to focus specifically on benchmarking data and describe their confidence in making comparisons for their company with benchmark data. The results indicated that 75% of the researchers who had less than 11 years in market research had confidence in using external benchmarking data; whereas only 58% of those with more than 11 years in market research had confidence in using the external benchmarking data for comparisons. The conclusion drawn from this piece of information is that it appears that the longer people are in market research, the less trust they have in using the external benchmark data to draw comparisons.

Since there are such low scores with respect to using external benchmark data, the question is what are the factors that people should be aware before using external benchmark data. The survey found the following factors and are ranked in order of importance by those who were surveyed:
1. 91% were concerned about the consistency of the scales and responses,
2. 83% were concerned with consistent question wording,
3. 77% were concerned with consistent screening,
4. 67% were concerned with the method of collecting the data (web, phone, etc),
5. 58% were concerned with type of sample that was used, and
6. 51% were concerned with the time period for data collection.

As an editorial comment I would note that many companies put extreme value on industry benchmarks. I had a consulting assignment several years ago with a high-tech company that had the lowest scores on their industry benchmark. The executives and managers were very concerned about their score. They believed, and rightly so, that thefact that the benchmark that showed they had the lowest scores in their industry would impact their sales. They asked for my help and within 3 years they were tied at the top of the industry benchmark. The problem was that they were scored low but did not have enough information to know where to make changes to their product nor support. Once we captured the information it was only a matter of time before they moved to the top.

The bottom line is that benchmarks are becoming more and more important to companies. Today many companies use benchmark data to assess their market position without understanding how the benchmark was created. In addition, the companies do not have the detail information necessary to accurately evaluate the meaning of the benchmark scores. The lesson is benchmarks are very valuable when used properly but can be VERY misleading when not properly understood.