The Problem with Assumptions

I just finished reading an article by Michael Lieberman in the November Issue of Quirk's Marketing Research Review. The article is titled Measuring and maximizing the ROI of a loyalty program. Needless to say, it caught my attention.

First let me say that Mr. Lieberman has presented an excellent case for showing how an ROI can be computed for a given loyalty program. He uses a technique referred to as Monte Carlo Simulation. This technique uses a series of random trials (events) to see how a system that is not deterministic will respond by going through the process a large number of times. By using some statistical distributions to represent the different ways elements of the system (customers) could act, the output provides a picture of those many trials. This technique has been used for a long time and has been well tested and accepted.

There are two very important aspects to performing a Monte Carlo simulation. The first is to understand the assumptions used and second is to compare the simulation results to actual results. Unfortunately, the magazine did not give Mr. Leiberman the space to fully demonstrate the power of Monte Carlo simulation and answer these two aspects.

The first aspect which was inadequately discussed was a clear and complete discussion of the assumptions in his model. The second aspect was to compare the simulation results to the real world that he was simulating. There was no there, there. The results of his simulation were not compared to actual data.

Some of the assumptions that should have been explained in the article might include:
1. the relationship between the dollar value of the purchases and the incentives given. Is the relationship linear, non-linear or ???
2. the variation of margins for different product mixes. Was this considered a constant?
3. the percentage of loyalty points that will be redeemed (if the data is categorized then the assumptions should be noted by group unless it is further assumed that the groups are all the same).
4. the belief that a customer's spending behavior will change because their spending has changed.

The second aspect is that results must ALWAYS be compared with the real world. Yes, computers can produce a lot of output that seems to represent the real world, but until it is tested and verified in the real world, it is just computer output. The fact is that the computer output may be 100 percent accurate or 10 percent accurate, but you will never know it unless it is checked against the actual system in place.

For example, when I worked at Xerox, we were investigating the idea of going to team service. One of the variables we needed to understand was the statistical distribution of the time to complete a service call. The assumption was that the service completion time was statistically distributed according to the negative exponential distribution. Before we ran our simulation model we built an analytical queuing model to get an idea what the optimum team size should be. We could not easily include meetings, lunches and parts chasing into our model so the only way to get an accurate assessment of the optimum team size was to build the simulation model to account for the vagaries that were not easily modeled analytically. As it turned out the service completion time distribution was not a simple negative distribution, it was a significantly different variant of the negative exponential distribution and one that made a difference in our results. While it did not change the optimum team size, it did impact the value of the team.

The point here is that you must carefully describe the model and ALL its assumptions so that you are not misled by the computer results. A friend of mine once said that GIGO does not stand for "Garbage - In Garbage Out." Rather it stands for "Garbage In - Gospel Out."

The bottom line is that Monte Carlo Simulations provide an excellent tool for pre-testing business policies. I believe that Michael Lieberman did do the work and managed the assumptions. Unfortunately, his article doesn't reflect that. Sometimes an editorial pen can kill you.