Controlled Experiments in Marketing

We all want marketing programs that work. But how do we prove that something is working?

The definition of effective marketing programs is that they produce results that are incremental to status quo. It is the difference between having the program in place and not running it that we are after, and this difference is rather elusive.

It is hard to measure what happened. It is much harder to measure what would have happened.

With so many things going on in the marketplace, attribution can be hard, especially, for an established business where many sales happen spontaneously. For example, if the company runs TV ads, radio ads, internet campaign, and sends out direct mail pieces, how fair it is attribute all sales that came from the mail piece phone number to direct mail? Many years of my marketing analytics experience suggest that attribution based on the "funnel tracking" (toll free numbers, clicks) greatly overstates the contribution of trackable advertising channels such as direct mail and internet ads compared to other channels, such as radio and TV.

Design of experiments offers a solution to this problem: use of control groups. Control groups are groups of customers that are excluded from the marketing communication. By measuring the total sales from the customers who are exposed to the communication and those who are not, we can correctly attribute incremental sales driven by this particular communication to it. In effect, we measure both what happened and what would have happened if we decided to not run the campaign, and take the difference.


Basic terminology used in controlled experiment design:

  1. The group that gets exposed to the marketing campaign is usually called treatment group or test group. A more precise name like mail group or campaign 1 group is also common.
  2. The control group sometimes is called a hold out group.
  3. The groups need to be very similar, also called matched or representative. Random assignment of subjects to the groups is the gold standard to achieve representative groups.


Continue reading on experiment design in marketing:

  1. How to Create Effective Control Groups
  2. Experiment Design in Practice: When Random Selection is Impossible
  3. Control Group Dos and Donts
  4. Experiment Design in Practice: Control Group Size
  5. Experiment Design in Practice: Adjustment to the Measurement
  6. Special Type of Marketing Experiment: A/B Test
  7. Universal Control Groups and Advanced Experiments in Marketing