Universal Control Groups and Advanced Experiments in Marketing

Universal control groups are control groups that are being held out of multiple marketing communications. They are used to measure the cumulative impact of all of the communications the group is excluded from. In compound experiments universal control groups are used in combination individual campaign control groups, providing very powerful tools for sales attribution.

Universal control groups are commonly used to achieve these goals:

  1. Measure cumulative impact of sequential marketing communications.
  2. Measure cumulative impact of concurrent marketing communications.
  3. Measure multiple location based tests in retail.

Measuring sequential marketing communications with universal control group

Marketers believe that advertising has effects that outlive the duration of campaigns, which often refers to as brand equity. Cumulative effect of marketing campaigns over time means that there is a “leftover” effect from past campaigns, which adds up over time and transforms into the base level of sales.

Universal control groups can be used to determine the cumulative impact of marketing campaigns run sequentially. In this case, universal control group is a control group that is held out of all marketing communications at the start of the marketing campaign sequence.

Reviewing the performance of your treatment groups vs universal control group can inform up whether your communications have cumulative effect or not.

If your marketing has cumulative effect, the distance between universal control and treatment group will grow as time passes:

When there is no cumulative effect, the sales level at the end of individual campaigns should return to the same baseline level as the universal control group.

Here is the most important thing about using universal control groups for measuring sequential communications:

If you think your marketing has cumulative effect, do not measure the impact of a single marketing campaign in the sequence using the universal control group that has been held out from other marketing campaigns.

To measure the effect of an individual campaign in the sequence, you need to create a separate control group that is only withheld from this particular communication. Why? Because your universal control group stops being representative of your treated group after it has been withheld from previous communications.

This is how you should measure the impact of individual campaign in the sequence:

When individual campaign is measured against universal control only, it is impossible to isolate the effect of the campaign from leftover effects of other campaigns your treatment group has been exposed to.

The following example shows the measurement of the Campaign Communication #3 against both universal control and campaign specific control group. Campaign specific control group has been held out of Campaign Communication #3 only. It was subject to all prior communications, and will be subject of subsequent communications.

When the results of the measurement are summarized, we can see that the respective control groups show the impact of the communications they have been held out of.

Below is the interpretation of the results produced by a compound experiment that has both universal and campaign control.

(Treated Group) – (Campaign Control) = Performance of Campaign #3 Only

(Treated Group) – (Universal Control) = Cumulative Performance of Campaigns 1-3 during Campaign 3 window

(Campaign Control) – (Universal Control) = Leftover effect of Campaigns 1-2 on baseline during Campaign 3

While this problem does not arise when there is no cumulative effect of multiple communications, however, the best practice is not to use universal control group to measure parts of the campaign.

Measuring cumulative impact of concurrent marketing communications using compound control groups

This is the classic case of sales channel attribution, where you conduct an advanced experiment that correctly identifies incremental effects of each communication with and without the presence of other communications.

This powerful design can help companies not only measure incremental impact of each channel on sales, but to optimize marketing spend across channels. Universal control group is being held off from all marketing and is used to determine baseline sales.

In this sample compound experiment, we look at the effects of three marketing vehicles: content ads, email, and catalog. The results are measured through several groups that act as treatment and control groups against each other. Please note that while treatment/control split for email and catalog can be executed randomly on an individual recipient basis, content ads are usually turned on and off based on geography/ISP basis.

Here are the sample results of average sales to each of the groups.

We can easily determine sales lift over baseline, or universal control group, for each group:

The real power of this design is in comparison across different treatment groups, which allows us not only to conduct true lift attribution, but to make conclusions about effectiveness of different channels in the presence of communications from other channels. This is really the gold mine for marketing optimization.

For example, this analysis of effectiveness of content ads shows increase in sales for groups with and without content ads in the presence of email, catalog, email and catalog, and no other communications.

From this analysis we can conclude that Content Ads result in incremental increase of 2 units of sales in the absence of email communication, and in increase of 1 unit of sales in the presence of email communications. Please note that determining the effect of content ads in the presence of email is based on control group that receives email, thus control group for any given element should never be “cleaned” of additional communications present in the marketplace other than the one being measured.

Measuring multiple location based tests in retail

We have reviewed advanced attribution and program assessment using universal control groups, yet the most common use of universal control groups is in store-based retail.

Retailers love to test. They test merchandise assortment, store layouts, visual presentation, everything. Solutions like APT Test&Learn for stores automate and simplify testing in the retail environment. It is not uncommon for a large retailer to execute up to 100 tests at the same time. Since the smallest unit for these tests is a store, it can be hard to create tests and control groups where test results do not interfere with each other.

For example, if the minimum sample size for a valid test is 20 stores, a retailer would need 800 stores to run 20 tests: for each test there will be 20 test stores, 20 control stores, times 20. However, if we believe all tests can be measured against the same control group, we can potentially have 20×20=400 test stores and only 20 control stores, thus reducing the number of stores in the program to 420. However, if we believe that we can test women’s lingerie layout and toy merchandise assortment in the same set of stores without the results interfering with each other, we can reduce the number of stores in the program further – thus, being able to conduct more test in the company footprint.

This is why many retailers assign non-interfering tests to the same test groups, while holding out universal control groups.