So, you have decided to measure your marketing campaign against a control group.
Congratulations! You are on the right path.
Now you need to figure out how many targets you need, particularly, how many targets needed in a control group.
In his post he describes the application of a test vs control design to breaking down how different channels of communication perform separately vs when used together. This is a good application of the controlled experiment methods.
The goal of control group selection is to keep the test (treatment) and control (holdout) groups representative of each other, both in terms of sample composition and in terms of measurement. The only difference between the groups should be where your treatment is applied.
Did you ever wonder why your results always seem to go the same way? Many analysis setups have subtle biases that drive conclusions in the same direction. Learn to identify and eliminate them for high quality results.
A/B testing is a special case of experimental design, aimed at determining a better performing option of communication. In practical terms, it means that all of the targets in the A/B test get exposure to some version of marketing communication while in test vs control design only the test targets get exposed.
One of the most important principles of this design is making sure the control is representative of the treatment group. This is typically achieved by random assignment. However, sometimes random selection of customers into the groups is impossible.
we must make sure the control group mimics the behavior of the test group. This phenomena is described as groups being matched or representative of each other.
Measurement of overlapping campaigns when using controlled experiment design has been a bit of a tricky point for many controlled experiment proponents. Some claim that you have to hold other campaigns when measuring, which is a difficult proposition, especially if the goal is to measure every campaign as BAU.
In compound experiments universal control groups are used in combination individual campaign control groups, providing very powerful tools for sales attribution.