I have collected the list of most important things to look out for when creating control groups. The bottom line, we must make sure the control group mimics the behavior of the test group. This phenomena is described as groups being matched or representative of each other.
- Control group must be representative of the treated group. This is most commonly achieved by random assignment of customers or subjects.
- In cases when random assignment is not possible, for example, your group has to cover the whole DMA or organizational unit, you want to find several units that are similar on parameters that impact your outcome/measure. In this case, it is best incorporate pre-test trends into the understanding of the test period outcome.
- If using matched groups is not possible for legal or organizational reasons, your best bet is to transform your control group to match your treatment group by selecting/dropping certain members and/or reweighing the outcome. This approach is called synthetic control group. This is the most analytically intensive way to measure test performance.
- The role of a control group is to show what would have happened given everything else that is going on in the marketplace except for the treatment you are holding them out from. There is no need to “rest” either of the groups or keep them clean from any program that impacts both test and control groups. This is one of the most important principles of the test design.
- Control group measures only the type of treatment withheld. If you withhold control from more than one treatment, it becomes a control for all of the withheld treatment put together.
- The measurement of the outcome should be identical for both groups. It is important to apply additional data transformation to the test group or its responses, you must do the same for the control group. Attributes like response window duration, type of action that is considered to be response, or whether a segment is excluded from the group should be identical.