Back in 2011 Avinash wrote an excellent blog post about the value of controlled experiments called Measuring Incrementality: Controlled Experiments to the Rescue! In his post he describes the application of a test vs control design to breaking down how different channels of communication perform separately vs when used together. This is a good application of the controlled experiment methods.
Avinash is clearly very impressed with the methodology. I was really surprised that he called it “advanced”. As far as methods of measurement goes, simple test vs control design like he describes does not require the use of advanced methods, which is really an advantage of its own. You want simple and elegant? That truly describes what test vs control is. You would need to have a little bit of statistical knowledge, and the rest can be figured out with some logical thinking.
Some quotes and my reaction:
It turns out if you completely stop marketing, and you are an established company, the impact is not that your revenue goes to zero! 🙂
Avinash Kaushik
No doubt! That is the point that most experienced marketing people tend to willfully ignore, and most digital marketing people seem to be completely unaware of. Lots of them are reading Avinash’s blog, so I am hoping they are paying attention.
It is possible to go back and use this incrementality to make solid, long-term new decisions for the business (and not to keep doing what you have forever until your business goes bankrupt).
I could not agree more.
It is possible to take politics, bickering, back stabbing and all that ridiculous stuff out of the picture. Okay maybe not all of it, but a lot of it.
Unfortunately, no. In fact, bringing up the results of controlled experiments usually makes a lot of people look not quite as good as they see themselves. Then, the claws come out. They pull out their direct response results (clicks, calls to toll free numbers, all sorts of funnel stuff), and start indignantly shaking them “So, are you saying all those click/calls/connects/sales are not real? Look, there are real customers who bought because of our program!”. – “No, I am not saying they are not real. I am saying they are not incremental“. Been there, done that.
It is possible to determine the value of doing Paid Search campaigns for brand terms where you already rank #1 via SEO. It is possible to understand if you should invest in Facebook at all. It is possible to understand how much to support your TV campaigns via Yahoo! display campaigns. It is possible to specifically nail down every incremental dollar added to the bottom-line of adding YouTube to your Search campaigns and then adding radio campaigns and then adding magazine ads and then adding Twitter. And along that chain it is possible to understand exactly when you’ve reached diminishing margins of return!
Yes, it is possible, if you do the right experiments.
Work as hard as you can, and then some, to ensure that there are as few “disturbances” in your test and control group. In the middle of the experiment don’t start a new paid search campaign or tweeting like a crazy duck to the same set of customers. Shout loudly until the entire company knows what you are up to (and beg for their co-operation).
Avinash Kaushik
This is not a concern for a well designed controlled experiment. The beauty of the design is that it acts as “the great equalizer”, as long as the disturbances impact both test and control group equally.
Twitter campaign? That’s absolutely fine. You will be measuring the incremental impact your test communication given the conditions on the market, which in your case include a Twitter campaign. Or a paid search campaign. Or a TV campaign. In fact, that’s exactly the reason why controlled experiments were invented – so you can control for other things happening on the market, like TV, Twitter or paid search.