If you are an analyst in a medium to large size company, you probably encountered this phenomenon. Most new programs that you assess are not profitable, and many existing programs have a low ROI. The implication is that as an analyst you have to deliver bad news to the business stakeholders.
I have seen this unfold many times, and I’ve always wondered why this was happening. The explanation is two-fold. First, analysts usually assess the marginal impact of the program. Second, optimized programs achieve low marginal ROI by definition.
Assessing the Marginal Impact of a Program
Assessment of the marginal effect is very common in analytical measurement. What it means is that your program is not evaluated in isolation, but when it is run on top of everything else your company is doing.
To illustrate how assessing marginal impact is different from assessing the total impact, let’s consider this simple example. Suppose your company runs only two kinds of marketing campaigns, email and TV advertising. You were able to split your targets into four matching groups, and they show the following impact of the programs on sales.
| Email Yes | Email No | |
| TV Yes | Both +5% | TV Only +4% |
| TV No | EMail Only +3% | Baseline +0% |
Let’s say your TV advertising needs an impact of +3% to be profitable, and your email marketing needs an increase in sales of +2%.
A quick look at the table above shows that not only are both vehicles profitable when run on their own, they also pay for themselves when run together.
However, in real life we never have a whole group of customers not targeted for advertising (aka baseline), therefore, we assess the marginal impact of advertising, which is the impact of one vehicle when run concurrently with everything else. Therefore, the marginal impact of our advertising channels when run concurrently is:
MI (TV) = (Both – Email Only) = 5% – 3% = +2% increase in sales
MI (Email) = (Both – TV Only) = 5% – 4% = +1% increase in sales
As you can see, both of the advertising vehicles have a negative marginal ROI, i.e. ROI when they are run concurrently with other programs.
This illustrates the difference between the total impact of both advertising vehicles, which is +5%, and their marginal impact, which is +2% and +1%.
When you run hundreds of campaigns, marginal impact is often the only way to assess the effect of your program. Unlike my simplistic example, most companies cannot afford to keep a completely clean baseline or a universal control group.
The ultimate answer on whether email or TV ads had a positive ROI in this example depends on the sizes of groups for each marketing communication. For example for TV, the blended sales lift is:
( 2%*(# of targets in Both) + 4%*(# of TV only targets) )/(total TV targets)
Therefore, assessing programs on their marginal impact produces lower effectiveness than when looking at them in isolation.
Optimizing Investment Reduces Marginal Effectiveness
There is a common misconception that in a well-run company every program delivers a high ROI. This is what most business executives are trying to achieve, but it is a misguided goal.
Whenever a program performs well and delivers a high ROI, it means that the company under-invested in that area. Getting a great return indicates that more money should be spent on this or similar programs, up until the point when the ROI on the next (aka marginal) program is reduced to the lowest acceptable level. This is what budget optimization means in practice.
Most companies invest in their programs to the max, and occasionally over-invest. As a result, it is typical for them to find a low or even negative ROI for their existing programs.
New programs face an even higher hurdle – they need to deliver beyond the costs and on top of all of the existing proven programs. This is why the success rates for new programs are usually pretty low. In my experience, less than 10% of new programs deliver a positive ROI. This is not a sign of weakness, though. It is a sign of the strength of the existing programs.
When an organization first engages in analytics, the goals usually include cost savings from the programs that have been over-invested in and reallocation of the budget to the programs with high profitability. Having these goals in mind is important when evaluating the analytical insights.
While delivering not so spectacular news to the business leaders, the analyst should remind the business about the goals of budget optimization and put their numbers into perspective. In the optimal scenario, most programs should deliver a fairly low ROI.
For an organization that is new to analytics, the maturation process means learning how to apply good judgment to the insights from the data.
Tom O’Toole wrote about the importance of accepting bad news and acting on them in his article The Best Approach to Data Analytics:
You may need to accept “inconvenient outcomes.” Data inevitably creates transparency and reveals business insights that can be unexpected, uncomfortable, and unwelcome. Data analytics will unearth inefficiencies and misconceptions that complicate leadership and disrupt conventional thinking. Business leaders who crush or ignore answers they don’t like will rapidly undercut the value of data analytics.
Tom O’Tool