Incrementality is one of the most crucial, and often overlooked, concepts in marketing. For anyone who has ever looked at the performance reporting from marketing channels like Facebook, Adwords or (especially) digital retargeting and felt a degree of skepticism, you’re not alone.
For obvious reasons, every marketing platform wants to show the best possible results to clients so that they continue to invest and scale that channel. That doesn’t mean they are fudging the results, but it also doesn’t mean they’re necessarily giving you a full picture of the true benefit that channel is providing. The main reason is that they (generally) aren’t giving you insight into the incremental benefit of that channel.
Technically speaking, incrementality is not a real word, although the term has found common usage throughout performance marketing as the nominalized version of incremental. Generally, incrementality is defined as “a measure of supplemental business resulting from a marketing tactic or set of marketing tactics which would not have otherwise occurred.” (https://marketing-dictionary.org/i/incrementality/)
In short, what you really want to understand when judging a marketing channel’s performance is how much incremental revenue (or customers, etc.) that channel is driving. If Facebook shows 100 conversions and $1,000 in revenue, can you be certain that if you had turned Facebook off, none of those conversions would have happened otherwise? Unless you are testing for incrementality in each channel, the answer is no.
The best way to get a read on the true incrementality that a channel provides is to run a test / control experiment. This is accomplished by setting up a control, or holdout group that can be used to compare against the group that receives marketing from that channel. A control group is simply a group of your target audience that is set aside and does not receive the marketing message that you are testing. You can then measure the response from that control group and compare it to the group that received the marketing to understand the true incremental value that the channel provided.
To use a simple example, let’s say you have a customer base of 10,000 people. You want to target them with a special promotion on Facebook, but you’re not convinced that Facebook’s conversion reporting is giving you a true read on the campaign’s value. Instead of uploading a custom audience containing all 10,000 customers, you instead upload an audience of 5,000 of those customers (your Test audience) and withhold the other 5,000 from being targeted on Facebook with the promotion (your Control audience). That control group of 5,000 customers can now be used as your baseline.
What you’re looking for when calculating incrementality is the difference in performance between the control group and the test audience. If you are most interested in Conversion Rate, for example, you’d simply subtract the conversion rate of the control group from the test audience.
Going back to our Facebook example, if you saw 750 conversions from the test audience and 200 from the holdout audience, the calculation would be the following:
Test Conversion Rate = 750 / 5000 = 15%
Control Conversion Rate = 200 / 5000 = 4%
Incremental Conversion Rate = 15% - 4% = 11%
The same approach can be applied to any metric you’re interested in, such as CPA, Revenue or ROAS.
The question that every marketer should be asking themselves on an ongoing basis is “Where should I be investing the next dollar in my marketing budget?” Without looking at the incrementality of individual channels, it’s easy to come to the wrong decision here.
Retargeting on Facebook may have a CPA of $20, and Direct Mail prospecting a CPA of $45, but it’s very likely that the incremental CPA in the two channels is very different from those numbers.
Retargeting in particular is notorious for having much higher incremental CPAs than is reported by pure last-click attribution, and investing more and more marketing budget into such a channel can lead to a mirage of “good” results that don’t translate to additional revenue for your business.
Be careful not to unbalance your incrementality test by adding an unfair advantage to one group that will skew the results. If you are doing a 50%/50% incrementality test on email for example, and then you run a separate Facebook campaign that targets all recent email recipients, you’ve added an additional touchpoint to the “test” group that the control group didn’t receive. While this is fine if you’ve designed the test accordingly, it’s no longer a clean test of just that original email.
It’s important that both the holdout and test groups are created from audiences that are equally likely to respond. Sometimes marketers will simply split an audience using excel, taking the top and bottom sections as the test and holdouts. That’s fine if it’s a truly randomly sorted dataset, but what if it was sorted by first purchase date? Or even state? Factors that you wouldn’t expect can quickly take your test from unbiased to biased.
If all of your oldest customers are in the test group and your newest customers in the control group, you’re likely to see different response rates even if they all received the same marketing message. Within Poplar, you can set holdouts at a campaign level, and Poplar ensures that a holdout group is truly randomly selected.
Incrementality testing is crucial, but only if you’re able to get statistically significant learnings. Creating a holdout of 10 customers and a test group of 20 won’t lead to any real learnings. Likewise a conversion rate of 5.1% for the test group and 5.0% for the control group is likely insignificant unless you have very large audiences for each. To understand whether your results meet significance thresholds, you can use one of many free tools available online, such as https://neilpatel.com/ab-testing-calculator/.
There are significant challenges managing a multi-channel marketing plan, and understanding the true value of a channel is one of the most important factors to consider when deciding where to invest additional budget. Grasping incrementality and how to measure it is a necessity when working with multiple channels where an overlap of targeting is almost impossible to avoid.
