Back to site
>>> article

Scale or Stop: Incrementality Tests That Prove iROAS ≥3.0 for Founders

For founders and DTC brands: run feasible incrementality tests, set iROAS decision rules (scale if iROAS ≥3.0) and convert lift into profit.

Decorative title card illustration

Incrementality testing is a randomized experiment that isolates the causal effect of your marketing spend by comparing exposed customers to a withheld control group. It answers one question your dashboards cannot: how many conversions actually happened because of the ad, not just near it. The output is net-new conversions and incremental ROAS (iROAS), and the takeaway is blunt. If you cannot show a causal baseline, you cannot prove the spend worked.

TL;DR:

  • Small sample sizes may prevent detection of lifts below 10%, requiring thousands of conversions per group for reliable results.
  • Contamination from cross-device, household sharing, or retargeting can erode control groups and bias uplift measurements.
  • Overlapping changes in creative, budget, or targeting during testing can invalidate causal attribution.
  • Proper test length, usually three to four weeks, is essential to capture delayed conversions in long-sales-cycle products.
  • Combining incrementality testing with media mix modeling and attribution maximizes strategic insights and validation.

Table of Contents

What Is Incrementality Testing and How Does It Work?

Every incrementality test rests on one comparison: a treatment group that sees your ads against a control group that does not. The gap between their conversion rates is the causal signal. Amplitude’s framework defines lift as the difference in conversions between the two groups, expressed either as an absolute number or as a percentage relative to control.

The math is simpler than most analysts expect:

  • Absolute lift = Treatment conversions − Control conversions
  • Relative lift = Absolute lift ÷ Control conversions
  • Incremental ROAS (iROAS) = Incremental revenue ÷ Ad spend

Say a campaign generates 1,200 conversions in the treatment group and the control group (identical size, no ad exposure) converts at a rate that projects to 1,000 conversions. If those 200 incremental conversions generate revenue well above the ad spend, iROAS reflects the effective return on investment, not whatever blended ROAS your ad platform is reporting.

That distinction is the whole point. Attribution models assign credit to a touchpoint based on rules, not evidence. They will happily credit a retargeting ad for a sale that would have happened anyway. Incrementality testing removes that guesswork by measuring what happens in a world where the ad simply did not exist.

Hands pouring coffee over grounds in dripper

Which Test Design Fits Your Campaign?

Choosing the wrong test design is the single fastest way to waste a testing budget. Three approaches dominate the field, and each fits a different data reality.

  1. Randomized user-level holdouts. Best for digital campaigns where you control audience assignment, such as paid social or email. You randomly withhold a percentage of eligible users from ad exposure and compare conversion rates directly. This is the gold standard for precision because randomization eliminates selection bias.
  2. Geo or market holdouts. Necessary when you cannot control exposure at the individual level, which covers most retail media, linear TV, out-of-home, and CTV campaigns. You suppress spend in selected markets and compare performance against matched markets that keep running as normal. Retail media in particular leans on this design because retailers need to prove net-new sales, not just diverted ones.
  3. Synthetic controls and causal modeling. The fallback when randomization was never set up and you are working retrospectively, or when a campaign runs nationally with no clean holdout market available. These methods construct a statistical “what would have happened” baseline from historical and comparable data rather than a live control group.

SearchEngineLand’s measurement primer frames these alongside media mix modeling as the four core approaches, and recommends combining them rather than betting everything on one.

The trade-offs are real. User-level holdouts give you the cleanest read but only work where you control targeting. Geo holdouts are more flexible but require more time and data to reach meaningful conclusions. Synthetic controls are the least intrusive on revenue but carry more modeling uncertainty.

Pro Tip: Before committing to a holdout size, calculate the revenue you are willing to sacrifice during the test window. A 10% holdout on a $500,000 monthly budget means $50,000 in spend generating zero attributable exposure. Treat that number as the price of the causal baseline, not a sunk cost.

How Incrementality Differs from A/B Testing, Attribution, and MMM

These four methods get lumped together constantly, and that confusion leads teams to run the wrong test for the wrong question.

  • A/B testing answers a narrow question: did variant A outperform variant B inside a single campaign? It never asks whether the campaign itself caused anything.
  • Attribution assigns credit across touchpoints using rules or algorithms, but a model can only distribute credit among the channels it observes. It cannot tell you what would have happened with zero ad exposure.
  • Incrementality testing answers that exact counterfactual: would these conversions have happened anyway? It is the only one of the four that produces a causal number.
  • Media mix modeling (MMM) works at a macro level, using aggregate spend and outcome data across quarters to estimate channel-level contribution, without needing individual-level tracking.

A workable program uses all three together. MMM sets strategic budget allocation across channels because it handles long time horizons and diminishing returns well. Attribution improves day-to-day bidding and creative decisions because it is fast and granular. Incrementality testing validates both, periodically checking whether the channels MMM and attribution favor are actually causing results, rather than just correlating with them.

How to Run an Incrementality Test That Actually Holds Up

A test that is not sized correctly will fail before it starts, no matter how clean the design. Run through these steps in order.

  1. Check feasibility first. Before building anything, calculate whether you have enough conversion volume to detect the lift you care about. Soku’s sample-size rule puts it at roughly 15.68 divided by the relative lift squared, per group, for 95% confidence and 80% power. Detecting a 10% lift needs about. Detecting smaller lifts requires substantially more conversions per group, while larger lifts can be detected with fewer conversions.
  2. Respect the volume floor. When conversion volume is low, platform-native lift tools may not produce reliable results. In such cases, a geo holdout or longer test period is preferable over a user-level test likely to be inconclusive.
  3. Design before you launch. Write the hypothesis, define the unit of randomization (user, household, or geo), set the holdout percentage, and check for contamination risk, meaning any way a control-group member could still see the ad through another device, channel, or account.
  4. Run long enough to catch delayed conversions. eMarketer’s guidance recommends a minimum of three to four weeks, and longer for products with extended consideration cycles. Cutting a test short before conversion lag plays out is one of the most common ways teams underestimate lift.
  5. Analyze with confidence intervals, not just a point estimate. Compute incremental conversions, relative lift, and iROAS, then report the confidence interval around each. A CI tells you what the test rules out, not just what it suggests. Pre-write your decision rule before you see results: for example, “scale spend if iROAS exceeds 3.0 and the lower bound of the CI stays above 2.0.”

Statistic to know: Detecting a lift smaller than about 10% on a typical campaign can require conversion volume in the thousands per group. If your campaign generates a few hundred conversions a month total, a tight lift detection is not realistic. Widen the test window or lower your minimum detectable lift instead of forcing a false read.

Larger holdouts increase statistical power but cost more revenue during the test. There is no way around that trade-off. Decide upfront how much statistical confidence is worth how much sacrificed revenue, and document it before launch so nobody re-litigates the decision after seeing the number.

Platform Tools, First-Party Data, and Privacy Constraints

Most major ad platforms now offer native lift testing, and understanding their mechanics prevents misreading the output.

  • Platform lift tools generally enforce minimum budget or audience thresholds, and those minimums exist because of the same conversion-volume math covered above, not arbitrary gatekeeping.
  • Google Ads’ Conversion Lift reports incremental conversions as the treatment-control difference, and it can include modeled “delayed incremental conversions” to account for conversions that land after the study window closes. For long-sales-cycle products, that modeling can meaningfully change your reported lift, so read the methodology notes before you act on the number.
  • First-party data, hashed customer uploads, and privacy-safe matching techniques are increasingly necessary as third-party signal loss continues, and they generally produce more stable control-group assignment than platform-default audiences.
  • Watch for cross-device and frequency contamination: a control-group member who sees the ad on a second device, or a household member who shares a login, quietly erodes the clean separation the whole test depends on.

Where Incrementality Tests Go Wrong

Most flawed tests fail for a small set of repeatable reasons, and most are avoidable with discipline rather than more budget.

  • Underpowered tests get misread as “no effect.” A null result from a small sample only proves the true lift is below your detection threshold; it does not prove the lift is zero.
  • Contamination quietly erodes control groups. Frequency capping, retargeting pools, and shared devices all leak exposure into supposed control users.
  • Multiple simultaneous changes ruin attribution of the result. Changing creative, budget, and audience in the same window makes it impossible to say what caused the lift.
  • Mid-test edits invalidate the baseline. Adjusting bids or targeting once a test is live resets the clock on your causal comparison.
  • Skipping confidence intervals hides the real uncertainty. A single point estimate without a CI invites overconfidence in a number that might swing widely.

Pro Tip: Document your test’s run dates, holdout size, and decision rule in writing before launch. When a stakeholder pushes back on a result they don’t like, that document is what keeps the analysis honest.

Turning Incrementality Data Into Profitability Decisions

A causal baseline is not an academic exercise. It is the number that protects next year’s budget from being reallocated based on noise. Google’s own measurement guidance treats the short-term revenue given up during a holdout as an investment, not a loss, because the alternative is scaling a channel that was never actually working.

In diagnostic and advisory work, Commerce Catalyst applies incrementality findings directly to unit-economics decisions, using tools like the DTC Unit Economics Calculator to translate iROAS into a marketing efficiency ratio (MER) the whole business can act on. A typical decision rule: if iROAS clears 3.0 with a CI lower bound above 2.0, scale; if it lands under 1.5, pause and re-diagnose before adding another dollar of spend.

Hands adjusting financial calculator

Why Founders Should Treat This as an Investment, Not an Expense

Most marketing advice treats measurement as overhead, something to tolerate after the “real” work of running campaigns. That framing gets the priority backward. A causal baseline is the cheapest insurance a growing brand can buy against scaling the wrong channel for a year on the strength of a flattering attribution report.

The conventional advice on this topic obsesses over test design and ignores feasibility. Analysts will debate geo holdouts versus synthetic controls for hours while running a test with a third of the conversion volume needed to detect anything. Power math is not optional homework. It is the gate that decides whether your test result means anything at all.

Turning Test Results Into a Profitability Plan

Running a clean incrementality test is only half the job. The harder part is converting a lift number into a budget decision that actually improves cash flow, and that’s the gap Commerce Catalyst was built to close for consumer brands doing $5M to $75M in revenue.

Commercecatalyst

The DTC Financial Health Assessment takes your existing test data, MER, and unit economics and turns them into a prioritized list of fixes, not a generic report. If your incrementality numbers already point to a specific channel problem, the 90-Day Profit Sprint is built to operationalize that finding fast, tightening cash flow and margin within a single quarter rather than a full fiscal year. For teams still figuring out where marketing efficiency is bleeding, reviewing how marketing efficiency connects to profit beforehand makes the diagnostic conversation sharper. Bring your last two quarters of spend and conversion data, and start with a diagnostic call.

Where to Go Deeper on Incrementality Testing

For technical follow-up beyond this guide, these sources go further into platform mechanics and statistical methodology:

Sources

Related tools

>>> next step

Want to see where your business actually stands?

Run the numbers through the diagnostic, or talk it through with someone who has been in your seat.

Get the Diagnostic Book a Founder Hour