(Please the bottom of this blog for my quick take on different incrementality measurement methods.)

Marketing analytics teams are embracing “state-of-the-art” incrementality methods to measure advertising effectiveness—especially approaches branded as causal or AI-driven.

Ironically, some of these methods can systematically understate…even invert…advertising impact, distort optimization, and lead teams to kill the very campaigns that are actually working.

Here are three common ways this happens.

 “Causal” Measurement Can Invert Effectiveness

Many modern measurement approaches enforce that conversions are only attributed to advertising if they occur after ad exposure.

On the surface, this sounds rigorous. In practice, it can create a fatal bias that actually inverts measured lift. Why?

• For exposed consumers, this approach of “time bound causality” discards all conversions that occurred before exposure. This biases downward the baseline of conversions among those exposed.

• For unexposed consumers, you retain their full baseline conversion history.

• You then compare a downward-biased baseline plus incremental conversions (exposed) to a full baseline of conversions (unexposed).

(Yes, there are ways around this, but this is typically how it works).

If the true incremental lift from advertising is modest—as it usually is—this lost baseline can overwhelm the incremental gain, producing measured negative lift even when ads are genuinely effective.

In a simulated data set where I set the true causal effect of advertising as increasing conversion rates by 1 percentage point in generating the data, a time-bound approach mistakenly concluded that advertising reduced conversions by nearly 20%.

The business consequence: campaigns that worked get labeled as failures—and the marketer dumps the creative and media strategy that actually was working.

The Post-Campaign Lift That Never Gets Measured

Standard practice is to measure conversions during the campaign plus a short lookback window.

What this misses is how advertising actually works.

Advertising often:

• Influences journey choices that can take months to conclude

• Changes brand memory structures that persist well after exposure which can influence behavior when consumers later enter a need state, even a year later.

Using the MMA’s brand as performance method, we observe equal or greater conversion lift in the six months after campaigns end for all 4 studies conducted.

Typical measurement therefore:

• Misses crediting the campaign for delayed lift in conversions

• ROI is understated

The result: Advertising budgets get treated as an expense to be managed down, not knowing the advertising paid back over time

Optimizing Toward the Wrong Tactics and Partners

Another common failure occurs when measurement systems don’t properly account for media covariance.

This is especially prevalent when:

• The same AdTech partner runs multiple tactics

• Those tactics reach highly overlapping audiences via shared identity graphs and exchanges

When exposures are correlated:

• Ineffective tactics can appear to “work” as highly correlated effective channels “subsidize” weaker ones

• Budget gets optimized toward the wrong partners and formats

The result: marketers continue to fund underperforming tactics

A Quick Diagnostic

If any of the following are familiar, your measurement approach—not your marketing—may be the issue:

• Are you seeing negative lift more often than random error suggests?

• Does your iROAS frequently look underwater?

• Do partner or tactic-level results feel counterintuitive or unstable?

The Good News

These problems might be consequences of how you are measuring incrementality not the effectiveness of your advertising per se.

I’ve spent substantial time diagnosing these biases and developing practical fixes. The encouraging part? They’re addressable!

Stay open to questioning your measurement system. You may discover that your advertising has been working better than the numbers suggest.

Quick takes…thoughts on…

Ghost ads by Google for YouTube…true Random testing, a gold standard. However, it reinforces that advertising on the platform is likely to be biased towards YouTube addicts (because very low bid win rates means an ID has to show up lots of times in the bid stream to have a high probability of being served an ad at least once. Maybe this is all digital.)

Other RCT testing… not really RCT.  As soon as you realize some IDs are not reached (and for systematic reasons) your RCT is blown and you need to use weighting and modeling.

Doubly robust estimators are a gold standard, right?  Not so fast! The good news…only one model (effects or propensity) has to be right.  The bad news?  You have no idea if either model is really right. Also, it is a model of counterfactuals so you do not report the real data anymore, just the model results.

Twinning.  For behavioral data, twinning is good but…you must twin the ad impressions onto the unexposed twin also or you get the inversion bias I refer to.

AI Causality measurement.  This usually involves directed graphs (Judea Pearl stuff) as well as machine learning to build all the equational linkages. It sounds good on paper but the directed graphs have a lot of assumptions built in, and can compound model error. Machine learning offers few generalizable insights because it is a non-equational approach…meaning if the results seem strange, there is no model where you can look at parameters and decide if they make sense or not.  I did not have good success with AI Causality on the Ally data set. (But I’m not giving up!)

Macro analysis (like MMM) of campaign test vs control data.  I tried these on my sandbox data sets and typical approaches I tried did not work very well.  However, I saw that looking at experimental data over time added some revealing patterns. I developed a new form of model with a different equation structure (inspired by Physics (forces and gravity/geodesics*) that showed very promising results.

Last click metrics…don’t get me started!!!

*A geodesic is the natural path an object follows when no non-gravitational forces act on it, like a satellite orbiting Earth. It struck me that a brand’s conversions or sales baselines are like its geodesic.