Skip to content
ZAHER WORLDSZAHER WORLDSZAHER WORLDSZAHER WORLDS
Performance · 9 min read

Run the holdout test. You will hate the result.

Attribution tells you which channel got the credit. Incrementality tells you which channel earned it. They are rarely the same channel.

OKOmar Kilani

Every platform reports itself generously. That is not a conspiracy, it is how last-touch works: whoever was standing closest to the purchase gets the sale written on their card.

A geo holdout is the cheapest honest answer available. Split comparable regions, switch a channel off in half of them, wait six weeks, compare. It is not perfect — seasonality, spillover and small samples all bite — but it is dramatically better than believing the dashboard.

In our experience, branded search is the channel that most often collapses under a holdout. It reports beautifully. Turn it off and revenue frequently does not move, because those people were coming anyway.

The second usual casualty is broad retargeting. Not all of it — but the segment made up of people who visited in the last three days and would have converted regardless is usually a large share of the reported return.

The uncomfortable part is that running these tests will occasionally prove that your agency's best-performing channel was never performing. We run them anyway, and we send you the result either way.

Got something worth remembering?

Tell us what you are building and what is in the way. If we are not the right fit we will say so in the first call, and point you at someone who is.

Sunday – Thursday, 9:00 – 18:00 AST