Two things I read this week landed on the same point, and it wasn't the one I expected.
The first was a case study from Haus, a measurement company, on a 26-week YouTube experiment they ran with TextNow. TextNow wanted to know whether awareness advertising, the kind that's hard to tie back to a sale, actually moved the business. The honest answer took months to appear. In the first five weeks there was almost no measurable lift. Most of the effect they eventually recorded, 87% of it, accumulated only after the first two months, and the campaign finished with a 1.99% lift in weekly paying users. That's one company and one channel, so I'd hold the numbers loosely. The shape is the useful part. The payoff was real and it was slow, and most measurement windows are too short to catch it.
The second was a piece by Karan Dhir arguing that a modern measurement function needs four different people, and that almost no company has all four. A measurement scientist to build models you can trust. An experimentation lead to produce causal evidence. A finance translator so the CFO believes the numbers. And a decision architect, the person who sets who gets to act on the findings, when, and what happens when the model and the gut disagree. His point is that when companies fold these roles together to save money, the work that quietly gets dropped is the deciding, not the modeling.
I keep noticing how much of the AI conversation right now is about production. Write the brief faster, ship more variations. The harder problem, the one both of these pieces circle, sits downstream of all that. It isn't making the campaign. It's knowing whether the campaign did anything, being willing to wait long enough to find out, and having someone whose actual job is to turn that answer into a decision.
That last part is the one I don't think you can hand to a model yet. It can tell you a number moved. It can't tell you whether to believe it, or whether two months is long enough, or whether to override the dashboard because you know something it doesn't. Those are judgment calls, and they belong to people who understand the business and are willing to be accountable when they get one wrong.
None of this is an argument against AI in measurement. Haus is a measurement company built on modeling, and the modeling is the point of it. My point is narrower. The tools got cheaper and faster, so the production bottleneck is mostly gone, and what's left is the human work of deciding what the outputs mean and what to do about them. That work is getting more valuable, not less, and going by Dhir's piece, it's still badly staffed.
I'll get to test this against reality soon. This fall I'm sitting down with a group of Vancouver marketing practitioners to hear how they're actually using AI day to day, and measurement is near the top of what I want to ask about. I suspect the gap between what the tools can do and who's deciding what to do with them is wider than the vendor decks let on.
Sources
- Haus, TextNow YouTube measurement case study: haus.io
- Karan Dhir, "The 4 roles every modern measurement function needs": karandhir.substack.com