Your AI Metric Is Meaningless Without a Benchmark

Measuring GenAI · Part 3 of 4. Previous → The $22-a-Month Sentiment Pipeline. Next → Separate Measurement from Interpretation.

By this point in the series I had good numbers on an internal support AI: how often it resolved things on its own, how often it deflected, how quality trended. And I nearly presented them the way most people do — against an internal target someone had made up.

The number floating around was “we should be resolving 40 to 60% of requests autonomously.” Nobody could tell me where it came from. It was a guess that had hardened into a goal. And measured against it, the product looked behind — which was fueling a dangerous conversation: maybe we should just rip this out and buy a vendor’s.

A number needs an anchor

Here’s the thing a metric can’t do on its own: tell you whether it’s good. “62% autonomous resolution” is neither impressive nor disappointing until you say compared to what. The anchor does all the interpretive work — and if the anchor is a guess, so is your conclusion.

So I went and found real ones. Public industry research exists for exactly this: analyst firms and large platform vendors publish benchmarks and forward targets for autonomous resolution and agentic support. I swapped the made-up internal number for a couple of those, cited, with dates.

The picture inverted. Against the credible public targets — including where a major analyst firm projected the industry would be several years out — the product wasn’t behind. It was already at or above where the field was expected to be at the end of the decade. Three years early.

Same numbers, opposite story

I want to be precise about what changed, because it’s the whole point: the numbers did not change. Not one data point moved. The only thing I swapped was the yardstick — from an uncited internal guess to defensible outside research. That single substitution flipped the leadership conversation from “is this good enough, should we replace it” to “this is ahead of the curve, let’s invest.”

The work, it turned out, wasn’t analytics at all. It was sourcing a credible benchmark. I spent more time finding the right published figures than I did computing our own.

The lesson

If you present an AI metric — or any metric — to a room that has to make a decision, the benchmark you choose matters more than the measurement you took. A made-up internal target isn’t neutral; it silently decides whether your work reads as success or failure. Go find the real yardstick. It’s usually public, it’s usually citable, and it’s almost always the difference between “we’re behind” and “we’re ahead.”

Once you’ve got trustworthy numbers and an honest benchmark, the last problem is presenting them without confusing everyone. That’s Part 4.