The Counterfeit Test for AI Metrics

We had an AI quality metric that behaved exactly the way any good metric should. It was stable across model tiers. It improved when we refined the system. It separated weaker specialist lenses from stronger ones. And it measured something tied to proprietary knowledge the base model did not have.

It measured grounding coverage: the percentage of findings that cited a valid control from our reliability catalog. Every control in the catalog traces back to real incidents and known failure patterns.

At first, the reasoning seemed hard to argue with. If a finding cited a control, the model must be using the specialist knowledge we’d given it, right? But models hallucinate, so we tried to prove ourselves wrong.

Why the metric looked convincing

Our original question was simple: do specialist agents outperform a bare model at finding reliability defects in code? We built fixtures with seeded bugs and distractor code, then ran a bare-model arm against a specialist arm. We repeated each configuration eight times because smaller samples had repeatedly misled us. Recall, the percentage of seeded defects the system found, did not give us a clear answer. The specialists produced only a modest overall lift, and that lift changed across model tiers and model releases. The underlying model was repricing the result on someone else’s schedule.

Grounding coverage looked more durable. It stayed nearly constant as the models changed, and it improved when we worked on the specialist prompts. Our reasoning was straightforward:

If a finding cites one of our proprietary controls, the model must be using our specialist knowledge.

But the metric didn’t actually test that. It counted the citation and assumed why it appeared.

That is a common problem with metrics. They do not just measure an output. We attach a story to the output about what caused it. A retrieval score goes up, so we assume the system found more relevant information. An agent finishes a benchmark, so we assume it planned and acted successfully. A personalized response mentions something about the user, so we assume the system understood them.

The result may be real. The explanation may still be wrong.

Build the cheapest counterfeit

Once we saw the assumption inside the metric, the right experiment became obvious. A useful control should not be random or deliberately incompetent. It should preserve the surface behavior the metric rewards while removing the capability the metric is supposed to represent. Our catalog-only baseline could emit valid control codes. It knew enough vocabulary to look grounded. But it lacked the knowledge that told the specialist what to detect, why it mattered, and which control actually applied.

If grounding coverage measured correct use of specialist knowledge, this counterfeit should have failed. It didn’t.

Its 0.99 score showed that the model needed access to the vocabulary, not mastery of the domain, to satisfy the metric.

We had not been measuring whether the model understood which control applied. We had been measuring whether the citation machinery was connected. Coverage was not useless. It still told us whether findings cited controls from the catalog. That made it a reasonable smoke test. The mistake was treating a plumbing check as evidence of understanding.

This is not unique to our system. Early work on AI citations often measured whether a response included sources before later work asked whether those sources actually supported the claims. One audit of deployed generative search engines found that roughly half of the citations it examined were unsupported even though the cited sources existed. A real source is not necessarily a supporting source. A valid control code is not necessarily the correct control. Existence is easier to measure than correctness, which makes it easy to mistake one for the other.

Find a metric the counterfeit cannot reproduce

The failed control told us what the next metric needed to do: distinguish plausible use of the catalog’s vocabulary from correct use of its knowledge. Our ground truth already recorded the control associated with each seeded defect. That let us measure citation accuracy:

When a finding matches a labeled defect and cites a control, how often is it the correct control?

Across three model tiers, the specialists scored between 89 and 95 percent. Each result included more than 1,100 matched findings.

The catalog-only baseline behaved very differently depending on how much domain knowledge the task required.

On the Go fixture, the defects were conventional enough that a short control description was sufficient. The counterfeit matched the specialist. As the domain became more ambiguous, the systems separated. On AI reliability, the catalog-only model attached the wrong control to nearly half its findings, even though its grounding coverage remained 0.99.

The original metric saw almost no difference. Citation accuracy exposed one. That does not make citation accuracy a complete measure of system value. It means that, for this particular claim, it could distinguish the specialist from the counterfeit.

It survived the first test.

The control was testing us too

There was another reason the counterfeit mattered. We had built a product around specialist knowledge, then found a proprietary metric that appeared to validate that knowledge. The metric improved when we worked on the system. It remained stable across model changes. It aligned neatly with the product thesis. Those are ideal conditions for motivated reasoning.

The catalog-only arm was not just adversarial toward the model. It was adversarial toward our explanation of the model. It asked whether the result actually required the cause we wanted to believe produced it.

The same verification process later caught an error in our internal summary. A claim about 22 of 23 specialists had become all 23 before publication. The protocol corrected its own author. That is not an embarrassment. It is evidence that the process works.

Trust does not come from believing the evaluator is unusually careful. It comes from building a process capable of rejecting the evaluator’s preferred conclusion.

Test the metric, not just the system

The usual evaluation question is whether the system can pass the test. The more revealing question is whether a fake can pass it too.

For a citation metric, the fake may attach real sources that do not support the claim. For retrieval, it may exploit keyword overlap instead of relevance. For an agent benchmark, it may hardcode the expected path. For personalization, it may repeat stored facts without changing its decisions.

The pattern is the same: Preserve the measurable surface. Remove the underlying value. Then run the metric.

If the counterfeit succeeds, the metric may still be useful, but only for a narrower claim. Rename it, limit it, and stop asking it to prove more than it can. We began with a metric that looked stable, proprietary, responsive, and difficult to game. A system with almost none of the relevant expertise scored 0.99.

The experiment did not just give us a better number. It changed our standard for believing numbers. A metric is not trustworthy because the real system passes. It becomes trustworthy when the fake fails.