In a December 2025 survey of 1,340 practitioners, quality was the number-one blocker to putting agents in production — cited ahead of latency and ahead of cost. The same survey found that 29.5% of teams run no evaluations at all. Which means nearly a third of the industry is shipping on hope, and most of the rest are shipping on something that feels like evaluation but doesn't behave like one when it matters.
This post is about the second group, because that's who we meet most often. The team has an eval suite. Someone built it in a sprint six months ago. It passes. And the agent still isn't trusted enough to take the human out of the loop — because the suite has one of the four defects below.
1. The rubric is a vibe
"Response is helpful and accurate" is not a scoring criterion; it's a horoscope. The test is simple: give the same output and the same rubric to two people independently. If they disagree more than they agree, the rubric is measuring the reviewer, not the system.
On a recent logistics engagement, the first labelling pass over a 340-invoice golden set came in at 0.71 inter-annotator agreement. That number told us something useful: not that the analysts were careless, but that two of our eight discrepancy categories were defined ambiguously. One rubric revision later, agreement was 0.87. We fixed the instruction, not the people.
2. The judge was never calibrated
LLM-as-judge is fine. We use it. But a judge you haven't measured against human review is a rumor with a dashboard. The procedure isn't exotic: hold out a slice of your golden set, have humans score it under the rubric, run the judge on the same slice, and report the agreement number.
The tell is in how a team answers one question: "how do you know your judge agrees with your experts?" Teams with calibrated judges answer with a number. Everyone else answers with an anecdote.
3. The golden set was never golden
A golden set built only from cases the team already handles well is a mirror, not a test. Ours are stratified deliberately: across formats, sources and failure categories, and they always include two awkward slices — cases the team already flagged as hard, and cases the team passed that turned out on review to contain errors.
An eval you can't re-run isn't an eval — it's a memory of one.
4. It ran once
The suite that ran before launch and never again is the most common defect of all. Models get upgraded, prompts get "improved" on a Friday, a retrieval index gets rebuilt — and the numbers everyone remembers are from a system that no longer exists. The fix is structural: the harness runs in CI, on every prompt and model change, and a regression gate fails the build the way a broken unit test would.
What fixing this actually costs
Less than the stalled pilot did. A rubric workshop is days. A stratified golden set is one to two weeks including the double-labelling. Judge calibration is a few days once the golden set exists. CI wiring is engineering hygiene. The reason teams don't do it isn't cost — it's that nobody owns it, because evaluation sits in the gap between the ML team, QA and the business owner.
That gap is, more or less, our whole AI Assurance practice. If you'd rather close it yourself, the four sections above are the order we'd do it in.