Why your agent benchmarks are lying to you

We deployed a coding agent that hit 94% on the industry benchmark. It failed in production on the first real edge case because the benchmark measured single-turn success and our actual work was multi-turn refinement. The model could not update its beliefs correctly when new evidence arrived, something no single-turn eval would catch. This is not a hypothetical. I have watched agents shine in demo and disintegrate on the messy input that production actually serves. The gap between what we…