AI agent evaluation: runtime graders for real tool use

1Key Takeaways

• July 14, 2026 is a useful timestamp for agent evaluation: Anthropic is teaching the practice, while Claude Managed Agents documents runtime capabilities like Outcomes.
• That distinction matters if you are trying to ship agents, not just compare models.
• Single-turn evals miss the job A single-turn LLM evaluation can tell you whether an answer looks plausible.
• It cannot verify whether an agent used the right tool, changed state correctly, delegated work safely, retried after failure, or stopped at….