How to Build an Evaluation Harness for Your AI Agent (So It Doesn't Break in Production)

The agent passed every test I threw at it by hand. Then a user asked it to "book the cheaper flight," it happily called book on the wrong flight ID, and nobody caught it for three days because the demo I kept running never once asked for the cheaper flight. That is the trap. When you test an agent by running it yourself, the demo is the test — you type the happy-path prompt, watch it work, and ship. But your users don't type your happy path, and your model provider silently ships a new…