How I generate LLM test cases that actually catch bugs

I got tired of hand-writing test cases for our agent, so I did the obvious 2026 thing: I pointed an LLM at our docs and asked it to generate them. In an afternoon I had a few thousand. I felt incredibly productive. They passed CI on every release. Production kept breaking anyway. When I finally looked closely, maybe a third of my generated tests were wrong, near-duplicates, or things no real user would ever ask. The set was too noisy to catch a real regression, so a green check meant nothing.…