We added synthetic data to our eval set. The pass rate rose, and so did our production incidents.

We needed a bigger eval set, so we generated one. A model wrote a few thousand test cases that looked like our traffic, we scored against them, the pass rate went up, and we felt good. Then production incidents went up too, on exactly the inputs the synthetic set said we handled. The test set had grown and its predictive value had dropped, at the same time. That is the trap with synthetic eval data, and it is not a tooling problem. Generating cases is easy now. Every framework will hand you a…