We refused to let an AI grade our AI, and our headline number got worse

Every AI eval tool will tell you how good your agent is. Almost none of them will tell you who decided. That sounds like a small distinction until you look at how the number is produced. You run your agent, you run a second model over the output, the second model says 87% and that 87% goes on a dashboard, into a board deck, into a decision about which model to ship. Nobody writes down that the 87% is a model's opinion about another model. The number arrives wearing the same clothes as a number…