LLM-as-judge disagrees with itself between runs

The flap I had a faithfulness gate on merge: judge scores every case, the mean has to clear 0.80. One Tuesday it failed at 0.79. I re-ran the identical job, no code change, no prompt change, and it passed at 0.82. Ran it a third time: 0.80 exactly. Nothing in the repo had moved. The judge was disagreeing with itself. A gate that returns a different verdict on the same inputs is worse than no gate. People stop believing the red, they re-run until it goes green, and now the check is a slot…