Our Quality Gate Was 24x Noisier Than What It Guarded

1Key Takeaways

• Originally published on hexisteme notes .
• In a public ML benchmark competition, we ran a submission policy: don't spend a submission — a scarce, rate-limited resource — unless a candidate change beats the current best by at least 2 percentage points on a fixed, 40-task local holdout set.
• It read like ordinary scientific discipline: don't act on noise, require a minimum effect size before you spend something scarce.
• Over a 125-day eligibility window, at one submission per day, exactly one….