Golden Datasets: Building and Maintaining One

Nearly every team that builds a golden dataset builds a good one. The problem is eighteen months later, when the person who wrote it has moved teams and nobody remaining can say why item 47 is graded the way it is — so nobody dares change it, and nobody trusts it either. How golden sets actually die Not through neglect, usually. Through four specific losses, each of which happens while the file itself looks perfectly healthy: Lost intent. An item’s expected output encodes a policy decision —…