We built the benchmark we'd want any AI estimation vendor to pass. Then we failed it.

Large IT projects run about 45% over budget on average. That figure comes from McKinsey and Oxford looking at more than 5,400 of them, and it is the reason a wave of tooling now promises to read your spec and tell you what it will cost. We were about to build one of those tools. Before writing the product, we wrote the test. The claim is falsifiable, which is the useful thing about it. The effort-estimation research community has been publishing datasets with real recorded effort for decades.…