Methods report, validation rig and mandatory-gated scoring
The scorer moved from dense partial credit to mandatory-gated task scores after a do-nothing submission measured 22.6 out of 100. Four known-answer probes and 34 named failure classes were published with the methods report.
What changed
- Scoring. The second scoring design assembled partial credit from every check a task defines. Many of those checks are prohibitions (no credential in a query string, reject a forged webhook signature, honour the advertised retry delay), and a submission that issues no requests satisfies all of them. Measured across the public suite, a do-nothing submission averaged 22.6 out of 100. The replacement,
task-score-v4-mandatory-gated, zeroes a task when any mandatory check fails and only then awards credit for the remaining checks. Every published number sits on this scorer; the entire sweep was rescored from banked verdicts without re-running an agent. - Validation rig. Every task is now probed with four submissions whose correct score is known in advance: one that does nothing, one that exits instantly, the reference solution, and a “naive” implementation written faithfully from the task’s own deliberately wrong documentation. A task ships only when the four land where they should.
- Failure catalogue. 34 named ways a benchmark of this class produces confident numbers that measure something else, each with a real example from our own suite.
- Early sweep. Four model tiers over the full public set, one run each, in early August. A quarter of the tasks were solved by nobody and another quarter by everybody, so the effective size of the suite on the strict metric was well under half; the top three tiers finished inside overlapping confidence intervals and separated only on cost per solve ($1.28 to $6.25).
Caveats recorded
- Three grader repairs were made during the early-August sweep itself; those tasks describe the benchmark as much as the models.
- Run-to-run variance is uncharacterised (one run per cell).
- The agent scaffold was not held fixed across that sweep.
The full write-up is the HeyMilo post We built a benchmark for AI integration work, then spent months proving it wrong.