It started as a connector problem
HeyMilo talks to hiring stacks. Every customer arrives with an applicant-tracking system, an HRIS, a handful of job boards, and the expectation that candidates, jobs, applications and notes flow between them without anyone thinking about it. Over the years that has meant more than a hundred integrations, against software written anywhere between 1993 and this year. The oldest lineage we connect to is UKG Pro, which first shipped in 1993 as UltiPro for LAN, a DOS-based HRMS/payroll product (Ultimate Software S-1, 1998).
The work has a shape. Someone reads the vendor’s documentation, writes a connector, and then finds out which parts of the documentation were true. Some vendors are legacy in the literal sense: session cookies, XML over POST, pagination that only works if you never ask for page two. Others are modern and still wrong in the places the docs are silent about. Either way, the specification you are handed lies somewhere, and the only way to find out where is to make requests and compare what comes back with what was promised.
When we started handing pieces of this work to coding agents, they failed in a way that looked exactly like success. The diff was clean. The tests passed. The connector pulled the wrong rows, or hammered a rate limit, or put a credential in a query string, and the only place any of that was visible was the vendor’s own request log.
That is the observation the benchmark is built on: for integration work, the evidence is in the traffic, not in the code.
Vendors that do not exist
The obvious way to benchmark this is to use real vendor APIs. We ruled that out early. Real vendors change, cannot be reset, and are in every model’s training data through SDKs, changelogs and forum threads. A benchmark built on them measures memory as much as skill, and decays as vendors ship.
So each vendor in Integration Bench is generated. A single declaration describes the vendor’s resources, auth chain, pagination style, webhook semantics, rate limits, the faults it can inject mid-run, and a budget of deliberate documentation errors. From that declaration we produce two things: a live API simulator that runs inside the task environment, and the documentation set the agent reads. The docs are wrong in the ways the declaration says they are wrong, and only there.
Twenty-three vendors exist this way across the full suite, fifteen of them in the public set, covering the applicant-tracking, HRIS and job-board surfaces our connector team actually sees. No SDK, tutorial or Stack Overflow answer about any of them can exist anywhere, because there is nothing to write about. Contamination is ruled out by construction rather than by an n-gram scan.
What the agent gets, and what it does not
A task is a ticket (PROBLEM.md), a starter repository, and the vendor documentation. The vendor runs live. The agent has to make the integration work: back-fill a canonical store, wire a webhook receiver, harden a sync against injected faults, or migrate records across schema versions. It runs in its provider’s own harness (Claude Code, Codex CLI, or OpenCode) at the highest reasoning effort exposed, with a 60-minute wall clock.
What the agent does not get is anything that would let it skip the work: the answer key, the reference solution, the vendor declaration, or the timeline of mutations and faults the simulator will apply during the run.
Grading from the vendor’s side of the wire
After the agent stops, we check three things, and none of them involves reading its code.
Data correctness. The canonical store after the run is compared against an answer key produced by a committed script that talked to the same vendor and recorded what it observed. Row counts, field values, stage transitions, deletions honoured.
Protocol discipline. Read out of the vendor’s request log: credentials never in a query string, advertised retry delays honoured, no full re-crawl on an incremental run, forged webhook signatures rejected, the include_stage flag sent on every call that needs it. These are the checks that decide whether a connector gets your API key revoked.
Fault survival. Whether the integration recovered from what the simulator did to it mid-run: a rate-limit window, a schema change, a webhook redelivery, an auth token that stopped working.
Every check is a programmatic assertion over a seeded, checkpointed world. Nothing is scored by a model and nothing depends on a rubric applied at grading time. A score is a function of a recorded run, which means anyone can recompute it offline from the published verdict without re-running the agent. We leaned on that property hard when we had to replace the scorer.
The submission that did nothing
Our second scoring design gave partial credit assembled from the checks a task defines. Many of those checks are prohibitions. A submission that issues no requests at all satisfies every prohibition, because a prohibition that never triggers is a prohibition that was never violated.
We measured it. Across the public suite, a submission that did nothing averaged 22.6 out of 100. Every partial-credit number we had reported was starting from there rather than from zero, and we would not have known if we had only ever pointed the instrument at models whose real score was the open question.
The fix is the scorer every published number now sits on: task-score-v4-mandatory-gated. A task’s hard checks are mandatory; failing any one of them zeroes the task. Only then is credit awarded for the remaining checks. The full sweep was rescored from banked verdicts, no run repeated.
The larger lesson became the validation rig. Every task is now probed with four submissions whose correct score we already know: one that does nothing, one that exits instantly, the reference solution, and a “naive” implementation written faithfully from the task’s own lying documentation. The naive probe is the important one. It turns “this task is hard” from a claim the author makes into something we measured. Along the way we catalogued 34 named ways a benchmark of this class produces confident numbers that measure something else, each with a real example from our own suite. That catalogue is in the methods write-up on heymilo.ai.
What the first sweeps showed
The early-August sweep over four model tiers was humbling about the suite rather than the models. A quarter of the public tasks were solved by nobody and another quarter by everybody, so more than half contributed nothing to a ranking. The top three tiers finished inside overlapping confidence intervals. Cost per solve ran from $1.28 to $6.25 against accuracy differences that were noise.
The rev 01 sweep that produced the current leaderboard ran 17 models, one official attempt each on every public task. The pattern from the earlier write-up held: webhook tasks are much harder than polling tasks, sparse documentation costs roughly twenty points, and models are far stronger at a conventional data pull than at keeping an integration coherent across events, faults and more than one source of truth. Read those as development-set numbers. The public set is the tasks we built and debugged the instrument against.
Why everything is published
The held-out set is the exam; more than half of it is fully authored and nothing about it is public. The public set is being published all the way down, in batches: every tool call the agent made, every request the vendor saw, every check the grader ran, the diff, and the harness’s raw transcript. The first ten tasks, 170 attempts, are out now.
We did that because the numbers on a leaderboard are the least interesting output of a benchmark like this. The interesting output is the run where a model reads a rate-limit header, ignores it, and gets throttled for the next four minutes; or the one where it notices the documentation is wrong, says so, and works around it. You can open those runs here.
If a task is wrong, a grader is unfair, or you want the held-out set for a model you are building, tell us.