81M integrations
for AI agents to write, against software as old as 1993.
HeyMilo connects hiring teams to their ATS, HRIS, CRM, SORs, and legacy systems. Integration Bench turns what that taught us into one test for AI coding agents: can it ship an integration that survives the real vendor?
Source
81M: Zapier app directory lists 9,000+ apps with a public API (2026). Ordered pairs, 9,000 × 8,999 = 80,991,000. 1993: UKG Pro's first release, UltiPro for LAN (Ultimate Software S-1, 1998).
How this started
We did not set out to build a benchmark. We set out to stop rewriting connectors.
The long version
Nov 2023
First connector ships.
A customer's screening results need to land in their ATS. One engineer, one vendor, one week of reading documentation and finding out which parts were true. Tooling: ChatGPT in a browser tab (GPT-4 Turbo shipped that month), code pasted into the connector by hand.
2024
The staffing and recruiting stack, one vendor at a time.
Bullhorn, Greenhouse, Lever, Avionté, JobDiva, Ceipal and the rest. Every connector follows the same pattern: the docs are wrong somewhere, and the only place you can see it is the vendor's request log. Tooling: GPT-4o (May) and Claude 3.5 Sonnet (June) move into the editor. They explain the docs and fill in boilerplate; a human still writes the connector.
2025
Enterprise HRIS, and the software nobody wants to integrate with.
Workday, SuccessFactors, UKG Pro, iCIMS. Then the long tail: SOAP endpoints, cookie-session auth, CSV drops over SFTP, pagination that breaks on page two. The oldest lineage is UKG Pro: it shipped in 1993 as a DOS payroll product and still runs payroll today. Tooling: Claude Code (Feb), then GPT-5 (Aug) and Claude Sonnet 4.5 (Sep) write whole files from a ticket. Every line still reviewed by a person.
Early 2026
Agents ship connectors end to end.
Claude Opus 4.6 (Feb) takes a ticket and returns a working connector, and review moves from reading the diff to reading the vendor's log. That is where the failures are: right-looking code that pulls the wrong rows, hammers a rate limit, or puts a credential in a query string. So we build vendors that do not exist, with request logs we control, and grade from the other side of the wire.
Aug 2026
Integration Bench rev 01.
15 invented vendors, 50 public tasks, 17 models, one official attempt each. Scores are read from the vendor's log, not the agent's code.
Now
100+ integrations shipped. First 10 tasks published in full.
Every tool call, vendor request, graded check and diff for 170 attempts is on this site. The remaining 40 public tasks follow in batches.
Open a trajectory
57.5%
of HTTP requests to web content now come from automated clients, not people.
Cloudflare Radar recorded the crossover in June 2026. A growing share of that traffic is agents reading docs, calling APIs and moving records between systems. The integration engineer is increasingly a model. Whether it integrates correctly is now the load-bearing question for any product that only works once connected.
Source: Cloudflare Radar, bot vs. human traffic, June 2026
Graded from the vendor's side of the wire
Most coding benchmarks run the repo's own tests. Integration work fails differently:
- The documentation is wrong somewhere.
- The vendor enforces limits and signatures the code never mentions.
- A connector can return the right rows and still get your API key revoked.
None of that shows in the diff. All of it shows in the traffic. That is where we read the score.
15
Invented vendors, none of them real.
Each one is generated from a single declaration into a live API simulator: its own auth, pagination, webhooks, rate limits and a budget of deliberate documentation errors. No SDK or forum thread about them exists anywhere. Contamination is ruled out by construction.
22.6
What a submission that does nothing scored under our first partial-credit design.
"Never put a credential in a query string" passes trivially if you make no requests. We measured that floor, then switched to mandatory-gated scoring: one failed hard check zeroes the task. Every published number uses that scorer.
170
Attempts published in full.
Every tool call, vendor request, graded check and the final diff, for 17 models on the first 10 released tasks. The score is a function of the recorded run. Anyone can recompute it.
Browse trajectoriesIntegration Bench · rev 01 · preliminary
Leaderboard
Mean task score over 50 public tasks, one official attempt per model, scorer task-score-v4-mandatory-gated. Development-set numbers: we built the instrument against these tasks, so they describe the benchmark as much as the models.
The plan
One method, applied in order to the software where integration work is hardest. Public tasks to develop on. Held-out tasks as the exam. Every run published.
Now · rev 01Integration Bench
50 public tasks, 15 invented ATS, HRIS and job-board vendors. Polling, webhooks, writeback. 200 held-out tasks in authoring; capability claims wait for those.
Leaderboard and method NextLegacy Integration Bench
The vendors nobody wants to integrate with: SOAP and XML-RPC, cookie-session auth, CSV over SFTP, undocumented pagination. The 1993-to-2010 slice of what HeyMilo connects to.
What is planned · Collaborate on it After that: systems of record. Bidirectional CRM sync, where a wrong write is the expensive failure. Not scheduled yet.
Release history and what changed in each revision
Claude Fable 5
GPT-5.6 Sol
DeepSeek V4 Pro (max)
Qwen 3.8 2.4T A95B
Grok 4.6
Kimi K3
