releases
Every revision of Integration Bench in order: task set, scorer, models evaluated, data published, and the caveats that apply. Held-out results and Legacy Integration Bench are listed as upcoming.
Upcoming
Next · Integration Bench
Held-out set: first capability results
More than half of the held-out set is fully authored. The protocol for filling the table is already published: one scorer version, one task-tree commit, strict solve rate as the headline with its interval, do-nothing floor reported next to any partial-credit score.
Late 2026
Legacy Integration Bench
SOAP and XML-RPC, cookie-session auth, CSV over SFTP, undocumented pagination.
History
1 Sep 2026 · rev 01 · data
First ten tasks published in full, and integrationbench.com
The first 10 public tasks are released in full: ticket, graded checks and all 170 official attempts as normalised JSON at data.integrationbench.com, with a viewer for the timeline, vendor request log, graded checks and diff of every run. This site launches as the benchmark's home.
20 Aug 2026 · rev 01
rev 01: 17 models on the public tasks
First public leaderboard. Seventeen models, one official attempt on each public task, each run in its vendor's own harness at the highest reasoning effort exposed. Scored with task-score-v4-mandatory-gated.
10 Aug 2026 · scorer v4
Methods report, validation rig and mandatory-gated scoring
The scorer moved from dense partial credit to mandatory-gated task scores after a do-nothing submission measured 22.6 out of 100. Four known-answer probes and 34 named failure classes were published with the methods report.