Skip to content
integration bench from HeyMilo

releases

Every revision of Integration Bench in order: task set, scorer, models evaluated, data published, and the caveats that apply. Held-out results and Legacy Integration Bench are listed as upcoming.

Upcoming

Next · Integration Bench Held-out set: first capability results Fully authored. Running preliminary evals now: verifier hardening, gold and floor probes. First capability results publish after that pass.
Late 2026 Legacy Integration Bench Evaluating agent capabilities on enterprise software that shipped before REST: SOAP services, EDI interchanges, fixed-width file drops over SFTP, terminal sessions, message queues, ERP staging tables. Invented vendors, documentation that lies, graded from the wire.

History

10 Sep 2026 GPT-6 Astra added GPT-6 Astra joins the public board at 65.40 Task Score. Same 50 tasks, same scorer, one official attempt per task at xhigh reasoning effort. All 50 attempts are published in full. 4 Sep 2026 Claude Fable 5.1, Gemini 3.8 Flash, and Muse Spark 1.3 added Claude Fable 5.1, Gemini 3.8 Flash, and Muse Spark 1.3 are added to the public board. Same 50 tasks, same scorer, one official attempt each. Fable 5.1 is first at 69.35. All new attempts are published in the same layout as the rest. 2 Sep 2026 Integration Bench released The 50 public tasks and their official runs are published in full. Each attempt ships the ticket, diff, vendor request log, graded checks, and raw transcript. Leaderboard, matrix, and per-model and per-task indexes go up at data.integrationbench.com; this site is the public home. 22 Aug 2026 15+ models evaluated 15+ models are run on the public 50, including Claude Fable 5 and GPT-5.6 Sol. One official attempt per model per task, provider harness, 60-minute wall clock, one scorer version. Claude models in Claude Code; GPT-5.6 Sol, Terra, and Luna in Codex; the rest in OpenCode. 28 Jul 2026 Tasks authored The 50 public tasks are authored against live synthetic vendors: applicant tracking, HRIS, and job boards; build, fix, harden, and migrate. Each task is internally evaluated, then verifiers and scoring are iterated: gold and floor probes, check coverage, request-log assertions. 15 Jul 2026 Conceptualized The bench is specified: synthetic vendors that run live in the task, a ticket plus starter repo plus vendor docs as the only inputs, and a grader that reads the request log and store state rather than the agent's code. Two sets are declared: a public development set and an unpublished held-out set.