Skip to content
integration bench from HeyMilo
  1. Home
  2. Releases
rev 01 · data

First ten tasks published in full, and integrationbench.com

The first 10 public tasks are released in full: ticket, graded checks and all 170 official attempts as normalised JSON at data.integrationbench.com, with a viewer for the timeline, vendor request log, graded checks and diff of every run. This site launches as the benchmark's home.

1 September 2026data

What shipped

  • Initial release: 10 tasks. task-0001, 0011, 0015, 0016, 0020, 0025, 0037, 0042, 0047 and 0049, chosen to span the polling, writeback and webhook surfaces and the six vendors they touch. Leaderboard scores are still the mean over the full public set; only the per-task detail and trajectories are staged. Further batches follow as we finish reviewing each task’s grader in public.

  • data.integrationbench.com. One static bundle per attempt: summary.json, a harness-independent trajectory.json, verdict.json, vendor-requests.json, patch.diff, the harness’s original transcript.jsonl and the ticket as problem.md. Plus leaderboard.json, matrix.json, manifest.json and per-model and per-task indexes. 1,208 files for the initial batch.

  • Normalised trajectories. Claude Code, Codex CLI and OpenCode transcripts are mapped to a single event schema (system, user, message, reasoning, tool_call with attached result, error) so runs can be compared step for step across harnesses. Reasoning blocks are included where the harness exposes them.

  • Scrubbing. Internal registry and gateway hostnames, project identifiers and runner home paths are rewritten, and response headers, cookies and forwarding headers are dropped. Vendor credentials that appear in transcripts and request logs are issued by the simulator for that run and are not secrets. Nothing else is edited.

  • Trajectory viewer. Timeline with visual and JSON modes per event, graded checks by layer with mandatory markers, vendor request log with rate-limit and auth-failure flags, file-by-file diff, and the raw transcript.

  • This site. Leaderboard, per-model and per-task pages, releases and a blog, all rendered from the same published JSON. Model pages show aggregate numbers over the full public set and per-attempt rows for the released ones.

Known issues

  • One Claude Code normalisation bug (tool results attached to the wrong event) was found while building the viewer and fixed before publication; all Claude Code trajectories were re-exported.
  • Vendor logs are captured by the simulator, so requests that never reached it (DNS failures, wrong host) do not appear.