Every number on this site is rendered from static JSON that you can fetch yourself. This post is the map.
What a run leaves behind
Each official attempt produces an evidence bundle: the harness transcript, the grader’s verdict, the diff the agent left, the vendor simulator’s request and webhook logs, the canonical database, the run manifest and hashes for the task tree and harness tree. The full bundle across all official attempts is about 1.4 GB; most of it is repository snapshots and canonical databases that are only useful to someone re-grading.
We publish the parts that let you understand and recompute a run, and keep the rest available on request. We are also releasing tasks in batches rather than all at once: the leaderboard is scored over the full public set, but per-task detail and trajectories go out ten or so at a time as we finish reviewing each grader in public. The first batch is 10 tasks and 170 attempts.
Three harnesses, one event schema
The three harnesses write very different transcripts. Claude Code emits assistant messages with content blocks and pairs tool results by id. Codex CLI streams item events with a type per item. OpenCode emits per-part events with token accounting attached. Comparing a Claude run against a GPT run step for step is impossible on the raw files.
trajectory.json normalises them to one ordered list of events:
system session start, model, tool count
user the ticket and any follow-up prompt
message assistant text
reasoning thinking blocks, where the harness exposes them
tool_call tool, input, and the attached result {ok, output, exit_code}
error harness or provider errors
Each event carries its index so links like #e42 are stable. The original transcript is published beside it as transcript.jsonl, so nothing is lost in the mapping and anyone can check it.
Scores are recomputed, not copied
summary.json carries the task score, but the score is derived from verdict.json, not read from a leaderboard. The exporter re-applies task-score-v4-mandatory-gated to every verdict: if any mandatory check fails the task is zero, otherwise the score is the fraction of remaining checks passed. Before release we compared the recomputed leaderboard against the published rev 01 numbers, model by model, and they agree. Verdicts that are missing (provider failures) are scored as zero and flagged.
What is scrubbed
Internal registry and gateway hostnames, project identifiers and runner home paths are rewritten. Response headers, cookies and forwarding headers are dropped from the vendor log. Vendor credentials that appear in transcripts are issued by the simulator for that run; they are not secrets and are left in place because several protocol checks are about how they were used. Nothing else is edited.
The files
https://www.integrationbench.com/data/v1/integration-bench/ (mirror: https://data.integrationbench.com/v1/integration-bench/)
manifest.json models, tasks, counts, generated_at
leaderboard.json one row per model with by-surface breakdown
matrix.json model × task scores
models/{model}.json aggregates over all scored tasks, rows for released attempts
tasks/{task}.json every model's result on a released task, the ticket, check names
attempts/{model}/{task}/
summary.json score, checks, cost, timing, hashes, provenance
trajectory.json normalised events
verdict.json every graded check with detail
vendor-requests.json the simulator's request log per vendor
patch.diff the diff the agent left
transcript.jsonl the harness's original transcript
problem.md the ticket
CORS is open and everything is cacheable. Paths are v1/<bench>/…, so Legacy Integration Bench will sit beside this one with their own task numbering, and a schema change ships as v2/ without breaking anyone’s scripts.
A few things worth doing with it
Filter every tool_call whose result has ok: false across the published runs and group by tool: the distribution of what agents fail at is not the distribution of what they do most. Join vendor-requests.json against the L2 checks in verdict.json to see exactly which request tripped a protocol failure. Diff two models’ patch.diff on the same task where one resolved it and the other did not; the difference is usually smaller than the score gap suggests.
If you want the full evidence bundle for an attempt, more tasks, or the held-out set, ask.