Blog

Notes from building Integration Bench: how the vendors are invented, how grading reads the vendor's logs, what went wrong with our first scorers, and what the published trajectories show about how coding agents integrate.

9 Sep 2026 What Integration Bench's Synthetic Vendors Resemble A practical map from Integration Bench's fictional vendors to familiar ATS, HRIS, staffing, scheduling, screening, and talent-software categories. 3 Sep 2026 A Model on Integration Bench Tried to Break Out of Its Container Muse Spark 1.2, stuck on a DNS problem in task-0040, spent half its run probing the Docker socket, calling unshare() and nsenter, writing its own UID map and drafting a DNS spoof. The Integration Bench sandbox denied every call. The task then failed on an ordinary integration bug. 2 Sep 2026 How Many API Integrations Are Possible in the World? 9,000 apps expose a public API, so 81 million ordered pairs are possible. Two apps only need to talk when one company runs both, and the average company runs 101 apps: about 900,000 integrations are worth writing. Far fewer exist. 1 Sep 2026 How We Built Integration Bench The construction, in order: why a connector team started grading coding agents, how vendors that do not exist get generated from one declaration, why the grader never reads the code, and what a submission that did nothing taught us about our own scorer. 27 Aug 2026 Publishing the Trajectories: The Data Behind This Site How the raw evidence bundle from every Integration Bench run becomes the JSON on data.integrationbench.com: one event schema across three harnesses, what gets scrubbed, how scores are recomputed, and how to use the files yourself.