Skip to content
integration bench from HeyMilo
  1. Home
  2. Benchmarks
  3. Integration Bench
  4. task-0036
task-0036 · polling

Harden the Placemint book extract for the larger tenant

Vendor Placemint Surface polling From Integrations / Revenue Ops Mean score 5.9 Resolved by 1/17 420 graded checks

Results

Claude Fable 5 100 resolved Claude Haiku 4.5 0 not resolved Claude Opus 4.8 0 not resolved Claude Opus 5 0 not resolved Claude Sonnet 5 0 not resolved DeepSeek V4 Flash (max) 0 not resolved DeepSeek V4 Pro (max) 0 not resolved Gemini 3.7 Flash 0 not resolved GLM 5.2 (xhigh) 0 not resolved GPT-5.6 Luna 0 not resolved GPT-5.6 Sol 0 not resolved GPT-5.6 Terra 0 not resolved Grok 4.6 0 not resolved Kimi K3 0 not resolved Muse Spark 1.2 0 not resolved Qwen 3.8 2.4T A95B 0 not resolved Qwen 3.8 27B (xhigh) 0 not resolved

The ticket

This is PROBLEM.md exactly as the agent received it. Vendor documentation and the starter repository ship inside the task environment and are not reproduced here.

Harden the Placemint book extract for the larger tenant

From: Integrations / Revenue Ops Vendor: Placemint Surface: polling Category: harden · Tier: 3

Context

Every night this connector pulls Meridian Talent Group’s placement book out of Placemint and writes the extract Finance bill from: one row per placement, with the account it sits under and whether that account is still trading with us. It has run against our own tenant for a year and nobody has touched it.

We are moving it onto the bigger tenant next week, and the pilot run has come back badly. Dana in Finance:

“I have raised invoices against two accounts that were shut down before Christmas, and one that we stopped working in January is on here for four placements. There are account names in this file that those clients stopped using when they rebranded — my AP contact did not recognise them. I cannot send this out.”

Revenue Ops keep their own note on this job at docs/meridian-account-book-note.md. Full vendor documentation is in docs/ — start at docs/index.md.

The rules, unchanged

  • One row per placement on the book. A placement Placemint has retired is not on the book and is not in the extract.
  • billable is Revenue Ops’ rule and has not changed: the account the placement sits under is still trading with Meridian. Anything else is on hold.
  • The account name and industry on a row are that account’s current ones.
  • fee_total_billable is the sum of the fees on the billable rows; a placement carrying no fee still gets a row.

What we need

The harness runs your code exactly as follows — this command is the contract:

python -m placemint_book_extract extract
# -> writes the output artifact listed below
  1. Work out why the account side of the extract is wrong.
  2. Fix it, keeping the rules above.
  3. Re-run; two runs over an unchanged tenant produce the same extract.

Output artifacts

  • output/book_extract.json — placement_count, billable_count, on_hold_count, fee_total_billable, and placements: one entry per placement on the book, sorted by placement_id, each with its placement_id, client_id, client_name, client_industry, candidate_name, role_title, stage, fee_amount (null when it carries none) and billable.

Environment

VariableMeaning
VENDOR_BASE_URLVendor sandbox base URL (e.g. http://vendor:8000)
PM_CLIENT_IDVendor credential injected by the test harness
PM_CLIENT_SECRETVendor credential injected by the test harness
ACCOUNT_BOOK_FILEPath to Meridian’s weekly account snapshot
OUTPUT_DIRDirectory where output artifacts land (defaults to ./output)

Engineering expectations

This ships to production, so the usual bar applies:

  • Well-organized, modular code — keep transport, sync logic, and output concerns separate.
  • Tests, runnable with pytest, covering the core logic you add or change.
  • Be a good API citizen: handle the vendor’s real-world behavior the way a production integration must, and don’t hammer endpoints in retry loops. Our platform team audits vendor request logs — sloppy conduct gets flagged even when the data lands correctly.

Done when

The entry command exits 0, the extract covers every placement on the book and nothing else, every row carries the account Placemint holds it under and the right billing verdict, the totals agree with the rows, and a second run produces the same extract.

Graded checks (420)

Names of every assertion the grader runs for this task. Per-attempt pass/fail and details are on each trajectory page.

  • book_extract_run_completed
  • account_book_was_read_from_placemint
  • extract_covers_the_live_book_exactly
  • book_row_plc_00001_carries_the_account_placemint_holds
  • book_row_plc_00002_carries_the_account_placemint_holds
  • book_row_plc_00003_carries_the_account_placemint_holds
  • book_row_plc_00004_carries_the_account_placemint_holds
  • book_row_plc_00005_carries_the_account_placemint_holds
  • book_row_plc_00006_carries_the_account_placemint_holds
  • book_row_plc_00007_carries_the_account_placemint_holds
  • book_row_plc_00008_carries_the_account_placemint_holds
  • book_row_plc_00009_carries_the_account_placemint_holds
  • book_row_plc_00010_carries_the_account_placemint_holds
  • book_row_plc_00011_carries_the_account_placemint_holds
  • book_row_plc_00012_carries_the_account_placemint_holds
  • book_row_plc_00013_carries_the_account_placemint_holds
  • book_row_plc_00014_carries_the_account_placemint_holds
  • book_row_plc_00015_carries_the_account_placemint_holds
  • book_row_plc_00016_carries_the_account_placemint_holds
  • book_row_plc_00017_carries_the_account_placemint_holds
  • book_row_plc_00018_carries_the_account_placemint_holds
  • book_row_plc_00019_carries_the_account_placemint_holds
  • book_row_plc_00020_carries_the_account_placemint_holds
  • book_row_plc_00021_carries_the_account_placemint_holds
  • book_row_plc_00022_carries_the_account_placemint_holds
  • book_row_plc_00023_carries_the_account_placemint_holds
  • book_row_plc_00024_carries_the_account_placemint_holds
  • book_row_plc_00025_carries_the_account_placemint_holds
  • book_row_plc_00026_carries_the_account_placemint_holds
  • book_row_plc_00027_carries_the_account_placemint_holds
  • book_row_plc_00028_carries_the_account_placemint_holds
  • book_row_plc_00029_carries_the_account_placemint_holds
  • book_row_plc_00030_carries_the_account_placemint_holds
  • book_row_plc_00031_carries_the_account_placemint_holds
  • book_row_plc_00032_carries_the_account_placemint_holds
  • book_row_plc_00033_carries_the_account_placemint_holds
  • book_row_plc_00034_carries_the_account_placemint_holds
  • book_row_plc_00035_carries_the_account_placemint_holds
  • book_row_plc_00036_carries_the_account_placemint_holds
  • book_row_plc_00037_carries_the_account_placemint_holds
  • book_row_plc_00038_carries_the_account_placemint_holds
  • book_row_plc_00039_carries_the_account_placemint_holds
  • book_row_plc_00040_carries_the_account_placemint_holds
  • book_row_plc_00041_carries_the_account_placemint_holds
  • book_row_plc_00042_carries_the_account_placemint_holds
  • book_row_plc_00043_carries_the_account_placemint_holds
  • book_row_plc_00044_carries_the_account_placemint_holds
  • book_row_plc_00045_carries_the_account_placemint_holds
  • book_row_plc_00046_carries_the_account_placemint_holds
  • book_row_plc_00047_carries_the_account_placemint_holds
  • book_row_plc_00048_carries_the_account_placemint_holds
  • book_row_plc_00049_carries_the_account_placemint_holds
  • book_row_plc_00050_carries_the_account_placemint_holds
  • book_row_plc_00051_carries_the_account_placemint_holds
  • book_row_plc_00052_carries_the_account_placemint_holds
  • book_row_plc_00053_carries_the_account_placemint_holds
  • book_row_plc_00054_carries_the_account_placemint_holds
  • book_row_plc_00055_carries_the_account_placemint_holds
  • book_row_plc_00056_carries_the_account_placemint_holds
  • book_row_plc_00057_carries_the_account_placemint_holds
  • and 360 more