Skip to content
  1. Home
  2. Benchmarks
  3. Integration Bench
  4. task-0026
task-0026 · polling · writeback

Push the Q1 close back into Placemint

Vendor Placemint Surface polling, writeback From Integrations / Revenue Ops Mean score 52.9 Resolved by 9/17 269 graded checks

Results

The ticket

This is PROBLEM.md exactly as the agent received it. Vendor documentation and the starter repository ship inside the task environment and are not reproduced here.

Push the Q1 close back into Placemint

From: Integrations / Revenue Ops Vendor: Placemint Surface: polling, writeback Category: build · Tier: 3

Context

When Finance closes a quarter, its two exports have to land back in Placemint so account managers see the invoiced numbers. The files are exactly what the billing tool produces:

  • input/invoices.csv — one row per invoice: invoice_ref, period, fee_pct (the commission rate for that engagement), status, issued_at.
  • input/placement_lines.csv — one row per placement being closed: line_ref, invoice_ref, placement_id, base_salary, close_reason.

Revenue Ops keep their own runbook for this job at docs/meridian-quarter-close-runbook.md.

Revenue Ops’ rules

These are ours, not Placemint’s.

  • A line is only ever written if its invoice is issued. A line whose invoice_ref is draft, void, or absent from invoices.csv altogether is held: nothing is written for it at all, this quarter or ever.
  • A line naming a placement Placemint has retired is retired, and nothing is written for it. Meridian does not invoice a retired placement.
  • A line naming a placement Placemint has never issued is unknown, and nothing is written for it.
  • Everything else is applied, and applying a line means:
    • close_reason of hired → the placement’s stage becomes placed and its fee becomes base_salary × fee_pct ÷ 100, rounded to two decimal places.
    • close_reason of withdrawn or declined → the placement’s stage becomes fell_through and its fee becomes 0. No commission is earned on a placement that did not complete, whatever the salary said.
    • the placement also gets one note, authored by billing@meridian.test, whose body is the invoice ref, the close reason and the fee to two decimal places, separated by single spaces — e.g. INV-0000-00 hired fee 0.00.
  • A placement is named by at most one line, and a line names exactly one placement.

Full vendor documentation is in docs/ — start at docs/index.md.

What we need

The harness runs your code exactly as follows — this command is the contract:

python -m placemint_quarter_close close-quarter
# -> writes the output artifact listed below
  1. Read both exports and decide every line.
  2. Settle the applied lines against Placemint — the fee, the stage and the note.
  3. Write the report described below.

Running the command a second time over the same two files must leave Placemint exactly as the first run left it.

Output artifacts

  • output/close_report.jsonline_count, applied_count, held_count, retired_count, unknown_count, and lines: one entry per row of placement_lines.csv, in file order, each with its line_ref, invoice_ref, placement_id, outcome (applied, held, retired or unknown), and — for an applied line — the fee_amount and stage that were written and the note_id Placemint gave the note.

Environment

VariableMeaning
VENDOR_BASE_URLVendor sandbox base URL (e.g. http://vendor:8000)
PM_CLIENT_IDVendor credential injected by the test harness
PM_CLIENT_SECRETVendor credential injected by the test harness
INVOICES_FILEPath to the invoice export
LINES_FILEPath to the placement-line export
OUTPUT_DIRDirectory where output artifacts land (defaults to ./output)
PAGE_LIMITOptional batch-size override

Engineering expectations

This ships to production, so the usual bar applies:

  • Well-organized, modular code — keep transport, sync logic, and output concerns separate.
  • Tests, runnable with pytest, covering the core logic you add or change.
  • Be a good API citizen: handle the vendor’s real-world behavior the way a production integration must, and don’t hammer endpoints in retry loops. Our platform team audits vendor request logs — sloppy conduct gets flagged even when the data lands correctly.

Done when

The entry command exits 0, every line in the report carries the outcome Revenue Ops’ rules give it, every applied placement carries the fee, the stage and the one note the close file says it should, every line that was not applied has left its placement exactly as it was, and a second run changes nothing.

Graded checks (269)

Names of every assertion the grader runs for this task. Per-attempt pass/fail and details are on each trajectory page.

  • quarter_close_completed
  • close_report_accounts_for_every_line
  • close_run_read_every_named_placement
  • line_L-0001_reported_applied
  • line_L-0002_reported_retired
  • line_L-0003_reported_applied
  • line_L-0004_reported_retired
  • line_L-0005_reported_retired
  • line_L-0006_reported_applied
  • line_L-0007_reported_retired
  • line_L-0008_reported_applied
  • line_L-0009_reported_applied
  • line_L-0010_reported_retired
  • line_L-0011_reported_unknown
  • line_L-0012_reported_applied
  • line_L-0013_reported_retired
  • line_L-0014_reported_retired
  • line_L-0015_reported_retired
  • line_L-0016_reported_applied
  • line_L-0017_reported_retired
  • line_L-0018_reported_retired
  • line_L-0019_reported_applied
  • line_L-0020_reported_applied
  • line_L-0021_reported_applied
  • line_L-0022_reported_unknown
  • line_L-0023_reported_applied
  • line_L-0024_reported_applied
  • line_L-0025_reported_retired
  • line_L-0026_reported_retired
  • line_L-0027_reported_retired
  • line_L-0028_reported_applied
  • line_L-0029_reported_applied
  • line_L-0030_reported_applied
  • line_L-0031_reported_retired
  • line_L-0032_reported_retired
  • line_L-0033_reported_unknown
  • line_L-0034_reported_retired
  • line_L-0035_reported_applied
  • line_L-0036_reported_applied
  • line_L-0037_reported_retired
  • line_L-0038_reported_applied
  • line_L-0039_reported_applied
  • line_L-0040_reported_retired
  • line_L-0041_reported_retired
  • line_L-0042_reported_applied
  • line_L-0043_reported_retired
  • line_L-0044_reported_retired
  • line_L-0045_reported_applied
  • line_L-0046_reported_retired
  • line_L-0047_reported_applied
  • line_L-0048_reported_applied
  • line_L-0049_reported_retired
  • line_L-0050_reported_applied
  • line_L-0051_reported_retired
  • line_L-0052_reported_applied
  • line_L-0053_reported_held
  • line_L-0054_reported_held
  • line_L-0055_reported_held
  • line_L-0056_reported_held
  • line_L-0057_reported_held
  • and 209 more