Skip to content
  1. Home
  2. Benchmarks
  3. Integration Bench
  4. task-0033
task-0033 · polling · writeback

Keep the RecruitOS pipeline and the Placemint ledger in step

Vendor RecruitOS, Placemint Surface polling, writeback From Integrations / Revenue Operations Mean score 52.9 Resolved by 9/17 153 graded checks

Results

The ticket

This is PROBLEM.md exactly as the agent received it. Vendor documentation and the starter repository ship inside the task environment and are not reproduced here.

Keep the RecruitOS pipeline and the Placemint ledger in step

From: Integrations / Revenue Operations Vendor: RecruitOS, Placemint Surface: polling, writeback Category: build · Tier: 2

Context

Recruiters at Northgate work the pipeline in RecruitOS. Everything finance and the client portal see comes out of Placemint. Nobody has joined the two up, so Revenue Ops keep a crosswalk by hand — input/placement_links.csv, one line per requisition we bill for — and this job stops the two drifting apart. Our RecruitOS tenant is read-only; Placemint is where we write.

One invocation is one cycle. Both systems can move between cycles, so each cycle has to account for both. The repository contains the first reconciler and the state it carries between cycles.

Revenue Ops keep their own runbook for this job at docs/northgate-placement-sync-runbook.md.

The rules Revenue Ops settled on

These are ours, not RecruitOS’s and not Placemint’s.

  • A RecruitOS application stage maps to a Placemint placement stage like this: appliedsourced, interviewinterviewing, offeroffered, hiredplaced, rejectedfell_through.
  • Every crosswalk line gets exactly one outcome per cycle, and the outcome vocabulary is closed:
    • unlinked — Placemint does not hold that placement. Report it for Ken and carry on; a line we cannot act on is not a failed cycle.
    • retired — Placemint holds it but the requisition has been closed out. Closed-out requisitions are finished business: nothing about them goes back to Placemint.
    • inbound — a Placemint account manager moved the placement themselves since our last cycle. Their move is the authoritative one for this cycle and the ATS stage is not pushed over it.
    • pushed — none of the above, and the mapped ATS stage is not the stage Placemint holds, so the mapped stage goes to Placemint.
    • in_sync — none of the above and nothing to do.
  • The first cycle has no previous cycle, so nothing counts as inbound on it.
  • STATE_DIR is ours and survives between cycles. OUTPUT_DIR does not.

Full vendor documentation is in docs/ — start at docs/index.md.

What we need

The harness runs your code exactly as follows — this command is the contract:

python -m northgate_placement_sync
# -> writes the three output artifacts listed below

Run once, that is cycle 1. Run again, that is cycle 2, and so on.

Output artifacts

  • output/result.jsoncycle, links, counts (one integer per outcome name above), ats_watermark and marketplace_watermark (the positions this cycle is leaving behind, as strings; empty is allowed).
  • output/import_report.csv — header row plus one row per crosswalk line, in crosswalk order, with columns application_id, placement_id, outcome, ats_stage (the stage RecruitOS holds), target_stage (that stage mapped), and resulting_stage (the stage Placemint holds for that placement when the cycle ends; empty for unlinked).
  • output/writeback_log.jsoncycle and writes: one entry per write this cycle sent to Placemint, each with placement_id, stage, status and idempotency_key.

Environment

VariableMeaning
VENDOR_BASE_URLRecruitOS sandbox base URL
PLACEMINT_BASE_URLPlacemint sandbox base URL
RO_CLIENT_ID, RO_CLIENT_SECRETRecruitOS credentials from the harness
PM_CLIENT_ID, PM_CLIENT_SECRETPlacemint credentials from the harness
CROSSWALK_FILEPath to Ken’s crosswalk
OUTPUT_DIRDirectory where output artifacts land
STATE_DIRDirectory that survives between cycles

Engineering expectations

This ships to production, so the usual bar applies:

  • Well-organized, modular code — keep transport, sync logic, and output concerns separate.
  • Tests, runnable with pytest, covering the core logic you add or change.
  • Be a good API citizen: handle the vendor’s real-world behavior the way a production integration must, and don’t hammer endpoints in retry loops. Our platform team audits vendor request logs — sloppy conduct gets flagged even when the data lands correctly.

Done when

The entry command exits 0 on every cycle, each cycle’s report accounts for every crosswalk line with the right outcome and stages, and Placemint ends each cycle holding the stage the rules above say it should for every linked placement.

Graded checks (153)

Names of every assertion the grader runs for this task. Per-attempt pass/fail and details are on each trajectory page.

  • placement_sync_cycle1_completed
  • cycle1_link_app_0030
  • cycle1_link_app_0007
  • cycle1_link_app_0038
  • cycle1_link_app_0071
  • cycle1_link_app_0070
  • cycle1_link_app_0032
  • cycle1_link_app_0020
  • cycle1_link_app_0031
  • cycle1_link_app_0021
  • cycle1_link_app_0064
  • cycle1_link_app_0065
  • cycle1_link_app_0044
  • cycle1_link_app_0010
  • cycle1_link_app_0001
  • cycle1_link_app_0035
  • cycle1_link_app_0062
  • cycle1_link_app_0022
  • cycle1_link_app_0037
  • cycle1_link_app_0067
  • cycle1_link_app_0061
  • cycle1_link_app_0034
  • cycle1_link_app_0052
  • cycle1_link_app_0047
  • cycle1_link_app_0033
  • cycle1_link_app_0046
  • cycle1_link_app_0009
  • cycle1_link_app_0051
  • cycle1_link_app_0050
  • cycle1_link_app_0060
  • cycle1_link_app_0003
  • cycle1_link_app_0002
  • cycle1_link_app_0023
  • cycle1_link_app_0066
  • cycle1_link_app_0039
  • cycle1_link_app_0008
  • cycle1_link_app_0063
  • cycle1_link_app_0006
  • cycle1_link_app_0004
  • cycle1_link_app_0045
  • cycle1_link_app_0011
  • cycle1_link_app_0036
  • cycle1_outcome_counts
  • cycle1_all_artifacts_match_the_complete_contract
  • cycle1_accepted_write_traffic_is_exact
  • cycle1_pushed_placements_carry_the_ats_stage
  • cycle1_retired_placements_not_written_to
  • cycle1_writeback_log_lists_the_accepted_writes
  • placement_sync_cycle2_completed
  • cycle2_link_app_0030
  • cycle2_link_app_0007
  • cycle2_link_app_0038
  • cycle2_link_app_0071
  • cycle2_link_app_0070
  • cycle2_link_app_0032
  • cycle2_link_app_0020
  • cycle2_link_app_0031
  • cycle2_link_app_0021
  • cycle2_link_app_0064
  • cycle2_link_app_0065
  • and 93 more