Skip to content
  1. Home
  2. Benchmarks
  3. Integration Bench
  4. task-0010
task-0010 · polling

Build the GlobalHire audit-period extract Compliance keeps asking us for

Vendor GlobalHire Surface polling From Integrations / Data Platform Mean score 94.1 Resolved by 16/17 81 graded checks

Results

The ticket

This is PROBLEM.md exactly as the agent received it. Vendor documentation and the starter repository ship inside the task environment and are not reproduced here.

Build the GlobalHire audit-period extract Compliance keeps asking us for

From: Integrations / Data Platform Vendor: GlobalHire Surface: polling Category: build · Track: python · Tier: 2

Context

Compliance reviews our GlobalHire tenant a few periods at a time. Every quarter they hand us a calendar — a list of the periods they are looking at — and ask what GlobalHire touched inside each one. Until now somebody has answered that by exporting the tenant to a spreadsheet and filtering it by hand, which took a week last quarter and disagreed with GlobalHire’s own UI twice.

We are replacing that with the job in repo/. Its first implementation was built from the old manual runbook and has not produced a trustworthy audit pack against the live tenant. Treat the repository as the system to diagnose and finish, not as a blank scaffold.

This quarter’s calendar is in input/ledger_windows.csv, in Compliance’s order. Each row is a period, half-open: starts_at counts, ends_at does not. The periods are the ones Compliance sampled, so treat the file as given — do not reorder it and do not merge rows.

Our own runbook for this extract is docs/ledger-window-runbook.md. Full vendor documentation is in docs/ — start at docs/index.md.

What we need

A record’s activity is its last modification. A record belongs to the one period that contains it, and to no period at all if it falls outside every row of the calendar. Each record’s outcome is deleted if GlobalHire has the record flagged deleted, created if its created and modified stamps are the same instant, and updated otherwise.

The harness runs your code exactly as follows — this command is the contract:

python -m gh_activity_ledger
# -> writes both artifacts below into OUTPUT_DIR

activity_ledger.csv

A header row plus one row per record placed in a period, ordered by period in calendar order, then entity, then record_id:

window_id,entity,record_id,outcome
W0,candidates,XX_00000,created

result.json

The summary the audit pack embeds. per_window is one entry per period in calendar order; tenant is how many records GlobalHire holds; outside_windows is how many of those the ledger placed in no period:

{"per_window": [{"window_id": "W0",
                 "candidates": {"total": 0, "live": 0, "deleted": 0},
                 "placements": {"total": 0, "live": 0, "deleted": 0},
                 "agencies":   {"total": 0, "live": 0, "deleted": 0}}],
 "tenant": {"candidates": 0, "placements": 0, "agencies": 0},
 "outside_windows": {"candidates": 0, "placements": 0, "agencies": 0}}

Environment

VariableMeaning
VENDOR_BASE_URLGlobalHire sandbox base URL
GH_API_KEYvendor credential injected by the harness
OUTPUT_DIRdirectory the two artifacts are written to (defaults to ./output)

Engineering expectations

This ships to production, so the usual bar applies:

  • Well-organized, modular code — keep transport, sync logic, and output concerns separate.
  • Tests, runnable with pytest, covering the core logic you add or change.
  • Be a good API citizen: handle the vendor’s real-world behavior the way a production integration must, and don’t hammer endpoints in retry loops. Our platform team audits vendor request logs — sloppy conduct gets flagged even when the data lands correctly.

Done when

The entry command exits 0, both artifacts are present and in the shapes above, every record GlobalHire holds appears under the one period its last activity falls in — and under no period when it falls outside the calendar — the summary’s counts and its live/deleted split agree with the ledger and with the tenant, and running the extract twice over an unchanged tenant produces the same two files both times.

Graded checks (81)

Names of every assertion the grader runs for this task. Per-attempt pass/fail and details are on each trajectory page.

  • activity_ledger_artifact_exact
  • activity_summary_artifact_exact
  • activity_ledger_run_completed
  • window_W1_candidates_activity_count
  • window_W1_placements_activity_count
  • window_W1_agencies_activity_count
  • window_W2_candidates_activity_count
  • window_W2_placements_activity_count
  • window_W2_agencies_activity_count
  • window_W3_candidates_activity_count
  • window_W3_placements_activity_count
  • window_W3_agencies_activity_count
  • window_W4_candidates_activity_count
  • window_W4_placements_activity_count
  • window_W4_agencies_activity_count
  • window_W5_candidates_activity_count
  • window_W5_placements_activity_count
  • window_W5_agencies_activity_count
  • window_W6_candidates_activity_count
  • window_W6_placements_activity_count
  • window_W6_agencies_activity_count
  • window_W7_candidates_activity_count
  • window_W7_placements_activity_count
  • window_W7_agencies_activity_count
  • window_W1_roster_covers_every_record_active_in_it
  • window_W1_roster_admits_nothing_active_outside_it
  • window_W2_roster_covers_every_record_active_in_it
  • window_W2_roster_admits_nothing_active_outside_it
  • window_W3_roster_covers_every_record_active_in_it
  • window_W3_roster_admits_nothing_active_outside_it
  • window_W4_roster_covers_every_record_active_in_it
  • window_W4_roster_admits_nothing_active_outside_it
  • window_W5_roster_covers_every_record_active_in_it
  • window_W5_roster_admits_nothing_active_outside_it
  • window_W6_roster_covers_every_record_active_in_it
  • window_W6_roster_admits_nothing_active_outside_it
  • window_W7_roster_covers_every_record_active_in_it
  • window_W7_roster_admits_nothing_active_outside_it
  • ledger_rows_placed_candidates
  • ledger_rows_placed_placements
  • ledger_rows_placed_agencies
  • records_outside_every_audit_period_candidates
  • records_outside_every_audit_period_placements
  • records_outside_every_audit_period_agencies
  • unledgered_gap_record_agy_00011
  • unledgered_gap_record_cand_00193
  • unledgered_gap_record_plc_00190
  • unledgered_gap_record_cand_01170
  • unledgered_gap_record_cand_03060
  • unledgered_gap_record_cand_04860
  • unledgered_gap_record_cand_09004
  • unledgered_gap_record_plc_00056
  • unledgered_gap_record_agy_09003
  • unledgered_gap_record_agy_00012
  • unledgered_gap_record_cand_00194
  • unledgered_gap_record_plc_00189
  • activity_outcome_agy_00007
  • activity_outcome_cand_02369
  • activity_outcome_cand_04210
  • activity_outcome_agy_00001
  • and 21 more