Skip to content
  1. Home
  2. Benchmarks
  3. Integration Bench
  4. task-0034
task-0034 · polling · writeback

Weekly crew top-up is putting people on the gate twice

Vendor CrewCall Surface polling, writeback From Integrations / Workforce Platform Mean score 82.3 Resolved by 14/17 165 graded checks

Results

The ticket

This is PROBLEM.md exactly as the agent received it. Vendor documentation and the starter repository ship inside the task environment and are not reproduced here.

Weekly crew top-up is putting people on the gate twice

From: Integrations / Workforce Platform Vendor: CrewCall Surface: polling, writeback Category: harden · Track: python · Tier: 3

Context

Ironvale Stadium Services staff their match days through Fairweather Crewing. Every Sunday night the agency sends over the week’s placement list and our job puts each of those people on our CrewCall tenant, because the stewards’ gate scanners resolve a worker id and nothing else. The job has been running since last season and it exits clean every week.

It is being moved onto a much larger stadium next month, and before that happens the ops lead wants the thing gone over. Two things came in from the desk after last Sunday’s run and neither has an explanation yet:

“Marisol scanned in on Saturday as a brand new starter with no history and no pay band, and the old record is still sitting there under her name. Rota has her twice now and I can’t tell the two apart.”

“Three of the returners from last winter never made it onto the tenant at all. The report says they were fine. They were turned away at the gate.”

Full vendor documentation is in docs/ — start at docs/index.md.

What we need

The same job, safe to run against a tenant several times this size.

Two rules the desk works to, which are ours and not CrewCall’s:

  • A crew member is the CrewCall worker holding the address on the agency’s line, compared case-insensitively. That is the only identity the agency and we share.
  • A worker CrewCall no longer holds is not that crew member. If somebody the tenant has let go turns up on the placement list again, they are a new signup: the stewards cannot scan a badge we no longer own.

Every placed crew member must come out of the run as exactly one worker on the tenant, and every row of the agency’s file must carry the id that crew member ends up with.

The test harness runs your code exactly as follows — this command is the contract:

python -m ironvale_topup

Output artifacts

  • output/placement_report.csv — one row per row of the agency’s file, in file order, with the columns placement_ref,shift_date,crew_email,worker_id,outcome. outcome is matched or created.
  • output/topup_summary.jsonrow_count, person_count, matched_count, created_count, roster_rows_seen, and people: one entry per crew member the file places, carrying person_key, crew_email, crew_name, placement_refs, outcome and worker_id.

Environment

VariableMeaning
VENDOR_BASE_URLVendor sandbox base URL (e.g. http://vendor:8000)
CC_API_KEYVendor credential injected by the test harness
INPUT_FILEThe agency’s weekly placement file
OUTPUT_DIRDirectory where output artifacts land (defaults to ./output)
PAGE_LIMITPage size the roster sweep asks for

Engineering expectations

This ships to production, so the usual bar applies:

  • Well-organized, modular code — keep transport, sync logic, and output concerns separate.
  • Tests, runnable with pytest, covering the core logic you add or change.
  • Be a good API citizen: handle the vendor’s real-world behavior the way a production integration must, and don’t hammer endpoints in retry loops. Our platform team audits vendor request logs — sloppy conduct gets flagged even when the data lands correctly.

Done when

The run exits 0, both artifacts are written, and the tenant holds each placed crew member exactly once.

Graded checks (165)

Names of every assertion the grader runs for this task. Per-attempt pass/fail and details are on each trajectory page.

  • topup_run_completed
  • topup_headline_counts_exact
  • topup_roster_rows_seen_matches_the_tenant
  • refused_roster_windows_were_read_not_skipped
  • roster_sweep_reran_from_the_first_window
  • roster_sweep_spanned_every_position_the_tenant_holds
  • accepted_signups_match_the_crew_crewcall_lacks
  • placed_amara_diallo_crew_test_identity_exact
  • placed_blaise_haas_crew_test_identity_exact
  • placed_blaise_petrov_crew_test_identity_exact
  • placed_charles_larsen_crew_test_identity_exact
  • placed_charles_wang_crew_test_identity_exact
  • placed_cyrus_wang_crew_test_identity_exact
  • placed_emmy_jansen_crew_test_identity_exact
  • placed_emmy_quinn_crew_test_identity_exact
  • placed_fatima_kowalski_crew_test_identity_exact
  • placed_florence_rossi_crew_test_identity_exact
  • placed_grace_ibarra_crew_test_identity_exact
  • placed_grace_yamada_crew_test_identity_exact
  • placed_hedy_oduya_crew_test_identity_exact
  • placed_jean_tan_crew_test_identity_exact
  • placed_katherine_haas_crew_test_identity_exact
  • placed_katherine_ibarra_crew_test_identity_exact
  • placed_katherine_quinn_crew_test_identity_exact
  • placed_linus_mbeki_crew_test_identity_exact
  • placed_linus_tan_crew_test_identity_exact
  • placed_linus_yamada_crew_test_identity_exact
  • placed_marie_ibarra_crew_test_identity_exact
  • placed_marisol_okafor_ironvale_test_identity_exact
  • placed_nadia_okonkwo_crew_test_identity_exact
  • placed_niamh_doherty_ironvale_test_identity_exact
  • placed_oliver_wang_crew_test_identity_exact
  • placed_peggy_zhang_crew_test_identity_exact
  • placed_richard_santos_crew_test_identity_exact
  • placed_sophie_nakamura_crew_test_identity_exact
  • placed_teodor_vasilev_ironvale_test_identity_exact
  • placed_ursula_mbeki_crew_test_identity_exact
  • placed_ursula_oduya_crew_test_identity_exact
  • placed_ursula_tan_crew_test_identity_exact
  • placed_vera_ueda_crew_test_identity_exact
  • placed_walter_tan_crew_test_identity_exact
  • placement_iv_2601_carries_the_right_worker
  • placement_iv_2602_carries_the_right_worker
  • placement_iv_2603_carries_the_right_worker
  • placement_iv_2604_carries_the_right_worker
  • placement_iv_2605_carries_the_right_worker
  • placement_iv_2606_carries_the_right_worker
  • placement_iv_2607_carries_the_right_worker
  • placement_iv_2608_carries_the_right_worker
  • placement_iv_2609_carries_the_right_worker
  • placement_iv_2610_carries_the_right_worker
  • placement_iv_2611_carries_the_right_worker
  • placement_iv_2612_carries_the_right_worker
  • placement_iv_2613_carries_the_right_worker
  • placement_iv_2614_carries_the_right_worker
  • placement_iv_2615_carries_the_right_worker
  • placement_iv_2616_carries_the_right_worker
  • placement_iv_2617_carries_the_right_worker
  • placement_iv_2618_carries_the_right_worker
  • placement_iv_2619_carries_the_right_worker
  • and 105 more