Skip to content
  1. Home
  2. Benchmarks
  3. Integration Bench
  4. task-0015
task-0015 · writeback

Bulk-import a migration batch into StaffLine

Vendor StaffLine Surface writeback (bulk create) From Integrations / Customer Success Engineering Mean score 5.9 Resolved by 1/17 26 graded checks

Results

The ticket

This is PROBLEM.md exactly as the agent received it. Vendor documentation and the starter repository ship inside the task environment and are not reproduced here.

Bulk-import a migration batch into StaffLine

From: Integrations / Customer Success Engineering Vendor: StaffLine (legacy staffing ATS) Surface: writeback (bulk create)

Context

A staffing customer is migrating a batch of candidates out of a spreadsheet and into StaffLine. They want the batch imported in bulk rather than one record at a time, and — because this feeds their own reporting — they need our canonical store to reflect exactly which candidates actually ended up in StaffLine, not just which ones we asked StaffLine to create. StaffLine shipped a bulk import endpoint for exactly this kind of migration.

The batch to import is staged in repo/input/candidate_batch.json: a JSON list of pending candidates, each with a stable client_ref (our own handle for that logical record) plus the candidate’s fields.

Full vendor documentation is in docs/ — start at docs/index.md.

Environment

VariableMeaning
VENDOR_BASE_URLStaffLine sandbox base URL (e.g. http://vendor:8000)
SL_APP_TOKEN / SL_HMAC_SECRETstatic application token + HMAC signing secret
DATABASE_URLsqlite URL for the canonical store
INPUT_FILEpath to the staged batch (defaults to ./input/candidate_batch.json)
OUTPUT_DIRwhere output files go (defaults to ./output)

What we need

The grader runs your package the same way every time — this is the contract:

python -m staffline_bulk push
python -m staffline_bulk dump

Engineering expectations

This ships to production, so the usual bar applies:

  • Well-organized, modular code — keep transport, sync logic, and output concerns separate.
  • Tests, runnable with pytest, covering the core logic you add or change.
  • Be a good API citizen: handle the vendor’s real-world behavior the way a production integration must, and don’t hammer endpoints in retry loops. Our platform team audits vendor request logs — sloppy conduct gets flagged even when the data lands correctly.

Done when

A push against the staged batch produces bulk_result.json with every item accounted for exactly once, created reflecting each record’s genuine, confirmed presence in StaffLine rather than the API’s immediate response; and a later push against the same batch and the same tenant leaves that result unchanged and creates nothing new.

Graded checks (26)

Names of every assertion the grader runs for this task. Per-attempt pass/fail and details are on each trajectory page.

  • push_completed_and_output_readable
  • mixed_status_bookkeeping_matches_fixture
  • mixed_multiple_bulk_calls_issued
  • transient_503_items_created_after_retry
  • transient_items_retried_by_same_client_ref
  • permanents_not_retried
  • retry_payload_preserves_original_fields
  • push_completed_and_output_readable
  • lying_ref_marked_not_created_and_independently_confirmed_absent
  • lying_ref_output_never_reports_an_id
  • creates_output_matches_true_vendor_state
  • lying_scenario_status_bookkeeping_matches_fixture
  • reconciliation_attempted_at_all
  • reconciliation_not_abandoned_after_one_read
  • first_push_matches_fixture
  • both_pushes_completed_and_readable
  • no_batch_replay_duplicates_exact_count
  • second_push_did_not_regress_first_push_result
  • no_batch_replay_duplicates
  • no_credentials_in_query_string
  • no_secrets_echoed_to_vendor
  • no_credentials_in_query_string
  • no_secrets_echoed_to_vendor
  • no_credentials_in_query_string
  • no_secrets_echoed_to_vendor
  • no_unnecessary_full_resync:candidate