Skip to content
  1. Home
  2. Benchmarks
  3. Integration Bench
  4. task-0001
task-0001 · polling

Sync StaffLine candidates, jobs, applications, and notes into our canonical store

Vendor StaffLine Surface polling (pull) From Integrations / Customer Success Engineering Mean score 77.9 Resolved by 14/17 30 graded checks

Results

The ticket

This is PROBLEM.md exactly as the agent received it. Vendor documentation and the starter repository ship inside the task environment and are not reproduced here.

Sync StaffLine candidates, jobs, applications, and notes into our canonical store

From: Integrations / Customer Success Engineering Vendor: StaffLine (legacy staffing ATS) Surface: polling (pull)

Context

A staffing customer just went live on StaffLine and we need their data flowing into our canonical store so the rest of the product (search, dedupe, reporting) can use it: a one-time back-fill of everything that exists today, plus an ongoing catch-up pass that keeps us current as records change upstream. StaffLine has no webhooks, so polling is this integration’s only freshness mechanism.

Full vendor documentation is in docs/ — start at docs/index.md.

What we need

The grader runs your package exactly two ways — these commands are the contract:

# Full back-fill
python -m staffline_sync

# Incremental catch-up
python -m staffline_sync --incremental

Canonical output shape

One JSON file per entity kind under OUTPUT_DIR, each a JSON array of rows sorted by source_id:

KeyMeaning
source_idthe StaffLine record id
dataevery other field as returned by the API
updated_atStaffLine’s own modification timestamp for the row
is_deletedtombstone flag

Don’t change this shape — the downstream consumer already depends on it.

Environment

VariableMeaning
VENDOR_BASE_URLStaffLine sandbox base URL (e.g. http://vendor:8000)
SL_APP_TOKEN / SL_HMAC_SECRETthis tenant’s application credentials
DATABASE_URLSQLite URL backing the canonical store (sqlite:////data/canonical.db)
OUTPUT_DIRwhere the JSON output files land (defaults to ./output)

Engineering expectations

This ships to production, so the usual bar applies:

  • Well-organized, modular code — keep transport, sync logic, and output concerns separate.
  • Tests, runnable with pytest, covering the core logic you add or change.
  • Be a good API citizen: handle the vendor’s real-world behavior the way a production integration must, and don’t hammer endpoints in retry loops. Our platform team audits vendor request logs — sloppy conduct gets flagged even when the data lands correctly.

Done when

A fresh back-fill populates all four canonical files from StaffLine’s current state, and — after StaffLine has accumulated its own edits, creates, and deletes — a later incremental pass converges all four files to match, with both python -m staffline_sync and python -m staffline_sync --incremental exiting 0.

Graded checks (30)

Names of every assertion the grader runs for this task. Per-attempt pass/fail and details are on each trajectory page.

  • app_exit_ok
  • backfill_row_count:candidates
  • backfill_row_count:jobs
  • backfill_row_count:applications
  • backfill_row_count:notes
  • applications_carry_stage
  • include_stage_on_every_applications_call
  • app_exit_ok
  • tombstone_sweep_complete
  • incremental_watermark_uses_server_clock
  • app_exit_ok
  • signing_timestamp_not_skew_corrected
  • no_credentials_in_query_string
  • no_secrets_echoed_to_vendor
  • no_credentials_in_query_string
  • no_secrets_echoed_to_vendor
  • no_credentials_in_query_string
  • no_secrets_echoed_to_vendor
  • no_unnecessary_full_resync:candidate
  • no_unnecessary_full_resync:job
  • no_unnecessary_full_resync:application
  • no_unnecessary_full_resync:note
  • no_unnecessary_full_resync:candidate
  • no_unnecessary_full_resync:job
  • no_unnecessary_full_resync:application
  • no_unnecessary_full_resync:note
  • no_unnecessary_full_resync:candidate
  • no_unnecessary_full_resync:job
  • no_unnecessary_full_resync:application
  • no_unnecessary_full_resync:note