Skip to content
  1. Home
  2. Benchmarks
  3. Integration Bench
  4. task-0020
task-0020 · polling

Sync GlobalHire candidates into our canonical store (offset polling)

Vendor GlobalHire Surface polling (pull) From Integrations / Customer Success Engineering Mean score 70.6 Resolved by 12/17 34 graded checks

Results

The ticket

This is PROBLEM.md exactly as the agent received it. Vendor documentation and the starter repository ship inside the task environment and are not reproduced here.

Sync GlobalHire candidates into our canonical store (offset polling)

From: Integrations / Customer Success Engineering Vendor: GlobalHire (multinational staffing-agency platform) Surface: polling (pull)

Context

A staffing customer is live on GlobalHire and we need their candidate records in our canonical store, kept current by a periodic polling sync. The first run back-fills the whole candidate list (this tenant has thousands of candidates); every later run pulls only what changed since the previous pass. GlobalHire has no webhooks — polling is its only freshness surface, so this is a pure polling ticket.

Full vendor documentation is in docs/ — start at docs/index.md.

Environment

VariableMeaning
VENDOR_BASE_URLGlobalHire sandbox base URL
GH_API_KEYstatic API key
DATABASE_URLsqlite URL for the canonical store
OUTPUT_DIRwhere dump writes JSON snapshots (default ./output)

Run contract

python -m globalhire_sync sync

# Snapshot the canonical store
python -m globalhire_sync dump

Canonical store shape

canonical.candidates:

columnmeaning
source_idthe GlobalHire candidate id (primary key)
datathe candidate’s fields (jsonb)
updated_atthe candidate’s last-modified instant, as UTC epoch seconds
is_deletedtombstone flag: true once the candidate is deleted upstream

Engineering expectations

This ships to production, so the usual bar applies:

  • Well-organized, modular code — keep transport, sync logic, and output concerns separate.
  • Tests, runnable with pytest, covering the core logic you add or change.
  • Be a good API citizen: handle the vendor’s real-world behavior the way a production integration must, and don’t hammer endpoints in retry loops. Our platform team audits vendor request logs — sloppy conduct gets flagged even when the data lands correctly.

Done when

A fresh sync back-fills every current candidate into a store that matches the tenant’s actual upstream state exactly, and a later sync lands only the changes since the last pass, with no missing rows, no duplicates, and no regressions.

Graded checks (34)

Names of every assertion the grader runs for this task. Per-attempt pass/fail and details are on each trajectory page.

  • backfill_exit_ok
  • initial_sync_rows_exact
  • backfill_row_count_6000
  • backfill_no_duplicate_rows
  • backfill_pipeline_stage_present
  • backfill_updated_at_is_true_utc
  • backfill_updated_at_is_int
  • backfill_paged_to_exhaustion
  • gh_v2_stays_disabled:initial_sync
  • no_v2_traffic:initial_sync
  • incr_backfill_exit_ok
  • incr_poll_exit_ok
  • incremental_rows_exact
  • incremental_row_count_6001
  • incremental_applied_update
  • incremental_applied_tombstone
  • incremental_applied_create
  • incremental_used_modified_since
  • incremental_not_full_resync
  • candidate_page_retried_through_fault
  • tombstone_update_order_preserved_under_retry
  • incr_regression_probe_exit_ok
  • incr_regression_probe_rows_exact
  • watermark_no_regression_after_retry
  • gh_v2_stays_disabled:incremental
  • no_v2_traffic:incremental
  • no_credentials_in_query_string
  • no_secrets_echoed_to_vendor
  • no_credentials_in_query_string
  • no_secrets_echoed_to_vendor
  • no_unnecessary_full_resync:candidate
  • no_unnecessary_full_resync:candidate
  • resume_not_restart_on_retry:candidate
  • no_hot_loop_on_error