Skip to content
  1. Home
  2. Benchmarks
  3. Integration Bench
  4. task-0022
task-0022 · polling

Harden the GlobalHire backfill for a production tenant

Vendor GlobalHire Surface polling (pull) From Integrations / Reliability Engineering Mean score 58.8 Resolved by 10/17 28 graded checks

Results

The ticket

This is PROBLEM.md exactly as the agent received it. Vendor documentation and the starter repository ship inside the task environment and are not reproduced here.

Harden the GlobalHire backfill for a production tenant

From: Integrations / Reliability Engineering Vendor: GlobalHire (multinational staffing-agency platform) Surface: polling (pull)

Context

We already have a GlobalHire polling connector (repo/, package globalhire_sync) that backfills the ~6,000-record candidate list into our canonical store and keeps it current with incremental catch-ups. Reliability wants it hardened before we run it against GlobalHire under real production load.

Full vendor documentation is in docs/ — start at docs/index.md.

Environment

VariableMeaning
VENDOR_BASE_URLGlobalHire sandbox base URL (e.g. http://vendor:8000)
GH_API_KEYstatic API key
DATABASE_URLsqlite URL for the canonical store
OUTPUT_DIRwhere dump writes the snapshot

What we need

The grader runs your package exactly two ways:

python -m globalhire_sync sync
python -m globalhire_sync dump

python -m globalhire_sync sync
python -m globalhire_sync dump

Canonical store shape

canonical.candidates:

columnmeaning
source_idthe GlobalHire candidate id (primary key)
datathe candidate’s fields (jsonb), verbatim from the wire
updated_atthe candidate’s last-modified instant, as UTC epoch seconds
is_deletedtombstone flag: true once the candidate is deleted upstream

Engineering expectations

This ships to production, so the usual bar applies:

  • Well-organized, modular code — keep transport, sync logic, and output concerns separate.
  • Tests, runnable with pytest, covering the core logic you add or change.
  • Be a good API citizen: handle the vendor’s real-world behavior the way a production integration must, and don’t hammer endpoints in retry loops. Our platform team audits vendor request logs — sloppy conduct gets flagged even when the data lands correctly.

Done when

A fresh sync back-fills every current candidate into a store that matches the tenant’s actual upstream state exactly, and a later sync lands only the changes since the last pass, with no record missed, duplicated, or corrupted under real vendor conditions.

Graded checks (28)

Names of every assertion the grader runs for this task. Per-attempt pass/fail and details are on each trajectory page.

  • app_exit_ok
  • backfill_rows_exact
  • backfill_row_count_6000
  • resume_not_restart
  • exactly_once
  • retry_after_honored_l3
  • incr_backfill_exit_ok
  • app_exit_ok
  • incremental_rows_exact
  • incremental_row_count_6001
  • incremental_applied_update
  • incremental_applied_tombstone
  • incremental_applied_create
  • resume_not_restart
  • exactly_once
  • no_retry_after_violation_on_incremental
  • incremental_used_modified_since
  • no_credentials_in_query_string
  • no_secrets_echoed_to_vendor
  • no_credentials_in_query_string
  • no_secrets_echoed_to_vendor
  • retry_after_honored
  • no_unnecessary_full_resync:candidate
  • resume_not_restart_on_retry:candidate
  • no_hot_loop_on_error
  • no_unnecessary_full_resync:candidate
  • resume_not_restart_on_retry:candidate
  • no_hot_loop_on_error