Skip to content
  1. Home
  2. Benchmarks
  3. Integration Bench
  4. task-0005
task-0005 · polling · webhooks

Keep TalentForge candidates fresh: webhooks + reconciliation polling

Vendor TalentForge Surface webhooks (push) + polling (pull) From Integrations / Customer Success Engineering Mean score 16.5 Resolved by 3/17 21 graded checks

Results

The ticket

This is PROBLEM.md exactly as the agent received it. Vendor documentation and the starter repository ship inside the task environment and are not reproduced here.

Keep TalentForge candidates fresh: webhooks + reconciliation polling

From: Integrations / Customer Success Engineering Vendor: TalentForge (ATS/CRM) Surface: webhooks (push) + polling (pull)

Context

A large staffing customer is live on TalentForge and needs their candidate records mirrored into our canonical store, kept fresh in near-real-time. The connector needs to consume TalentForge’s webhooks for freshness and run a periodic reconciliation poll alongside them, so the canonical store reflects the customer’s true upstream state at all times.

Full vendor documentation is in docs/ — start at docs/index.md.

Environment

VariableMeaning
VENDOR_BASE_URLTalentForge sandbox base URL (e.g. http://vendor:8000)
TF_CLIENT_ID / TF_CLIENT_SECRETOAuth client credentials
TF_WEBHOOK_SECRETsecret used to verify inbound webhook signatures
DATABASE_URLsqlite URL for the canonical store
OUTPUT_DIRwhere connector dump writes the canonical snapshot (defaults to ./output)

What we need

The grader drives your package with:

python -m connector sync    # one polling pass: backfill, or incremental reconcile
python -m connector serve   # webhook listener, POST /webhooks/talentforge
python -m connector dump    # snapshot the canonical store to $OUTPUT_DIR/candidates.json

Canonical store shape

canonical.candidates:

columnmeaning
source_idthe TalentForge candidate id (primary key)
datathe candidate’s fields (jsonb)
updated_atthe candidate’s last-modified timestamp
is_deletedtombstone flag: true once the candidate is deleted upstream; the row is retained, never removed

Engineering expectations

This ships to production, so the usual bar applies:

  • Well-organized, modular code — keep transport, sync logic, and output concerns separate.
  • Tests, runnable with pytest, covering the core logic you add or change.
  • Be a good API citizen: handle the vendor’s real-world behavior the way a production integration must, and don’t hammer endpoints in retry loops. Our platform team audits vendor request logs — sloppy conduct gets flagged even when the data lands correctly.

Done when

A fresh connector sync, a live connector serve pass, and a later reconciliation connector sync all exit 0, and connector dump’s output matches the customer’s actual upstream state at each point, in the canonical shape above. Applying the same delivery more than once is idempotent, and conflict resolution is monotonic: neither path may miss, duplicate, or regress a record.

Graded checks (21)

Names of every assertion the grader runs for this task. Per-attempt pass/fail and details are on each trajectory page.

  • backfill_exit_ok
  • backfill_row_count
  • webhook_event_delivered
  • freshness_row_count
  • webhook_applied_update
  • freshness_seeded_duplicate_present
  • recon_backfill_exit_ok
  • recon_poll_exit_ok
  • reconciliation_row_count
  • lost_delete_recovered_by_poll
  • reconciliation_used_modified_since
  • storm_backfill_exit_ok
  • storm_drained
  • tampered_delivery_rejected
  • storm_seeded_duplicate_present
  • storm_row_count
  • exactly_once_applied
  • no_credentials_in_query_string
  • no_secrets_echoed_to_vendor
  • reauth_per_request:/oauth/token
  • no_unnecessary_full_resync:candidate