Skip to content
  1. Home
  2. Benchmarks
  3. Integration Bench
  4. task-0024
task-0024 · polling · webhooks

TalentLoop connector: unified freshness across webhook-covered and poll-only entities

Vendor TalentLoop Surface webhooks (push) and polling (read), all four entities From Integrations / Customer Success Engineering Mean score 0.0 Resolved by 0/17 40 graded checks

Results

The ticket

This is PROBLEM.md exactly as the agent received it. Vendor documentation and the starter repository ship inside the task environment and are not reproduced here.

TalentLoop connector: unified freshness across webhook-covered and poll-only entities

From: Integrations / Customer Success Engineering Vendor: TalentLoop (event-first modern ATS) Surface: webhooks (push) and polling (read), all four entities

Context

A customer runs TalentLoop as their ATS. We need a canonical mirror of ALL FOUR of their entities — candidates, jobs, applications, and notes — that stays fresh on an ongoing basis, not just at initial setup.

Full vendor documentation is in docs/ — start at docs/index.md.

Environment

VariableMeaning
VENDOR_BASE_URLTalentLoop sandbox base URL
TL_CLIENT_ID / TL_CLIENT_SECRETOAuth2 client credentials
TL_WEBHOOK_SECRETHMAC secret for webhook verification
DATABASE_URLsqlite URL for the canonical store
OUTPUT_DIRwhere dump writes JSON snapshots (default ./output)

Report / outputs

output/candidates.json / output/jobs.json / output/applications.json / output/notes.json (written by dump)

The canonical mirrors, each a JSON array of {source_id, data, updated_at, is_deleted} rows sorted by source_id.

Run contract

python -m talentloop_selective backfill
python -m talentloop_selective poll
python -m talentloop_selective serve [--max-events N] [--idle-timeout S] [--max-runtime S]
python -m talentloop_selective dump

Engineering expectations

This ships to production, so the usual bar applies:

  • Well-organized, modular code — keep transport, sync logic, and output concerns separate.
  • Tests, runnable with pytest, covering the core logic you add or change.
  • Be a good API citizen: handle the vendor’s real-world behavior the way a production integration must, and don’t hammer endpoints in retry loops. Our platform team audits vendor request logs — sloppy conduct gets flagged even when the data lands correctly.

Done when

A later poll reflects upstream changes across all four entities — the mirror stays current on an ongoing basis, not just at the initial backfill. Webhook-driven freshness holds under real delivery conditions, with a tampered delivery never touching the store. A deletion in any of the four entities lands as is_deleted: true in the canonical mirror, regardless of which discovery path finds it first.

Graded checks (40)

Names of every assertion the grader runs for this task. Per-attempt pass/fail and details are on each trajectory page.

  • backfill_exit_ok
  • backfill_candidates_rows_exact
  • backfill_jobs_rows_exact
  • backfill_applications_rows_exact
  • backfill_notes_rows_exact
  • webhook_events_delivered
  • freshness_candidates_rows_exact
  • freshness_applications_rows_exact
  • webhook_applied_delete::cand_0007
  • webhook_applied_update::cand_0055
  • webhook_applied_delete::app_0009
  • backfill_exit_ok
  • backfill_candidates_rows_exact
  • backfill_jobs_rows_exact
  • backfill_applications_rows_exact
  • backfill_notes_rows_exact
  • second_poll_exit_ok
  • poll_recur_candidates_rows_exact
  • poll_recur_applications_rows_exact
  • poll_recur_jobs_rows_exact
  • poll_recur_notes_rows_exact
  • poll_recur_applied_job_status
  • poll_recur_applied_note_body
  • tamper_backfill_exit_ok
  • tamper_drained
  • tampered_delivery_present
  • tampered_delivery_rejected
  • tamper_candidates_rows_exact
  • tamper_applications_rows_exact
  • exactly_once_delete_applied
  • webhook_bad_signature_rejected
  • webhook_stale_timestamp_rejected
  • no_credentials_in_query_string
  • no_secrets_echoed_to_vendor
  • webhook_bad_signature_rejected
  • webhook_stale_timestamp_rejected
  • reauth_per_request:/token
  • no_unnecessary_full_resync:candidate
  • no_unnecessary_full_resync:job
  • no_unnecessary_full_resync:application