Skip to content
  1. Home
  2. Benchmarks
  3. Integration Bench
  4. task-0023
task-0023 · polling · webhooks

TalentLoop connector: keep a candidate/application mirror deletion-accurate

Vendor TalentLoop Surface polling (read) and webhooks (push) From Integrations / Customer Success Engineering Mean score 0.0 Resolved by 0/17 26 graded checks

Results

The ticket

This is PROBLEM.md exactly as the agent received it. Vendor documentation and the starter repository ship inside the task environment and are not reproduced here.

TalentLoop connector: keep a candidate/application mirror deletion-accurate

From: Integrations / Customer Success Engineering Vendor: TalentLoop (event-first modern ATS) Surface: polling (read) and webhooks (push)

Context

A customer runs TalentLoop as their ATS. We need a canonical mirror of their candidates and applications that stays accurate when a record is deleted upstream — not just when one is created or updated. The mirror can’t depend on a single discovery path: it must converge whether the connector learns about a deletion through a consumed webhook event or through a polling pass with no webhook listener running at all.

Full vendor documentation is in docs/ — start at docs/index.md.

Environment

VariableMeaning
VENDOR_BASE_URLTalentLoop sandbox base URL
TL_CLIENT_ID / TL_CLIENT_SECRETOAuth2 client credentials
TL_WEBHOOK_SECRETHMAC secret for webhook verification
DATABASE_URLsqlite URL for the canonical store
OUTPUT_DIRwhere dump writes JSON snapshots (default ./output)

Report / outputs

output/candidates.json / output/applications.json (written by dump)

The canonical mirrors, each a JSON array of {source_id, data, updated_at, is_deleted} rows sorted by source_id. A deleted record appears with is_deleted: true in your canonical output.

Run contract

python -m talentloop_deletes backfill
python -m talentloop_deletes poll
python -m talentloop_deletes serve [--max-events N] [--idle-timeout S] [--max-runtime S]
python -m talentloop_deletes dump

Engineering expectations

This ships to production, so the usual bar applies:

  • Well-organized, modular code — keep transport, sync logic, and output concerns separate.
  • Tests, runnable with pytest, covering the core logic you add or change.
  • Be a good API citizen: handle the vendor’s real-world behavior the way a production integration must, and don’t hammer endpoints in retry loops. Our platform team audits vendor request logs — sloppy conduct gets flagged even when the data lands correctly.

Done when

A candidate or application deleted upstream lands as is_deleted: true in the canonical mirror, whether discovered through the webhook path or through a poll-only run, and every command exits 0 with the mirror matching the tenant’s actual upstream state.

Graded checks (26)

Names of every assertion the grader runs for this task. Per-attempt pass/fail and details are on each trajectory page.

  • backfill_exit_ok
  • backfill_candidates_rows_exact
  • backfill_applications_rows_exact
  • webhook_delete_event_delivered
  • freshness_candidates_rows_exact
  • freshness_applications_rows_exact
  • webhook_applied_delete
  • poll_backfill_exit_ok
  • poll_backfill_candidates_rows_exact
  • poll_exit_ok
  • poll_reconcile_candidates_rows_exact
  • poll_reconcile_applications_rows_exact
  • poll_only_reconciled_delete_via_410
  • no_other_candidate_incorrectly_tombstoned
  • tamper_backfill_exit_ok
  • tamper_drained
  • tampered_delivery_present
  • tampered_delivery_rejected
  • tamper_candidates_rows_exact
  • tamper_applications_rows_exact
  • exactly_once_delete_applied
  • no_credentials_in_query_string
  • no_secrets_echoed_to_vendor
  • reauth_per_request:/token
  • no_unnecessary_full_resync:candidate
  • no_unnecessary_full_resync:application