Skip to content
  1. Home
  2. Benchmarks
  3. Integration Bench
  4. task-0038
task-0038 · polling · webhooks

TalentLoop connector: make webhook-driven freshness durable against a misbehaving receiver

Vendor TalentLoop Surface webhooks (push) and polling (read) From Integrations / Customer Success Engineering Mean score 0.0 Resolved by 0/17 37 graded checks

Results

The ticket

This is PROBLEM.md exactly as the agent received it. Vendor documentation and the starter repository ship inside the task environment and are not reproduced here.

TalentLoop connector: make webhook-driven freshness durable against a misbehaving receiver

From: Integrations / Customer Success Engineering Vendor: TalentLoop (event-first modern ATS) Surface: webhooks (push) and polling (read)

Context

A customer runs TalentLoop as their ATS. Our connector keeps our platform’s mirror of their candidates, jobs, applications, and notes current. The business requirement is non-negotiable: no candidate or application change — a pipeline update, a note left after an interview, a withdrawn application — may go missing from our mirror, even one our own receiver mishandles or never gets a chance to see at all. A recruiter acting on stale data is a customer-visible failure, not an edge case we can shrug off.

Full vendor documentation is in docs/ — start at docs/index.md.

Environment

VariableMeaning
VENDOR_BASE_URLTalentLoop sandbox base URL
TL_CLIENT_ID / TL_CLIENT_SECRETOAuth2 client credentials
TL_WEBHOOK_SECRETHMAC secret for webhook verification
DATABASE_URLsqlite URL for the canonical store
OUTPUT_DIRwhere dump writes JSON snapshots (default ./output)

Report / outputs

output/candidates.json / output/jobs.json / output/applications.json / output/notes.json (written by dump)

The canonical mirrors, each a JSON array of {source_id, data, updated_at, is_deleted} rows sorted by source_id.

Run contract

python -m talentloop_reliable backfill
python -m talentloop_reliable poll
python -m talentloop_reliable serve [--max-events N] [--idle-timeout S] [--max-runtime S]
python -m talentloop_reliable recover_missed_events
python -m talentloop_reliable dump

Engineering expectations

This ships to production, so the usual bar applies:

  • Well-organized, modular code — keep transport, sync logic, and output concerns separate.
  • Tests, runnable with pytest, covering the core logic you add or change.
  • Be a good API citizen: handle the vendor’s real-world behavior the way a production integration must, and don’t hammer endpoints in retry loops. Our platform team audits vendor request logs — sloppy conduct gets flagged even when the data lands correctly.

Done when

Every candidate and application change lands in the canonical mirror, whether it arrives through the webhook listener or has to be picked up through the platform’s own undelivered-event listing — including a change whose push attempt never happened at all. By the end of a run, nothing this tenant’s platform still considers undelivered is left outstanding. The job/note mirror keeps recurring on every poll, not just at initial backfill, unaffected by any of the above.

Graded checks (37)

Names of every assertion the grader runs for this task. Per-attempt pass/fail and details are on each trajectory page.

  • backfill_exit_ok
  • backfill_candidates_rows_exact
  • backfill_applications_rows_exact
  • webhook_events_acked::cand_0007_delete
  • webhook_events_acked::cand_0055_update
  • webhook_events_acked::app_0009_delete
  • poll_exit_ok
  • job_note_polling_unaffected
  • webhook_freshness_baseline::cand_0007
  • webhook_freshness_baseline::cand_0055
  • webhook_freshness_baseline::app_0009
  • webhook_freshness_baseline::no_regression_candidates
  • webhook_freshness_baseline::no_regression_applications
  • ack_token_recomputed_per_attempt::cand_0007_delete
  • ack_token_recomputed_per_attempt::cand_0055_update
  • ack_token_recomputed_per_attempt::app_0009_delete
  • dlq_backfill_exit_ok
  • ordinary_events_unaffected_by_drop_fault::cand_0007_delete
  • ordinary_events_unaffected_by_drop_fault::cand_0055_update
  • recover_missed_events_exit_ok
  • dead_letter_recovered_via_drain::cand_0007
  • dead_letter_recovered_via_drain::cand_0055
  • dead_letter_recovered_via_drain::app_0009
  • dead_letter_recovered_via_drain::no_regression_candidates
  • dead_letter_recovered_via_drain::no_regression_applications
  • dead_letter_recovered_delete_applied::app_0009
  • dlq_drained_empty
  • no_credentials_in_query_string
  • no_secrets_echoed_to_vendor
  • webhook_bad_signature_rejected
  • webhook_stale_timestamp_rejected
  • no_credentials_in_query_string
  • no_secrets_echoed_to_vendor
  • reauth_per_request:/token
  • no_unnecessary_full_resync:candidate
  • no_unnecessary_full_resync:job
  • reauth_per_request:/token