Skip to content
  1. Home
  2. Benchmarks
  3. Integration Bench
  4. task-0012
task-0012 · polling · writeback · webhooks

Build the full TalentLoop connector: polling, webhooks, and writeback

Vendor TalentLoop Surface polling (pull), webhooks (push), writeback (POST) From Integrations / Customer Success Engineering Mean score 37.2 Resolved by 7/17 123 graded checks

Results

The ticket

This is PROBLEM.md exactly as the agent received it. Vendor documentation and the starter repository ship inside the task environment and are not reproduced here.

Build the full TalentLoop connector: polling, webhooks, and writeback

From: Integrations / Customer Success Engineering Vendor: TalentLoop (event-first ATS) Surface: polling (pull), webhooks (push), writeback (POST)

Context

We’re bringing TalentLoop on as a full two-way integration: candidates, jobs, applications, and notes need to stay in sync in both directions, and this has to hold up in production, not just in a clean demo. The repository contains the first implementation of all three surfaces (repo/, package talentloop_summit) against TalentLoop’s sandbox.

Full vendor documentation is in docs/ — start at docs/index.md.

The repository contains a starter implementation spanning authentication, persistence and the five command surfaces below. Inspect it as a whole; no module should be assumed correct merely because another component calls it.

Environment

VariableMeaning
VENDOR_BASE_URLTalentLoop sandbox base URL (e.g. http://vendor:8000)
TL_CLIENT_ID / TL_CLIENT_SECRETOAuth client-credentials
TL_WEBHOOK_SECRETwebhook signing secret
INPUT_FILEstaged writeback batch (defaults to input/pending_writeback.json)
OUTPUT_DIRwhere output files go (defaults to ./output)

What we need

The grader runs your package the same way every time — this is the contract:

python -m talentloop_summit backfill   # full crawl, all 4 entities
python -m talentloop_summit serve      # webhook listener
python -m talentloop_summit poll       # incremental sweep — must run repeatedly, not just once
python -m talentloop_summit push       # drain the staged writeback batch
python -m talentloop_summit dump       # write output/*.json for inspection

Output format

dump writes output/{candidates,jobs,applications,notes}.json, each a sorted JSON array of canonical rows: {"source_id", "data", "updated_at", "is_deleted"}. push writes output/writeback_result.json: {"events": [...]}, one entry per staged item ({"client_ref", "ok", "record"} on success, {"client_ref", "ok": false, "error"} on failure), sorted by client_ref. Do not change these shapes.

Engineering expectations

This ships to production, so the usual bar applies:

  • Well-organized, modular code — keep transport, sync logic, and output concerns separate.
  • Tests, runnable with pytest, covering the core logic you add or change.
  • Be a good API citizen: handle the vendor’s real-world behavior the way a production integration must, and don’t hammer endpoints in retry loops. Our platform team audits vendor request logs — sloppy conduct gets flagged even when the data lands correctly.

Done when

backfill, serve, poll, and push all exit 0, and after a full sync cycle the canonical store and writeback output match TalentLoop’s actual upstream state for all four entities — every update and delete correctly reflected regardless of which surface first revealed it, no duplicate or missing records, and the declared output and idempotency contracts holding across repeated runs.

Graded checks (123)

Names of every assertion the grader runs for this task. Per-attempt pass/fail and details are on each trajectory page.

  • backfill_exit_ok
  • backfill_row_count:candidate
  • backfill_fields_exact:candidate
  • backfill_row_count:job
  • backfill_fields_exact:job
  • backfill_row_count:application
  • backfill_fields_exact:application
  • backfill_row_count:note
  • backfill_fields_exact:note
  • webhook_events_delivered
  • tampered_delivery_rejected
  • poll_exit_ok
  • freshness_row_count:candidate
  • freshness_fields_exact:candidate
  • freshness_row_count:job
  • freshness_fields_exact:job
  • freshness_row_count:application
  • freshness_fields_exact:application
  • freshness_row_count:note
  • freshness_fields_exact:note
  • webhook_applied_delete::cand_0007
  • webhook_applied_update::cand_0055
  • webhook_applied_delete::app_0009
  • poll_applied_update::job_0003
  • poll_applied_update::note_0004
  • backfill_exit_ok
  • backfill_row_count:candidate
  • backfill_fields_exact:candidate
  • backfill_row_count:job
  • backfill_fields_exact:job
  • backfill_row_count:application
  • backfill_fields_exact:application
  • backfill_row_count:note
  • backfill_fields_exact:note
  • fault_actually_dropped_event
  • poll_exit_ok
  • reconcile_row_count:job
  • reconcile_fields_exact:job
  • reconcile_row_count:application
  • reconcile_fields_exact:application
  • reconcile_row_count:note
  • reconcile_fields_exact:note
  • reconcile_row_count:candidate
  • reconcile_fields_exact:candidate
  • reconciled_delete_applied
  • reconciled_via_410
  • no_other_candidate_incorrectly_tombstoned
  • multidrop_backfill_exit_ok
  • multidrop_poll_cp4_exit_ok
  • multidrop_poll_cp5_exit_ok
  • fault_dropped_update_event
  • fault_dropped_appdelete_event
  • multidrop_row_count:job
  • multidrop_fields_exact:job
  • multidrop_row_count:note
  • multidrop_fields_exact:note
  • multidrop_row_count:candidate
  • multidrop_fields_exact:candidate
  • multidrop_row_count:application
  • multidrop_fields_exact:application
  • and 63 more