Skip to content
  1. Home
  2. Benchmarks
  3. Integration Bench
  4. task-0016
task-0016 · polling · writeback

HireWire connector: push stage-change events + keep an incremental poll fresh

Vendor HireWire Surface writeback (PATCH + POST) and polling (incremental read) From Integrations / Customer Success Engineering Mean score 93.2 Resolved by 17/17 48 graded checks

Results

The ticket

This is PROBLEM.md exactly as the agent received it. Vendor documentation and the starter repository ship inside the task environment and are not reproduced here.

HireWire connector: push stage-change events + keep an incremental poll fresh

From: Integrations / Customer Success Engineering Vendor: HireWire (a scrappy-startup ATS) Surface: writeback (PATCH + POST) and polling (incremental read)

Context

We run a two-way integration with HireWire for a staffing customer. Our product both reads candidates out of HireWire on a schedule and writes stage-change activity back into it. A starter connector is in repo/; inspect the complete read and write paths against the contract below.

Full vendor documentation is in docs/ — start at docs/index.md.

Environment

VariableMeaning
VENDOR_BASE_URLHireWire sandbox base URL (e.g. http://vendor:8000)
HW_API_KEYstatic API key
INPUT_FILEpath to the staged writeback batch (defaults to input/pending_events.json)
OUTPUT_DIRwhere output files go (defaults to ./output)

Report / outputs

output/writeback_result.json (written by push)

{
  "events": [
    {"client_ref": "evt-1", "ok": true,  "candidate": { ...current state of the patched candidate... },
     "event": { ...the created event record... }},
    {"client_ref": "evt-3", "ok": false, "error": {"status": 422, "field_errors": {"event_type": ["is required"]}}}
  ]
}

events is sorted by client_ref. On success, candidate/event carry the patched candidate’s current state and the created event record; on a rejected write, ok is false and error carries the vendor’s status and field_errors body instead.

output/candidates.json (written by poll)

A JSON array of canonical candidate rows, sorted by source_id:

[ {"source_id": "cand_0001", "data": { ...raw record... }, "updated_at": 1773482460, "is_deleted": false} ]

source_id is the candidate id, data is the full record HireWire returned, updated_at is the last-modified timestamp as an integer, and is_deleted mirrors the candidate’s soft-delete flag (retained as a tombstone row, not dropped).

Run contract

python -m hirewire_connector push   # writes output/writeback_result.json
python -m hirewire_connector poll   # writes output/candidates.json
python -m hirewire_connector dump   # rewrites output/candidates.json from the current canonical store

Engineering expectations

This ships to production, so the usual bar applies:

  • Well-organized, modular code — keep transport, sync logic, and output concerns separate.
  • Tests, runnable with pytest, covering the core logic you add or change.
  • Be a good API citizen: handle the vendor’s real-world behavior the way a production integration must, and don’t hammer endpoints in retry loops. Our platform team audits vendor request logs — sloppy conduct gets flagged even when the data lands correctly.

Done when

push, a repeated push, and poll (initial backfill and a later incremental run) all exit 0, with output/writeback_result.json and output/candidates.json matching HireWire’s actual state at each point — every staged write landed exactly once with rejections reported rather than fatal or duplicated, and the candidate store current with no data lost or re-fetched needlessly.

Graded checks (48)

Names of every assertion the grader runs for this task. Per-attempt pass/fail and details are on each trajectory page.

  • push_exit_ok
  • result_readable
  • recorded_event_fields_exact::evt-1
  • recorded_event_fields_exact::evt-2
  • recorded_event_fields_exact::evt-3
  • no_unexpected_recorded_events
  • all_event_refs_present_once
  • successful_refs_correct
  • failed_refs_correct
  • malformed_item_reported_as_422
  • one_accepted_patch::evt-1
  • one_accepted_event_post::evt-1
  • one_accepted_patch::evt-2
  • one_accepted_event_post::evt-2
  • malformed_event_never_accepted
  • confirmed_via_get_by_id::evt-1
  • confirmed_via_get_by_id::evt-2
  • did_not_confirm_by_relisting
  • first_push_exit_ok
  • second_push_exit_ok
  • result_readable
  • retry_recorded_event_fields_unchanged::evt-1
  • retry_recorded_event_fields_unchanged::evt-2
  • retry_recorded_event_fields_unchanged::evt-3
  • event_ids_stable_across_retry
  • retries_carry_idempotency_key
  • backfill_exit_ok
  • backfill_store_readable
  • backfill_no_duplicate_rows
  • backfill_updated_at_is_int
  • backfill_paged_to_exhaustion
  • incr_poll_exit_ok
  • incremental_row_count
  • incremental_fields_exact
  • incremental_applied_update
  • incremental_applied_tombstone
  • incremental_applied_create
  • incremental_used_modified_since
  • watermark_sent_as_epoch_seconds
  • incremental_not_full_resync
  • no_credentials_in_query_string
  • no_secrets_echoed_to_vendor
  • no_credentials_in_query_string
  • no_secrets_echoed_to_vendor
  • no_credentials_in_query_string
  • no_secrets_echoed_to_vendor
  • idempotent_write_retries
  • no_unnecessary_full_resync:candidate