Skip to content
  1. Home
  2. Benchmarks
  3. Integration Bench
  4. task-0040
task-0040 · writeback · webhooks

Push interview reschedules upstream into Interviewly

Vendor Interviewly Surface writeback (reschedule) + webhooks (confirmation) From Integrations / Customer Success Engineering Mean score 0.0 Resolved by 0/17 20 graded checks

Results

The ticket

This is PROBLEM.md exactly as the agent received it. Vendor documentation and the starter repository ship inside the task environment and are not reproduced here.

Push interview reschedules upstream into Interviewly

From: Integrations / Customer Success Engineering Vendor: Interviewly (interview scheduling) Surface: writeback (reschedule) + webhooks (confirmation)

Context

We need to push interview reschedules from our system into Interviewly using its documented write operation.

Full vendor documentation is in docs/ — start at docs/index.md.

Environment

VariableMeaning
VENDOR_BASE_URLInterviewly sandbox base URL
IV_CLIENT_ID / IV_CLIENT_SECRETOAuth client credentials
IV_WEBHOOK_SECRETwebhook signing secret
SERVE_HOST / SERVE_PORTwebhook listener bind address
INPUT_FILEstaged reschedules to push (defaults to input/pending_reschedules.json)
OUTPUT_DIRwhere output files go (defaults to ./output)

What we need

python -m interviewly_writeback push   # push the staged reschedules
python -m interviewly_writeback serve  # webhook listener + confirmation handling
python -m interviewly_writeback dump   # re-write output files from the current store

Output format

$OUTPUT_DIR/writeback_result.json is {"reschedules": [...]}, sorted by client_ref, one record per staged item: client_ref, interview_id, requested_scheduled_at, idempotency_key (derive it as reschedule:<client_ref> — downstream reconciliation keys on that exact format), status (pending, committed, or failed), confirmation_event_id (set once confirmed, otherwise null), ok (bool), and error (the vendor’s failure body, or null).

$OUTPUT_DIR/interviews.json is the same canonical shape as our other Interviewly connectors: a source_id-sorted array of {source_id, data, updated_at, is_deleted}.

Engineering expectations

This ships to production, so the usual bar applies:

  • Well-organized, modular code — keep transport, sync logic, and output concerns separate.
  • Tests, runnable with pytest, covering the core logic you add or change.
  • Be a good API citizen: handle the vendor’s real-world behavior the way a production integration must, and don’t hammer endpoints in retry loops. Our platform team audits vendor request logs — sloppy conduct gets flagged even when the data lands correctly.

Done when

Every well-formed staged reschedule reaches committed in writeback_result.json, matching what actually landed upstream, exactly once even under a repeated push of the same batch; the malformed entry is reported as a failure and never applied.

Graded checks (20)

Names of every assertion the grader runs for this task. Per-attempt pass/fail and details are on each trajectory page.

  • push_exit_ok
  • 202_treated_as_provisional_not_committed
  • committed_on_confirming_event
  • confirmation_event_id_recorded
  • writeback_row_fields_exact::resched-1
  • writeback_row_fields_exact::resched-2
  • first_push_exit_ok
  • first_push_committed
  • second_push_exit_ok
  • still_committed_after_retry
  • confirmation_event_id_stable_across_retry
  • both_pushes_reached_vendor
  • retry_reused_same_idempotency_key
  • no_credentials_in_query_string
  • no_secrets_echoed_to_vendor
  • no_credentials_in_query_string
  • no_secrets_echoed_to_vendor
  • reauth_per_request:/oauth/token
  • reauth_per_request:/oauth/token
  • idempotent_write_retries