Skip to content
  1. Home
  2. Benchmarks
  3. Integration Bench
  4. task-0028
task-0028 · polling

Harden the CrewCall snapshot connector for production

Vendor CrewCall Surface polling (pull) From Integrations / Customer Success Engineering Mean score 70.6 Resolved by 12/17 44 graded checks

Results

The ticket

This is PROBLEM.md exactly as the agent received it. Vendor documentation and the starter repository ship inside the task environment and are not reproduced here.

Harden the CrewCall snapshot connector for production

From: Integrations / Customer Success Engineering Vendor: CrewCall (high-churn temp-staffing marketplace) Surface: polling (pull)

Context

We already have a CrewCall connector that snapshots workers, gigs, and assignments faithfully against a quiet sandbox. The first production exports were missing records visible in CrewCall, and one scheduled run exited nonzero. It needs to be safe to schedule unattended.

Full vendor documentation is in docs/ — start at docs/index.md.

Environment

VariableMeaning
VENDOR_BASE_URLCrewCall sandbox base URL (e.g. http://vendor:8000)
CC_API_KEYthe credential the connector authenticates with
OUTPUT_DIRwhere output files go (defaults to ./output)

What we need

The grader runs your package the same way every time — this is the contract:

python -m crewcall_sync sync

Output format (unchanged)

Same canonical shape as the existing connector already produces: source_id, data (the full raw wire object), updated_at (ISO-8601 Z string), and is_deleted (tombstone flag). Do not change these shapes.

Engineering expectations

This ships to production, so the usual bar applies:

  • Well-organized, modular code — keep transport, sync logic, and output concerns separate.
  • Tests, runnable with pytest, covering the core logic you add or change.
  • Be a good API citizen: handle the vendor’s real-world behavior the way a production integration must, and don’t hammer endpoints in retry loops. Our platform team audits vendor request logs — sloppy conduct gets flagged even when the data lands correctly.

Done when

A fresh python -m crewcall_sync sync produces the correct, duplicate-free snapshot of workers, gigs, and assignments at the time of the run, matching the tenant’s actual upstream state and failing clearly rather than publishing an incomplete result.

Graded checks (44)

Names of every assertion the grader runs for this task. Per-attempt pass/fail and details are on each trajectory page.

  • sync_exit_ok
  • resume_not_restart
  • no_retry_after_violation_under_faults
  • worker_rows_exact
  • worker_expected_count
  • gig_rows_exact
  • gig_expected_count
  • assignment_rows_exact
  • assignment_expected_count
  • worker_no_duplicate_ids
  • gig_no_duplicate_ids
  • assignment_no_duplicate_ids
  • exactly_once
  • sync_exit_ok
  • resume_not_restart
  • no_retry_after_violation_under_faults
  • worker_rows_exact
  • worker_expected_count
  • gig_rows_exact
  • gig_expected_count
  • assignment_rows_exact
  • assignment_expected_count
  • worker_no_duplicate_ids
  • gig_no_duplicate_ids
  • assignment_no_duplicate_ids
  • exactly_once
  • checkpoint2_create_landed
  • checkpoint2_delete_tombstoned
  • checkpoint2_update_landed
  • checkpoint2_multiple_full_passes
  • no_credentials_in_query_string
  • no_secrets_echoed_to_vendor
  • no_credentials_in_query_string
  • no_secrets_echoed_to_vendor
  • no_unnecessary_full_resync:worker
  • resume_not_restart_on_retry:worker
  • no_unnecessary_full_resync:gig
  • no_unnecessary_full_resync:assignment
  • no_hot_loop_on_error
  • no_unnecessary_full_resync:worker
  • resume_not_restart_on_retry:worker
  • no_unnecessary_full_resync:gig
  • no_unnecessary_full_resync:assignment
  • no_hot_loop_on_error