Skip to content
  1. Home
  2. Benchmarks
  3. Integration Bench
  4. task-0002
task-0002 · polling

Legal-hold export: TalentForge custodians and everything on file against them

Vendor TalentForge Surface polling (pull) From Integrations / Customer Success Engineering Mean score 87.7 Resolved by 15/17 64 graded checks

Results

The ticket

This is PROBLEM.md exactly as the agent received it. Vendor documentation and the starter repository ship inside the task environment and are not reproduced here.

Legal-hold export: TalentForge custodians and everything on file against them

From: Integrations / Customer Success Engineering Vendor: TalentForge (enterprise ATS/CRM) Surface: polling (pull) Category: build · Track: python · Tier: 2

Context

A staffing customer of ours is in litigation and their counsel has issued a preservation notice. It names a set of people whose TalentForge records — and every recruiter note filed against them — have to be collected into one reviewable file and handed to the other side’s e-discovery vendor. Counsel will re-request this at intervals while the matter runs, so it has to be a job we can re-run against a tenant that has moved on, not something an account manager assembles by hand in the TalentForge UI once.

The preservation list is in the repo at input/legal_hold_roster.csv (matter_ref,custodian_email). Email addresses are all we get: counsel has no TalentForge ids and is never going to have any.

A handover note from the first manual production is at input/HANDOVER-legal-hold.md.

Full vendor documentation is in docs/ — start at docs/index.md.

Scope rules

These come from our legal team. They are not derivable from anything in TalentForge, so take them as given:

  • A hold attaches to an address, not to a record. If more than one person in the tenant carries a roster address, every one of them is a custodian. We do not get to decide which one counsel meant.
  • Scope is the roster, not the state of the file. Archived, closed and withdrawn people are in scope exactly like active ones, and their notes come with them. A hold does not lapse because someone’s file did.
  • Completeness is the deliverable. Review proceeds on the assumption that the export is everything the tenant holds against each custodian.
  • An address we cannot place is a finding. Report it; never drop it.

What we need

The grader runs your package exactly this way — this command is the contract:

python -m talentforge_legal_hold export

It writes one artifact, $OUTPUT_DIR/legal_hold_export.json:

KeyMeaning
roster_row_countrows read from the roster CSV
custodian_countcustodians in this export
note_countnotes across all custodians
custodians[]one object per custodian (below)
unmatched_roster_emails[]roster addresses that match nobody in the tenant

Each custodian object:

KeyMeaning
matter_ref / roster_emailthe roster row this custodian came from
candidate_idthe TalentForge record id
given_name, family_name, phone, pipeline_status, is_deletedas the tenant holds them
created_at, updated_atUTC ISO 8601 seconds (2020-01-02T03:04:05Z)
notes[]note_id, author, body, created_at — the note’s own timestamp, carried across unchanged

A custodian with nothing on file still gets a row, with "notes": [] and the rest of their details filled in — counsel asked to be told about those people too. Ordering inside the file is already handled by the starter’s writer, so build the lists in whatever order suits you.

Environment

VariableMeaning
VENDOR_BASE_URLTalentForge sandbox base URL (e.g. http://vendor:8000)
TF_CLIENT_ID / TF_CLIENT_SECRETthis tenant’s OAuth client credentials
ROSTER_PATHthe preservation roster CSV
OUTPUT_DIRwhere legal_hold_export.json lands (defaults to ./output)

Engineering expectations

This ships to production, so the usual bar applies:

  • Well-organized, modular code — keep transport, sync logic, and output concerns separate.
  • Tests, runnable with pytest, covering the core logic you add or change.
  • Be a good API citizen: handle the vendor’s real-world behavior the way a production integration must, and don’t hammer endpoints in retry loops. Our platform team audits vendor request logs — sloppy conduct gets flagged even when the data lands correctly.

Done when

python -m talentforge_legal_hold export exits 0 and legal_hold_export.json describes exactly the people the roster names, as TalentForge holds them at the moment it runs, with everything on file against each of them and nothing that belongs to anybody else.

Graded checks (64)

Names of every assertion the grader runs for this task. Per-attempt pass/fail and details are on each trajectory page.

  • lh_backfill_export_written
  • lh_backfill_full_export_contract_exact
  • lh_custodian_set_matches_roster_resolution
  • lh_shared_email_keeps_every_holder
  • lh_unresolvable_roster_rows_reported
  • lh_roster_row_count_declared
  • lh_custodian_record_cand_0012
  • lh_custodian_record_cand_0017
  • lh_custodian_record_cand_0042
  • lh_custodian_record_cand_0052
  • lh_custodian_record_cand_0071
  • lh_custodian_record_cand_0099
  • lh_custodian_record_cand_0147
  • lh_custodian_record_cand_0151
  • lh_custodian_record_cand_0157
  • lh_custodian_record_cand_0158
  • lh_custodian_record_cand_0171
  • lh_custodian_record_cand_0236
  • lh_custodian_record_cand_0280
  • lh_custodian_record_cand_0301
  • lh_custodian_record_cand_0322
  • lh_custodian_record_cand_0351
  • lh_custodian_record_cand_0355
  • lh_custodian_record_cand_0378
  • lh_custodian_record_cand_0398
  • lh_note_payloads_carried_across_verbatim
  • lh_note_ids_exact_per_custodian
  • lh_childless_custodians_exported_with_empty_notes
  • lh_export_holds_no_repeated_note
  • lh_declared_note_total_matches_the_tenant
  • lh_parent_cursor_walk_reached_the_last_page
  • lh_parent_pages_not_rewalked
  • lh_custodian_detail_read_for_every_custodian
  • lh_no_detail_read_outside_the_roster
  • lh_custodian_detail_not_reread
  • lh_every_custodian_subresource_requested
  • lh_no_subresource_sweep_outside_the_roster
  • lh_multipage_note_tails_requested
  • lh_note_pages_not_rewalked
  • lh_resweep_export_written
  • lh_resweep_full_export_contract_exact
  • lh_resweep_custodian_set_tracks_the_moved_tenant
  • lh_newly_created_custodian_on_the_ninth_page_resolved
  • lh_resweep_unresolvable_roster_rows_reported
  • lh_soft_deleted_custodian_retained_with_notes
  • lh_resweep_moved_record_cand_0017
  • lh_resweep_moved_record_cand_0042
  • lh_resweep_moved_record_cand_0099
  • lh_resweep_moved_record_cand_0900
  • lh_resweep_custodian_fields_match_the_moved_tenant
  • lh_resweep_note_ids_exact_per_custodian
  • lh_resweep_parent_walk_reached_the_page_that_did_not_exist_before
  • lh_resweep_custodian_detail_read_for_every_custodian
  • lh_resweep_no_detail_read_outside_the_roster
  • lh_resweep_every_custodian_subresource_requested
  • lh_resweep_multipage_note_tails_requested
  • no_credentials_in_query_string
  • no_secrets_echoed_to_vendor
  • no_credentials_in_query_string
  • no_secrets_echoed_to_vendor
  • and 4 more