fix(executions): claim-token fencing and lease-style recovery of lost claims

[psteinroe]

Claims and claim-dependent writes were fenced only by locked_by = orchestrator_id, and every call retried ambiguous connection errors. As a result, a lost claim response could orphan a batch until restart, and an old request from an earlier claim could change a newer claim of the same execution. This PR adds a per-claim fencing token and recovers orphaned claims the way a lease expires. It also fixes returnExecutions retries, which never re-sent anything.

Claim token

Every claim sets a fresh claim_token (gen_random_uuid()). Settlement, saveStep, registerEventWait and clearWaitingState require it alongside locked_by. Before this, a retried or late request from an earlier claim could act on a newer claim by the same orchestrator. Tests reproduce both harmful cases:

• A replayed released result released the running claim. That decremented attempts, cleared the lock, and let a second worker run the execution at the same time.
• A replayed registerEventWait call created a fresh subscription and released the running claim. That claim's own result was then dropped, and the execution waited for an event that never came.

dispatchCustomEvents stays on locked_by. It only inserts deduplicated delivery rows and never changes the source's claim, which existing tests already cover. attempts can't serve as the token because suspension decrements it.

Claims are not retried after ambiguous errors; orphans are released by the heartbeat

A connection error doesn't tell us whether the statement committed. getExecutions therefore retries only rollback errors (40001, 40P01, 55P03), and a failed poll is followed by the next poll. A claim that committed but whose response was lost is recovered like an expired lease:

• Each worker tracks the executions it holds: claimed and not yet settled, including results buffered for flush.
• Every heartbeat sends those ids. It releases rows locked by its orchestrator that are not held and were locked more than one heartbeat interval (30s) ago. The grace covers a claim whose response is still in flight. The release clears locked_by/locked_at exactly like stale-orchestrator recovery.
• The heartbeat also retries only rollback errors. A retry would compare an old snapshot of held ids against claims made after it, and could release one of them. A failed heartbeat is followed by the next one.

This is how other queues handle it. None of them makes the claim itself idempotent; they bound how long an orphaned claim can stay locked:

• Graphile Worker retries its claim only on serialization, deadlock and can't-connect errors (40001, 40P01, 57P03, EHOSTUNREACH, ETIMEDOUT), and releases any job locked for more than 4 hours.
• pg-boss doesn't retry queries. Orphaned active jobs fail by expiry, or by an optional per-job heartbeat timeout where workers touch the jobs they hold. That's the closest match to this design. pg-boss also recently added a per-attempt fence on retry_count for the stale-settle problem (#925).
• SQS and Temporal recover a lost receive or poll response through the visibility timeout or task timeout.

Earlier revisions made the claim retry idempotent with a client token and an advisory lock. That was dropped in favour of this simpler lease-style recovery.

Settlement retries

returnExecutions built one postgres.js query and passed () => query to the retry loop. A postgres.js query runs only once, so every retry re-awaited the same rejection with backoff up to 60s until shutdown. The query is now built inside the retry callback.

Verification

• Integration tests, each failing without the change:
  • the heartbeat releases a claim the orchestrator does not hold: a claim made for the orchestrator but never handed to its worker survives a heartbeat within the grace. Past the grace it is released and run by the worker, while the execution the worker holds stays locked throughout.
  • retries a settlement whose connection was terminated: the backend is killed with pg_terminate_backend mid-statement.
  • fences a replayed release after the same orchestrator re-claims the execution
  • does not let a replayed registration release a later claim
• just lint, just format, bun run typecheck and the full bun test are clean.

Follow-ups

None. The broader concern about connection-error retries of writes that aren't idempotent (#55) was resolved by #63.