Fix #21: start the fetch lease when the instance is claimed; run the queue-race validations

[waldemort-auto]

Opened on behalf of Affan Dar (@affandar).

Fixes #21. This is the provider side of microsoft/duroxide#46 (see also microsoft/duroxide#47).

Requires duroxide 0.1.32, the release that contains microsoft/duroxide#67. It is on crates.io, and both manifests use it from there. There is no [patch] entry.

The SDK drafts microsoft/duroxide-node#24 and microsoft/duroxide-python#26 build against this branch. duroxide-node built from it also passed the end-to-end PilotSwarm check described in microsoft/duroxide#67.

1. fetch_orchestration_item never returns an already-expired lease (migration 0025)

Problem

The lease was p_now_ms + p_lock_timeout_ms. The caller reads p_now_ms before the call: before taking a pooled connection, and before the advisory-lock wait inside the call. When those waits were longer than the lock timeout, the item was returned with a lease that had already expired. Every ack then failed with Invalid lock token. That includes the poison ack at max_attempts, so the message was retried at the lock-timeout interval for as long as the contention lasted.

Change

Migration 0025 redefines fetch_orchestration_item exactly as in 0019, with one change. The lease is computed when the instance is claimed:

v_lease_base_ms := GREATEST(
    p_now_ms + (EXTRACT(EPOCH FROM (clock_timestamp() - statement_timestamp())) * 1000)::BIGINT,
    (EXTRACT(EPOCH FROM clock_timestamp()) * 1000)::BIGINT
);
v_locked_until := v_lease_base_ms + p_lock_timeout_ms;

• The first term covers waits inside the call and stays in the caller's clock. ack_orchestration_item checks the lease against the caller's clock.
• The second term covers a p_now_ms that was already stale when the call started.
• GREATEST can only make the lease longer. When the database clock is ahead of the worker's, a crashed worker's item is recovered later by that skew.

Nothing else changes. The locking, the candidate selection, the re-check and the predicates stay as in 0019. migrations/0025_diff.md was generated with scripts/generate_migration_diff.sh 25.

Making the claim non-blocking, bounding the candidate loop and refreshing the caller clock in the predicates are performance improvements. They are tracked in #34.

Compatibility

• Signature and result shape are unchanged. There is no Rust change. CREATE OR REPLACE keeps the function's OID and grants.
• 0022 and 0024 are used by open pull requests. The runner applies every pending embedded version in order, so the gap is harmless.
• fix(orchestrator): never return an already-expired lock lease #22 should be closed rather than merged. Its 0024 also redefines this function, and the runner would apply it after 0025.
• As with any new migration, a binary without 0025 refuses a database that has it.

Tests

tests/fetch_lease_regression_tests.rs reproduces both cases from #21:

Test	On main	With 0025
stale_caller_timestamp_never_yields_an_expired_lease: p_now_ms 10 s stale on arrival, both procedure branches	Fail: lease −5,020 ms when returned	Pass: about 5,000 ms left; the ack commits
hot_instance_contention_never_yields_an_expired_lease_and_poison_commits: 16 dispatchers wait 4 s behind a session holding the instance (lock timeout 2 s); the message is at max_attempts	Fail: lease −2,006 ms when returned; the poison ack fails with Invalid lock token	Pass: about 2,000 ms left; the poison ack commits and the queue drains (the fetches still wait about 4 s behind the holder)

2. Run the queue-race provider validations on PostgreSQL

Wraps the provider validations added by microsoft/duroxide#67, so that PostgreSQL runs them. They cover race replay, continue-as-new transition delivery for both version stamps, a poisoned continue-as-new successor, duplicate starts, hidden-start attempt preservation, and the delayed-abandon locked-row contract. PostgreSQL already met them, so no provider code changes.

Test results

PostgreSQL 15.19 in a disposable container, Rust 1.97.1. The first two rows were run against duroxide 0.1.32 from crates.io. The others were run earlier, against microsoft/duroxide#67.

Check	Result
cargo nextest run --all-features	359 passed, 0 failed, 9 skipped
cargo package	passes
cargo test --workspace	367 passed, 0 failed, 9 ignored
Long-running ignored tests (-- --ignored, live Entra test excluded)	8 passed, 0 failed
The 2 regression tests without migration 0025 (schemas at migration 0023)	both fail as shown above
Every test schema that applied 0025	fetch_orchestration_item has the new lease and the unchanged advisory lock
cargo fmt --check	clean
cargo clippy --all-targets --all-features -- -D warnings	no findings in files this PR changes; 5 existing findings on main

Commits

Commit	Summary
cd974bb	fix(orchestrator): start the fetch lease when the instance is claimed
80ffb5a	test: run the queue-race provider validations on PostgreSQL
8df07b9	build: build against the duroxide v0.1.32 tag
f15f9c6	build: use duroxide 0.1.32 from crates.io