Fix queue and signal race replay, exact continue-as-new carry, input loss between executions, and KV key order

[waldemort-auto]

Opened on behalf of Affan Dar (@affandar).

Fixes #66. Fixes #40. Fixes #57.

Related, not fixed here: #58 (this PR only adds collision notices to the failure commit), #4 / #5 (messages queued before a brand-new instance starts; Fix 5 covers only the gap between two executions).

Summary

This PR fixes a family of race bugs around dropped queue reads and signal waits, the carry-forward of queue messages at continue_as_new, inputs that arrive while an instance is between two executions, the history written when a continue_as_new successor is poisoned, and a replay failure caused by the order of KV prune clears. Several of them lose messages silently or make replay fail with a nondeterminism error.

Fix	Problem	Change	Gated
1. Queue reads	A dropped dequeue_event future kept its FIFO position until its cancellation row was replayed, so live execution and replay could pick different winners (nondeterminism), skip a message or read one twice	A dropped read is cancelled at the drop, or when it binds if it was dropped first	Yes
2. Signal waits	A dropped wait still counted as open: a later wait could receive the stale value (replay failure), or an instance could hang forever	A dropped bound wait is cancelled at the drop, and a value already given to its slot is discarded	Yes
3. Cancellation order	Cancellation rows were written in HashSet order, so two runs of the same turn could write different histories	Sorted by id within each type; replay compares these rows as a set, so older histories are unaffected	No
4. Continue-as-new carry	Unread messages were counted, not tracked: a dropped or held read counted as a consumer and a message could be lost	The engine records arrival positions and the positions reads actually returned, and carries exactly the unread ones (still at most 100)	Yes
5. Inputs between executions	A fetch that saw the ContinuedAsNew history but not yet the successor's start acknowledged (deleted) every input in the batch	Such a batch is abandoned with the unregistered-handler backoff, keeping its attempt; a start that never appears ends in poison after max_attempts	No
6. Duplicate starts	A duplicate client start could hide the continue-as-new start, or replace a running instance's handler; colliding sub-orchestration parents were notified only for finished instances	A start for an instance with history is ignored; colliding parents are notified for running instances too, except the instance's own parent call with a known, equal parent execution id	No
7. SQLite abandon	abandon_orchestration_item undid attempts and applied its delay to every row of the instance, including rows the fetch never locked	Both apply only to the locked rows, as on PostgreSQL; the lock delete validates the token	No
8. KV key order	prune_kv_values_updated_before and get_kv_all_keys used HashMap order, which differs between replays, so replay failed with kv clear mismatch	Both use key order; replay accepts a recorded run of clears in any order, so histories from older runtimes still replay	No
9. Poisoned successor	When the successor's batch was poisoned, OrchestrationFailed was appended to the previous execution after its ContinuedAsNew row, so the execution had two terminal rows and the runtime still read it as "continued as new"	The successor is committed as its own failed execution N+1 (OrchestrationStarted from the CAN start, or from execution N's name and parent when the start is missing, then the failure). Histories written by older runtimes are still treated as terminal	No

Compatibility

• Fixes 1, 2 and 4 change which branch wins a race, so they are pinned per execution by the duroxide_version stamp on the execution's first history row. Executions started before 0.1.31 keep the old behaviour, including its known defects, until they finish or continue as new. An unparsable stamp keeps the old behaviour and logs a warning.
• Fixes 3, 5, 6, 7, 8 and 9 change no recorded decision and apply to every execution. Fix 9 changes only the history written when a successor is poisoned; client status (Failed) and the parent's SubOrchFailed are unchanged.
• The crate version is 0.1.31 so the tests exercise the real threshold. RELEASE_POLICY.md reserves 0.1.31: no release numbered 0.1.31 or later may ship without this change.
• API: TurnResult::ContinueAsNew gains the field unconsumed_queue_arrivals. Code that builds or exhaustively matches the variant must add the field or use ...
• Provider contract: abandon_orchestration_item must apply its attempt undo and its delay only to the rows that fetch locked. New public validations: provider_validations::race_replay::*, poison_message::orchestration_ignore_attempt_preserves_hidden_start and poison_message::orchestration_delayed_abandon_preserves_unlocked_rows.
• Operators: see the migration guide (drain recipes, core-runtime rollback across 0.1.31, and the new warnings).

Testing

Check	Result
Each commit on its own: cargo fmt --check, Clippy -D warnings	clean
cargo test --workspace --all-features on the pinned Rust 1.97.1	1,263 passed, 0 failed
Exhaustive generators (about 588,000 enumerated race cases; every 0.1.31 case asserted for live/replay equality and FIFO; the 0.1.30 half matches a digest recorded on the unmodified 0.1.30 engine)	pass, digests unchanged
Upgrade fixture: histories recorded with the unmodified 0.1.30 engine replay and resume unchanged	pass
PostgreSQL 15 via duroxide-pg with the new validations wrapped (microsoft/duroxide-pg#35)	367 passed, 0 failed; 8 long-running tests passed
duroxide-node, 1,000 messages across repeated continue_as_new (in-memory SQLite, file SQLite, PostgreSQL) plus two varied-timer cases	every message delivered exactly once
Mutation check: each decision this PR adds was broken on purpose, one at a time	45 of 45 caught by a test (37 race-fix decisions, 3 KV-order decisions, 5 poisoned-successor decisions); an end-to-end test also fails if a poisoned child stops notifying its parent

Developed on Rust 1.99; after rebasing onto main, every commit and the full workspace suite were re-checked on the repository's pinned Rust 1.97.1.

Commits

• 1ed4085 replay: cancel dropped queue reads with execution-pinned policy (4 files, +60 / -5)
• 7d986e0 replay: preserve bound-slot signal admission and cancellation (1 file, +31 / -15)
• 632065d continue-as-new: carry exactly the unread queue positions (3 files, +53 / -4)
• 615b59d dispatcher: preserve CAN admission and terminal input handling (4 files, +102 / -15)
• e39ae3b sqlite: scope abandon attempt undo and delay to locked rows (1 file, +19 / -29)
• f607838 test: cover queue signal and CAN replay boundaries (20 files, +14,376 / -5)
• 91589c2 docs: describe the race policy cutover and deferred work (10 files, +343 / -57)
• a33bf24 kv: clear and list keys in key order (4 files, +110 / -4)
• 67415d0 poison: record a continue-as-new successor as its own execution (8 files, +497 / -62)

Downstream

Repository	PR	Status
duroxide-pg	microsoft/duroxide-pg#35	Draft: claim-time fetch lease fix plus wrappers for the new provider validations; 367 passed, 0 failed
duroxide-node	microsoft/duroxide-node#24	Draft: moves to 0.1.31 with 3 regression tests; 163 passed, 0 failed
duroxide-python	microsoft/duroxide-python#26	Draft: the same change for Python; 166 passed, 0 failed

The SDK and provider PRs stay drafts until this PR is released; each builds against this branch through a temporary [patch.crates-io].

End to end in microsoft/PilotSwarm. PilotSwarm main (77c68e1a) ran on one machine twice: with the released duroxide-node 0.2.0, and with duroxide-node built from this PR and microsoft/duroxide-pg#35.

Check	duroxide-node 0.2.0	Built from this PR
race(dequeueEvent, scheduleTimer) loops, 300 messages each (PostgreSQL ×5, in-memory SQLite ×3)	8 of 8 fail with nondeterministic: history schedule but no emitted action	8 of 8 complete; every message once, in order
1,000 messages, timers 10-50 ms, continue_as_new every 200 cycles (×2)	2 of 2 fail with the same error before any message is applied	2 of 2 complete; 1,000 applied, 0 lost, 0 reordered, 0 duplicates
PilotSwarm local suite	2,271 passed, 22 failed, 18 skipped	identical, the same 22 tests (they need a GitHub Copilot model the test machine does not have)

Deferred follow-ups are listed in design/queue-race-d2-follow-ups.md.