Deterministic simulation of multi-worker computations

[frankmcsherry]

Deterministic simulation of multi-worker computations

Adds timely::simulate: several workers hosted on the calling thread, with message delivery controlled by the caller.
An execution is a pure function of the applied sequence of Decisions (step a worker; deliver up to N messages from one worker to another).
Any decision sequence is valid, since deliveries clamp to what is pending, so schedule traces are recordable, replayable, and shrinkable by construction.

Design

Delivery control lives in the existing serializing allocators rather than in a separate simulation allocator.
A Gate sits on the receive side of ProcessAllocator and TcpAllocator.
It physically receives all bytes.
When built open, the default, bytes pass through as before.
When built holding, it splits received bytes into framed messages, files them by source worker, and surfaces a message only when released.
Held messages raise no events, so operators see only what has been logically delivered.
Per-source FIFO order is preserved; the driver chooses only the interleaving across sources, which is the schedule space a real transport can produce.
Messages a worker sends to itself are never held, matching real allocators.

Whether a gate holds is a builder setting, ProcessBuilder::holding, fixed when the allocator is built.
Simulation builds ordinary Bytes process allocators on one thread with holding gates.
Building those sequentially on one thread deadlocked, because each build waits on its peers, so ProcessBuilder::build_all fulfills every obligation before completing any allocator.

Commits:

• communication: the Gate, the holding builder setting, and build_all.
• timely: the Simulation/Decision driver and tests: multi-worker barriers asserting progress safety under seeded random schedules, and a property-testing grind whose oracles check frontier safety, conservation, quiescence, and same-seed determinism.

Why this is worth carrying in-tree

This harness produced the deterministic reproduction of TimelyDataflow/differential-dataflow#801, a batch-loss bug in DD's join that had resisted reproduction except as a ~20% flake under CPU contention with three workers.
Under the simulator the bug reproduced on a single worker with no chaos at all, deterministically on every seed, which made the bisection to first divergence (and the fix, differential-dataflow#802) tractable.
The follow-up plan is a DD regression test running the #801 computation under seeded schedules against a sequential oracle, once this is released.

Validation

• Full workspace tests pass, including the new tests.
• Release-mode grind passes, plus the #[ignore]d chaos_big_grind at 2,000 runs.
• No new clippy or rustdoc warnings.

🤖 Generated with Claude Code

https://claude.ai/code/session_01LEpSYjkqw2zfhGSMAXNXyD