DDIR: run the examples and tests through the server; retire the ddir harness

[frankmcsherry]

The single-program harness (interactive/examples/ddir.rs) duplicated the server's job: build one dataflow, fill its inputs, close epochs. This retargets the examples, the test suites and the AoC suite onto the server, and — since the repo had grown two server front ends, both building a binary named ddir_server — ends with exactly one: the live ddir-server crate. Net −451 lines with the example driver gone.

What the server gained

• feed <prog> <in#> from <recipe-or-file> — fills a positional input from a source the server reads itself, sharded across the workers (each worker feeds row % peers == index through its own handle), so no row crosses the wire. The source is a recipe an import already accepts (random:nodes=N,edges=E[,arity=A][,seed=S][,churn=C], iota:N) or a text file of whitespace-separated integer rows. A churn=C recipe keeps changing the input on every tick — the harness's batch/rounds loop — through the same per-input generator that drives generated sources (now one map per program, not one slot).
• load <name> from <path> [explain=<arity>[,debug]] [name=binding …] — installs a program file (.ddp pipe syntax, else applicative), read on the session thread like an inline body. explain= applies the explanation rewrite before optimization and replaces the reserved --explain; the query input is the one after the program's own, the demand sets are peekable exports.
• peek <trace> [key] and tick [n] reporting wall-clock time alongside the epoch.
• server::evaluate(backend, timely::Config, program, inputs) — the data-in/data-out entry point the test suites use: install, feed, tick, snapshot, the path a live install takes. It replaces the vec and corgi backends' private evaluate harnesses, so the corgi gate (1–4 workers and over serializing channels) and the explanation tests all run against the server on both backends.
• The session scripts, the AoC suite (run.sh [vec|corgi], 33/33 on both) and the READMEs pipe into cargo run -p ddir-server with DDIR_WORKERS / DDIR_BACKEND. install is spelled load throughout, the crate's word. The crate now runs on the same global allocator the example driver did.
• The corgi typer accepts the if(1, $1, 0) idiom (a literal condition compiles only its live branch), which was the one AoC part corgi could not run.

Deleted: the harness, the example driver, survey_sources, the diagnostics dev-dependency on interactive, and the harness's --sync=K. The last one cannot be reproduced yet: several epochs in flight make leave_dynamic see a two-element stamp, and it asks for a singleton.

What other workloads will need from the server

Opening the discussion, from what this migration ran into:

1. Input formats. feed … from reads integer rows only. Anything with strings, decimals or dates needs a transcription step (as AoC does) or a richer loader.
2. Output capture. Results come back via peek (one data line per row) or tail (a standing change stream), both as text on the session. A client that wants an export as data — a file, a typed channel — has nothing yet.
3. The open-loop regime. tick n sync=k is a dozen lines in the server but blocked in DD's dynamic scope as above.
4. Sharded loading of large files. Every worker reads the whole file and keeps its shard; fine at 2M rows, not at 200M.
5. Programs over several inputs fill them one feed … from at a time; multi-input round-robin from one file (what EDGES_FILE did) is gone on purpose.
6. Timely's own flags. The crate is configured by environment (DDIR_WORKERS, DDIR_BACKEND, bind addresses); anything execute_from_args used to take (-w, -n, -p, hosts) is not reachable, so multi-process runs are not either.

🤖 Generated with Claude Code

https://claude.ai/code/session_01QDsLC46QQW9aBrksaWad6T