Skip to content

Conformance suite: any pipeline binary against the protocol (#519) - #528

Merged
winding-lines merged 2 commits into
mainfrom
conformance-519
Nov 17, 2026
Merged

winding-lines merged 2 commits into
mainfrom
conformance-519

Conversation

@winding-lines

Copy link
Copy Markdown
Contributor

Closes #519.

What

tests/conformance.py checks any pipeline binary against docs/pipeline-protocol.md. It takes the binary's path and a JSON manifest that says what the pipeline is (steps, partitions, keys, the datasets each step reads and writes, plus optional env that slows a step and fixture commands for tables). It drives the binary only the way runners do: it inserts a run row as the scheduler starts one (running, owner scheduler, no heartbeat) and runs <binary> resume <run>. It never runs run --partitions. Over Postgres the pipeline is registered with morca register. Over SQLite, where no scheduler runs, the suite registers it itself.

Each check prints ok, FAIL (with the rule and what was wrong), xfail, XPASS or skip, and cites its doc section as docs/pipeline-protocol.md#<anchor>.

pixi run conformance <binary> <manifest>

Checks (pg and sqlite each for the run checks)

  • static: usage.unknown, describe.exit, describe.json (one line, one object, any spacing), describe.fields, describe.manifest, describe.hash, describe.morca, describe.protocol, resume.no_store, keys.not_keyed, keep-log.append, put-log
  • sensor: sense.lines, sense.no_warehouse
  • Postgres: register, register.compact
  • runs: resume.succeeds, resume.run_row, resume.steps, resume.attempts, resume.order, resume.logs, claim.done, claim.statuses (queued/failed/deferred), claim.live, claim.stale (incl. interrupt), resume.other_pipeline, resume.other_code (+ --allow-new-code), keep-log.crash, statuses
  • with slow_env: kill.resume (heartbeat stays fresh during a step longer than the stale limit, kill -9, resume skips what succeeded and retries what was interrupted), cancel.between, cancel.due (Postgres only; the doc says the SQLite beat handles no cancels), lost.stops
  • with tables: tables.commits (orca.* summary keys incl. run token), tables.reads, tables.crash_resume, tables.retry_finds_commit, tables.pinned (a crash after commit is simulated in the rows, a new input snapshot is committed, then resume)

Known-failing

Not checked: the outdated flag (what a step does with it is up to the step), timeouts (no example step declares one), HTTP store and the EC2/dista-only commands.

CI, keeping shard boundaries

  • test-conformance (in test-executor-core): toy, daily and fanout from the multicall binary, then tests/chaos.sh in its new CHAOS_BIN mode over Postgres and SQLite. In that mode every run, the reference one included, starts via conformance.py new-run + resume. About 2 minutes locally.
  • test-conformance-tables (in test-data-sensors): the sensor tests/sensors/clean.mojo (reads raw.events, commits to lake.clean).

Registry hashes are unchanged (toy 0xa3bcc571ad7c6939, daily 0x81ead348d0a21c1f, fanout 0x1f0ebf1f2a93d993). The manifests pin them.

Locally (macOS, ouromo/data envs) every target passes: 47 ok, 2 xfail, 1 skip each for toy, daily and fanout; 52 ok, 2 xfail, 8 skip for clean; and chaos in both stores. I also ran the suite against deliberately broken binaries: a wrapper that steals live claims fails claim.live, and a changed run token fails tables.retry_finds_commit.

🤖 Generated with Claude Code

https://claude.ai/code/session_017TycszSbyw92Efu4ZeCYL6

winding-lines and others added 2 commits November 16, 2026 21:14
tests/conformance.py drives a pipeline binary as morca's runners and
scheduler do (a run row as the scheduler starts it, then `resume`; never
`run --partitions`) and checks it against docs/pipeline-protocol.md, over
SQLite and Postgres, given a JSON manifest of what the pipeline is. Each
check cites the doc section it tests. register.compact (#524) and
describe.protocol (#525) are marked known-failing.

`pixi run conformance <binary> <manifest>`; test-conformance (in
test-executor-core) runs it on toy, daily and fanout and runs
tests/chaos.sh in its new CHAOS_BIN mode; test-conformance-tables (in
test-data-sensors) runs it on the sensor tests/sensors/clean.mojo.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017TycszSbyw92Efu4ZeCYL6
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017TycszSbyw92Efu4ZeCYL6
@winding-lines
winding-lines merged commit 851007f into main Nov 17, 2026
35 checks passed
@winding-lines
winding-lines deleted the conformance-519 branch November 17, 2026 05:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Conformance suite: test any pipeline binary against the protocol

1 participant