Repository navigation
Conformance suite: any pipeline binary against the protocol (#519) - #528
Merged
Merged
Conversation
tests/conformance.py drives a pipeline binary as morca's runners and scheduler do (a run row as the scheduler starts it, then `resume`; never `run --partitions`) and checks it against docs/pipeline-protocol.md, over SQLite and Postgres, given a JSON manifest of what the pipeline is. Each check cites the doc section it tests. register.compact (#524) and describe.protocol (#525) are marked known-failing. `pixi run conformance <binary> <manifest>`; test-conformance (in test-executor-core) runs it on toy, daily and fanout and runs tests/chaos.sh in its new CHAOS_BIN mode; test-conformance-tables (in test-data-sensors) runs it on the sensor tests/sensors/clean.mojo. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017TycszSbyw92Efu4ZeCYL6
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017TycszSbyw92Efu4ZeCYL6
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #519.
What
tests/conformance.pychecks any pipeline binary against docs/pipeline-protocol.md. It takes the binary's path and a JSON manifest that says what the pipeline is (steps, partitions, keys, the datasets each step reads and writes, plus optional env that slows a step and fixture commands for tables). It drives the binary only the way runners do: it inserts a run row as the scheduler starts one (running, ownerscheduler, no heartbeat) and runs<binary> resume <run>. It never runsrun --partitions. Over Postgres the pipeline is registered withmorca register. Over SQLite, where no scheduler runs, the suite registers it itself.Each check prints
ok,FAIL(with the rule and what was wrong),xfail,XPASSorskip, and cites its doc section asdocs/pipeline-protocol.md#<anchor>.Checks (pg and sqlite each for the run checks)
usage.unknown,describe.exit,describe.json(one line, one object, any spacing),describe.fields,describe.manifest,describe.hash,describe.morca,describe.protocol,resume.no_store,keys.not_keyed,keep-log.append,put-logsense.lines,sense.no_warehouseregister,register.compactresume.succeeds,resume.run_row,resume.steps,resume.attempts,resume.order,resume.logs,claim.done,claim.statuses(queued/failed/deferred),claim.live,claim.stale(incl. interrupt),resume.other_pipeline,resume.other_code(+--allow-new-code),keep-log.crash,statusesslow_env:kill.resume(heartbeat stays fresh during a step longer than the stale limit, kill -9, resume skips what succeeded and retries what was interrupted),cancel.between,cancel.due(Postgres only; the doc says the SQLite beat handles no cancels),lost.stopstables.commits(orca.* summary keys incl. run token),tables.reads,tables.crash_resume,tables.retry_finds_commit,tables.pinned(a crash after commit is simulated in the rows, a new input snapshot is committed, then resume)Known-failing
register.compact:morca registerrefuses describe printed compactly (the{"pipeline":prefix). Until register: parse describe's JSON instead of matching its spacing #524 lands. The suite itself accepts any spacing.describe.protocol: describe names no protocol version. Until A pipeline-protocol version, separate from MORCA_VERSION #525 lands.Not checked: the outdated flag (what a step does with it is up to the step), timeouts (no example step declares one), HTTP store and the EC2/dista-only commands.
CI, keeping shard boundaries
test-conformance(intest-executor-core): toy, daily and fanout from the multicall binary, thentests/chaos.shin its newCHAOS_BINmode over Postgres and SQLite. In that mode every run, the reference one included, starts viaconformance.py new-run+resume. About 2 minutes locally.test-conformance-tables(intest-data-sensors): the sensortests/sensors/clean.mojo(reads raw.events, commits to lake.clean).Registry hashes are unchanged (toy 0xa3bcc571ad7c6939, daily 0x81ead348d0a21c1f, fanout 0x1f0ebf1f2a93d993). The manifests pin them.
Locally (macOS, ouromo/data envs) every target passes: 47 ok, 2 xfail, 1 skip each for toy, daily and fanout; 52 ok, 2 xfail, 8 skip for clean; and chaos in both stores. I also ran the suite against deliberately broken binaries: a wrapper that steals live claims fails
claim.live, and a changed run token failstables.retry_finds_commit.🤖 Generated with Claude Code
https://claude.ai/code/session_017TycszSbyw92Efu4ZeCYL6