Skip to content

docs: the pipeline-binary protocol - #523

Merged
winding-lines merged 1 commit into
mainfrom
pipeline-protocol-518
Nov 17, 2026
Merged

winding-lines merged 1 commit into
mainfrom
pipeline-protocol-518

Conversation

@winding-lines

Copy link
Copy Markdown
Contributor

Closes #518.

This adds docs/pipeline-protocol.md, the contract between the prebuilt morca (scheduler, runners, UI) and a pipeline binary. It's the first step towards pure-Rust pipelines (#522). Every rule cites the code at morca-mojo 0.34.0 and is marked contract (something outside the binary relies on it) or incidental (what the Mojo binary happens to do).

It covers:

  • Subcommands, with a table of which placement needs which. A first implementation for default/control needs describe, resume, sense/keys for sensors and keyed pipelines, and keep-log.
  • describe's JSON: every field and its readers, with defaults. The registry hash is an opaque string the scheduler compares; how Mojo computes it is incidental.
  • Environment: each MORCA_* variable, who sets it on each runner, and what the binary does with it.
  • The run store: the rows, each RunStore method with its SQL, and the rules for claiming, interrupting, heartbeats, ownership checks, the commit fence, cancels, timeouts, the outdated notice, DB retries, migrations and the HTTP store.
  • Tables: the orca.* summary keys and their readers, find_commit's matching rule, recorded reads/commits and pinned reads.
  • Step outputs: the dista codec bytes. No morca reader decodes them; only the pipeline's own resumes and reruns read them back.
  • Partition key formats, logs, versions and exit codes.

The README's docs table links to it. Docs only, so it's labelled no-changelog.

Mismatches found while writing it (not fixed here)

  1. describe's prefix check depends on whitespace. morca register refuses output that doesn't start with exactly {"pipeline": (src/morca/_internal/registering.mojo:70). built_with searches for "morca": " (src/morca/_internal/version.mojo:22-23). A serde_json binary's compact output would fail both. The doc tells implementers to match the spacing. Parsing the JSON instead would be more robust; that's worth considering before morca-sdk: whole pipelines in Rust, with no Mojo toolchain #521.
  2. MORCA_VERSION is the only version check. A Rust binary would have to claim a morca-mojo version to register. A separate protocol version in describe would fix that. It's noted in the doc's Versions section as a suggestion, not a rule.
  3. The state.mojo docstring names the wrong schema. It says everything lives in the morca schema (src/morca/_internal/state.mojo:3), but every query uses orca. Its claim docstring (:193-197) also leaves out deferred, which the SQL accepts.
  4. The keep-log log line contradicts what the code does. When keep-log exits nonzero, the scheduler prints keep-log exited N keeping <stage> but then removes the stage file (src/morca/_internal/tick.mojo:51-56). The worker does keep it (worker.mojo:472-478).
  5. keep-log's crash handling never runs for local and control runs. On those runners MORCA_RUN isn't set, so keep-log can't fail a crashed run's attempts. The scheduler does that itself at reap time (scheduler.mojo:942-970), so nothing is lost. EC2 sets MORCA_RUN and relies on it.
  6. SQLite runs can't be stopped mid-step by a cancel. The SQLite heartbeat enforces timeouts but not cancel_at (sqlite_store.mojo:111-160). A cancel only takes effect between attempts.

🤖 Generated with Claude Code

https://claude.ai/code/session_017TycszSbyw92Efu4ZeCYL6

docs/pipeline-protocol.md writes down what the scheduler, the runners and
the UI need from a pipeline binary: the subcommands and which placement
needs each, describe's JSON and the registry hash, the environment, the
run store's rows and rules (claim, heartbeat, ownership, fence, interrupt,
cancel, timeouts, the outdated notice), the tables' summary keys,
find_commit and pinned reads, the step output encoding, versions and exit
codes. Each rule cites the code and says whether it is contract or
incidental.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017TycszSbyw92Efu4ZeCYL6
@winding-lines winding-lines added documentation Improvements or additions to documentation no-changelog Needs no changelog fragment (CI, docs or tests only) labels Nov 17, 2026
@winding-lines
winding-lines merged commit 7930a17 into main Nov 17, 2026
30 checks passed
@winding-lines
winding-lines deleted the pipeline-protocol-518 branch November 17, 2026 04:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation no-changelog Needs no changelog fragment (CI, docs or tests only)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Pipeline-binary protocol: document what the scheduler needs from a pipeline binary

1 participant