From 8a2f6f98e02eeb001e8fcd8a5b5fdf0286497c41 Mon Sep 17 00:00:00 2001 From: Marius S <39998+winding-lines@users.noreply.github.com> Date: Mon, 16 Nov 2026 20:50:21 -0800 Subject: [PATCH] docs: the pipeline-binary protocol (#518) docs/pipeline-protocol.md writes down what the scheduler, the runners and the UI need from a pipeline binary: the subcommands and which placement needs each, describe's JSON and the registry hash, the environment, the run store's rows and rules (claim, heartbeat, ownership, fence, interrupt, cancel, timeouts, the outdated notice), the tables' summary keys, find_commit and pinned reads, the step output encoding, versions and exit codes. Each rule cites the code and says whether it is contract or incidental. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_017TycszSbyw92Efu4ZeCYL6 --- README.md | 1 + docs/pipeline-protocol.md | 983 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 984 insertions(+) create mode 100644 docs/pipeline-protocol.md diff --git a/README.md b/README.md index 429db547..866ad1ef 100644 --- a/README.md +++ b/README.md @@ -54,6 +54,7 @@ them comes with a note in the changelog. Everything under `morca._internal` | [docs/morca-toml.md](docs/morca-toml.md) | `morca.toml`: every key, precedence, `morca config show` and `export`, and [fills](docs/morca-toml.md#fills) | | [docs/placement.md](docs/placement.md) | where a pipeline's runs run: `default`, `ec2`, `control` and `worker:`, and the control host's limits | | [docs/workers.md](docs/workers.md) | remote workers: `morca worker`, worker tokens, a worker on another machine, the run's state over the API | +| [docs/pipeline-protocol.md](docs/pipeline-protocol.md) | the pipeline-binary protocol: the subcommands, describe's JSON, the environment, the run store's rules and the tables' summary keys a pipeline binary must keep to | | [docs/rust-workers.md](docs/rust-workers.md) | steps written in Rust: the `morca-worker` crate, `morca.rust_step`, the C ABI, the schema file both sides read, and Parquet files a worker writes itself | | [docs/watchdog.md](docs/watchdog.md) | the watchdog's rules and thresholds, what a trip pauses, `morca watchdog status`, `ack` and `snooze` | | [docs/alerts.md](docs/alerts.md) | alerts to your phone through ntfy: what is sent, `MORCA_NOTIFY_*`, `morca notify` | diff --git a/docs/pipeline-protocol.md b/docs/pipeline-protocol.md new file mode 100644 index 00000000..c2718641 --- /dev/null +++ b/docs/pipeline-protocol.md @@ -0,0 +1,983 @@ +# The pipeline-binary protocol + +The prebuilt `morca` binary (the scheduler, the UI, `morca worker` and the +runners) never links a pipeline's code. It starts a **pipeline binary** and +deals with it only through four things: the binary's command line, its +environment, the run store (Postgres, SQLite, or the UI's worker API) and the +Iceberg tables. This page writes that contract down, so that a second +implementation, such as a pipeline binary written in Rust (#521), can be +built from it. The conformance suite (#519) tests against it. + +Today the Mojo side of the contract is `morca_main` (`src/morca/_cli.mojo`), +`Pipeline` (`src/morca/pipeline.mojo`), `Ctx` (`src/morca/ctx.mojo`) and the +`RunStore` implementations under `src/morca/_internal/`. Each rule below cites +the code it comes from as `path:line`, against morca-mojo 0.34.0. + +**Contract or incidental.** Each rule is marked: + +- **Contract**: the scheduler, the UI or another process relies on it. A + second implementation must do the same, and a change to it changes this + page and the conformance suite in the same PR. +- **Incidental**: what the Mojo binary happens to do today. Nothing outside + the binary depends on it, so a second implementation may differ. + +If a rule isn't marked, it is contract. + +## Contents + +- [A run's life](#a-runs-life) +- [Subcommands](#subcommands), and [which placement needs which](#which-placement-needs-which) +- [`describe` and the registry hash](#describe-and-the-registry-hash) +- [Environment](#environment) +- [The run store](#the-run-store) +- [Tables](#tables) +- [Step inputs and outputs](#step-inputs-and-outputs) +- [Logs](#logs) +- [Versions](#versions) +- [Exit codes](#exit-codes) +- [A first implementation](#a-first-implementation) + +## A run's life + +1. **Registration.** `morca register ` runs ` describe` and + stores the JSON it prints as the pipeline's registration + (`src/morca/_internal/registering.mojo:69-71`). From then on the scheduler + reads the DAG, partitions, schedule and retries from that stored JSON + (`src/morca/_internal/db/registered.mojo:76-100`). It doesn't run the + binary again to find them out. +2. **Queueing.** The scheduler decides which partition keys a run covers (from + the schedule, a sensor, a fill or a user) and inserts the run row itself, + with the registered `registry_hash` and the keys comma-joined in + `runs.partitions` (`src/morca/_internal/db/starting.mojo:253-270`). +3. **Starting.** It marks the run `running` with `owner = 'scheduler'` and no + heartbeat, counts a start (`src/morca/_internal/db/starting.mojo:297-310`), + and launches **` resume `** (`src/morca/_internal/starting.mojo:479-481`). + Every placement starts a run this way. Nothing outside the binary ever + runs `run --partitions`. +4. **Claiming.** The binary claims the run in the store, which makes it the + run's owner, and keeps a heartbeat going while it runs. +5. **Steps.** For each step and partition that hasn't succeeded yet, it + records an attempt, runs the step, and records the result. Reads and + commits to tables are recorded as they happen, and every commit carries + the `orca.*` summary keys. +6. **Ending.** It records the run's outcome (`succeeded`, `failed`, + `deferred` or `cancelled`), clears its heartbeat and exits. + +The scheduler learns how a run ended from the run row, not from the process. +The exit code and the output matter only when the process never claimed +the run, or crashed (see [Exit codes](#exit-codes)). + +## Subcommands + +`morca_main` parses `argv[1]` (`src/morca/_cli.mojo:148-323`). An unknown +command, or a known one with the wrong number of arguments, prints the usage +and exits 2 (`src/morca/_cli.mojo:106-110`, `:322-323`). + +Two checks run before any command except `sample` (**incidental**, but they +keep a broken binary from being registered): + +- `MORCA_DATABASE_URL` and `MORCA_STATE_URL` both set is a usage error, exit + 2 (`src/morca/_cli.mojo:156-157`). +- A scheduled pipeline parses its cron expression and loads its zone, so a + bad schedule fails every command (`src/morca/_cli.mojo:160-161`). + +### Which placement needs which + +A placement says where a pipeline's runs start ([placement.md](placement.md)). +A first implementation can leave out every subcommand its placements don't +need. + +| Subcommand | Called by | `default` (local) | `control` | `ec2` | `worker:` | kubernetes runner | +|---|---|---|---|---|---|---| +| `describe` | `morca register`, `morca locate` | yes | yes | yes | yes | yes | +| `resume [--allow-new-code]` | the runner, to start every run | yes | yes | yes | yes | yes | +| `sense` | the scheduler, for a sensor pipeline | sensor only | sensor only | sensor only | sensor only | sensor only | +| `keys` | the scheduler and UI, for a keyed pipeline | keyed only | keyed only | keyed only | keyed only | keyed only | +| `sample --out ... --pid ...` | the runner, beside each run | optional | optional | optional | optional | optional | +| `keep-log ` | the runner, after the run's process ends | with secrets | yes | yes | yes | no | +| `put-log ` | the EC2 instance script | no | no | yes | no | no | +| `pool-wait`, `pool-next` | the EC2 warm pool | no | no | pools only | no | no | +| `dista-status`, `dista-drain` | the scheduler, scaling an EC2 run | no | no | with dista tasks | no | no | +| no arguments, `DISTA_ROLE` set | the runner, for a dista head or worker | with dista tasks | with dista tasks | with dista tasks | with dista tasks | with dista tasks | +| `run`, `status`, `fires` | people and tests | no | no | no | no | no | + +"Optional" means a runner that can't start `sample` only logs it +(`src/morca/_internal/starting.mojo:512-527`), and the run's Machines card +stays empty. "With secrets" means the local runner hands the driver log to +`keep-log` only for a run with secrets, so they're masked; without secrets +it keeps the log itself (`src/morca/_internal/tick.mojo:91-100`). + +### `describe` + +` describe` prints the DAG as one line of JSON and exits 0 +(`src/morca/_cli.mojo:164-166`). See [`describe` and the registry +hash](#describe-and-the-registry-hash) for its fields. + +- The output must **start with the bytes `{"pipeline": `** (with that one + space), or `morca register` refuses the binary + (`src/morca/_internal/registering.mojo:70`). A JSON serialiser's compact + output (`{"pipeline":"x"`) is refused. +- `describe` is killed after `MORCA_BINARY_SECONDS` (default 300) + (`src/morca/_internal/proc.mojo:202-208`). It needs no database and no + warehouse. + +### `resume [--allow-new-code]` + +Continues a run at whatever isn't done (`src/morca/_cli.mojo:285-310`, +`:373-424`). The runner starts every run this way. `--allow-new-code` is +passed when the run row has `allow_new_code` set +(`src/morca/_internal/starting.mojo:479-481`). + +It needs `MORCA_DATABASE_URL` or `MORCA_STATE_URL`; with neither it's a usage +error, exit 2 (`src/morca/_cli.mojo:286-287`). In order, it: + +1. migrates the schema (`store.migrate()`, `src/morca/_cli.mojo:379`; see + [Migrations](#migrations)); +2. reads the run (`run_info`) and refuses, exit 3, a run of another pipeline + (`src/morca/_cli.mojo:380-389`) or of another registry hash unless + `--allow-new-code` was given (`:390-400`); +3. **claims** the run as `process_owner()`, with `MORCA_STALE_SECONDS` as the + stale limit (`:401-402`); +4. records the version it ran (`version`, then `ran_version`, `:403-405`); +5. **interrupts** what a previous process left running (`interrupt`, `:406`); +6. loads the steps that already succeeded, with their outputs (`prior`, `:407`); +7. prints `resuming run with steps done and interrupted` + (`:408-416`) (**incidental**); +8. parses `runs.partitions` back into keys and runs what's left + (`_execute`, `:417-424`; see [Executing](#executing)). + +### `run --partitions [--allow-new-code]` + +Starts a new run and runs it (`src/morca/_cli.mojo:250-284`, `:334-370`). +No runner uses it, so for a second implementation it is a convenience for +people and tests, not contract. + +- `--partitions` takes `KEY[..KEY][,...]` for a calendar pipeline, or names, + `@all` or `@file:` for a keyed one (`src/morca/_cli.mojo:94-103`). + A key that doesn't parse is a usage error, exit 2 (`:265-270`). +- With `MORCA_STATE_URL` it's refused, exit 2: a worker only resumes the run + it was given (`:271-275`). +- Without `MORCA_DATABASE_URL` the run is recorded in memory only + (`MemoryStore`, `:277-279`). Its run numbers are 0 or negative, so its + commits are never recognised as a retry's (`Ctx.made`, + `src/morca/ctx.mojo:256-257`). +- If the pipeline isn't registered with a binary yet, `run` registers it + (`register`, with no binary path). If it is, and the registered hash + isn't this binary's, it exits 2 unless given `--allow-new-code` + (`src/morca/_cli.mojo:344-360`). +- It creates the run with trigger `manual`, prints `run `, and runs it + (`:365-370`). Nothing parses the `run ` line (**incidental**). + +### `status [ | --before ] [--status [,...]]` + +Lists the latest 20 runs, or one run's steps +(`src/morca/_cli.mojo:311-321`, `:427-470`). For people only +(**incidental**). + +### `sense` + +For a sensor pipeline (`"sensor": true` in describe). The scheduler runs it +every `MORCA_SENSE_SECONDS` (default 60) +(`src/morca/_internal/sensing.mojo:58-67`), and `morca register` runs it once +for a sensor (`src/morca/_internal/registering.mojo:144-156`). + +It prints one line per **sensed dataset**: each dataset a step reads and no +step writes, once, in step order, then a keyed pipeline's keys dataset +(`src/morca/pipeline.mojo:305-325`). Each line is + +```text + +``` + +(`src/morca/_cli.mojo:232-249`): + +- `snapshot` is the table's current snapshot ID in `MORCA_WAREHOUSE`, or `-1` + when there's no table or no snapshot; +- `orca` means the current snapshot's summary has an `orca.run` key, and + `other` means it doesn't; +- `partitions` is that summary's `orca.partitions`, or `-` when it's empty + or missing. + +The scheduler splits each non-blank line on whitespace and needs exactly +four fields, or the whole poll fails (`src/morca/_internal/sensors.mojo:179-196`). +It acts only on lines with `snapshot >= 0` and `other` +(`src/morca/_internal/sensing.mojo:62-66`). For a dataset that morca's own +steps wrote, it uses `orca.commits` instead. A nonzero exit is recorded as +the pipeline's sense error, with the output +(`src/morca/_internal/sensing.mojo:283-296`). + +`sense` with sensed datasets and no `MORCA_WAREHOUSE` raises, exit 3 +(`src/morca/_cli.mojo:235-236`). How the current snapshot is found is under +[The warehouse](#the-warehouse). + +### `keys` + +For a keyed pipeline (`"partitions": {"kind": "keyed", ...}`). It prints the +keys dataset's snapshot ID on the first line, then one key per line +(`src/morca/_cli.mojo:222-231`). On a pipeline that isn't keyed it's a usage +error, exit 2. + +The scheduler parses the first line as an integer and takes every later +non-empty line as a key. Empty output is an error +(`src/morca/_internal/queueing.mojo:78-89`). The keys are stored in +`orca.keys` (`:98-113`). It runs from the sensor loop +(`src/morca/_internal/sensing.mojo:315-319`) and for backfills and the UI +(`src/morca/_internal/queueing.mojo:197`), killed after `MORCA_BINARY_SECONDS`. + +### `fires [] [--after