diff --git a/README.md b/README.md index 429db547..866ad1ef 100644 --- a/README.md +++ b/README.md @@ -54,6 +54,7 @@ them comes with a note in the changelog. Everything under `morca._internal` | [docs/morca-toml.md](docs/morca-toml.md) | `morca.toml`: every key, precedence, `morca config show` and `export`, and [fills](docs/morca-toml.md#fills) | | [docs/placement.md](docs/placement.md) | where a pipeline's runs run: `default`, `ec2`, `control` and `worker:`, and the control host's limits | | [docs/workers.md](docs/workers.md) | remote workers: `morca worker`, worker tokens, a worker on another machine, the run's state over the API | +| [docs/pipeline-protocol.md](docs/pipeline-protocol.md) | the pipeline-binary protocol: the subcommands, describe's JSON, the environment, the run store's rules and the tables' summary keys a pipeline binary must keep to | | [docs/rust-workers.md](docs/rust-workers.md) | steps written in Rust: the `morca-worker` crate, `morca.rust_step`, the C ABI, the schema file both sides read, and Parquet files a worker writes itself | | [docs/watchdog.md](docs/watchdog.md) | the watchdog's rules and thresholds, what a trip pauses, `morca watchdog status`, `ack` and `snooze` | | [docs/alerts.md](docs/alerts.md) | alerts to your phone through ntfy: what is sent, `MORCA_NOTIFY_*`, `morca notify` | diff --git a/docs/pipeline-protocol.md b/docs/pipeline-protocol.md new file mode 100644 index 00000000..c2718641 --- /dev/null +++ b/docs/pipeline-protocol.md @@ -0,0 +1,983 @@ +# The pipeline-binary protocol + +The prebuilt `morca` binary (the scheduler, the UI, `morca worker` and the +runners) never links a pipeline's code. It starts a **pipeline binary** and +deals with it only through four things: the binary's command line, its +environment, the run store (Postgres, SQLite, or the UI's worker API) and the +Iceberg tables. This page writes that contract down, so that a second +implementation, such as a pipeline binary written in Rust (#521), can be +built from it. The conformance suite (#519) tests against it. + +Today the Mojo side of the contract is `morca_main` (`src/morca/_cli.mojo`), +`Pipeline` (`src/morca/pipeline.mojo`), `Ctx` (`src/morca/ctx.mojo`) and the +`RunStore` implementations under `src/morca/_internal/`. Each rule below cites +the code it comes from as `path:line`, against morca-mojo 0.34.0. + +**Contract or incidental.** Each rule is marked: + +- **Contract**: the scheduler, the UI or another process relies on it. A + second implementation must do the same, and a change to it changes this + page and the conformance suite in the same PR. +- **Incidental**: what the Mojo binary happens to do today. Nothing outside + the binary depends on it, so a second implementation may differ. + +If a rule isn't marked, it is contract. + +## Contents + +- [A run's life](#a-runs-life) +- [Subcommands](#subcommands), and [which placement needs which](#which-placement-needs-which) +- [`describe` and the registry hash](#describe-and-the-registry-hash) +- [Environment](#environment) +- [The run store](#the-run-store) +- [Tables](#tables) +- [Step inputs and outputs](#step-inputs-and-outputs) +- [Logs](#logs) +- [Versions](#versions) +- [Exit codes](#exit-codes) +- [A first implementation](#a-first-implementation) + +## A run's life + +1. **Registration.** `morca register ` runs ` describe` and + stores the JSON it prints as the pipeline's registration + (`src/morca/_internal/registering.mojo:69-71`). From then on the scheduler + reads the DAG, partitions, schedule and retries from that stored JSON + (`src/morca/_internal/db/registered.mojo:76-100`). It doesn't run the + binary again to find them out. +2. **Queueing.** The scheduler decides which partition keys a run covers (from + the schedule, a sensor, a fill or a user) and inserts the run row itself, + with the registered `registry_hash` and the keys comma-joined in + `runs.partitions` (`src/morca/_internal/db/starting.mojo:253-270`). +3. **Starting.** It marks the run `running` with `owner = 'scheduler'` and no + heartbeat, counts a start (`src/morca/_internal/db/starting.mojo:297-310`), + and launches **` resume `** (`src/morca/_internal/starting.mojo:479-481`). + Every placement starts a run this way. Nothing outside the binary ever + runs `run --partitions`. +4. **Claiming.** The binary claims the run in the store, which makes it the + run's owner, and keeps a heartbeat going while it runs. +5. **Steps.** For each step and partition that hasn't succeeded yet, it + records an attempt, runs the step, and records the result. Reads and + commits to tables are recorded as they happen, and every commit carries + the `orca.*` summary keys. +6. **Ending.** It records the run's outcome (`succeeded`, `failed`, + `deferred` or `cancelled`), clears its heartbeat and exits. + +The scheduler learns how a run ended from the run row, not from the process. +The exit code and the output matter only when the process never claimed +the run, or crashed (see [Exit codes](#exit-codes)). + +## Subcommands + +`morca_main` parses `argv[1]` (`src/morca/_cli.mojo:148-323`). An unknown +command, or a known one with the wrong number of arguments, prints the usage +and exits 2 (`src/morca/_cli.mojo:106-110`, `:322-323`). + +Two checks run before any command except `sample` (**incidental**, but they +keep a broken binary from being registered): + +- `MORCA_DATABASE_URL` and `MORCA_STATE_URL` both set is a usage error, exit + 2 (`src/morca/_cli.mojo:156-157`). +- A scheduled pipeline parses its cron expression and loads its zone, so a + bad schedule fails every command (`src/morca/_cli.mojo:160-161`). + +### Which placement needs which + +A placement says where a pipeline's runs start ([placement.md](placement.md)). +A first implementation can leave out every subcommand its placements don't +need. + +| Subcommand | Called by | `default` (local) | `control` | `ec2` | `worker:` | kubernetes runner | +|---|---|---|---|---|---|---| +| `describe` | `morca register`, `morca locate` | yes | yes | yes | yes | yes | +| `resume [--allow-new-code]` | the runner, to start every run | yes | yes | yes | yes | yes | +| `sense` | the scheduler, for a sensor pipeline | sensor only | sensor only | sensor only | sensor only | sensor only | +| `keys` | the scheduler and UI, for a keyed pipeline | keyed only | keyed only | keyed only | keyed only | keyed only | +| `sample --out ... --pid ...` | the runner, beside each run | optional | optional | optional | optional | optional | +| `keep-log ` | the runner, after the run's process ends | with secrets | yes | yes | yes | no | +| `put-log ` | the EC2 instance script | no | no | yes | no | no | +| `pool-wait`, `pool-next` | the EC2 warm pool | no | no | pools only | no | no | +| `dista-status`, `dista-drain` | the scheduler, scaling an EC2 run | no | no | with dista tasks | no | no | +| no arguments, `DISTA_ROLE` set | the runner, for a dista head or worker | with dista tasks | with dista tasks | with dista tasks | with dista tasks | with dista tasks | +| `run`, `status`, `fires` | people and tests | no | no | no | no | no | + +"Optional" means a runner that can't start `sample` only logs it +(`src/morca/_internal/starting.mojo:512-527`), and the run's Machines card +stays empty. "With secrets" means the local runner hands the driver log to +`keep-log` only for a run with secrets, so they're masked; without secrets +it keeps the log itself (`src/morca/_internal/tick.mojo:91-100`). + +### `describe` + +` describe` prints the DAG as one line of JSON and exits 0 +(`src/morca/_cli.mojo:164-166`). See [`describe` and the registry +hash](#describe-and-the-registry-hash) for its fields. + +- The output must **start with the bytes `{"pipeline": `** (with that one + space), or `morca register` refuses the binary + (`src/morca/_internal/registering.mojo:70`). A JSON serialiser's compact + output (`{"pipeline":"x"`) is refused. +- `describe` is killed after `MORCA_BINARY_SECONDS` (default 300) + (`src/morca/_internal/proc.mojo:202-208`). It needs no database and no + warehouse. + +### `resume [--allow-new-code]` + +Continues a run at whatever isn't done (`src/morca/_cli.mojo:285-310`, +`:373-424`). The runner starts every run this way. `--allow-new-code` is +passed when the run row has `allow_new_code` set +(`src/morca/_internal/starting.mojo:479-481`). + +It needs `MORCA_DATABASE_URL` or `MORCA_STATE_URL`; with neither it's a usage +error, exit 2 (`src/morca/_cli.mojo:286-287`). In order, it: + +1. migrates the schema (`store.migrate()`, `src/morca/_cli.mojo:379`; see + [Migrations](#migrations)); +2. reads the run (`run_info`) and refuses, exit 3, a run of another pipeline + (`src/morca/_cli.mojo:380-389`) or of another registry hash unless + `--allow-new-code` was given (`:390-400`); +3. **claims** the run as `process_owner()`, with `MORCA_STALE_SECONDS` as the + stale limit (`:401-402`); +4. records the version it ran (`version`, then `ran_version`, `:403-405`); +5. **interrupts** what a previous process left running (`interrupt`, `:406`); +6. loads the steps that already succeeded, with their outputs (`prior`, `:407`); +7. prints `resuming run with steps done and interrupted` + (`:408-416`) (**incidental**); +8. parses `runs.partitions` back into keys and runs what's left + (`_execute`, `:417-424`; see [Executing](#executing)). + +### `run --partitions [--allow-new-code]` + +Starts a new run and runs it (`src/morca/_cli.mojo:250-284`, `:334-370`). +No runner uses it, so for a second implementation it is a convenience for +people and tests, not contract. + +- `--partitions` takes `KEY[..KEY][,...]` for a calendar pipeline, or names, + `@all` or `@file:` for a keyed one (`src/morca/_cli.mojo:94-103`). + A key that doesn't parse is a usage error, exit 2 (`:265-270`). +- With `MORCA_STATE_URL` it's refused, exit 2: a worker only resumes the run + it was given (`:271-275`). +- Without `MORCA_DATABASE_URL` the run is recorded in memory only + (`MemoryStore`, `:277-279`). Its run numbers are 0 or negative, so its + commits are never recognised as a retry's (`Ctx.made`, + `src/morca/ctx.mojo:256-257`). +- If the pipeline isn't registered with a binary yet, `run` registers it + (`register`, with no binary path). If it is, and the registered hash + isn't this binary's, it exits 2 unless given `--allow-new-code` + (`src/morca/_cli.mojo:344-360`). +- It creates the run with trigger `manual`, prints `run `, and runs it + (`:365-370`). Nothing parses the `run ` line (**incidental**). + +### `status [ | --before ] [--status [,...]]` + +Lists the latest 20 runs, or one run's steps +(`src/morca/_cli.mojo:311-321`, `:427-470`). For people only +(**incidental**). + +### `sense` + +For a sensor pipeline (`"sensor": true` in describe). The scheduler runs it +every `MORCA_SENSE_SECONDS` (default 60) +(`src/morca/_internal/sensing.mojo:58-67`), and `morca register` runs it once +for a sensor (`src/morca/_internal/registering.mojo:144-156`). + +It prints one line per **sensed dataset**: each dataset a step reads and no +step writes, once, in step order, then a keyed pipeline's keys dataset +(`src/morca/pipeline.mojo:305-325`). Each line is + +```text + +``` + +(`src/morca/_cli.mojo:232-249`): + +- `snapshot` is the table's current snapshot ID in `MORCA_WAREHOUSE`, or `-1` + when there's no table or no snapshot; +- `orca` means the current snapshot's summary has an `orca.run` key, and + `other` means it doesn't; +- `partitions` is that summary's `orca.partitions`, or `-` when it's empty + or missing. + +The scheduler splits each non-blank line on whitespace and needs exactly +four fields, or the whole poll fails (`src/morca/_internal/sensors.mojo:179-196`). +It acts only on lines with `snapshot >= 0` and `other` +(`src/morca/_internal/sensing.mojo:62-66`). For a dataset that morca's own +steps wrote, it uses `orca.commits` instead. A nonzero exit is recorded as +the pipeline's sense error, with the output +(`src/morca/_internal/sensing.mojo:283-296`). + +`sense` with sensed datasets and no `MORCA_WAREHOUSE` raises, exit 3 +(`src/morca/_cli.mojo:235-236`). How the current snapshot is found is under +[The warehouse](#the-warehouse). + +### `keys` + +For a keyed pipeline (`"partitions": {"kind": "keyed", ...}`). It prints the +keys dataset's snapshot ID on the first line, then one key per line +(`src/morca/_cli.mojo:222-231`). On a pipeline that isn't keyed it's a usage +error, exit 2. + +The scheduler parses the first line as an integer and takes every later +non-empty line as a key. Empty output is an error +(`src/morca/_internal/queueing.mojo:78-89`). The keys are stored in +`orca.keys` (`:98-113`). It runs from the sensor loop +(`src/morca/_internal/sensing.mojo:315-319`) and for backfills and the UI +(`src/morca/_internal/queueing.mojo:197`), killed after `MORCA_BINARY_SECONDS`. + +### `fires [] [--after