From 5dbd5fb05fbd0c67fdb4c97aef485e6bab36a343 Mon Sep 17 00:00:00 2001 From: Marius S <39998+winding-lines@users.noreply.github.com> Date: Mon, 16 Nov 2026 21:47:45 -0800 Subject: [PATCH 1/2] register: parse describe's JSON and check its protocol version (#524, #525) morca register parses describe's output with json_mlake instead of matching the `{"pipeline": ` prefix and searching for `"morca": "`, so compact JSON registers. describe now says `"protocol": 1`, which register checks; a describe with `protocol` and no `morca` registers, one with neither registers unchecked as before, and morca-mojo binaries keep `pipeline` first and `morca` last for older schedulers. Closes #524, closes #525. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_017TycszSbyw92Efu4ZeCYL6 --- changes/530-describe-protocol.minor.md | 1 + docs/binaries.md | 3 + docs/pipeline-protocol.md | 58 ++++++++++++------ src/morca/_internal/registering.mojo | 20 ++++-- src/morca/_internal/version.mojo | 84 +++++++++++++++++++++----- src/morca/pipeline.mojo | 10 +-- tests/conformance.py | 12 ++-- tests/versions.sh | 18 +++++- 8 files changed, 157 insertions(+), 49 deletions(-) create mode 100644 changes/530-describe-protocol.minor.md diff --git a/changes/530-describe-protocol.minor.md b/changes/530-describe-protocol.minor.md new file mode 100644 index 00000000..193ee6df --- /dev/null +++ b/changes/530-describe-protocol.minor.md @@ -0,0 +1 @@ +- **describe's protocol version**: `describe` says `"protocol": 1` and `morca register` checks it, parsing describe's JSON so compact JSON registers. (#530) diff --git a/docs/binaries.md b/docs/binaries.md index acc6ab1d..c0f6b2e7 100644 --- a/docs/binaries.md +++ b/docs/binaries.md @@ -77,6 +77,9 @@ The binary and the pipelines it runs come from one morca-mojo: be checked, and registers as before. Runs aren't checked: a pipeline registered under one morca keeps running after the scheduler moves to the next, as it always has (an older binary on a newer schema, AGENTS.md). +- **A pipeline not built with morca-mojo** (#521) says `"protocol"` and no + `"morca"`: register checks the protocol version it speaks instead + ([pipeline-protocol.md](pipeline-protocol.md#versions)). - **The project image** (`deploy/project.Dockerfile`) stops when the runtime's `morca version` isn't the one in the project's environment, and says which `--build-arg MORCA_VERSION` to build with. diff --git a/docs/pipeline-protocol.md b/docs/pipeline-protocol.md index a1b3857c..e9f60189 100644 --- a/docs/pipeline-protocol.md +++ b/docs/pipeline-protocol.md @@ -114,10 +114,14 @@ it keeps the log itself (`src/morca/_internal/tick.mojo:91-100`). (`src/morca/_cli.mojo:164-166`). See [`describe` and the registry hash](#describe-and-the-registry-hash) for its fields. -- The output must **start with the bytes `{"pipeline": `** (with that one - space), or `morca register` refuses the binary - (`src/morca/_internal/registering.mojo:70`). A JSON serialiser's compact - output (`{"pipeline":"x"`) is refused. +- The output must be **one JSON object with a non-empty string + `pipeline`**, or `morca register` refuses the binary + (`src/morca/_internal/version.mojo`, `parse_describe`). It's parsed, so + whitespace and key order don't matter: a JSON serialiser's compact output + (`{"pipeline":"x",...}`) registers (#524). morca before 0.35.0 required + the output to start with the bytes `{"pipeline": ` (one space); a binary + that must register with those writes that prefix + ([Versions](#versions)). - `describe` is killed after `MORCA_BINARY_SECONDS` (default 300) (`src/morca/_internal/proc.mojo:202-208`). It needs no database and no warehouse. @@ -314,12 +318,16 @@ Only for a pipeline whose describe lists `tasks`. ## `describe` and the registry hash -`describe` prints one JSON object (`src/morca/pipeline.mojo:215-290`). The -registration keeps it as `orca.pipelines.describe` (jsonb), and the scheduler -and UI read it there with SQL, so key order and whitespace don't matter to -them. They do matter to the two text checks: the prefix `{"pipeline": ` (see -[`describe`](#describe)) and the `"morca": ""` key, which is found by -searching the text from the end (`src/morca/_internal/version.mojo:19-30`). +`describe` prints one JSON object (`src/morca/pipeline.mojo:215-292`). The +registration parses it and keeps it as `orca.pipelines.describe` (jsonb), and +the scheduler and UI read it there with SQL, so key order and whitespace +don't matter. A field morca doesn't know is kept and ignored, so a new +optional field never breaks an older scheduler. + +morca before 0.35.0 checked two things in the text instead: the prefix +`{"pipeline": ` and the `"morca": ""` key, found by searching from +the end. morca-mojo binaries keep both (`pipeline` first, `morca` last), so +older schedulers still register them. An example, from a test's stand-in binary (`tests/fake_sensor:10`): @@ -332,7 +340,7 @@ An example, from a test's stand-in binary (`tests/fake_sensor:10`): "input": "morca.month.Month", "output": "Int", "reads": ["raw.a", "raw.b"], "writes": ["lake.ab"]}], "tasks": [], "source": {"repo": "", "commit": "", "dirty": false}, - "morca": "0.34.0"} + "protocol": 1, "morca": "0.34.0"} ``` | Field | Written at | Read by | Default when absent | @@ -352,7 +360,8 @@ An example, from a test's stand-in binary (`tests/fake_sensor:10`): | `steps[]` | `pipeline.mojo:252-279` | see below | required | | `tasks` | `pipeline.mojo:280-283` | dista task names; non-empty means the run gets a cluster (`db/registered.mojo:95`, `db/config_show.mojo:13`) | `[]` | | `source` | `pipeline.mojo:284-286` | `repo`, `commit`, `dirty`: the row of `orca.pipeline_versions` (`db/runs.mojo:50-86`), and whether a run is [outdated](#the-stop-notice-a-newer-version-leads) | empty | -| `morca` | `pipeline.mojo:288` | the morca it was built with, checked at registration ([Versions](#versions)) | not checked | +| `protocol` | `pipeline.mojo:289` | the version of this protocol the binary speaks, checked at registration ([Versions](#versions)) | not checked | +| `morca` | `pipeline.mojo:290` | the morca-mojo it was built with, checked at registration; only for binaries built with morca-mojo ([Versions](#versions)) | not checked | Paths in the table are under `src/morca/` and `src/morca/_internal/`. @@ -366,7 +375,7 @@ Each element of `steps`, in run order: | `function`, `input`, `output` | shown in the UI only (`ui/step_cards.mojo:266-273`) (**incidental** values: Mojo type and linkage names) | | `timeout_seconds` | only when set; shown in the UI. The binary enforces it ([Timeouts](#timeouts-and-a-cancel-that-ran-out)) | | `version` | only when set; the run's data version (`migrations.mojo:9-23`), so a fill redoes keys made at another version | -| `reads`, `writes` | dataset names `ns.table`. `writes` includes `replaces`. Used for sensing, lineage, cycles at registration and appending writes (`db/sensing.mojo:28-33`, `registering.mojo:78`, `db/queueing.mojo:231-237`) | +| `reads`, `writes` | dataset names `ns.table`. `writes` includes `replaces`. Used for sensing, lineage, cycles at registration and appending writes (`db/sensing.mojo:28-33`, `registering.mojo:90`, `db/queueing.mojo:231-237`) | | `replaces` | only when non-empty: datasets whose partitions a rerun replaces instead of appending to (`db/queueing.mojo:231-237`) | ### The registry hash @@ -916,14 +925,24 @@ driver log and metrics (`driver.log`, `metrics/`, `timeline.json`). ## Versions +- **The protocol.** describe's `"protocol": ` is the version of **this + protocol** the binary speaks; it is 1 (`PROTOCOL_VERSION`, + `src/morca/_internal/version.mojo`). `morca register` refuses a binary + whose `protocol` isn't an integer from 1 to the version it speaks, + whatever `MORCA_MIXED_VERSIONS` says, and registers one without the key + unchecked (#525). The version goes up only for a change an older binary + can't follow; a new optional field or subcommand doesn't raise it. - **`MORCA_VERSION`.** describe's last key, `"morca": ""`, is the - morca-mojo the binary was built with (`src/morca/pipeline.mojo:287-288`). + morca-mojo the binary was built with (`src/morca/pipeline.mojo:290`). `morca register` refuses a binary of another version unless `MORCA_MIXED_VERSIONS=1`, and registers one without the key unchecked - (`src/morca/_internal/version.mojo:19-48`, [binaries.md](binaries.md#versions)). - A second implementation should say which version of **this protocol** it - implements here, and morca should check that instead. Until then, it - must print the `morca` of the scheduler it targets. + (`src/morca/_internal/version.mojo`, [binaries.md](binaries.md#versions)). + A binary not built with morca-mojo says `protocol` and leaves `morca` out. +- **Older schedulers.** morca before 0.35.0 reads neither `protocol` nor + the JSON: it needs the bytes `{"pipeline": ` first, and checks the last + `"morca": ""` if there is one. A binary that must register with + such a scheduler prints describe that way, and the `morca` of that + scheduler. - **The registry hash** says which DAG a run ran ([above](#the-registry-hash)). - **The source.** describe's `source` (`repo`, `commit`, `dirty`) and `MORCA_IMAGE` make the run's version row, which the UI shows and the @@ -970,7 +989,8 @@ What the runners do with it: For `default` and `control` placement on Postgres or SQLite, with no dista tasks, a pipeline binary needs: -1. `describe`, with the fields above and the `{"pipeline": ` prefix. +1. `describe`: one JSON object with the fields above and `"protocol": 1` + (no `morca`, unless it's built with morca-mojo). 2. `resume [--allow-new-code]`: claim, version, interrupt, prior, execute with a heartbeat, ownership checks at every attempt, the commit fence, timeouts and cancels, and ending the run. diff --git a/src/morca/_internal/registering.mojo b/src/morca/_internal/registering.mojo index 83cc930d..ec6c7f53 100644 --- a/src/morca/_internal/registering.mojo +++ b/src/morca/_internal/registering.mojo @@ -31,7 +31,13 @@ from morca._internal.proc import ( capture, ) from morca._internal.secrets import secrets_json -from morca._internal.version import MISMATCH_HINT, mismatch, mixed_allowed +from morca._internal.version import ( + MISMATCH_HINT, + mismatch, + mixed_allowed, + parse_describe, + protocol_refusal, +) from morca._internal.sensors import parse_sense from morca._internal.scheduler import Scheduler from morca._internal.registered import ( @@ -67,10 +73,16 @@ def register_pipeline( Returns the pipeline's name.""" var path = absolute(binary) var got = capture([path, "describe"], binary_seconds()) - if got[0] != 0 or not got[1].startswith('{"pipeline": '): + if got[0] != 0: raise Error("morca: ", path, " describe exited ", got[0], ": ", got[1]) - # The pipeline and this morca from one release (#455). - var other = mismatch(path, got[1]) + # Parsed, so spacing and key order don't matter (#524). + var d = parse_describe(path, got[1]) + # A protocol this morca speaks (#525), and for a morca-mojo binary, + # the pipeline and this morca from one release (#455). + var refusal = protocol_refusal(path, d) + if refusal: + raise Error("morca: ", refusal) + var other = mismatch(path, d) if other and not mixed_allowed(): raise Error("morca: ", other, ": ", MISMATCH_HINT) if other: diff --git a/src/morca/_internal/version.mojo b/src/morca/_internal/version.mojo index 6a8380b4..cf65ab57 100644 --- a/src/morca/_internal/version.mojo +++ b/src/morca/_internal/version.mojo @@ -8,32 +8,86 @@ differ, so a release bumps it with them. `morca version` prints it, each pipeline's `describe` says it (`"morca"`), and `morca register` refuses a pipeline built from another one (`mismatch`), unless `MORCA_MIXED_VERSIONS=1`, when it only warns. + +`PROTOCOL_VERSION` is the version of the pipeline-binary protocol +(docs/pipeline-protocol.md) this morca speaks, which describe says as +`"protocol"` (#525). A binary not built with morca-mojo (#521) says that +and no `"morca"`; `morca register` refuses a protocol it doesn't speak. """ from std.os import getenv +from json import Value, loads comptime MORCA_VERSION = "0.34.0" """morca-mojo's version: pixi.toml's `[package] version`.""" +comptime PROTOCOL_VERSION = 1 +"""The pipeline-binary protocol's version (docs/pipeline-protocol.md): +describe's `"protocol"`. Raised only when a change to the protocol would +break a binary written to the one before.""" -def built_with(describe: String) -> String: - """The morca version a pipeline's `describe` (JSON) names, or empty for - a binary from before describe said it (morca-mojo 0.32.0 and older).""" - comptime key: StaticString = '"morca": "' - var at = describe.rfind(key) - if at < 0: - return "" - var start = at + key.byte_length() - var end = describe.find('"', start) - if end < 0: + +def parse_describe(binary: String, describe: String) raises -> Value: + """`describe`'s JSON, checked to be an object with a non-empty string + `pipeline`; whitespace and key order don't matter (#524).""" + var d: Value + try: + d = loads(describe) + except e: + raise Error( + "morca: ", binary, " describe didn't print JSON (", e, "): ", describe + ) + if ( + not d.is_object() + or "pipeline" not in d + or not d["pipeline"].is_string() + or not d["pipeline"].string_value() + ): + raise Error( + "morca: ", + binary, + " describe printed no object with a `pipeline` name: ", + describe, + ) + return d^ + + +def built_with(d: Value) raises -> String: + """The morca version a pipeline's parsed `describe` names, or empty for + a binary from before describe said it (morca-mojo 0.32.0 and older) or + one not built with morca-mojo.""" + if "morca" in d and d["morca"].is_string(): + return d["morca"].string_value() + return "" + + +def protocol_refusal(binary: String, d: Value) raises -> String: + """Why this morca can't register the pipeline `binary`, whose parsed + `describe` this is, for the protocol it says it speaks, or empty when + it speaks this one or doesn't say (#525).""" + if "protocol" not in d: return "" - return String(describe[byte=start:end]) + var p = d["protocol"].copy() + if not p.is_int() or p.int_value() < 1: + return String( + binary, "'s describe has a protocol that isn't a version number" + ) + if p.int_value() > PROTOCOL_VERSION: + return String( + binary, + " speaks pipeline protocol ", + p.int_value(), + " and this morca speaks ", + PROTOCOL_VERSION, + ": upgrade morca", + ) + return "" -def mismatch(binary: String, describe: String) -> String: - """Why the pipeline `binary`, whose `describe` this is, wasn't built from - this morca, or empty when it was (or doesn't say).""" - var theirs = built_with(describe) +def mismatch(binary: String, d: Value) raises -> String: + """Why the pipeline `binary`, whose parsed `describe` this is, wasn't + built from this morca, or empty when it was (or doesn't say).""" + var theirs = built_with(d) if not theirs or theirs == MORCA_VERSION: return "" return String( diff --git a/src/morca/pipeline.mojo b/src/morca/pipeline.mojo index cc97077d..65abe3a8 100644 --- a/src/morca/pipeline.mojo +++ b/src/morca/pipeline.mojo @@ -38,7 +38,7 @@ from morca.civil import Civil, now from morca.partitions import Monthly, Partitions from morca.schedule import Fill, NoSchedule, Previous, Schedule, ScheduleLike from morca._internal.source import source_commit, source_dirty, source_repo -from morca._internal.version import MORCA_VERSION +from morca._internal.version import MORCA_VERSION, PROTOCOL_VERSION from morca.step import Gather, Mapped, StepLike, deferred_for, deferred_why from morca._internal.store import ( DEFERRED, @@ -216,8 +216,8 @@ struct Pipeline[ def describe() -> String: """The DAG as JSON: the pipeline's name, its registry hash, its partitions, each step's name, kind, upstream, function, types and - datasets in run order, its dista tasks, its source and the morca - version it was built with.""" + datasets in run order, its dista tasks, its source, the pipeline + protocol it speaks and the morca version it was built with.""" Self.check() var out = String("{") out.write('"pipeline": ', quote(Self.name), ", ") @@ -284,7 +284,9 @@ struct Pipeline[ out.write('], "source": {"repo": ', quote(source_repo())) out.write(', "commit": ', quote(source_commit())) out.write(', "dirty": ', "true" if source_dirty() else "false", "}") - # Last, so `morca._internal.version.built_with` finds it from the end. + # Next to last, so `{"pipeline": ` stays the start and `"morca"` the + # last key, which a scheduler before #524 looks for at the end. + out.write(', "protocol": ', PROTOCOL_VERSION) out.write(', "morca": ', quote(MORCA_VERSION), "}") return out^ diff --git a/tests/conformance.py b/tests/conformance.py index ba9e3990..d390b6f3 100755 --- a/tests/conformance.py +++ b/tests/conformance.py @@ -56,11 +56,8 @@ DOC = "docs/pipeline-protocol.md" -# Checks known to fail until an issue lands: id -> issue. -KNOWN = { - "register.compact": "#524", - "describe.protocol": "#525", -} +# Checks known to fail until an issue lands: id -> issue. None now. +KNOWN: dict[str, str] = {} STALE = 2.0 """MORCA_STALE_SECONDS for every command: the suite is the scheduler.""" @@ -541,6 +538,9 @@ def morca_version(): def protocol(): need("protocol" in d, "describe names no protocol version") + p = d["protocol"] + need(isinstance(p, int) and not isinstance(p, bool) and p >= 1, + f"`protocol` is {p!r}, not a version number") self.check("describe.protocol", "versions", "describe says which version of the protocol it implements", @@ -1218,7 +1218,7 @@ def describe_fields(d): f"`partitions` is {parts}") for k, t in (("max_runs", int), ("run_retries", int), ("keys_per_run", int), ("workers", int), ("sensor", bool), ("tasks", list), - ("source", dict), ("morca", str)): + ("source", dict), ("protocol", int), ("morca", str)): if k in d: need(isinstance(d[k], t) and not (t is int and isinstance(d[k], bool)), f"`{k}` is {d[k]!r}") diff --git a/tests/versions.sh b/tests/versions.sh index d9331753..f1232433 100755 --- a/tests/versions.sh +++ b/tests/versions.sh @@ -205,4 +205,20 @@ echo "$out" | grep -qF "toy-old was built with morca 0.0.1 and this morca is $wa [ "$(q "SELECT count(*) FROM orca.pipelines WHERE name = 'toy_old'")" = 0 ] || fail "the refused pipeline is registered" out=$(MORCA_MIXED_VERSIONS=1 ./build/morca register ./build/toy-old 2>&1) || fail "MORCA_MIXED_VERSIONS=1 register: $out" echo "$out" | grep -qF "morca: warning: $PWD/build/toy-old was built with morca 0.0.1" || fail "no warning: $out" -echo "ok versions: a run links the commit it ran, the pipeline lists its versions, register checks the morca" +# The protocol version (#525): a binary not built with morca-mojo says +# `protocol` and no `morca`, in compact JSON (#524), and registers; one of +# a protocol this morca doesn't speak is refused, MORCA_MIXED_VERSIONS or not. +./build/toy describe | jq -e '.protocol == 1' > /dev/null || fail "describe's protocol: $(./build/toy describe | jq -c .protocol)" +./build/toy describe | jq -c '.pipeline = "toy_sdk" | del(.morca)' > build/toy-sdk.json +printf '#!/bin/sh\ncat "%s"\n' "$PWD/build/toy-sdk.json" > build/toy-sdk +chmod +x build/toy-sdk +out=$(./build/morca register ./build/toy-sdk 2>&1) || fail "register a protocol-only describe: $out" +echo "$out" | grep -qF warning && fail "a protocol-only describe warned: $out" +[ "$(q "SELECT count(*) FROM orca.pipelines WHERE name = 'toy_sdk'")" = 1 ] || fail "the protocol-only pipeline isn't registered" +jq -c '.pipeline = "toy_next" | .protocol = 2' build/toy-sdk.json > build/toy-next.json +printf '#!/bin/sh\ncat "%s"\n' "$PWD/build/toy-next.json" > build/toy-next +chmod +x build/toy-next +out=$(MORCA_MIXED_VERSIONS=1 ./build/morca register ./build/toy-next 2>&1) && fail "registered protocol 2: $out" +echo "$out" | grep -qF "toy-next speaks pipeline protocol 2 and this morca speaks 1" || fail "the protocol refusal: $out" +[ "$(q "SELECT count(*) FROM orca.pipelines WHERE name = 'toy_next'")" = 0 ] || fail "the protocol-2 pipeline is registered" +echo "ok versions: a run links the commit it ran, the pipeline lists its versions, register checks the morca and the protocol" From 9cc811420ef428784e0bcffad068686d7f7d1aeb Mon Sep 17 00:00:00 2001 From: Marius S <39998+winding-lines@users.noreply.github.com> Date: Mon, 16 Nov 2026 21:48:56 -0800 Subject: [PATCH 2/2] Format version.mojo Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_017TycszSbyw92Efu4ZeCYL6 --- src/morca/_internal/version.mojo | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/src/morca/_internal/version.mojo b/src/morca/_internal/version.mojo index cf65ab57..d61bf21c 100644 --- a/src/morca/_internal/version.mojo +++ b/src/morca/_internal/version.mojo @@ -35,7 +35,12 @@ def parse_describe(binary: String, describe: String) raises -> Value: d = loads(describe) except e: raise Error( - "morca: ", binary, " describe didn't print JSON (", e, "): ", describe + "morca: ", + binary, + " describe didn't print JSON (", + e, + "): ", + describe, ) if ( not d.is_object()