Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changes/530-describe-protocol.minor.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
- **describe's protocol version**: `describe` says `"protocol": 1` and `morca register` checks it, parsing describe's JSON so compact JSON registers. (#530)
3 changes: 3 additions & 0 deletions docs/binaries.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,9 @@ The binary and the pipelines it runs come from one morca-mojo:
be checked, and registers as before. Runs aren't checked: a pipeline
registered under one morca keeps running after the scheduler moves to the
next, as it always has (an older binary on a newer schema, AGENTS.md).
- **A pipeline not built with morca-mojo** (#521) says `"protocol"` and no
`"morca"`: register checks the protocol version it speaks instead
([pipeline-protocol.md](pipeline-protocol.md#versions)).
- **The project image** (`deploy/project.Dockerfile`) stops when the runtime's
`morca version` isn't the one in the project's environment, and says which
`--build-arg MORCA_VERSION` to build with.
Expand Down
58 changes: 39 additions & 19 deletions docs/pipeline-protocol.md
Original file line number Diff line number Diff line change
Expand Up @@ -114,10 +114,14 @@ it keeps the log itself (`src/morca/_internal/tick.mojo:91-100`).
(`src/morca/_cli.mojo:164-166`). See [`describe` and the registry
hash](#describe-and-the-registry-hash) for its fields.

- The output must **start with the bytes `{"pipeline": `** (with that one
space), or `morca register` refuses the binary
(`src/morca/_internal/registering.mojo:70`). A JSON serialiser's compact
output (`{"pipeline":"x"`) is refused.
- The output must be **one JSON object with a non-empty string
`pipeline`**, or `morca register` refuses the binary
(`src/morca/_internal/version.mojo`, `parse_describe`). It's parsed, so
whitespace and key order don't matter: a JSON serialiser's compact output
(`{"pipeline":"x",...}`) registers (#524). morca before 0.35.0 required
the output to start with the bytes `{"pipeline": ` (one space); a binary
that must register with those writes that prefix
([Versions](#versions)).
- `describe` is killed after `MORCA_BINARY_SECONDS` (default 300)
(`src/morca/_internal/proc.mojo:202-208`). It needs no database and no
warehouse.
Expand Down Expand Up @@ -314,12 +318,16 @@ Only for a pipeline whose describe lists `tasks`.

## `describe` and the registry hash

`describe` prints one JSON object (`src/morca/pipeline.mojo:215-290`). The
registration keeps it as `orca.pipelines.describe` (jsonb), and the scheduler
and UI read it there with SQL, so key order and whitespace don't matter to
them. They do matter to the two text checks: the prefix `{"pipeline": ` (see
[`describe`](#describe)) and the `"morca": "<version>"` key, which is found by
searching the text from the end (`src/morca/_internal/version.mojo:19-30`).
`describe` prints one JSON object (`src/morca/pipeline.mojo:215-292`). The
registration parses it and keeps it as `orca.pipelines.describe` (jsonb), and
the scheduler and UI read it there with SQL, so key order and whitespace
don't matter. A field morca doesn't know is kept and ignored, so a new
optional field never breaks an older scheduler.

morca before 0.35.0 checked two things in the text instead: the prefix
`{"pipeline": ` and the `"morca": "<version>"` key, found by searching from
the end. morca-mojo binaries keep both (`pipeline` first, `morca` last), so
older schedulers still register them.

An example, from a test's stand-in binary (`tests/fake_sensor:10`):

Expand All @@ -332,7 +340,7 @@ An example, from a test's stand-in binary (`tests/fake_sensor:10`):
"input": "morca.month.Month", "output": "Int",
"reads": ["raw.a", "raw.b"], "writes": ["lake.ab"]}],
"tasks": [], "source": {"repo": "", "commit": "", "dirty": false},
"morca": "0.34.0"}
"protocol": 1, "morca": "0.34.0"}
```

| Field | Written at | Read by | Default when absent |
Expand All @@ -352,7 +360,8 @@ An example, from a test's stand-in binary (`tests/fake_sensor:10`):
| `steps[]` | `pipeline.mojo:252-279` | see below | required |
| `tasks` | `pipeline.mojo:280-283` | dista task names; non-empty means the run gets a cluster (`db/registered.mojo:95`, `db/config_show.mojo:13`) | `[]` |
| `source` | `pipeline.mojo:284-286` | `repo`, `commit`, `dirty`: the row of `orca.pipeline_versions` (`db/runs.mojo:50-86`), and whether a run is [outdated](#the-stop-notice-a-newer-version-leads) | empty |
| `morca` | `pipeline.mojo:288` | the morca it was built with, checked at registration ([Versions](#versions)) | not checked |
| `protocol` | `pipeline.mojo:289` | the version of this protocol the binary speaks, checked at registration ([Versions](#versions)) | not checked |
| `morca` | `pipeline.mojo:290` | the morca-mojo it was built with, checked at registration; only for binaries built with morca-mojo ([Versions](#versions)) | not checked |

Paths in the table are under `src/morca/` and `src/morca/_internal/`.

Expand All @@ -366,7 +375,7 @@ Each element of `steps`, in run order:
| `function`, `input`, `output` | shown in the UI only (`ui/step_cards.mojo:266-273`) (**incidental** values: Mojo type and linkage names) |
| `timeout_seconds` | only when set; shown in the UI. The binary enforces it ([Timeouts](#timeouts-and-a-cancel-that-ran-out)) |
| `version` | only when set; the run's data version (`migrations.mojo:9-23`), so a fill redoes keys made at another version |
| `reads`, `writes` | dataset names `ns.table`. `writes` includes `replaces`. Used for sensing, lineage, cycles at registration and appending writes (`db/sensing.mojo:28-33`, `registering.mojo:78`, `db/queueing.mojo:231-237`) |
| `reads`, `writes` | dataset names `ns.table`. `writes` includes `replaces`. Used for sensing, lineage, cycles at registration and appending writes (`db/sensing.mojo:28-33`, `registering.mojo:90`, `db/queueing.mojo:231-237`) |
| `replaces` | only when non-empty: datasets whose partitions a rerun replaces instead of appending to (`db/queueing.mojo:231-237`) |

### The registry hash
Expand Down Expand Up @@ -916,14 +925,24 @@ driver log and metrics (`driver.log`, `metrics/`, `timeline.json`).

## Versions

- **The protocol.** describe's `"protocol": <n>` is the version of **this
protocol** the binary speaks; it is 1 (`PROTOCOL_VERSION`,
`src/morca/_internal/version.mojo`). `morca register` refuses a binary
whose `protocol` isn't an integer from 1 to the version it speaks,
whatever `MORCA_MIXED_VERSIONS` says, and registers one without the key
unchecked (#525). The version goes up only for a change an older binary
can't follow; a new optional field or subcommand doesn't raise it.
- **`MORCA_VERSION`.** describe's last key, `"morca": "<version>"`, is the
morca-mojo the binary was built with (`src/morca/pipeline.mojo:287-288`).
morca-mojo the binary was built with (`src/morca/pipeline.mojo:290`).
`morca register` refuses a binary of another version unless
`MORCA_MIXED_VERSIONS=1`, and registers one without the key unchecked
(`src/morca/_internal/version.mojo:19-48`, [binaries.md](binaries.md#versions)).
A second implementation should say which version of **this protocol** it
implements here, and morca should check that instead. Until then, it
must print the `morca` of the scheduler it targets.
(`src/morca/_internal/version.mojo`, [binaries.md](binaries.md#versions)).
A binary not built with morca-mojo says `protocol` and leaves `morca` out.
- **Older schedulers.** morca before 0.35.0 reads neither `protocol` nor
the JSON: it needs the bytes `{"pipeline": ` first, and checks the last
`"morca": "<version>"` if there is one. A binary that must register with
such a scheduler prints describe that way, and the `morca` of that
scheduler.
- **The registry hash** says which DAG a run ran ([above](#the-registry-hash)).
- **The source.** describe's `source` (`repo`, `commit`, `dirty`) and
`MORCA_IMAGE` make the run's version row, which the UI shows and the
Expand Down Expand Up @@ -970,7 +989,8 @@ What the runners do with it:
For `default` and `control` placement on Postgres or SQLite, with no dista
tasks, a pipeline binary needs:

1. `describe`, with the fields above and the `{"pipeline": ` prefix.
1. `describe`: one JSON object with the fields above and `"protocol": 1`
(no `morca`, unless it's built with morca-mojo).
2. `resume <run> [--allow-new-code]`: claim, version, interrupt, prior,
execute with a heartbeat, ownership checks at every attempt, the commit
fence, timeouts and cancels, and ending the run.
Expand Down
20 changes: 16 additions & 4 deletions src/morca/_internal/registering.mojo
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,13 @@ from morca._internal.proc import (
capture,
)
from morca._internal.secrets import secrets_json
from morca._internal.version import MISMATCH_HINT, mismatch, mixed_allowed
from morca._internal.version import (
MISMATCH_HINT,
mismatch,
mixed_allowed,
parse_describe,
protocol_refusal,
)
from morca._internal.sensors import parse_sense
from morca._internal.scheduler import Scheduler
from morca._internal.registered import (
Expand Down Expand Up @@ -67,10 +73,16 @@ def register_pipeline(
Returns the pipeline's name."""
var path = absolute(binary)
var got = capture([path, "describe"], binary_seconds())
if got[0] != 0 or not got[1].startswith('{"pipeline": '):
if got[0] != 0:
raise Error("morca: ", path, " describe exited ", got[0], ": ", got[1])
# The pipeline and this morca from one release (#455).
var other = mismatch(path, got[1])
# Parsed, so spacing and key order don't matter (#524).
var d = parse_describe(path, got[1])
# A protocol this morca speaks (#525), and for a morca-mojo binary,
# the pipeline and this morca from one release (#455).
var refusal = protocol_refusal(path, d)
if refusal:
raise Error("morca: ", refusal)
var other = mismatch(path, d)
if other and not mixed_allowed():
raise Error("morca: ", other, ": ", MISMATCH_HINT)
if other:
Expand Down
89 changes: 74 additions & 15 deletions src/morca/_internal/version.mojo
Original file line number Diff line number Diff line change
Expand Up @@ -8,32 +8,91 @@ differ, so a release bumps it with them. `morca version` prints it, each
pipeline's `describe` says it (`"morca"`), and `morca register` refuses a
pipeline built from another one (`mismatch`), unless
`MORCA_MIXED_VERSIONS=1`, when it only warns.

`PROTOCOL_VERSION` is the version of the pipeline-binary protocol
(docs/pipeline-protocol.md) this morca speaks, which describe says as
`"protocol"` (#525). A binary not built with morca-mojo (#521) says that
and no `"morca"`; `morca register` refuses a protocol it doesn't speak.
"""

from std.os import getenv
from json import Value, loads

comptime MORCA_VERSION = "0.34.0"
"""morca-mojo's version: pixi.toml's `[package] version`."""

comptime PROTOCOL_VERSION = 1
"""The pipeline-binary protocol's version (docs/pipeline-protocol.md):
describe's `"protocol"`. Raised only when a change to the protocol would
break a binary written to the one before."""

def built_with(describe: String) -> String:
"""The morca version a pipeline's `describe` (JSON) names, or empty for
a binary from before describe said it (morca-mojo 0.32.0 and older)."""
comptime key: StaticString = '"morca": "'
var at = describe.rfind(key)
if at < 0:
return ""
var start = at + key.byte_length()
var end = describe.find('"', start)
if end < 0:

def parse_describe(binary: String, describe: String) raises -> Value:
"""`describe`'s JSON, checked to be an object with a non-empty string
`pipeline`; whitespace and key order don't matter (#524)."""
var d: Value
try:
d = loads(describe)
except e:
raise Error(
"morca: ",
binary,
" describe didn't print JSON (",
e,
"): ",
describe,
)
if (
not d.is_object()
or "pipeline" not in d
or not d["pipeline"].is_string()
or not d["pipeline"].string_value()
):
raise Error(
"morca: ",
binary,
" describe printed no object with a `pipeline` name: ",
describe,
)
return d^


def built_with(d: Value) raises -> String:
"""The morca version a pipeline's parsed `describe` names, or empty for
a binary from before describe said it (morca-mojo 0.32.0 and older) or
one not built with morca-mojo."""
if "morca" in d and d["morca"].is_string():
return d["morca"].string_value()
return ""


def protocol_refusal(binary: String, d: Value) raises -> String:
"""Why this morca can't register the pipeline `binary`, whose parsed
`describe` this is, for the protocol it says it speaks, or empty when
it speaks this one or doesn't say (#525)."""
if "protocol" not in d:
return ""
return String(describe[byte=start:end])
var p = d["protocol"].copy()
if not p.is_int() or p.int_value() < 1:
return String(
binary, "'s describe has a protocol that isn't a version number"
)
if p.int_value() > PROTOCOL_VERSION:
return String(
binary,
" speaks pipeline protocol ",
p.int_value(),
" and this morca speaks ",
PROTOCOL_VERSION,
": upgrade morca",
)
return ""


def mismatch(binary: String, describe: String) -> String:
"""Why the pipeline `binary`, whose `describe` this is, wasn't built from
this morca, or empty when it was (or doesn't say)."""
var theirs = built_with(describe)
def mismatch(binary: String, d: Value) raises -> String:
"""Why the pipeline `binary`, whose parsed `describe` this is, wasn't
built from this morca, or empty when it was (or doesn't say)."""
var theirs = built_with(d)
if not theirs or theirs == MORCA_VERSION:
return ""
return String(
Expand Down
10 changes: 6 additions & 4 deletions src/morca/pipeline.mojo
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ from morca.civil import Civil, now
from morca.partitions import Monthly, Partitions
from morca.schedule import Fill, NoSchedule, Previous, Schedule, ScheduleLike
from morca._internal.source import source_commit, source_dirty, source_repo
from morca._internal.version import MORCA_VERSION
from morca._internal.version import MORCA_VERSION, PROTOCOL_VERSION
from morca.step import Gather, Mapped, StepLike, deferred_for, deferred_why
from morca._internal.store import (
DEFERRED,
Expand Down Expand Up @@ -216,8 +216,8 @@ struct Pipeline[
def describe() -> String:
"""The DAG as JSON: the pipeline's name, its registry hash, its
partitions, each step's name, kind, upstream, function, types and
datasets in run order, its dista tasks, its source and the morca
version it was built with."""
datasets in run order, its dista tasks, its source, the pipeline
protocol it speaks and the morca version it was built with."""
Self.check()
var out = String("{")
out.write('"pipeline": ', quote(Self.name), ", ")
Expand Down Expand Up @@ -284,7 +284,9 @@ struct Pipeline[
out.write('], "source": {"repo": ', quote(source_repo()))
out.write(', "commit": ', quote(source_commit()))
out.write(', "dirty": ', "true" if source_dirty() else "false", "}")
# Last, so `morca._internal.version.built_with` finds it from the end.
# Next to last, so `{"pipeline": ` stays the start and `"morca"` the
# last key, which a scheduler before #524 looks for at the end.
out.write(', "protocol": ', PROTOCOL_VERSION)
out.write(', "morca": ', quote(MORCA_VERSION), "}")
return out^

Expand Down
12 changes: 6 additions & 6 deletions tests/conformance.py
Original file line number Diff line number Diff line change
Expand Up @@ -56,11 +56,8 @@

DOC = "docs/pipeline-protocol.md"

# Checks known to fail until an issue lands: id -> issue.
KNOWN = {
"register.compact": "#524",
"describe.protocol": "#525",
}
# Checks known to fail until an issue lands: id -> issue. None now.
KNOWN: dict[str, str] = {}

STALE = 2.0
"""MORCA_STALE_SECONDS for every command: the suite is the scheduler."""
Expand Down Expand Up @@ -541,6 +538,9 @@ def morca_version():

def protocol():
need("protocol" in d, "describe names no protocol version")
p = d["protocol"]
need(isinstance(p, int) and not isinstance(p, bool) and p >= 1,
f"`protocol` is {p!r}, not a version number")

self.check("describe.protocol", "versions",
"describe says which version of the protocol it implements",
Expand Down Expand Up @@ -1218,7 +1218,7 @@ def describe_fields(d):
f"`partitions` is {parts}")
for k, t in (("max_runs", int), ("run_retries", int), ("keys_per_run", int),
("workers", int), ("sensor", bool), ("tasks", list),
("source", dict), ("morca", str)):
("source", dict), ("protocol", int), ("morca", str)):
if k in d:
need(isinstance(d[k], t) and not (t is int and isinstance(d[k], bool)),
f"`{k}` is {d[k]!r}")
Expand Down
18 changes: 17 additions & 1 deletion tests/versions.sh
Original file line number Diff line number Diff line change
Expand Up @@ -205,4 +205,20 @@ echo "$out" | grep -qF "toy-old was built with morca 0.0.1 and this morca is $wa
[ "$(q "SELECT count(*) FROM orca.pipelines WHERE name = 'toy_old'")" = 0 ] || fail "the refused pipeline is registered"
out=$(MORCA_MIXED_VERSIONS=1 ./build/morca register ./build/toy-old 2>&1) || fail "MORCA_MIXED_VERSIONS=1 register: $out"
echo "$out" | grep -qF "morca: warning: $PWD/build/toy-old was built with morca 0.0.1" || fail "no warning: $out"
echo "ok versions: a run links the commit it ran, the pipeline lists its versions, register checks the morca"
# The protocol version (#525): a binary not built with morca-mojo says
# `protocol` and no `morca`, in compact JSON (#524), and registers; one of
# a protocol this morca doesn't speak is refused, MORCA_MIXED_VERSIONS or not.
./build/toy describe | jq -e '.protocol == 1' > /dev/null || fail "describe's protocol: $(./build/toy describe | jq -c .protocol)"
./build/toy describe | jq -c '.pipeline = "toy_sdk" | del(.morca)' > build/toy-sdk.json
printf '#!/bin/sh\ncat "%s"\n' "$PWD/build/toy-sdk.json" > build/toy-sdk
chmod +x build/toy-sdk
out=$(./build/morca register ./build/toy-sdk 2>&1) || fail "register a protocol-only describe: $out"
echo "$out" | grep -qF warning && fail "a protocol-only describe warned: $out"
[ "$(q "SELECT count(*) FROM orca.pipelines WHERE name = 'toy_sdk'")" = 1 ] || fail "the protocol-only pipeline isn't registered"
jq -c '.pipeline = "toy_next" | .protocol = 2' build/toy-sdk.json > build/toy-next.json
printf '#!/bin/sh\ncat "%s"\n' "$PWD/build/toy-next.json" > build/toy-next
chmod +x build/toy-next
out=$(MORCA_MIXED_VERSIONS=1 ./build/morca register ./build/toy-next 2>&1) && fail "registered protocol 2: $out"
echo "$out" | grep -qF "toy-next speaks pipeline protocol 2 and this morca speaks 1" || fail "the protocol refusal: $out"
[ "$(q "SELECT count(*) FROM orca.pipelines WHERE name = 'toy_next'")" = 0 ] || fail "the protocol-2 pipeline is registered"
echo "ok versions: a run links the commit it ran, the pipeline lists its versions, register checks the morca and the protocol"
Loading