From fa08d3b57c00e8fe3edbb6001854fd50170d3549 Mon Sep 17 00:00:00 2001 From: Marius S <39998+winding-lines@users.noreply.github.com> Date: Sun, 15 Nov 2026 21:22:23 -0800 Subject: [PATCH 1/2] Move the modules over Ctx and Pipeline out of _internal `_internal/cli.mojo` imported `morca.pipeline`, and `_internal/lakehost.mojo` and `_internal/rust_ffi.mojo` imported `morca.ctx`, all of which import `_internal`: cycles between the packages. They move, unchanged, beside the public modules that use them (`morca._cli`, `morca._lakehost`, `morca._rust_ffi`); `Ctx` and `Pipeline` keep their module paths, so registry hashes don't change. `scripts/check_layers.py` checks the layers in the customs job. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_017TycszSbyw92Efu4ZeCYL6 --- .github/workflows/ci.yml | 3 + AGENTS.md | 7 +- DESIGN.md | 2 +- scripts/check_layers.py | 161 ++++++++++++++++++ src/morca/__init__.mojo | 2 +- src/morca/{_internal/cli.mojo => _cli.mojo} | 0 src/morca/_internal/store.mojo | 2 +- src/morca/_lake/__init__.mojo | 2 +- src/morca/_lake/calls.mojo | 2 +- src/morca/_lake/client.mojo | 2 +- src/morca/_lake/host.mojo | 2 +- src/morca/_lake/impl.mojo | 2 +- .../lakehost.mojo => _lakehost.mojo} | 0 .../rust_ffi.mojo => _rust_ffi.mojo} | 0 src/morca/cli.mojo | 4 +- src/morca/rust_step.mojo | 2 +- src/morca/tables/__init__.mojo | 2 +- tests/keyed/names.mojo | 2 +- tests/sensors/source.mojo | 2 +- 19 files changed, 184 insertions(+), 15 deletions(-) create mode 100644 scripts/check_layers.py rename src/morca/{_internal/cli.mojo => _cli.mojo} (100%) rename src/morca/{_internal/lakehost.mojo => _lakehost.mojo} (100%) rename src/morca/{_internal/rust_ffi.mojo => _rust_ffi.mojo} (100%) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 0db529e4..c1db5ba7 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -879,6 +879,9 @@ jobs: steps: - uses: actions/checkout@v4 - uses: magmalake/customs@v0.2.1 + # And the packages as layers: `_internal` imports no public module + # that imports `_internal` back (AGENTS.md). + - run: python3 scripts/check_layers.py # The one check to require: every Linux job above passed, or, for a # prose-only PR, was skipped. (macOS runs only on the nightly, which this diff --git a/AGENTS.md b/AGENTS.md index feca241d..085213df 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -25,7 +25,12 @@ go in the PR. and the plumbing that calls it) imports only `morca._common`, and `morca._internal` and the public modules may import both. `_lake` never imports `_internal`, `morca.ctx` or another public module, even for one - helper: what both need goes in `_common`. + helper: what both need goes in `_common`. Public modules may import + `_internal`, but `_internal` imports only the public modules that don't + (`morca.civil`, `morca.day`, ...), never `morca.ctx`, `morca.pipeline` or + `morca`: code over `Ctx` or `Pipeline` lives beside them (`morca._cli`, + `morca._lakehost`, `morca._rust_ffi`). `scripts/check_layers.py` checks + the layers in CI. - **Import from the defining module.** `from m import x` names the module that defines `x`, not one that imports it and passes it on; only a package's `__init__.mojo` may re-export. That's why `morca.tables`, diff --git a/DESIGN.md b/DESIGN.md index 59356337..886980a9 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -113,7 +113,7 @@ One schema, `morca`, created and migrated by the binary at start-up (a `schema_v - **Statuses.** A run is `queued`, `running`, `succeeded`, `failed`, `deferred` (a step asked to wait) or `cancelled`. A step is `pending`, `running`, `succeeded`, `failed`, `deferred` or `skipped` (its upstream didn't succeed). - **One writer per run.** A process that runs a run is its `owner` (`host:pid`) and holds it by its heartbeat: a thread with a connection of its own (threads.mojo's `TypedPool`) updates `runs.heartbeat` five times per `MORCA_STALE_SECONDS` (default 30), and `resume` refuses a run whose heartbeat is younger than that. A run whose process died has a stale heartbeat and can be resumed; a run that ended clears its heartbeat, so a failed run can be resumed at once. Every attempt's first transaction also checks that this process still owns the run, so a process that lost its run to a resume (it was paused, or the database was unreachable) stops at its next step instead of racing the new owner. No advisory locks for this, so a pooled connection is fine; migrations take a transaction-scoped one. *Verified* by `tests/chaos.sh`. **Commits to Iceberg are fenced.** The ownership check at each attempt leaves the whole of a step in between, and a process whose heartbeat went stale while it was alive (the database unreachable for longer than the stale limit, a paused VM) may meanwhile have lost its run to a restart that reruns the step. So right before each commit to a table, `morca.tables` checks (`Ctx.fence`) that this process still owns the run and that its heartbeat is younger than half the stale limit; otherwise the attempt fails without committing. No other process can have claimed the run then, nor claim it during the commit unless the commit takes longer than the other half. A database that can't be reached is waited for until that half has passed, then the commit is refused too. *Verified* by `tests/test_tables.mojo` (`test_a_lost_run_doesnt_commit`, `test_a_stale_heartbeat_doesnt_commit`). - **Each state change is one transaction.** Starting an attempt inserts it and marks the step running; finishing it records the output, reads and commits, and marks the step. A kill between the two leaves an attempt `running` with a stale heartbeat, which resume marks `failed` before retrying. -- **The pipeline binary sees a `RunStore`, not SQL** (#82). `morca._internal.store`'s `Store` is the executor's view (attempts, skips, pinned reads, reads and commits through `Ctx`'s `Recorder`); `RunStore` adds the rest of what `morca._internal.cli` asks: `migrate`, `register`, `scheduled_hash`, `version`, `create_run`, `claim`, `ran_version`, `interrupt`, `prior`, `run_info`, `runs`, `steps`, and for the run it owns `own`, `start_heartbeat`/`stop_heartbeat`, `finish_run` and `cancel_requested`. `morca._internal.state.PgStore` implements it over `State`; the heartbeat is the store's (`morca._internal.heartbeat.Heartbeat[B]`, one thread beating a `Beater` on a connection of its own). The scheduler and the UI stay on `State`'s SQL: claims with `SKIP LOCKED`, the listings and the lineage queries are queries, and a trait over them would grow a method per page (`docs/future-directions.md`). +- **The pipeline binary sees a `RunStore`, not SQL** (#82). `morca._internal.store`'s `Store` is the executor's view (attempts, skips, pinned reads, reads and commits through `Ctx`'s `Recorder`); `RunStore` adds the rest of what `morca._cli` asks: `migrate`, `register`, `scheduled_hash`, `version`, `create_run`, `claim`, `ran_version`, `interrupt`, `prior`, `run_info`, `runs`, `steps`, and for the run it owns `own`, `start_heartbeat`/`stop_heartbeat`, `finish_run` and `cancel_requested`. `morca._internal.state.PgStore` implements it over `State`; the heartbeat is the store's (`morca._internal.heartbeat.Heartbeat[B]`, one thread beating a `Beater` on a connection of its own). The scheduler and the UI stay on `State`'s SQL: claims with `SKIP LOCKED`, the listings and the lineage queries are queries, and a trait over them would grow a method per page (`docs/future-directions.md`). ## Single-node morca on SQLite diff --git a/scripts/check_layers.py b/scripts/check_layers.py new file mode 100644 index 00000000..660cb38b --- /dev/null +++ b/scripts/check_layers.py @@ -0,0 +1,161 @@ +#!/usr/bin/env python3 +"""Checks that morca's packages form layers (AGENTS.md, "No circular imports"). + +customs checks cycles between modules; this checks the packages above them: + +- `morca._common` imports nothing of morca's but `_common`; +- `morca._lake` imports only `_lake` and `_common`; +- `morca._internal` imports `_internal`, `_lake`, `_common`, and only those + public modules that import no `_internal` module, directly or through + others (`morca.civil`, `morca.day`, ...): never `morca.ctx`, + `morca.pipeline` or `morca` itself, which would close a cycle through + `_internal`. + +Run `python3 scripts/check_layers.py [REPO]` (default: this repo). Exits 1 on a finding. +""" + +import os +import re +import sys + +SRC = "src" + + +def blank(text): + """The text with comments and string literals blanked, lines kept.""" + out = [] + i, n = 0, len(text) + while i < n: + c = text[i] + if c == "#": + j = text.find("\n", i) + j = n if j < 0 else j + i = j + continue + if c in "\"'": + q = text[i : i + 3] + if q in ('"""', "'''"): + j = text.find(q, i + 3) + j = n if j < 0 else j + 3 + else: + j = i + 1 + while j < n and text[j] != c and text[j] != "\n": + j += 2 if text[j] == "\\" else 1 + j += 1 + out.append("".join(ch if ch == "\n" else " " for ch in text[i:j])) + i = j + continue + out.append(c) + i += 1 + return "".join(out) + + +def modules(): + """Module name to file, for each module of the `morca` package.""" + mods = {} + root = os.path.join(SRC, "morca") + for d, _, files in os.walk(root): + if not os.path.exists(os.path.join(d, "__init__.mojo")): + continue + pkg = os.path.relpath(d, SRC).replace(os.sep, ".") + for f in files: + if not f.endswith(".mojo"): + continue + name = pkg if f == "__init__.mojo" else pkg + "." + f[:-5] + mods[name] = os.path.join(d, f) + return mods + + +IMPORT = re.compile( + r"^[ \t]*(?:from\s+(\.*[\w.]*)\s+import\s+(\([^)]*\)|[^\n]*(?:\\\n[^\n]*)*)" + r"|import\s+([^\n]*))", + re.M, +) + + +def imports(name, path, mods): + """(line, imported module) for each morca module `name` imports.""" + text = blank(open(path, encoding="utf-8").read()) + pkg = name if path.endswith("__init__.mojo") else name.rpartition(".")[0] + found = [] + for m in IMPORT.finditer(text): + line = text.count("\n", 0, m.start()) + 1 + if m.group(1) is not None: + base = m.group(1) + if base.startswith("."): + dots = len(base) - len(base.lstrip(".")) + parts = pkg.split(".") + parts = parts[: len(parts) - (dots - 1)] if dots > 1 else parts + rest = base.lstrip(".") + base = ".".join(parts + ([rest] if rest else [])) + targets = [base] + names = re.sub(r"[()\\]", " ", m.group(2)) + for item in names.split(","): + word = item.split() + if word: + targets.append(base + "." + word[0]) + else: + targets = [t.split()[0] for t in m.group(3).split(",") if t.split()] + for t in targets: + while t and t not in mods: + t = t.rpartition(".")[0] + if t and t != name and (line, t) not in found: + found.append((line, t)) + return found + + +def layer(name): + parts = name.split(".") + if len(parts) > 1 and parts[1] in ("_common", "_lake", "_internal"): + return parts[1] + return "public" + + +def main(): + here = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..") + os.chdir(sys.argv[1] if len(sys.argv) > 1 else here) + mods = modules() + graph = {n: imports(n, p, mods) for n, p in mods.items()} + + # Public modules that reach `_internal`, directly or through others. + reaches = {n for n in mods if layer(n) == "public" and any( + layer(t) == "_internal" for _, t in graph[n])} + changed = True + while changed: + changed = False + for n in mods: + if layer(n) == "public" and n not in reaches and any( + t in reaches for _, t in graph[n]): + reaches.add(n) + changed = True + + allowed = { + "_common": {"_common"}, + "_lake": {"_lake", "_common"}, + "_internal": {"_internal", "_lake", "_common"}, + } + bad = 0 + for n in sorted(mods): + lay = layer(n) + if lay == "public": + continue + for line, t in graph[n]: + tl = layer(t) + if tl in allowed[lay]: + continue + if lay == "_internal" and tl == "public" and t not in reaches: + continue + why = ( + f"{t} imports morca._internal, directly or through others" + if tl == "public" and lay == "_internal" + else f"morca.{lay} may import only " + + ", ".join("morca." + a for a in sorted(allowed[lay])) + ) + print(f"{mods[n]}:{line}: {n} imports {t}: {why}") + bad += 1 + print(f"check_layers: {len(mods)} modules: {bad} imports across layers") + return 1 if bad else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/src/morca/__init__.mojo b/src/morca/__init__.mojo index c4c9455a..2954dc92 100644 --- a/src/morca/__init__.mojo +++ b/src/morca/__init__.mojo @@ -73,7 +73,7 @@ stores and their SQL, the runners, the UI and its API, the command line's subcommands. They change without notice; a pipeline shouldn't import them. """ -from morca._internal.cli import morca_main, report +from morca._cli import morca_main, report from morca.ctx import Ctx from morca.civil import Civil, now, parse_time, utc from morca.cron import CronExpr diff --git a/src/morca/_internal/cli.mojo b/src/morca/_cli.mojo similarity index 100% rename from src/morca/_internal/cli.mojo rename to src/morca/_cli.mojo diff --git a/src/morca/_internal/store.mojo b/src/morca/_internal/store.mojo index 9ca91543..279b4207 100644 --- a/src/morca/_internal/store.mojo +++ b/src/morca/_internal/store.mojo @@ -7,7 +7,7 @@ process, for a run that lives as long as it does; `morca._internal.state.PgStore writes them to Postgres, so a run can be resumed by another process. `RunStore` is a `Store` that also keeps runs: the rest of what a pipeline -binary (`morca._internal.cli`) asks of persistence, from registering the pipeline to +binary (`morca._cli`) asks of persistence, from registering the pipeline to ending its run, so the binary depends on this trait and nothing else. Both backends' SQL is in `morca._internal.db` (`morca._internal.db.runs`, `morca._internal.db.steps`, `morca._internal.db.sqlite_runs`), as is the scheduler's and the UI's. diff --git a/src/morca/_lake/__init__.mojo b/src/morca/_lake/__init__.mojo index 25942926..bb870cac 100644 --- a/src/morca/_lake/__init__.mojo +++ b/src/morca/_lake/__init__.mojo @@ -8,6 +8,6 @@ Nothing here imports `morca._internal` or knows a `Ctx`: `morca._internal` (and `morca.tables`) import this package, never the other way round, and -the host block over a step's `Ctx` is `morca._internal.lakehost`'s. What the +the host block over a step's `Ctx` is `morca._lakehost`'s. What the two share lives in `morca._common`. """ diff --git a/src/morca/_lake/calls.mojo b/src/morca/_lake/calls.mojo index 47adde4b..5768b2d6 100644 --- a/src/morca/_lake/calls.mojo +++ b/src/morca/_lake/calls.mojo @@ -16,7 +16,7 @@ The table, reader and writer handles stay integers: each is owned by a value of the caller's (`morca.tables.Table`, `BatchReader`, `FileWriter`), held as a field or an argument across the call. -**The host** is anything `Hosting`: `morca._internal.lakehost.HostBlock` +**The host** is anything `Hosting`: `morca._lakehost.HostBlock` over the step's `Ctx`, or `NoHost` outside any step. A call whose callbacks change the `Ctx` (a pin, a fence, a commit) takes only a `writable` host, one over a `mut` borrow of it, and says so at compile time. diff --git a/src/morca/_lake/client.mojo b/src/morca/_lake/client.mojo index 4c144a53..6a47d0d4 100644 --- a/src/morca/_lake/client.mojo +++ b/src/morca/_lake/client.mojo @@ -1,7 +1,7 @@ """The program's side of liblake: finding and loading the library, and batches crossing as Arrow C Data. No iceberg here, and no `Ctx`: `morca._lake.calls` calls the library through this, and the host block over -a step's `Ctx` is the program's (`morca._internal.lakehost`). +a step's `Ctx` is the program's (`morca._lakehost`). **Loading.** The library is opened the first time a table is used, in the step or the dista task that uses it, never at program start (dista forks diff --git a/src/morca/_lake/host.mojo b/src/morca/_lake/host.mojo index fa471cff..633b6a1a 100644 --- a/src/morca/_lake/host.mojo +++ b/src/morca/_lake/host.mojo @@ -1,6 +1,6 @@ """The library's view of the step: the methods of `morca.Ctx` that the tables code uses, each a call through the host block (`morca._lake.abi`). The -program fills the block (`morca._internal.lakehost.HostBlock`).""" +program fills the block (`morca._lakehost.HostBlock`).""" from morca._lake.abi import ( CheckDeclaredFn, diff --git a/src/morca/_lake/impl.mojo b/src/morca/_lake/impl.mojo index 3773cd4a..a18e88a6 100644 --- a/src/morca/_lake/impl.mojo +++ b/src/morca/_lake/impl.mojo @@ -83,7 +83,7 @@ def _backoff(attempt: Int): half. Two writers that both wrote the same metadata version both back out (iceberg's filesystem catalog deletes both files), so retrying at once can clash again and again; the random wait lets one get ahead. - Each process seeds `std.random` itself (`morca._internal.cli.seed_random`). + Each process seeds `std.random` itself (`morca._cli.seed_random`). """ var base = 0.01 * Float64(1 << min(attempt - 1, 6)) sleep(base * random_float64(0.5, 1.5)) diff --git a/src/morca/_internal/lakehost.mojo b/src/morca/_lakehost.mojo similarity index 100% rename from src/morca/_internal/lakehost.mojo rename to src/morca/_lakehost.mojo diff --git a/src/morca/_internal/rust_ffi.mojo b/src/morca/_rust_ffi.mojo similarity index 100% rename from src/morca/_internal/rust_ffi.mojo rename to src/morca/_rust_ffi.mojo diff --git a/src/morca/cli.mojo b/src/morca/cli.mojo index de8afbe9..e1c06b93 100644 --- a/src/morca/cli.mojo +++ b/src/morca/cli.mojo @@ -7,7 +7,7 @@ def main() raises: `morca_main` and `report` are public (they're exported from `morca` too). The command line itself, its subcommands and stores, is internal: -`morca._internal.cli`. +`morca._cli`. """ -from morca._internal.cli import morca_main, report +from morca._cli import morca_main, report diff --git a/src/morca/rust_step.mojo b/src/morca/rust_step.mojo index 264b3a26..452b775b 100644 --- a/src/morca/rust_step.mojo +++ b/src/morca/rust_step.mojo @@ -85,7 +85,7 @@ from arrow_mlake.carrow_import import ImportedStream, release_c_array from morca.ctx import Ctx from morca._internal.handover import outdated -from morca._internal.rust_ffi import StopFn, WorkerCall, WorkerLib +from morca._rust_ffi import StopFn, WorkerCall, WorkerLib from morca._lake.abi import Dec, Slots from morca._lake import calls from morca._lake.client import Api diff --git a/src/morca/tables/__init__.mojo b/src/morca/tables/__init__.mojo index b1b80c93..67f48b3f 100644 --- a/src/morca/tables/__init__.mojo +++ b/src/morca/tables/__init__.mojo @@ -142,7 +142,7 @@ from dista import Codec from parquet import RecordBatch from morca.ctx import Ctx -from morca._internal.lakehost import HostBlock +from morca._lakehost import HostBlock from morca._lake.abi import ( MODE_APPEND, MODE_BY_STEP, diff --git a/tests/keyed/names.mojo b/tests/keyed/names.mojo index cbe900dc..41cd3714 100644 --- a/tests/keyed/names.mojo +++ b/tests/keyed/names.mojo @@ -17,7 +17,7 @@ from iceberg.transforms import PartitionSpec from iceberg.values import Datum from iceberg.write import WriteOptions, write_data_files -from morca._internal.cli import seed_random +from morca._cli import seed_random comptime SCHEMA = ( '{"schema-id":0,"type":"struct","fields":[' diff --git a/tests/sensors/source.mojo b/tests/sensors/source.mojo index 8db51742..7a625640 100644 --- a/tests/sensors/source.mojo +++ b/tests/sensors/source.mojo @@ -14,7 +14,7 @@ from iceberg.schema import Schema from iceberg.transforms import PartitionSpec from iceberg.write import WriteOptions, write_data_files -from morca._internal.cli import seed_random +from morca._cli import seed_random from rows import SCHEMA, rows From 36d68dce48fc2819b742376a5404e3a40692ef85 Mon Sep 17 00:00:00 2001 From: Marius S <39998+winding-lines@users.noreply.github.com> Date: Sun, 15 Nov 2026 21:22:43 -0800 Subject: [PATCH 2/2] Add the changelog fragment Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_017TycszSbyw92Efu4ZeCYL6 --- changes/511-package-layers.patch.md | 1 + 1 file changed, 1 insertion(+) create mode 100644 changes/511-package-layers.patch.md diff --git a/changes/511-package-layers.patch.md b/changes/511-package-layers.patch.md new file mode 100644 index 00000000..6a1b36ba --- /dev/null +++ b/changes/511-package-layers.patch.md @@ -0,0 +1 @@ +- **Package layers**: the modules over `Ctx` and `Pipeline` moved out of `morca._internal`, which no longer imports a module that imports it back, and CI checks the layers (#511).