Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -879,6 +879,9 @@ jobs:
steps:
- uses: actions/checkout@v4
- uses: magmalake/customs@v0.2.1
# And the packages as layers: `_internal` imports no public module
# that imports `_internal` back (AGENTS.md).
- run: python3 scripts/check_layers.py

# The one check to require: every Linux job above passed, or, for a
# prose-only PR, was skipped. (macOS runs only on the nightly, which this
Expand Down
7 changes: 6 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,12 @@ go in the PR.
and the plumbing that calls it) imports only `morca._common`, and
`morca._internal` and the public modules may import both. `_lake` never
imports `_internal`, `morca.ctx` or another public module, even for one
helper: what both need goes in `_common`.
helper: what both need goes in `_common`. Public modules may import
`_internal`, but `_internal` imports only the public modules that don't
(`morca.civil`, `morca.day`, ...), never `morca.ctx`, `morca.pipeline` or
`morca`: code over `Ctx` or `Pipeline` lives beside them (`morca._cli`,
`morca._lakehost`, `morca._rust_ffi`). `scripts/check_layers.py` checks
the layers in CI.
- **Import from the defining module.** `from m import x` names the module
that defines `x`, not one that imports it and passes it on; only a
package's `__init__.mojo` may re-export. That's why `morca.tables`,
Expand Down
2 changes: 1 addition & 1 deletion DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -113,7 +113,7 @@ One schema, `morca`, created and migrated by the binary at start-up (a `schema_v
- **Statuses.** A run is `queued`, `running`, `succeeded`, `failed`, `deferred` (a step asked to wait) or `cancelled`. A step is `pending`, `running`, `succeeded`, `failed`, `deferred` or `skipped` (its upstream didn't succeed).
- **One writer per run.** A process that runs a run is its `owner` (`host:pid`) and holds it by its heartbeat: a thread with a connection of its own (threads.mojo's `TypedPool`) updates `runs.heartbeat` five times per `MORCA_STALE_SECONDS` (default 30), and `resume` refuses a run whose heartbeat is younger than that. A run whose process died has a stale heartbeat and can be resumed; a run that ended clears its heartbeat, so a failed run can be resumed at once. Every attempt's first transaction also checks that this process still owns the run, so a process that lost its run to a resume (it was paused, or the database was unreachable) stops at its next step instead of racing the new owner. No advisory locks for this, so a pooled connection is fine; migrations take a transaction-scoped one. *Verified* by `tests/chaos.sh`. **Commits to Iceberg are fenced.** The ownership check at each attempt leaves the whole of a step in between, and a process whose heartbeat went stale while it was alive (the database unreachable for longer than the stale limit, a paused VM) may meanwhile have lost its run to a restart that reruns the step. So right before each commit to a table, `morca.tables` checks (`Ctx.fence`) that this process still owns the run and that its heartbeat is younger than half the stale limit; otherwise the attempt fails without committing. No other process can have claimed the run then, nor claim it during the commit unless the commit takes longer than the other half. A database that can't be reached is waited for until that half has passed, then the commit is refused too. *Verified* by `tests/test_tables.mojo` (`test_a_lost_run_doesnt_commit`, `test_a_stale_heartbeat_doesnt_commit`).
- **Each state change is one transaction.** Starting an attempt inserts it and marks the step running; finishing it records the output, reads and commits, and marks the step. A kill between the two leaves an attempt `running` with a stale heartbeat, which resume marks `failed` before retrying.
- **The pipeline binary sees a `RunStore`, not SQL** (#82). `morca._internal.store`'s `Store` is the executor's view (attempts, skips, pinned reads, reads and commits through `Ctx`'s `Recorder`); `RunStore` adds the rest of what `morca._internal.cli` asks: `migrate`, `register`, `scheduled_hash`, `version`, `create_run`, `claim`, `ran_version`, `interrupt`, `prior`, `run_info`, `runs`, `steps`, and for the run it owns `own`, `start_heartbeat`/`stop_heartbeat`, `finish_run` and `cancel_requested`. `morca._internal.state.PgStore` implements it over `State`; the heartbeat is the store's (`morca._internal.heartbeat.Heartbeat[B]`, one thread beating a `Beater` on a connection of its own). The scheduler and the UI stay on `State`'s SQL: claims with `SKIP LOCKED`, the listings and the lineage queries are queries, and a trait over them would grow a method per page (`docs/future-directions.md`).
- **The pipeline binary sees a `RunStore`, not SQL** (#82). `morca._internal.store`'s `Store` is the executor's view (attempts, skips, pinned reads, reads and commits through `Ctx`'s `Recorder`); `RunStore` adds the rest of what `morca._cli` asks: `migrate`, `register`, `scheduled_hash`, `version`, `create_run`, `claim`, `ran_version`, `interrupt`, `prior`, `run_info`, `runs`, `steps`, and for the run it owns `own`, `start_heartbeat`/`stop_heartbeat`, `finish_run` and `cancel_requested`. `morca._internal.state.PgStore` implements it over `State`; the heartbeat is the store's (`morca._internal.heartbeat.Heartbeat[B]`, one thread beating a `Beater` on a connection of its own). The scheduler and the UI stay on `State`'s SQL: claims with `SKIP LOCKED`, the listings and the lineage queries are queries, and a trait over them would grow a method per page (`docs/future-directions.md`).

## Single-node morca on SQLite

Expand Down
1 change: 1 addition & 0 deletions changes/511-package-layers.patch.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
- **Package layers**: the modules over `Ctx` and `Pipeline` moved out of `morca._internal`, which no longer imports a module that imports it back, and CI checks the layers (#511).
161 changes: 161 additions & 0 deletions scripts/check_layers.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,161 @@
#!/usr/bin/env python3
"""Checks that morca's packages form layers (AGENTS.md, "No circular imports").

customs checks cycles between modules; this checks the packages above them:

- `morca._common` imports nothing of morca's but `_common`;
- `morca._lake` imports only `_lake` and `_common`;
- `morca._internal` imports `_internal`, `_lake`, `_common`, and only those
public modules that import no `_internal` module, directly or through
others (`morca.civil`, `morca.day`, ...): never `morca.ctx`,
`morca.pipeline` or `morca` itself, which would close a cycle through
`_internal`.

Run `python3 scripts/check_layers.py [REPO]` (default: this repo). Exits 1 on a finding.
"""

import os
import re
import sys

SRC = "src"


def blank(text):
"""The text with comments and string literals blanked, lines kept."""
out = []
i, n = 0, len(text)
while i < n:
c = text[i]
if c == "#":
j = text.find("\n", i)
j = n if j < 0 else j
i = j
continue
if c in "\"'":
q = text[i : i + 3]
if q in ('"""', "'''"):
j = text.find(q, i + 3)
j = n if j < 0 else j + 3
else:
j = i + 1
while j < n and text[j] != c and text[j] != "\n":
j += 2 if text[j] == "\\" else 1
j += 1
out.append("".join(ch if ch == "\n" else " " for ch in text[i:j]))
i = j
continue
out.append(c)
i += 1
return "".join(out)


def modules():
"""Module name to file, for each module of the `morca` package."""
mods = {}
root = os.path.join(SRC, "morca")
for d, _, files in os.walk(root):
if not os.path.exists(os.path.join(d, "__init__.mojo")):
continue
pkg = os.path.relpath(d, SRC).replace(os.sep, ".")
for f in files:
if not f.endswith(".mojo"):
continue
name = pkg if f == "__init__.mojo" else pkg + "." + f[:-5]
mods[name] = os.path.join(d, f)
return mods


IMPORT = re.compile(
r"^[ \t]*(?:from\s+(\.*[\w.]*)\s+import\s+(\([^)]*\)|[^\n]*(?:\\\n[^\n]*)*)"
r"|import\s+([^\n]*))",
re.M,
)


def imports(name, path, mods):
"""(line, imported module) for each morca module `name` imports."""
text = blank(open(path, encoding="utf-8").read())
pkg = name if path.endswith("__init__.mojo") else name.rpartition(".")[0]
found = []
for m in IMPORT.finditer(text):
line = text.count("\n", 0, m.start()) + 1
if m.group(1) is not None:
base = m.group(1)
if base.startswith("."):
dots = len(base) - len(base.lstrip("."))
parts = pkg.split(".")
parts = parts[: len(parts) - (dots - 1)] if dots > 1 else parts
rest = base.lstrip(".")
base = ".".join(parts + ([rest] if rest else []))
targets = [base]
names = re.sub(r"[()\\]", " ", m.group(2))
for item in names.split(","):
word = item.split()
if word:
targets.append(base + "." + word[0])
else:
targets = [t.split()[0] for t in m.group(3).split(",") if t.split()]
for t in targets:
while t and t not in mods:
t = t.rpartition(".")[0]
if t and t != name and (line, t) not in found:
found.append((line, t))
return found


def layer(name):
parts = name.split(".")
if len(parts) > 1 and parts[1] in ("_common", "_lake", "_internal"):
return parts[1]
return "public"


def main():
here = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..")
os.chdir(sys.argv[1] if len(sys.argv) > 1 else here)
mods = modules()
graph = {n: imports(n, p, mods) for n, p in mods.items()}

# Public modules that reach `_internal`, directly or through others.
reaches = {n for n in mods if layer(n) == "public" and any(
layer(t) == "_internal" for _, t in graph[n])}
changed = True
while changed:
changed = False
for n in mods:
if layer(n) == "public" and n not in reaches and any(
t in reaches for _, t in graph[n]):
reaches.add(n)
changed = True

allowed = {
"_common": {"_common"},
"_lake": {"_lake", "_common"},
"_internal": {"_internal", "_lake", "_common"},
}
bad = 0
for n in sorted(mods):
lay = layer(n)
if lay == "public":
continue
for line, t in graph[n]:
tl = layer(t)
if tl in allowed[lay]:
continue
if lay == "_internal" and tl == "public" and t not in reaches:
continue
why = (
f"{t} imports morca._internal, directly or through others"
if tl == "public" and lay == "_internal"
else f"morca.{lay} may import only "
+ ", ".join("morca." + a for a in sorted(allowed[lay]))
)
print(f"{mods[n]}:{line}: {n} imports {t}: {why}")
bad += 1
print(f"check_layers: {len(mods)} modules: {bad} imports across layers")
return 1 if bad else 0


if __name__ == "__main__":
sys.exit(main())
2 changes: 1 addition & 1 deletion src/morca/__init__.mojo
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ stores and their SQL, the runners, the UI and its API, the command line's
subcommands. They change without notice; a pipeline shouldn't import them.
"""

from morca._internal.cli import morca_main, report
from morca._cli import morca_main, report
from morca.ctx import Ctx
from morca.civil import Civil, now, parse_time, utc
from morca.cron import CronExpr
Expand Down
File renamed without changes.
2 changes: 1 addition & 1 deletion src/morca/_internal/store.mojo
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ process, for a run that lives as long as it does; `morca._internal.state.PgStore
writes them to Postgres, so a run can be resumed by another process.

`RunStore` is a `Store` that also keeps runs: the rest of what a pipeline
binary (`morca._internal.cli`) asks of persistence, from registering the pipeline to
binary (`morca._cli`) asks of persistence, from registering the pipeline to
ending its run, so the binary depends on this trait and nothing else.
Both backends' SQL is in `morca._internal.db` (`morca._internal.db.runs`, `morca._internal.db.steps`,
`morca._internal.db.sqlite_runs`), as is the scheduler's and the UI's.
Expand Down
2 changes: 1 addition & 1 deletion src/morca/_lake/__init__.mojo
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,6 @@

Nothing here imports `morca._internal` or knows a `Ctx`: `morca._internal`
(and `morca.tables`) import this package, never the other way round, and
the host block over a step's `Ctx` is `morca._internal.lakehost`'s. What the
the host block over a step's `Ctx` is `morca._lakehost`'s. What the
two share lives in `morca._common`.
"""
2 changes: 1 addition & 1 deletion src/morca/_lake/calls.mojo
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ The table, reader and writer handles stay integers: each is owned by a
value of the caller's (`morca.tables.Table`, `BatchReader`, `FileWriter`),
held as a field or an argument across the call.

**The host** is anything `Hosting`: `morca._internal.lakehost.HostBlock`
**The host** is anything `Hosting`: `morca._lakehost.HostBlock`
over the step's `Ctx`, or `NoHost` outside any step. A call whose callbacks
change the `Ctx` (a pin, a fence, a commit) takes only a `writable` host,
one over a `mut` borrow of it, and says so at compile time.
Expand Down
2 changes: 1 addition & 1 deletion src/morca/_lake/client.mojo
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
"""The program's side of liblake: finding and loading the library, and
batches crossing as Arrow C Data. No iceberg here, and no `Ctx`:
`morca._lake.calls` calls the library through this, and the host block over
a step's `Ctx` is the program's (`morca._internal.lakehost`).
a step's `Ctx` is the program's (`morca._lakehost`).

**Loading.** The library is opened the first time a table is used, in the
step or the dista task that uses it, never at program start (dista forks
Expand Down
2 changes: 1 addition & 1 deletion src/morca/_lake/host.mojo
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
"""The library's view of the step: the methods of `morca.Ctx` that the tables
code uses, each a call through the host block (`morca._lake.abi`). The
program fills the block (`morca._internal.lakehost.HostBlock`)."""
program fills the block (`morca._lakehost.HostBlock`)."""

from morca._lake.abi import (
CheckDeclaredFn,
Expand Down
2 changes: 1 addition & 1 deletion src/morca/_lake/impl.mojo
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,7 @@ def _backoff(attempt: Int):
half. Two writers that both wrote the same metadata version both back
out (iceberg's filesystem catalog deletes both files), so retrying at
once can clash again and again; the random wait lets one get ahead.
Each process seeds `std.random` itself (`morca._internal.cli.seed_random`).
Each process seeds `std.random` itself (`morca._cli.seed_random`).
"""
var base = 0.01 * Float64(1 << min(attempt - 1, 6))
sleep(base * random_float64(0.5, 1.5))
Expand Down
File renamed without changes.
File renamed without changes.
4 changes: 2 additions & 2 deletions src/morca/cli.mojo
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ def main() raises:

`morca_main` and `report` are public (they're exported from `morca` too).
The command line itself, its subcommands and stores, is internal:
`morca._internal.cli`.
`morca._cli`.
"""

from morca._internal.cli import morca_main, report
from morca._cli import morca_main, report
2 changes: 1 addition & 1 deletion src/morca/rust_step.mojo
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ from arrow_mlake.carrow_import import ImportedStream, release_c_array

from morca.ctx import Ctx
from morca._internal.handover import outdated
from morca._internal.rust_ffi import StopFn, WorkerCall, WorkerLib
from morca._rust_ffi import StopFn, WorkerCall, WorkerLib
from morca._lake.abi import Dec, Slots
from morca._lake import calls
from morca._lake.client import Api
Expand Down
2 changes: 1 addition & 1 deletion src/morca/tables/__init__.mojo
Original file line number Diff line number Diff line change
Expand Up @@ -142,7 +142,7 @@ from dista import Codec
from parquet import RecordBatch

from morca.ctx import Ctx
from morca._internal.lakehost import HostBlock
from morca._lakehost import HostBlock
from morca._lake.abi import (
MODE_APPEND,
MODE_BY_STEP,
Expand Down
2 changes: 1 addition & 1 deletion tests/keyed/names.mojo
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ from iceberg.transforms import PartitionSpec
from iceberg.values import Datum
from iceberg.write import WriteOptions, write_data_files

from morca._internal.cli import seed_random
from morca._cli import seed_random

comptime SCHEMA = (
'{"schema-id":0,"type":"struct","fields":['
Expand Down
2 changes: 1 addition & 1 deletion tests/sensors/source.mojo
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ from iceberg.schema import Schema
from iceberg.transforms import PartitionSpec
from iceberg.write import WriteOptions, write_data_files

from morca._internal.cli import seed_random
from morca._cli import seed_random
from rows import SCHEMA, rows


Expand Down
Loading