From 98ec5dfe790da6e99e7502eaa68dbea1784b404d Mon Sep 17 00:00:00 2001 From: Marius S <39998+winding-lines@users.noreply.github.com> Date: Tue, 17 Nov 2026 00:05:22 -0800 Subject: [PATCH 1/2] Pipeline fire= replaces fills=; describe says schedule.fire (#370) `Pipeline[..., fire=Previous]` picks the partition a scheduled fire runs. `fills=` stays accepted for one release (setting both doesn't compile), and describe writes `schedule.fills` beside `schedule.fire` with the same value, so older schedulers still read it. The scheduler and the UI read `fire`, falling back to `fills`. The Rust SDK gets `fire_current()` and emits both keys. Registry hashes are unchanged: the schedule isn't hashed. morca.toml's `fills` (the automatic backfill) keeps its name. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_017TycszSbyw92Efu4ZeCYL6 --- DESIGN.md | 12 ++--- changes/370-fire.minor.md | 1 + docs/pipeline-protocol.md | 60 +++++++++++----------- docs/rust-pipelines.md | 2 +- rust/morca-sdk/src/pipeline.rs | 22 +++++--- src/morca/_cli.mojo | 2 +- src/morca/_internal/api/pipelines.mojo | 2 + src/morca/_internal/db/pipelines_view.mojo | 3 +- src/morca/_internal/db/registered.mojo | 3 +- src/morca/_internal/ui/pipeline.mojo | 6 +-- src/morca/keyed.mojo | 2 +- src/morca/pipeline.mojo | 42 +++++++++++---- src/morca/schedule.mojo | 16 +++--- tests/compile_fail/fire_and_fills.mojo | 16 ++++++ tests/compile_fail_keyed/keyed_fire.mojo | 17 ++++++ tests/hourly.mojo | 4 +- tests/hourly.sh | 2 +- tests/test_cron.mojo | 47 ++++++++++++++--- tests/test_steps.mojo | 6 +-- tests/weekly.mojo | 4 +- tests/weekly.sh | 2 +- 21 files changed, 189 insertions(+), 82 deletions(-) create mode 100644 changes/370-fire.minor.md create mode 100644 tests/compile_fail/fire_and_fills.mojo create mode 100644 tests/compile_fail_keyed/keyed_fire.mojo diff --git a/DESIGN.md b/DESIGN.md index 153cea56..03c84650 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -243,7 +243,7 @@ Gate: a schedule fills each month's partition as it falls due, and a backfill ov ### How it fits together - **A partition key is a type.** `Month` and a new `Day` share a `Key` trait (parse, write, next, compare). `Monthly[start]` and `Daily[start]` are partition sets over them, with the range syntax the command line already takes (`2024-01..2024-03`, comma lists). `Weekly[start]` (#140) is over `Week`, an ISO 8601 week written `YYYY-Www` (`2026-W46`, Monday 2026-11-09 to Sunday 2026-11-15): the year is the ISO week-numbering year, the year of the week's Thursday, so 2024-12-30 is `2025-W01` and 2021-01-03 `2020-W53`, and a year has 52 weeks or 53. Year and week are zero-padded, so weeks order as text, like the other time keys; a week ends at Monday 00:00 on the schedule's clock, and a sensor maps a week to the months it spans and a month to the weeks with a day in it. `Pipeline` takes `partitions=`, and its first step must take that key (a `Mapped` step) or a `Seq` of it (a `Gather` step), checked at compile time like any other edge. A pipeline that leaves it out is `Monthly` with no start, as in phase 1, and can't be scheduled or backfilled from the beginning. -- **A schedule is declared with the pipeline**: `schedule=Cron["0 3 2 * *", "America/New_York"]` and `fills=Previous` (the key before the one the fire falls in) or `Current`. `describe` prints the partitions and the schedule, so the scheduler learns both without linking the pipeline. +- **A schedule is declared with the pipeline**: `schedule=Cron["0 3 2 * *", "America/New_York"]` and `fire=Previous` (the key before the one the fire falls in) or `Current`. `describe` prints the partitions and the schedule, so the scheduler learns both without linking the pipeline. - **Registration.** `morca register ./build/taxi` runs `taxi describe` and stores the binary's path with the DAG in `pipelines` (`binary_path`, and `since`, when its schedule starts counting). The scheduler reads the pipelines it schedules from Postgres, and their schedules, partitions and limits from the stored JSON in SQL; there is no configuration file and no JSON parser. Registering a new build moves the pipeline's queued runs to its registry hash, and its failed runs waiting for a retry that have no step done. A retry of a run with steps done by other code would be refused by the binary's `resume`, so the scheduler fails it instead of starting it, with an error on the run (`runs.error`, shown by ` status ` and the run's page) saying to resume it with the new code; a local run whose process exits before claiming it fails with the exit code and the last line it printed as its error. Only `morca register` moves a pipeline's code: ` run` records a pipeline that isn't registered yet, but leaves a registered one's hash and `describe` alone, and refuses (exit code 2) a binary whose code isn't the registered one's unless it's given `--allow-new-code`; such a run counts as its own code's, so it makes keys done for the pipeline only if its code has the pipeline's current data version (#508). *Verified* by `tests/new_code.sh` and `tests/manual_run.sh`. - **A fire is a row.** `fires` (pipeline, fire time, partition, run) has a unique (pipeline, fire time), and the scheduler inserts the fire and its `queued` run in one transaction, so a scheduler killed and restarted mid-tick doesn't fire twice. Backfills insert queued runs the same way, with trigger `backfill`. - **One live run per partition.** Every path that queues a run (a fire, a sensor, a rebuild's level, the UI's start, a backfill) checks for a live run of the same keys, in the transaction that would queue it and under a lock on the pipeline's row, so two runs never run one partition at once and both append. A fire records the live run as its run, a sensor leaves the keys to it (and records it as the snapshot's run if no other key is left), a rebuild's level waits until it has ended, the UI refuses with the run's ID, and a backfill skips the key. A partition that's done is not protected: a later fire, sensor, rebuild or start runs it again, and its steps append again unless they replace it (`replaces=`, "Replacing partitions"), so `morca register` warns of a schedule that fires more than once for one partition (`0 */6 * * *` on `Daily`). *Verified* by `tests/one_run_per_key.sh` and, for sensors and rebuilds, `tests/sensors.sh`. @@ -260,7 +260,7 @@ Gate: a schedule fills each month's partition as it falls due, and a backfill ov | # | PR | Gate | Size | |---|---|---|---| | 1 | **Partition keys and sets.** The `Key` trait, `Day`, `Monthly`, `Daily`, `partitions=` on `Pipeline`, the first step's input checked against the key, `describe` with partitions, the command line parsing keys of the pipeline's type. | A daily toy pipeline runs over a range of days; a pipeline whose first step takes a `Month` under `Daily` doesn't compile; phase 1's tests pass unchanged. | ~600 | -| 2 | **Cron and time zones.** Cron parsing and next-fire computation in a time zone; a TZif reader with POSIX `TZ` rules; `schedule=`, `fills=` and `catchup=` on `Pipeline`, in `describe`. | Next fires match a table of known ones, including both daylight-saving transitions in `America/New_York` and `Europe/London`; the cron parser and the `TZ` string parser are fuzzed. | ~1,000 | +| 2 | **Cron and time zones.** Cron parsing and next-fire computation in a time zone; a TZif reader with POSIX `TZ` rules; `schedule=`, `fire=` and `catchup=` on `Pipeline`, in `describe`. | Next fires match a table of known ones, including both daylight-saving transitions in `America/New_York` and `Europe/London`; the cron parser and the `TZ` string parser are fuzzed. | ~1,000 | | 3 | **The scheduler.** The `morca` binary: `register`, `tick`, `scheduler` (tick once a second), `backfill`; `fires`, queued runs, starting runs as subprocesses, `max_runs`, run retries with backoff, missed fires and `catchup`. | With `MORCA_NOW` stepped month by month, the toy pipeline's partitions are filled once each, as they fall due; a scheduler killed mid-tick and restarted fires nothing twice; a backfill of 24 months never has more than `max_runs` running. | ~1,200 | | 4 | **The taxi backfill.** The taxi pipeline gets `partitions=Monthly["2011-01"]` and a monthly schedule; the load handles the 2011–2022 files' schemas. | Phase 2's gate, by hand: `morca backfill taxi 2011-01..2024-12` with `max_runs=4` loads every month once, and the schedule then fills the next month when it falls due (with `MORCA_NOW`). | ~300 | @@ -425,7 +425,7 @@ Months and days are the only partition keys so far, and the `Key` trait assumes - **The `Key` trait splits in two.** `Key` keeps what every key has: `parse`, `write_to`, equality and an order (`__lt__`, used for sorting and display only). `TimeKey(Key)` adds `next`, `prev` and `containing`; `Month` and `Day` implement it. Everything that does date arithmetic (ranges, `fills_for`, the scheduler's fires, the sensors' day↔month mapping) asks for `TimeKey`, so using it on a key that isn't a date is a compile error rather than a runtime surprise. - **`Name` is the new key**: a string of 1 to 128 bytes from `[A-Za-z0-9._:/-]`, without two dots in a row, and with a `/` only between other bytes, one at a time (so `owner/name` is a name, and `/a`, `a/` and `a//b` aren't). Since neither a comma nor `..` can appear in one, the command line and the stored `runs.partitions` text stay unambiguous. It orders by bytes. Where a name becomes one segment of a path, an attempt's log directory, it's written with `/` as `%2F` (`morca.name.path_segment`), and the UI's links encode it the same way, so `owner/name`'s page is `/pipelines/

/keys/owner%2Fname`. `/` was added after gharchive.example needed repositories as keys. - **`Keyed[dataset, column]` is the partition set**: `partitions=Keyed["av2.logs", "log_id"]`. Its keys are the distinct values of `column` in the current snapshot of the warehouse dataset `dataset`. Keys live in data rather than in the source because there are thousands of them and they grow, and because "which logs exist" then has snapshots and lineage like anything else. The first step takes a `Name`, or a `Seq[Name]` for a first gather step, checked at compile time as for `Monthly`. `describe` writes `{"kind": "keyed", "dataset": ..., "column": ...}`. -- **Keyed pipelines are refused things that need a calendar**, at compile time: `schedule=` and `fills=`. A keyed pipeline runs by `run`, by `backfill`, or as a sensor of its keys dataset (below). A cron schedule over names has no partition to fill, and guessing at one (all keys? new keys?) is a sensor's job done worse. +- **Keyed pipelines are refused things that need a calendar**, at compile time: `schedule=` and `fire=`. A keyed pipeline runs by `run`, by `backfill`, or as a sensor of its keys dataset (below). A cron schedule over names has no partition to fill, and guessing at one (all keys? new keys?) is a sensor's job done worse. - **` keys`** prints the keys dataset's current snapshot ID and then its keys, one per line, sorted. A value of the column that isn't a name, or is null, isn't a key: it's left out, and `keys` says on stderr how many there were, with the first few, so one odd row doesn't make every key unreadable. Like `sense`, it exists so the scheduler, which links no iceberg, can learn a pipeline's keys. It reads the one column through `morca.tables`, so only binaries with a keyed pipeline import iceberg for it. - **On the command line**, `--partitions` for a keyed pipeline takes a comma-separated list of names, or `@all` for every current key, or `@file:` for a list one per line, which scripts and the labelrefinery side need for samples. A range (`A..B`) is refused with a message saying keyed partitions take lists. A key that isn't in the current keys snapshot is refused, so a typo doesn't make a run of nothing. - **Many keys per run.** A run over a list of keys is already what phase 1 does: a mapped step runs once per key, one after another in the driver, each free to use dista inside, and a gather step runs once over all of them. Keyed pipelines are where that matters, since one run per key would mean 20,000 driver processes and as many dista runtimes. So the pipeline gets `keys_per_run: Int = 1`, and for a keyed pipeline the default to recommend is in the tens to hundreds. Parallelism is then `max_runs` runs at once, each using dista inside its steps. Time-partitioned pipelines keep one key per run by default; the parameter works for them too. @@ -440,12 +440,12 @@ Months and days are the only partition keys so far, and the `Key` trait assumes | # | PR | Gate | Size | |---|---|---|---| -| 1 | **Keys that aren't dates.** The `Key`/`TimeKey` split; `Name`; `Keyed[dataset, column]`; the compile-time refusals (`schedule=`, `fills=`, the first step's key type); ` keys`; `--partitions` lists, `@all` and `@file:`, with ranges and unknown keys refused; `keys_per_run`; `describe` with keyed partitions. | A keyed toy pipeline over a warehouse dataset of 50 names runs `--partitions @all` with a mapped and a gather step; `keys` prints the 50 sorted; a keyed pipeline with `schedule=` and one whose first step takes a `Month` don't compile (`tests/compile_fail/`); `a..b` and an unknown name are refused with exit code 2; phases 1 to 4's tests pass unchanged. | ~700 | +| 1 | **Keys that aren't dates.** The `Key`/`TimeKey` split; `Name`; `Keyed[dataset, column]`; the compile-time refusals (`schedule=`, `fire=`, the first step's key type); ` keys`; `--partitions` lists, `@all` and `@file:`, with ranges and unknown keys refused; `keys_per_run`; `describe` with keyed partitions. | A keyed toy pipeline over a warehouse dataset of 50 names runs `--partitions @all` with a mapped and a gather step; `keys` prints the 50 sorted; a keyed pipeline with `schedule=` and one whose first step takes a `Month` don't compile (`tests/compile_fail/`); `a..b` and an unknown name are refused with exit code 2; phases 1 to 4's tests pass unchanged. | ~700 | | 2 | **Backfills and sensors over keys.** "Done" per key; backfill choosing undone keys and batching them by `keys_per_run`, also for time partitions; the keys dataset sensed for a keyed sensor, and new keys batched into runs; equality mapping across keyed pipelines, and the refusal across kinds at registration; the keyed pipeline page and the per-key page. | The section's gate on a 20,000-key dataset of generated names and a step that sleeps briefly: `morca backfill keyed @all` with `keys_per_run=100` and `max_runs=4` queues 200 runs, never runs more than 4, and leaves every key done once; a backfill run again queues nothing; failing 3 keys and backfilling again queues one run of those 3; appending 500 names to the dataset starts 5 runs of the new names and none of the old; the per-key query is checked against the steps table directly. | ~900 | These need iceberg for the keys dataset, so like the lineage and sensor tests they run in the `data` environment. -As built in PR 1: `Keyed` is in `morca.keyed`, the one module of it that imports iceberg, and each partition set reads its own `--partitions` (`Partitions.parse`), fills a schedule's partition only if it has a calendar (`Partitions.fill`, which `fills_for` calls), and lists its keys only if it's keyed (`Partitions.keys`). `fills=` is refused as `fills=Current`, since the default can't be told from `fills=Previous` written out. A keyed pipeline's registry hash covers its keys dataset and column; a time-partitioned pipeline's is unchanged. The scheduler takes a keyed backfill as a list of names; `@all` there comes with PR 2, through ` keys`. +As built in PR 1: `Keyed` is in `morca.keyed`, the one module of it that imports iceberg, and each partition set reads its own `--partitions` (`Partitions.parse`), picks a fire's partition only if it has a calendar (`Partitions.fill`, which `fills_for` calls), and lists its keys only if it's keyed (`Partitions.keys`). `fire=` is refused as `fire=Current`, since the default can't be told from `fire=Previous` written out. A keyed pipeline's registry hash covers its keys dataset and column; a time-partitioned pipeline's is unchanged. The scheduler takes a keyed backfill as a list of names; `@all` there comes with PR 2, through ` keys`. As built in PR 2: the scheduler reads a keyed pipeline's keys by running ` keys` (`current_keys`), for `@all`, for checking names, for a keyed sensor and for the UI's counts. "Done" is one SQL query (`DONE_SQL`) over `steps` and `runs`, with an index on a step's partition key (migration 7); a pipeline with no mapped step has a key done when a run over it succeeded. (Since #508 the runs counted are those of any code with the current data version, not of the current hash; resuming a run with a different binary is still refused.) A backfill takes a lock on the pipeline's row while it chooses and queues keys, so two at once don't both queue one. A keyed sensor's keys dataset is in `sense`'s output, so a snapshot something else made is polled for like any other, and one a morca pipeline committed is found in `commits`. A sensor's snapshot offers every current key (the keys dataset's) or the names it holds (another keyed pipeline's), and the offered keys that aren't done or live are queued in the transaction that records the snapshot as seen. The refusal across kinds applies to sensors: a keyed sensor reading a dataset a time-partitioned pipeline writes, or the other way round, isn't registered. Its own keys dataset is the exception (added after gharchive.example ran into it): a time-partitioned pipeline may write it, since a snapshot of it offers its keys whoever made it, and `tests/keyed_sensor.sh` checks that a daily pipeline's commit there starts runs of the new keys only, with lineage through it. @@ -624,7 +624,7 @@ The page is written last, from what worked. Writing it is how this phase finds w `magmalake/gharchive.example`, a project of its own made with `pixi shelf add morca`: -1. **`gharchive`**, daily-partitioned (`Daily`, `schedule="0 3 * * *"`, `fills=Previous`): each day's 24 hourly files from data.gharchive.org, read with objectstore's HTTP, parsed as JSON lines and written to `gharchive.events` (Iceberg, partitioned by day). Backfilled from 2023-01-01, a few TB of input. +1. **`gharchive`**, daily-partitioned (`Daily`, `schedule="0 3 * * *"`, `fire=Previous`): each day's 24 hourly files from data.gharchive.org, read with objectstore's HTTP, parsed as JSON lines and written to `gharchive.events` (Iceberg, partitioned by day). Backfilled from 2023-01-01, a few TB of input. 2. **`mojo_repos`**, a sensor on `gharchive.events`: finds Mojo repositories in each day's events and in GitHub's code search (files ending `.mojo` or `.🔥`, and `pixi.toml`/`mojoproject.toml` naming a Mojo package), and adds the new ones to `mojo.repos`. How repositories are found is the example's code, so the page shows it rather than morca hiding it. 3. **`mojo_history`**, keyed by repository (`Keyed["mojo.repos", "repo"]`, `keys_per_run=50`, `sensor=True`): clones each repository as a dista task, reads its history and writes `mojo.commits` and `mojo.migrations`. New repositories start runs by themselves; a daily refresh brings known ones up to date. 4. **The console**, a flare app reading the `mojo.*` tables: libraries by size and age, Mojo versions adopted over time, and a page per repository. diff --git a/changes/370-fire.minor.md b/changes/370-fire.minor.md new file mode 100644 index 00000000..2056f706 --- /dev/null +++ b/changes/370-fire.minor.md @@ -0,0 +1 @@ +- **`fire=` replaces `fills=`**: `Pipeline[..., fire=Previous]` picks the partition a fire runs and `describe` says `schedule.fire`; `fills=` and `schedule.fills` still work for one release. (#370) diff --git a/docs/pipeline-protocol.md b/docs/pipeline-protocol.md index e9f60189..0bd1d089 100644 --- a/docs/pipeline-protocol.md +++ b/docs/pipeline-protocol.md @@ -188,7 +188,7 @@ for a sensor (`src/morca/_internal/registering.mojo:144-156`). It prints one line per **sensed dataset**: each dataset a step reads and no step writes, once, in step order, then a keyed pipeline's keys dataset -(`src/morca/pipeline.mojo:305-325`). Each line is +(`src/morca/pipeline.mojo:327-347`). Each line is ```text @@ -231,7 +231,7 @@ non-empty line as a key. Empty output is an error ### `fires [] [--after