Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -234,15 +234,15 @@ What PR 6 needs from the tins, checked at PR 2, and what it turned out to need:

Gate: a schedule fills each month's partition as it falls due, and a backfill over 2011–2024 runs with a bounded number at once.

- **Partition definitions** on the pipeline: `partitions=Monthly["2011-01"]` or `Daily["2024-01-01"]`, so the scheduler knows the set of keys and which are due.
- **Partition definitions** on the pipeline: `partitions=Monthly["2011-01"]`, `Weekly["2026-W40"]`, `Daily["2024-01-01"]` or `Hourly["2026-11-11T00"]`, so the scheduler knows the set of keys and which are due.
- **Cron**: five-field expressions with ranges, steps and lists, and a time zone. Time zones are read from the system's TZif files (`/usr/share/zoneinfo`), which is a small, documented format. A schedule says which partition a fire fills, e.g. "at 03:00 on the 2nd, the previous month".
- **The `morca` scheduler binary**: registers pipelines from their `describe`, ticks once a second, creates runs that are due, starts them as subprocesses, retries failed runs with exponential backoff and jitter, and enforces `max_runs` per pipeline by counting `running` rows in the same transaction that starts one.
- **Backfills**: one run per partition over a range, queued, started `max_runs` at a time. This is also how partitions run in parallel, in place of phase 1's `max_parallel`.
- **Missed fires** after the scheduler was down are caught up once each, not dropped and not fired many times; a pipeline can say `catchup=False`.

### How it fits together

- **A partition key is a type.** `Month` and a new `Day` share a `Key` trait (parse, write, next, compare). `Monthly[start]` and `Daily[start]` are partition sets over them, with the range syntax the command line already takes (`2024-01..2024-03`, comma lists). `Pipeline` takes `partitions=`, and its first step must take that key (a `Mapped` step) or a `Seq` of it (a `Gather` step), checked at compile time like any other edge. A pipeline that leaves it out is `Monthly` with no start, as in phase 1, and can't be scheduled or backfilled from the beginning.
- **A partition key is a type.** `Month` and a new `Day` share a `Key` trait (parse, write, next, compare). `Monthly[start]` and `Daily[start]` are partition sets over them, with the range syntax the command line already takes (`2024-01..2024-03`, comma lists). `Weekly[start]` (#140) is over `Week`, an ISO 8601 week written `YYYY-Www` (`2026-W46`, Monday 2026-11-09 to Sunday 2026-11-15): the year is the ISO week-numbering year, the year of the week's Thursday, so 2024-12-30 is `2025-W01` and 2021-01-03 `2020-W53`, and a year has 52 weeks or 53. Year and week are zero-padded, so weeks order as text, like the other time keys; a week ends at Monday 00:00 on the schedule's clock, and a sensor maps a week to the months it spans and a month to the weeks with a day in it. `Pipeline` takes `partitions=`, and its first step must take that key (a `Mapped` step) or a `Seq` of it (a `Gather` step), checked at compile time like any other edge. A pipeline that leaves it out is `Monthly` with no start, as in phase 1, and can't be scheduled or backfilled from the beginning.
- **A schedule is declared with the pipeline**: `schedule=Cron["0 3 2 * *", "America/New_York"]` and `fills=Previous` (the key before the one the fire falls in) or `Current`. `describe` prints the partitions and the schedule, so the scheduler learns both without linking the pipeline.
- **Registration.** `morca register ./build/taxi` runs `taxi describe` and stores the binary's path with the DAG in `pipelines` (`binary_path`, and `since`, when its schedule starts counting). The scheduler reads the pipelines it schedules from Postgres, and their schedules, partitions and limits from the stored JSON in SQL; there is no configuration file and no JSON parser. Registering a new build moves the pipeline's queued runs to its registry hash, and its failed runs waiting for a retry that have no step done. A retry of a run with steps done by other code would be refused by the binary's `resume`, so the scheduler fails it instead of starting it, with an error on the run (`runs.error`, shown by `<binary> status <run>` and the run's page) saying to resume it with the new code; a local run whose process exits before claiming it fails with the exit code and the last line it printed as its error. Only `morca register` moves a pipeline's code: `<binary> run` records a pipeline that isn't registered yet, but leaves a registered one's hash and `describe` alone, and refuses (exit code 2) a binary whose code isn't the registered one's unless it's given `--allow-new-code`; such a run counts as its own code's, so it makes keys done for the pipeline only if its code has the pipeline's current data version (#508). *Verified* by `tests/new_code.sh` and `tests/manual_run.sh`.
- **A fire is a row.** `fires` (pipeline, fire time, partition, run) has a unique (pipeline, fire time), and the scheduler inserts the fire and its `queued` run in one transaction, so a scheduler killed and restarted mid-tick doesn't fire twice. Backfills insert queued runs the same way, with trigger `backfill`.
Expand Down
1 change: 1 addition & 0 deletions changes/512-weekly.minor.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
- **Weekly partitions**: `Weekly[start]` partitions a pipeline by ISO week, keys `YYYY-Www` such as `2026-W46` (#512).
4 changes: 2 additions & 2 deletions docs/morca-toml.md
Original file line number Diff line number Diff line change
Expand Up @@ -816,7 +816,7 @@ The default, `fills = "since_registration"`, is what morca did before. A schedul

On every tick, after the other runs of every pipeline of its [priority](#priority) have started, the scheduler queues and starts fill runs for it:

- **The range** is the pipeline's first partition through the last one that has ended on its schedule's wall clock (UTC without a schedule), at least `fill_lag` ago: yesterday for a daily pipeline, last month for a monthly one. The partition still going is the schedule's. See [The lag](#the-lag).
- **The range** is the pipeline's first partition through the last one that has ended on its schedule's wall clock (UTC without a schedule), at least `fill_lag` ago: yesterday for a daily pipeline, last week for a weekly one, last month for a monthly one. The partition still going is the schedule's. See [The lag](#the-lag).
- **`keys_per_run` partitions per fill run** (one by default), in `fill_order`; a run's own partitions are in ascending order. The run's trigger is `fill` (`orca.runs.trigger`), so the runs page can filter for them.
- **A partition is taken** when it's done in a run of any of the pipeline's code, when a live run has it, or when a fill run of it was ever queued. A fill never runs a partition twice. A fill run that fails is retried like a scheduled run, up to `run_retries`. A partition whose fill run failed past its retries, or was cancelled, is left to a person: `morca backfill`, or resume in the UI. Running done partitions again stays `morca backfill --rerun`, or [as a fill](#a-backfill-as-a-fill) with `--as-fill`.
- **Fill runs don't count against `max_runs`.** At most `fill_max_runs` fill runs are going, beside up to `max_runs` other runs.
Expand All @@ -825,7 +825,7 @@ On every tick, after the other runs of every pipeline of its [priority](#priorit

### The lag

`fill_lag` (default `0`) holds a partition back for a while after it ends, for a source that publishes its data late. A partition is filled once its end is at least `fill_lag` in the past: a daily day D from D+1 00:00 plus the lag, a monthly month from the 1st of the next month plus the lag, both on the schedule's wall clock (UTC without a schedule).
`fill_lag` (default `0`) holds a partition back for a while after it ends, for a source that publishes its data late. A partition is filled once its end is at least `fill_lag` in the past: a daily day D from D+1 00:00 plus the lag, a weekly (ISO) week from the next Monday 00:00 plus the lag, a monthly month from the 1st of the next month plus the lag, both on the schedule's wall clock (UTC without a schedule).

GH Archive puts each hour's file up some minutes after the hour. With no lag, the fill starts the day that just ended at 00:00 UTC, before its last hour is there. Its attempts fail until the file appears, and enough failures in an hour trip the [watchdog](watchdog.md), which pauses the pipeline. With `fill_lag = "2h"` the day is filled from 02:00 UTC.

Expand Down
8 changes: 5 additions & 3 deletions docs/ui.md
Original file line number Diff line number Diff line change
Expand Up @@ -218,11 +218,13 @@ each pipeline of `/` and on `/api/pipelines/<name>`'s `pipeline`, as
rounded down to a tenth; the page shows it rounded down to a whole
percent, so 100% means all of them). `src/morca/_internal/coverage.mojo` defines it:

- **Hourly, daily or monthly**: partitions done of those in the range so
- **Hourly, daily, weekly or monthly**: partitions done of those in the range so
far, from the first partition through the last one that has ended on the
schedule's clock (UTC without a schedule): the last hour for an hourly
pipeline, yesterday for a daily one, last month for a monthly one. An
hourly pipeline's Coverage card shows a day of hours (`?cal=YYYY-MM-DD`). That's the range a fill works
pipeline, yesterday for a daily one, last week (ISO, Monday to Sunday) for
a weekly one, last month for a monthly one. An hourly pipeline's Coverage
card shows a day of hours (`?cal=YYYY-MM-DD`), a weekly one's the 52 or 53
weeks of an ISO year (`?cal=YYYY`). That's the range a fill works
through; today's day, still going, is in neither number, nor is one
that ended less than the pipeline's
[`fill_lag`](morca-toml.md#the-lag) ago.
Expand Down
4 changes: 3 additions & 1 deletion pixi.toml
Original file line number Diff line number Diff line change
Expand Up @@ -203,6 +203,8 @@ test-concurrency = { cmd = "scripts/with-postgres.sh tests/concurrency.sh" }
test-one-run-per-key = { cmd = "scripts/with-postgres.sh tests/one_run_per_key.sh" }
# Hourly partitions: */5 fires, hour backfills, the day-of-hours calendar.
test-hourly = { cmd = "scripts/with-postgres.sh tests/hourly.sh" }
# Weekly (ISO week) partitions: Monday fires, week backfills, the year of weeks.
test-weekly = { cmd = "scripts/with-postgres.sh tests/weekly.sh" }
# morca.toml (morca.toml, morca.config): the parser, what register reads from
# the file and records, config show and export, and the env on each runner.
test-toml = { cmd = "scripts/mojo-build.sh -I src $DEP_INC tests/test_toml.mojo -o build/test-toml && ./build/test-toml" }
Expand Down Expand Up @@ -286,7 +288,7 @@ test-executor-core = { depends-on = ["build-shared", "test-sample", "test-steps"
test-executor-workers = { depends-on = ["build-shared", "test-ec2", "test-autoscale", "test-no-workers", "test-pool", "test-hcloud", "test-pool-scale", "test-pool-scale-ec2", "test-pool-image", "test-run-cluster", "test-secrets", "test-placement", "test-grants", "test-pressure", "test-lost-workers"] }
test-executor = { depends-on = ["test-executor-core", "test-executor-workers"] }
# The scheduler, the UI, and what a registration and its code control:
test-scheduler-ui = { depends-on = ["build-shared", "test-toml", "test-upgrade", "test-config", "test-fill", "test-fill-version", "test-fill-requests", "test-priority", "test-pool-fills", "test-run-locks", "test-metrics", "test-scheduler", "test-ui", "test-fleet", "test-system", "test-actions", "test-calendar", "test-ui-login", "test-ui-drain", "test-remote", "test-worker-api", "test-pause", "test-older-binary", "test-watchdog", "test-notify", "test-versions", "test-manual-run", "test-new-code", "test-defer", "test-dead-runs", "test-adopt", "test-overlap", "test-outdated", "test-concurrency", "test-one-run-per-key", "test-hourly", "test-sense-errors", "test-stop", "test-binary-timeouts"] }
test-scheduler-ui = { depends-on = ["build-shared", "test-toml", "test-upgrade", "test-config", "test-fill", "test-fill-version", "test-fill-requests", "test-priority", "test-pool-fills", "test-run-locks", "test-metrics", "test-scheduler", "test-ui", "test-fleet", "test-system", "test-actions", "test-calendar", "test-ui-login", "test-ui-drain", "test-remote", "test-worker-api", "test-pause", "test-older-binary", "test-watchdog", "test-notify", "test-versions", "test-manual-run", "test-new-code", "test-defer", "test-dead-runs", "test-adopt", "test-overlap", "test-outdated", "test-concurrency", "test-one-run-per-key", "test-hourly", "test-weekly", "test-sense-errors", "test-stop", "test-binary-timeouts"] }
test = { depends-on = ["test-executor", "test-scheduler-ui"] }
toy = { cmd = "scripts/mojo-build.sh -I src $DEP_INC examples/toy.mojo -o build/toy" }
fanout = { cmd = "scripts/mojo-build.sh -I src $DEP_INC examples/fanout.mojo -o build/fanout" }
Expand Down
1 change: 1 addition & 0 deletions scripts/pipelines.sh
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ chaos_pipeline:tests/chaos_pipeline.mojo
defer:tests/defer_pipeline.mojo
hourly:tests/hourly.mojo
six-hourly:tests/six_hourly.mojo
weekly:tests/weekly.mojo
stop:tests/stop_pipeline.mojo
longpoll:tests/outdated_pipeline.mojo
longpoll2:tests/outdated_pipeline.mojo:run["longpoll2"]
Expand Down
6 changes: 4 additions & 2 deletions src/morca/__init__.mojo
Original file line number Diff line number Diff line change
Expand Up @@ -37,9 +37,9 @@ comes with a minor-version note in the changelog (changes/README.md).
| `morca.step` | `Step`, `Mapped`, `Gather`, `NoStep`, `StepKind`, `StepLike`, `defer` |
| `morca.pipeline` | `Pipeline`, and what a run returns: `RunResult`, `StepRun`, `SUCCEEDED`, `FAILED`, `SKIPPED` |
| `morca.ctx` | `Ctx`, what a step is given: its run, partition, attempt, logs and dista runtime |
| `morca.partitions` | `Monthly`, `Daily`, `Hourly`, `Partitions`, `parse_partitions` |
| `morca.partitions` | `Monthly`, `Weekly`, `Daily`, `Hourly`, `Partitions`, `parse_partitions` |
| `morca.keyed` | `Keyed`: partitions named by a dataset's column, not dates |
| `morca.month`, `morca.day`, `morca.hour`, `morca.key`, `morca.name` | the partition keys: `Month`, `Day`, `Hour`, `Name` (`Name.text` is the name as a `String`), `Key`, `TimeKey`, and their parsers |
| `morca.month`, `morca.week`, `morca.day`, `morca.hour`, `morca.key`, `morca.name` | the partition keys: `Month`, `Week`, `Day`, `Hour`, `Name` (`Name.text` is the name as a `String`), `Key`, `TimeKey`, and their parsers |
| `morca.schedule`, `morca.cron`, `morca.tz` | `Cron`, `Current`, `Previous`, `Fill`, `NoSchedule`, `Schedule`, `ScheduleLike`; `CronExpr`; `Zone`, `PosixTZ` |
| `morca.civil` | `Civil`, `now`, `parse_time`, `utc` |
| `morca.tasks` | `Tasks`, `TaskSet`, `NoTasks`, `workers`: a step's dista tasks |
Expand Down Expand Up @@ -81,6 +81,7 @@ from morca.day import Day
from morca.hour import Hour
from morca.key import Key, TimeKey, parse_keys
from morca.month import Month, days_in, parse_months
from morca.week import Week, weeks_in
from morca._internal.handover import (
code_version,
desired_code_version,
Expand All @@ -93,6 +94,7 @@ from morca.partitions import (
Hourly,
Monthly,
Partitions,
Weekly,
parse_partitions,
)
from morca.pipeline import Pipeline
Expand Down
71 changes: 65 additions & 6 deletions src/morca/_internal/calendar.mojo
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,8 @@ a monthly pipeline, month) is in, read from the database alone.
`morca ui` draws it twice, as the picker on the Start runs form and as the
read-only coverage on a pipeline's page, and serves it as JSON at
`/pipelines/<name>/calendar?month=2023-01`. Daily pipelines have a month of
days at a time; monthly ones a year of months; hourly ones a day of hours
days at a time; monthly ones a year of months; weekly ones an ISO year of
its 52 or 53 weeks (`?month=2026`); hourly ones a day of hours
(`?month=2023-01-31`). The states, in the order
they win:

Expand All @@ -19,7 +20,8 @@ they win:
- `open`: none of the above, so a backfill would queue it; shown as "not
done", since it may have been done under an older data version.

Hourly, daily and monthly keys are zero-padded, so they compare as text.
Hourly, daily, weekly and monthly keys are zero-padded, so they compare as
text.
"""

from std.collections import Dict
Expand All @@ -32,6 +34,7 @@ from morca.day import Day
from morca.hour import Hour
from morca._internal.json import quote
from morca.month import Month, days_in
from morca.week import Week, weeks_in
from morca._internal.queueing import done_keys
from morca._internal.state import State
from morca._internal.timekeys import unit_of
Expand All @@ -47,17 +50,19 @@ struct Calendar(Movable):
var pipeline: String
var kind: String
"""`daily` (cells are days of one month), `monthly` (months of one
year) or `hourly` (hours of one day)."""
year), `weekly` (the weeks of one ISO year) or `hourly` (hours of one
day)."""
var period: String
"""`YYYY-MM` for days, `YYYY` for months, `YYYY-MM-DD` for hours."""
"""`YYYY-MM` for days, `YYYY` for months and weeks, `YYYY-MM-DD` for
hours."""
var prev: String
"""The period before, empty if it ends before the pipeline starts."""
var next: String
"""The period after, empty if it starts after today."""
var start: String
var today: String
"""Today's key: `YYYY-MM-DD`, `YYYY-MM` or, for hours, this hour's
`YYYY-MM-DDTHH`."""
"""Today's key: `YYYY-MM-DD`, `YYYY-MM`, this week's `YYYY-Www` or, for
hours, this hour's `YYYY-MM-DDTHH`."""
var weekday: Int
"""Of a month's first day, with Monday as 0."""
var cells: List[Cell]
Expand Down Expand Up @@ -117,6 +122,8 @@ def calendar(
that are only open are `behind`."""
if kind == "hourly":
return _hours(db, pipeline, start, period, upstream)
if kind == "weekly":
return _weeks(db, pipeline, start, period, upstream)
if kind != "daily" and kind != "monthly":
raise Error(pipeline, " has ", kind, " partitions, not dated ones")
var t = Civil.of(now())
Expand Down Expand Up @@ -196,6 +203,58 @@ def _hours(
return cal^


def _weeks(
mut db: State,
pipeline: String,
start: String,
period: String,
upstream: Dict[String, Bool],
) raises -> Calendar:
"""A weekly pipeline's calendar: the weeks of the ISO year `period`
(`YYYY`, or the ISO year of a `YYYY-MM` month's first day or a
`YYYY-Www` week; this week's if empty)."""
var t = Civil.of(now())
var cal = Calendar(pipeline, "weekly")
cal.start = start
var today = Week.containing(t.year, t.month, t.day, t.hour)
cal.today = String(today)
var y: Int
if not period:
y = today.year
elif period.byte_length() == 4:
try:
y = Int(period)
except:
raise Error("not a year (YYYY): '", period, "'")
if y < 0:
raise Error("not a year (YYYY): '", period, "'")
elif period.byte_length() == 8:
y = Week.parse(period).year
else:
var m = Month.parse(period)
y = Week.containing(m.year, m.month, 1, 0).year
cal.period = _pad(y, 4)
var keys = List[String]()
for w in range(1, weeks_in(y) + 1):
keys.append(String(Week(y, w)))
cal.prev = _pad(y - 1, 4) if (
y > 0 and (not start or String(Week(y - 1, weeks_in(y - 1))) >= start)
) else String()
cal.next = (
_pad(y + 1, 4) if String(Week(y + 1, 1)) <= cal.today else String()
)
cal.cells = _cells(
db,
pipeline,
start,
cal.today,
keys,
done_keys(db, pipeline, keys),
upstream,
)
return cal^


def _frame_month(mut cal: Calendar, year: Int, month: Int) -> List[String]:
"""Sets `cal`'s period, neighbours and weekday for the days of `month`
of `year`, from its `start` and `today`, and returns their keys."""
Expand Down
Loading
Loading