diff --git a/DESIGN.md b/DESIGN.md index 886980a9..153cea56 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -234,7 +234,7 @@ What PR 6 needs from the tins, checked at PR 2, and what it turned out to need: Gate: a schedule fills each month's partition as it falls due, and a backfill over 2011–2024 runs with a bounded number at once. -- **Partition definitions** on the pipeline: `partitions=Monthly["2011-01"]` or `Daily["2024-01-01"]`, so the scheduler knows the set of keys and which are due. +- **Partition definitions** on the pipeline: `partitions=Monthly["2011-01"]`, `Weekly["2026-W40"]`, `Daily["2024-01-01"]` or `Hourly["2026-11-11T00"]`, so the scheduler knows the set of keys and which are due. - **Cron**: five-field expressions with ranges, steps and lists, and a time zone. Time zones are read from the system's TZif files (`/usr/share/zoneinfo`), which is a small, documented format. A schedule says which partition a fire fills, e.g. "at 03:00 on the 2nd, the previous month". - **The `morca` scheduler binary**: registers pipelines from their `describe`, ticks once a second, creates runs that are due, starts them as subprocesses, retries failed runs with exponential backoff and jitter, and enforces `max_runs` per pipeline by counting `running` rows in the same transaction that starts one. - **Backfills**: one run per partition over a range, queued, started `max_runs` at a time. This is also how partitions run in parallel, in place of phase 1's `max_parallel`. @@ -242,7 +242,7 @@ Gate: a schedule fills each month's partition as it falls due, and a backfill ov ### How it fits together -- **A partition key is a type.** `Month` and a new `Day` share a `Key` trait (parse, write, next, compare). `Monthly[start]` and `Daily[start]` are partition sets over them, with the range syntax the command line already takes (`2024-01..2024-03`, comma lists). `Pipeline` takes `partitions=`, and its first step must take that key (a `Mapped` step) or a `Seq` of it (a `Gather` step), checked at compile time like any other edge. A pipeline that leaves it out is `Monthly` with no start, as in phase 1, and can't be scheduled or backfilled from the beginning. +- **A partition key is a type.** `Month` and a new `Day` share a `Key` trait (parse, write, next, compare). `Monthly[start]` and `Daily[start]` are partition sets over them, with the range syntax the command line already takes (`2024-01..2024-03`, comma lists). `Weekly[start]` (#140) is over `Week`, an ISO 8601 week written `YYYY-Www` (`2026-W46`, Monday 2026-11-09 to Sunday 2026-11-15): the year is the ISO week-numbering year, the year of the week's Thursday, so 2024-12-30 is `2025-W01` and 2021-01-03 `2020-W53`, and a year has 52 weeks or 53. Year and week are zero-padded, so weeks order as text, like the other time keys; a week ends at Monday 00:00 on the schedule's clock, and a sensor maps a week to the months it spans and a month to the weeks with a day in it. `Pipeline` takes `partitions=`, and its first step must take that key (a `Mapped` step) or a `Seq` of it (a `Gather` step), checked at compile time like any other edge. A pipeline that leaves it out is `Monthly` with no start, as in phase 1, and can't be scheduled or backfilled from the beginning. - **A schedule is declared with the pipeline**: `schedule=Cron["0 3 2 * *", "America/New_York"]` and `fills=Previous` (the key before the one the fire falls in) or `Current`. `describe` prints the partitions and the schedule, so the scheduler learns both without linking the pipeline. - **Registration.** `morca register ./build/taxi` runs `taxi describe` and stores the binary's path with the DAG in `pipelines` (`binary_path`, and `since`, when its schedule starts counting). The scheduler reads the pipelines it schedules from Postgres, and their schedules, partitions and limits from the stored JSON in SQL; there is no configuration file and no JSON parser. Registering a new build moves the pipeline's queued runs to its registry hash, and its failed runs waiting for a retry that have no step done. A retry of a run with steps done by other code would be refused by the binary's `resume`, so the scheduler fails it instead of starting it, with an error on the run (`runs.error`, shown by ` status ` and the run's page) saying to resume it with the new code; a local run whose process exits before claiming it fails with the exit code and the last line it printed as its error. Only `morca register` moves a pipeline's code: ` run` records a pipeline that isn't registered yet, but leaves a registered one's hash and `describe` alone, and refuses (exit code 2) a binary whose code isn't the registered one's unless it's given `--allow-new-code`; such a run counts as its own code's, so it makes keys done for the pipeline only if its code has the pipeline's current data version (#508). *Verified* by `tests/new_code.sh` and `tests/manual_run.sh`. - **A fire is a row.** `fires` (pipeline, fire time, partition, run) has a unique (pipeline, fire time), and the scheduler inserts the fire and its `queued` run in one transaction, so a scheduler killed and restarted mid-tick doesn't fire twice. Backfills insert queued runs the same way, with trigger `backfill`. diff --git a/changes/512-weekly.minor.md b/changes/512-weekly.minor.md new file mode 100644 index 00000000..18ccb645 --- /dev/null +++ b/changes/512-weekly.minor.md @@ -0,0 +1 @@ +- **Weekly partitions**: `Weekly[start]` partitions a pipeline by ISO week, keys `YYYY-Www` such as `2026-W46` (#512). diff --git a/docs/morca-toml.md b/docs/morca-toml.md index bc51fed9..138442ce 100644 --- a/docs/morca-toml.md +++ b/docs/morca-toml.md @@ -816,7 +816,7 @@ The default, `fills = "since_registration"`, is what morca did before. A schedul On every tick, after the other runs of every pipeline of its [priority](#priority) have started, the scheduler queues and starts fill runs for it: -- **The range** is the pipeline's first partition through the last one that has ended on its schedule's wall clock (UTC without a schedule), at least `fill_lag` ago: yesterday for a daily pipeline, last month for a monthly one. The partition still going is the schedule's. See [The lag](#the-lag). +- **The range** is the pipeline's first partition through the last one that has ended on its schedule's wall clock (UTC without a schedule), at least `fill_lag` ago: yesterday for a daily pipeline, last week for a weekly one, last month for a monthly one. The partition still going is the schedule's. See [The lag](#the-lag). - **`keys_per_run` partitions per fill run** (one by default), in `fill_order`; a run's own partitions are in ascending order. The run's trigger is `fill` (`orca.runs.trigger`), so the runs page can filter for them. - **A partition is taken** when it's done in a run of any of the pipeline's code, when a live run has it, or when a fill run of it was ever queued. A fill never runs a partition twice. A fill run that fails is retried like a scheduled run, up to `run_retries`. A partition whose fill run failed past its retries, or was cancelled, is left to a person: `morca backfill`, or resume in the UI. Running done partitions again stays `morca backfill --rerun`, or [as a fill](#a-backfill-as-a-fill) with `--as-fill`. - **Fill runs don't count against `max_runs`.** At most `fill_max_runs` fill runs are going, beside up to `max_runs` other runs. @@ -825,7 +825,7 @@ On every tick, after the other runs of every pipeline of its [priority](#priorit ### The lag -`fill_lag` (default `0`) holds a partition back for a while after it ends, for a source that publishes its data late. A partition is filled once its end is at least `fill_lag` in the past: a daily day D from D+1 00:00 plus the lag, a monthly month from the 1st of the next month plus the lag, both on the schedule's wall clock (UTC without a schedule). +`fill_lag` (default `0`) holds a partition back for a while after it ends, for a source that publishes its data late. A partition is filled once its end is at least `fill_lag` in the past: a daily day D from D+1 00:00 plus the lag, a weekly (ISO) week from the next Monday 00:00 plus the lag, a monthly month from the 1st of the next month plus the lag, both on the schedule's wall clock (UTC without a schedule). GH Archive puts each hour's file up some minutes after the hour. With no lag, the fill starts the day that just ended at 00:00 UTC, before its last hour is there. Its attempts fail until the file appears, and enough failures in an hour trip the [watchdog](watchdog.md), which pauses the pipeline. With `fill_lag = "2h"` the day is filled from 02:00 UTC. diff --git a/docs/ui.md b/docs/ui.md index 4be5ee3b..c5210d8b 100644 --- a/docs/ui.md +++ b/docs/ui.md @@ -218,11 +218,13 @@ each pipeline of `/` and on `/api/pipelines/`'s `pipeline`, as rounded down to a tenth; the page shows it rounded down to a whole percent, so 100% means all of them). `src/morca/_internal/coverage.mojo` defines it: -- **Hourly, daily or monthly**: partitions done of those in the range so +- **Hourly, daily, weekly or monthly**: partitions done of those in the range so far, from the first partition through the last one that has ended on the schedule's clock (UTC without a schedule): the last hour for an hourly - pipeline, yesterday for a daily one, last month for a monthly one. An - hourly pipeline's Coverage card shows a day of hours (`?cal=YYYY-MM-DD`). That's the range a fill works + pipeline, yesterday for a daily one, last week (ISO, Monday to Sunday) for + a weekly one, last month for a monthly one. An hourly pipeline's Coverage + card shows a day of hours (`?cal=YYYY-MM-DD`), a weekly one's the 52 or 53 + weeks of an ISO year (`?cal=YYYY`). That's the range a fill works through; today's day, still going, is in neither number, nor is one that ended less than the pipeline's [`fill_lag`](morca-toml.md#the-lag) ago. diff --git a/pixi.toml b/pixi.toml index cb02b906..af9ddf46 100644 --- a/pixi.toml +++ b/pixi.toml @@ -203,6 +203,8 @@ test-concurrency = { cmd = "scripts/with-postgres.sh tests/concurrency.sh" } test-one-run-per-key = { cmd = "scripts/with-postgres.sh tests/one_run_per_key.sh" } # Hourly partitions: */5 fires, hour backfills, the day-of-hours calendar. test-hourly = { cmd = "scripts/with-postgres.sh tests/hourly.sh" } +# Weekly (ISO week) partitions: Monday fires, week backfills, the year of weeks. +test-weekly = { cmd = "scripts/with-postgres.sh tests/weekly.sh" } # morca.toml (morca.toml, morca.config): the parser, what register reads from # the file and records, config show and export, and the env on each runner. test-toml = { cmd = "scripts/mojo-build.sh -I src $DEP_INC tests/test_toml.mojo -o build/test-toml && ./build/test-toml" } @@ -286,7 +288,7 @@ test-executor-core = { depends-on = ["build-shared", "test-sample", "test-steps" test-executor-workers = { depends-on = ["build-shared", "test-ec2", "test-autoscale", "test-no-workers", "test-pool", "test-hcloud", "test-pool-scale", "test-pool-scale-ec2", "test-pool-image", "test-run-cluster", "test-secrets", "test-placement", "test-grants", "test-pressure", "test-lost-workers"] } test-executor = { depends-on = ["test-executor-core", "test-executor-workers"] } # The scheduler, the UI, and what a registration and its code control: -test-scheduler-ui = { depends-on = ["build-shared", "test-toml", "test-upgrade", "test-config", "test-fill", "test-fill-version", "test-fill-requests", "test-priority", "test-pool-fills", "test-run-locks", "test-metrics", "test-scheduler", "test-ui", "test-fleet", "test-system", "test-actions", "test-calendar", "test-ui-login", "test-ui-drain", "test-remote", "test-worker-api", "test-pause", "test-older-binary", "test-watchdog", "test-notify", "test-versions", "test-manual-run", "test-new-code", "test-defer", "test-dead-runs", "test-adopt", "test-overlap", "test-outdated", "test-concurrency", "test-one-run-per-key", "test-hourly", "test-sense-errors", "test-stop", "test-binary-timeouts"] } +test-scheduler-ui = { depends-on = ["build-shared", "test-toml", "test-upgrade", "test-config", "test-fill", "test-fill-version", "test-fill-requests", "test-priority", "test-pool-fills", "test-run-locks", "test-metrics", "test-scheduler", "test-ui", "test-fleet", "test-system", "test-actions", "test-calendar", "test-ui-login", "test-ui-drain", "test-remote", "test-worker-api", "test-pause", "test-older-binary", "test-watchdog", "test-notify", "test-versions", "test-manual-run", "test-new-code", "test-defer", "test-dead-runs", "test-adopt", "test-overlap", "test-outdated", "test-concurrency", "test-one-run-per-key", "test-hourly", "test-weekly", "test-sense-errors", "test-stop", "test-binary-timeouts"] } test = { depends-on = ["test-executor", "test-scheduler-ui"] } toy = { cmd = "scripts/mojo-build.sh -I src $DEP_INC examples/toy.mojo -o build/toy" } fanout = { cmd = "scripts/mojo-build.sh -I src $DEP_INC examples/fanout.mojo -o build/fanout" } diff --git a/scripts/pipelines.sh b/scripts/pipelines.sh index 9ac4cf5a..42f56dfa 100755 --- a/scripts/pipelines.sh +++ b/scripts/pipelines.sh @@ -41,6 +41,7 @@ chaos_pipeline:tests/chaos_pipeline.mojo defer:tests/defer_pipeline.mojo hourly:tests/hourly.mojo six-hourly:tests/six_hourly.mojo +weekly:tests/weekly.mojo stop:tests/stop_pipeline.mojo longpoll:tests/outdated_pipeline.mojo longpoll2:tests/outdated_pipeline.mojo:run["longpoll2"] diff --git a/src/morca/__init__.mojo b/src/morca/__init__.mojo index 2954dc92..83e36d1d 100644 --- a/src/morca/__init__.mojo +++ b/src/morca/__init__.mojo @@ -37,9 +37,9 @@ comes with a minor-version note in the changelog (changes/README.md). | `morca.step` | `Step`, `Mapped`, `Gather`, `NoStep`, `StepKind`, `StepLike`, `defer` | | `morca.pipeline` | `Pipeline`, and what a run returns: `RunResult`, `StepRun`, `SUCCEEDED`, `FAILED`, `SKIPPED` | | `morca.ctx` | `Ctx`, what a step is given: its run, partition, attempt, logs and dista runtime | -| `morca.partitions` | `Monthly`, `Daily`, `Hourly`, `Partitions`, `parse_partitions` | +| `morca.partitions` | `Monthly`, `Weekly`, `Daily`, `Hourly`, `Partitions`, `parse_partitions` | | `morca.keyed` | `Keyed`: partitions named by a dataset's column, not dates | -| `morca.month`, `morca.day`, `morca.hour`, `morca.key`, `morca.name` | the partition keys: `Month`, `Day`, `Hour`, `Name` (`Name.text` is the name as a `String`), `Key`, `TimeKey`, and their parsers | +| `morca.month`, `morca.week`, `morca.day`, `morca.hour`, `morca.key`, `morca.name` | the partition keys: `Month`, `Week`, `Day`, `Hour`, `Name` (`Name.text` is the name as a `String`), `Key`, `TimeKey`, and their parsers | | `morca.schedule`, `morca.cron`, `morca.tz` | `Cron`, `Current`, `Previous`, `Fill`, `NoSchedule`, `Schedule`, `ScheduleLike`; `CronExpr`; `Zone`, `PosixTZ` | | `morca.civil` | `Civil`, `now`, `parse_time`, `utc` | | `morca.tasks` | `Tasks`, `TaskSet`, `NoTasks`, `workers`: a step's dista tasks | @@ -81,6 +81,7 @@ from morca.day import Day from morca.hour import Hour from morca.key import Key, TimeKey, parse_keys from morca.month import Month, days_in, parse_months +from morca.week import Week, weeks_in from morca._internal.handover import ( code_version, desired_code_version, @@ -93,6 +94,7 @@ from morca.partitions import ( Hourly, Monthly, Partitions, + Weekly, parse_partitions, ) from morca.pipeline import Pipeline diff --git a/src/morca/_internal/calendar.mojo b/src/morca/_internal/calendar.mojo index 73d16d14..8410a68e 100644 --- a/src/morca/_internal/calendar.mojo +++ b/src/morca/_internal/calendar.mojo @@ -4,7 +4,8 @@ a monthly pipeline, month) is in, read from the database alone. `morca ui` draws it twice, as the picker on the Start runs form and as the read-only coverage on a pipeline's page, and serves it as JSON at `/pipelines//calendar?month=2023-01`. Daily pipelines have a month of -days at a time; monthly ones a year of months; hourly ones a day of hours +days at a time; monthly ones a year of months; weekly ones an ISO year of +its 52 or 53 weeks (`?month=2026`); hourly ones a day of hours (`?month=2023-01-31`). The states, in the order they win: @@ -19,7 +20,8 @@ they win: - `open`: none of the above, so a backfill would queue it; shown as "not done", since it may have been done under an older data version. -Hourly, daily and monthly keys are zero-padded, so they compare as text. +Hourly, daily, weekly and monthly keys are zero-padded, so they compare as +text. """ from std.collections import Dict @@ -32,6 +34,7 @@ from morca.day import Day from morca.hour import Hour from morca._internal.json import quote from morca.month import Month, days_in +from morca.week import Week, weeks_in from morca._internal.queueing import done_keys from morca._internal.state import State from morca._internal.timekeys import unit_of @@ -47,17 +50,19 @@ struct Calendar(Movable): var pipeline: String var kind: String """`daily` (cells are days of one month), `monthly` (months of one - year) or `hourly` (hours of one day).""" + year), `weekly` (the weeks of one ISO year) or `hourly` (hours of one + day).""" var period: String - """`YYYY-MM` for days, `YYYY` for months, `YYYY-MM-DD` for hours.""" + """`YYYY-MM` for days, `YYYY` for months and weeks, `YYYY-MM-DD` for + hours.""" var prev: String """The period before, empty if it ends before the pipeline starts.""" var next: String """The period after, empty if it starts after today.""" var start: String var today: String - """Today's key: `YYYY-MM-DD`, `YYYY-MM` or, for hours, this hour's - `YYYY-MM-DDTHH`.""" + """Today's key: `YYYY-MM-DD`, `YYYY-MM`, this week's `YYYY-Www` or, for + hours, this hour's `YYYY-MM-DDTHH`.""" var weekday: Int """Of a month's first day, with Monday as 0.""" var cells: List[Cell] @@ -117,6 +122,8 @@ def calendar( that are only open are `behind`.""" if kind == "hourly": return _hours(db, pipeline, start, period, upstream) + if kind == "weekly": + return _weeks(db, pipeline, start, period, upstream) if kind != "daily" and kind != "monthly": raise Error(pipeline, " has ", kind, " partitions, not dated ones") var t = Civil.of(now()) @@ -196,6 +203,58 @@ def _hours( return cal^ +def _weeks( + mut db: State, + pipeline: String, + start: String, + period: String, + upstream: Dict[String, Bool], +) raises -> Calendar: + """A weekly pipeline's calendar: the weeks of the ISO year `period` + (`YYYY`, or the ISO year of a `YYYY-MM` month's first day or a + `YYYY-Www` week; this week's if empty).""" + var t = Civil.of(now()) + var cal = Calendar(pipeline, "weekly") + cal.start = start + var today = Week.containing(t.year, t.month, t.day, t.hour) + cal.today = String(today) + var y: Int + if not period: + y = today.year + elif period.byte_length() == 4: + try: + y = Int(period) + except: + raise Error("not a year (YYYY): '", period, "'") + if y < 0: + raise Error("not a year (YYYY): '", period, "'") + elif period.byte_length() == 8: + y = Week.parse(period).year + else: + var m = Month.parse(period) + y = Week.containing(m.year, m.month, 1, 0).year + cal.period = _pad(y, 4) + var keys = List[String]() + for w in range(1, weeks_in(y) + 1): + keys.append(String(Week(y, w))) + cal.prev = _pad(y - 1, 4) if ( + y > 0 and (not start or String(Week(y - 1, weeks_in(y - 1))) >= start) + ) else String() + cal.next = ( + _pad(y + 1, 4) if String(Week(y + 1, 1)) <= cal.today else String() + ) + cal.cells = _cells( + db, + pipeline, + start, + cal.today, + keys, + done_keys(db, pipeline, keys), + upstream, + ) + return cal^ + + def _frame_month(mut cal: Calendar, year: Int, month: Int) -> List[String]: """Sets `cal`'s period, neighbours and weekday for the days of `month` of `year`, from its `start` and `today`, and returns their keys.""" diff --git a/src/morca/_internal/fill.mojo b/src/morca/_internal/fill.mojo index 58578350..575f76d4 100644 --- a/src/morca/_internal/fill.mojo +++ b/src/morca/_internal/fill.mojo @@ -19,7 +19,8 @@ from its first partition through the last one that has ended at least partition's data some time after it ends: GH Archive puts each hour's file up minutes after the hour, so at 00:00 UTC the day that just ended isn't all there yet. A Daily day D fills from D+1 00:00 plus the lag (on the -schedule's clock), a Monthly month from the 1st of the next plus the lag. +schedule's clock), a Weekly week from the next Monday 00:00 plus the lag, a +Monthly month from the 1st of the next plus the lag. Until then the partition is outside the range, for the fill and for the coverage alike (`range_keys`), so it isn't counted as missing; the pipeline page's Fill line says when it fills (`FillStatus.lag_key`). The default, @@ -319,7 +320,10 @@ def fill_refusal( " is keyed: its keys come from its keys dataset, which its", " sensor or `morca backfill ", pipeline, - " @all` runs; fills are for Hourly, Daily and Monthly partitions", + ( + " @all` runs; fills are for Hourly, Daily, Weekly and Monthly" + " partitions" + ), ) if sensor: return String( @@ -329,7 +333,9 @@ def fill_refusal( " may not be there; backfill it by hand", ) if not is_time_kind(kind): - return String(pipeline, " has no Hourly, Daily or Monthly partitions") + return String( + pipeline, " has no Hourly, Daily, Weekly or Monthly partitions" + ) if not start: return String(pipeline, " has no first partition to fill from") return "" diff --git a/src/morca/_internal/registered.mojo b/src/morca/_internal/registered.mojo index 233bca6e..55ce6db7 100644 --- a/src/morca/_internal/registered.mojo +++ b/src/morca/_internal/registered.mojo @@ -20,6 +20,7 @@ from morca._internal.fill import ( from morca.cron import CronExpr from morca.day import Day from morca.hour import Hour +from morca.week import Week from morca._internal.ec2 import Ec2Spec from morca.month import Month from morca.schedule import Schedule @@ -217,6 +218,9 @@ def fill_key(kind: String, fills: String, local: Civil) raises -> String: if kind == "daily": var d = Day.containing(local.year, local.month, local.day, 0) return String(d.prev() if fills == "previous" else d^) + if kind == "weekly": + var w = Week.containing(local.year, local.month, local.day, 0) + return String(w.prev() if fills == "previous" else w^) if kind == "monthly": var m = Month.containing(local.year, local.month, local.day, 0) return String(m.prev() if fills == "previous" else m^) diff --git a/src/morca/_internal/sensors.mojo b/src/morca/_internal/sensors.mojo index 86500356..4a0c34a0 100644 --- a/src/morca/_internal/sensors.mojo +++ b/src/morca/_internal/sensors.mojo @@ -49,16 +49,19 @@ from morca._internal.lineage import flows from morca._internal.db.live import live_runs from morca.month import Month, days_in from morca.name import sort_texts +from morca.week import Week from morca._internal.state import State from morca._internal.timekeys import parse_time_keys from morca._internal.warehouse import Sensed def key_kind(key: StringSlice) -> String: - """`monthly` for a `YYYY-MM` key, `daily` for `YYYY-MM-DD`, `hourly` - for `YYYY-MM-DDTHH`, or empty.""" + """`monthly` for a `YYYY-MM` key, `weekly` for `YYYY-Www`, `daily` for + `YYYY-MM-DD`, `hourly` for `YYYY-MM-DDTHH`, or empty.""" if key.byte_length() == 7: return "monthly" + if key.byte_length() == 8 and key[byte=5] == "W": + return "weekly" if key.byte_length() == 10: return "daily" if key.byte_length() == 13 and key[byte=10] == "T": @@ -82,7 +85,8 @@ def map_keys( """`keys`, of any time kind, as keys of `kind`: a key as the longer one it falls in (an hour as its day or month, a day as its month), or as each shorter one it has (a month as its days or hours, a day as its - hours). Those before `start` are dropped, and the rest are sorted, once + hours). Weeks and months overlap: a week is the months its days fall + in, a month the weeks with a day in it. Those before `start` are dropped, and the rest are sorted, once each.""" var out = List[String]() for k in keys: @@ -94,6 +98,8 @@ def map_keys( mapped.append(String(Day.parse(k))) elif have == kind and have == "hourly": mapped.append(String(Hour.parse(k))) + elif have == "weekly" or kind == "weekly": + mapped = _map_weekly(k, have, kind) elif have == "daily" and kind == "monthly": mapped.append(String(Day.parse(k).month_of())) elif have == "hourly" and kind == "monthly": @@ -127,6 +133,42 @@ def map_keys( return out^ +def _map_weekly(k: String, have: String, kind: String) raises -> List[String]: + """`map_keys` of one key when it or `kind` is weekly.""" + var out = List[String]() + if have == "weekly": + var w = Week.parse(k) + if kind == "weekly": + out.append(String(w)) + elif kind == "monthly": + for m in w.months(): + out.append(String(m)) + elif kind == "daily": + for d in w.days(): + out.append(String(d)) + elif kind == "hourly": + for d in w.days(): + for h in range(24): + out.append(String(Hour(d.year, d.month, d.day, h))) + else: + raise Error("morca: not a partition key: '", k, "'") + elif have == "daily": + var d = Day.parse(k) + out.append(String(Week.containing(d.year, d.month, d.day, 0))) + elif have == "hourly": + var h = Hour.parse(k) + out.append(String(Week.containing(h.year, h.month, h.day, h.hour))) + elif have == "monthly": + var m = Month.parse(k) + for d in range(1, days_in(m.year, m.month) + 1): + var w = String(Week.containing(m.year, m.month, d, 0)) + if w not in out: + out.append(w^) + else: + raise Error("morca: not a partition key: '", k, "'") + return out^ + + def join_keys(keys: List[String]) -> String: var out = String() for i in range(len(keys)): diff --git a/src/morca/_internal/timekeys.mojo b/src/morca/_internal/timekeys.mojo index 2e513bef..84fdd1bf 100644 --- a/src/morca/_internal/timekeys.mojo +++ b/src/morca/_internal/timekeys.mojo @@ -1,21 +1,31 @@ -"""Time keys by their partitions' kind (`monthly`, `daily`, `hourly`), as +"""Time keys by their partitions' kind (`monthly`, `weekly`, `daily`, +`hourly`), as text: for code that has a pipeline's kind as a string, not its type.""" from morca.day import Day from morca.hour import Hour from morca.key import parse_keys from morca.month import Month +from morca.week import Week def is_time_kind(kind: StringSlice) -> Bool: - """Whether `kind` is a calendar's: `monthly`, `daily` or `hourly`.""" - return kind == "monthly" or kind == "daily" or kind == "hourly" + """Whether `kind` is a calendar's: `monthly`, `weekly`, `daily` or + `hourly`.""" + return ( + kind == "monthly" + or kind == "weekly" + or kind == "daily" + or kind == "hourly" + ) def unit_of(kind: StringSlice) -> String: - """`month`, `day` or `hour`; `key` for any other kind.""" + """`month`, `week`, `day` or `hour`; `key` for any other kind.""" if kind == "monthly": return "month" + if kind == "weekly": + return "week" if kind == "daily": return "day" if kind == "hourly": @@ -29,6 +39,8 @@ def example_of(kind: StringSlice) -> String: return "2024-01-01T00" if kind == "daily": return "2024-01-01" + if kind == "weekly": + return "2024-W01" return "2024-01" @@ -44,6 +56,9 @@ def parse_time_keys( elif kind == "daily": for k in parse_keys[Day](s, first): out.append(String(k)) + elif kind == "weekly": + for k in parse_keys[Week](s, first): + out.append(String(k)) else: for k in parse_keys[Month](s, first): out.append(String(k)) @@ -58,6 +73,8 @@ def containing_key( return String(Hour(year, month, day, hour)) if kind == "daily": return String(Day(year, month, day)) + if kind == "weekly": + return String(Week.containing(year, month, day, hour)) return String(Month(year, month)) @@ -67,4 +84,6 @@ def prev_key(kind: StringSlice, key: StringSlice) raises -> String: return String(Hour.parse(key).prev()) if kind == "daily": return String(Day.parse(key).prev()) + if kind == "weekly": + return String(Week.parse(key).prev()) return String(Month.parse(key).prev()) diff --git a/src/morca/_internal/ui/coverage.mojo b/src/morca/_internal/ui/coverage.mojo index 2480c10a..f9229a33 100644 --- a/src/morca/_internal/ui/coverage.mojo +++ b/src/morca/_internal/ui/coverage.mojo @@ -7,8 +7,9 @@ a strip of every month of its range linking to their years). Both are in the page; the links work without scripts (`?cov=month|year`, with `?cal=` the month or year), and with them the tabs switch in place and the choice is kept in `localStorage`, as the Steps tabs are. Year is the default. A -monthly pipeline's card is its year of months, as before, and an hourly -one's a day of hours (`?cal=YYYY-MM-DD`). For a sensor, +monthly pipeline's card is its year of months, as before, a weekly one's +an ISO year of weeks (`?cal=YYYY`), and an hourly one's a day of hours +(`?cal=YYYY-MM-DD`). For a sensor, the days what it follows has but it hasn't done, failed or started are `behind` (`morca._internal.follow`), and `follow_html` is its Progress card's Following line. @@ -33,6 +34,7 @@ from morca._internal.ui.layout import act_button from morca.day import Day from morca.hour import Hour from morca.month import Month, days_in +from morca.week import Week from morca._internal.views.commands import shell_quote from morca._internal.state import State @@ -73,7 +75,7 @@ def state_words(state: String) -> String: def _cal_title(c: Calendar) raises -> String: - if c.kind == "monthly" or c.kind == "hourly": + if c.kind == "monthly" or c.kind == "hourly" or c.kind == "weekly": return c.period var y = c.period[byte=:4] var m = Int(c.period[byte=5:7]) @@ -170,13 +172,17 @@ def _month_html( view: String, follows: List[String], ): - """A month of days (a year of months for a monthly pipeline, a day of - hours for an hourly one), as the card has always drawn it.""" + """A month of days (a year of months for a monthly pipeline, of weeks + for a weekly one, a day of hours for an hourly one), as the card has + always drawn it.""" var daily = c.kind == "daily" var hourly = c.kind == "hourly" + var weekly = c.kind == "weekly" h.raw('
String: nxt = String(Day.parse(keys[j]).next()) elif unit == "hour": nxt = String(Hour.parse(keys[j]).next()) + elif unit == "week": + nxt = String(Week.parse(keys[j]).next()) else: nxt = String(Month.parse(keys[j]).next()) if nxt != keys[j + 1]: diff --git a/src/morca/key.mojo b/src/morca/key.mojo index 643924ac..3f3f259e 100644 --- a/src/morca/key.mojo +++ b/src/morca/key.mojo @@ -1,7 +1,7 @@ """`Key`: what a partition key is, whatever it counts. A key is written as text (`2024-01`, `2024-01-31`, `log-0f3a`), read back -by `parse`, and ordered, for sorting and display. `Month`, `Day` and +by `parse`, and ordered, for sorting and display. `Month`, `Week`, `Day` and `Hour` are `TimeKey`s too: each has a next and a previous key, and the key a date falls in, which ranges, schedules' fills and sensors' day-to-month mapping need. `Name` (`morca.name`) is a key that isn't a date, so asking it for its diff --git a/src/morca/partitions.mojo b/src/morca/partitions.mojo index ec3cd10b..99c9b62c 100644 --- a/src/morca/partitions.mojo +++ b/src/morca/partitions.mojo @@ -5,12 +5,13 @@ comptime Taxi = Pipeline[ "taxi", Download, Load, Compact, partitions=Monthly["2011-01"] ] comptime Weather = Pipeline["weather", Fetch, partitions=Daily["2020-01-01"]] +comptime Stats = Pipeline["stats", Count, partitions=Weekly["2026-W40"]] comptime Poll = Pipeline["poll", Poll, partitions=Hourly["2026-11-11T00"]] comptime Labels = Pipeline["labels", Label, partitions=Keyed["av2.logs", "log_id"]] ``` -A pipeline's first step takes its key (`Month` for `Monthly`, `Day` for -`Daily`, `Hour` for `Hourly`, `Name` for `morca.keyed.Keyed`), or a `Seq` of them for a first +A pipeline's first step takes its key (`Month` for `Monthly`, `Week` for +`Weekly` (ISO weeks, `2026-W46`), `Day` for `Daily`, `Hour` for `Hourly`, `Name` for `morca.keyed.Keyed`), or a `Seq` of them for a first gather step. `start` is the first partition: keys before it are refused, and phase 2's scheduler and backfills count from it. A pipeline without `partitions=` is `Monthly[]`, with no start. @@ -25,6 +26,7 @@ from morca.day import Day from morca.hour import Hour from morca.key import Key, TimeKey, parse_keys from morca.month import Month +from morca.week import Week trait Partitions: @@ -33,7 +35,8 @@ trait Partitions: @staticmethod def kind() -> StaticString: - """`monthly`, `daily`, `hourly` or `keyed`, as `describe` writes it.""" + """`monthly`, `weekly`, `daily`, `hourly` or `keyed`, as `describe` writes it. + """ ... @staticmethod @@ -116,6 +119,43 @@ struct Monthly[first: StaticString = ""](Partitions): raise Error("morca: monthly partitions have no keys dataset") +struct Weekly[first: StaticString = ""](Partitions): + """A partition per ISO week, Monday to Sunday, from `first` (`YYYY-Www`) + if given.""" + + comptime K = Week + + @staticmethod + def kind() -> StaticString: + return "weekly" + + @staticmethod + def start() -> StaticString: + return Self.first + + @staticmethod + def dataset() -> StaticString: + return "" + + @staticmethod + def column() -> StaticString: + return "" + + @staticmethod + def parse(s: StringSlice, check: Bool) raises -> List[Week]: + return parse_keys[Week](s, Self.first) + + @staticmethod + def fill( + year: Int, month: Int, day: Int, hour: Int, previous: Bool + ) raises -> Week: + return _fill[Week](year, month, day, hour, previous) + + @staticmethod + def keys() raises -> Tuple[Int64, List[String]]: + raise Error("morca: weekly partitions have no keys dataset") + + struct Daily[first: StaticString = ""](Partitions): """A partition per day, from `first` (`YYYY-MM-DD`) if given.""" diff --git a/src/morca/week.mojo b/src/morca/week.mojo new file mode 100644 index 00000000..4b49aab8 --- /dev/null +++ b/src/morca/week.mojo @@ -0,0 +1,121 @@ +"""`Week`: the partition key of a weekly pipeline. + +A week is an ISO 8601 week, Monday to Sunday, written `YYYY-Www` +(`2026-W46`, the week from Monday 2026-11-09 to Sunday 2026-11-15), and a +range of them `2026-W01..2026-W52`, both ends included. `YYYY` is the ISO +week-numbering year, the year of the week's Thursday: week 1 is the week +with the year's first Thursday (the one with January 4th), so 2024-12-30 is +in `2025-W01` and 2021-01-03 in `2020-W53`. A year has 52 weeks, or 53 when +it starts on a Thursday, or is a leap year starting on a Wednesday. The year +and week are zero-padded, so weeks order as text. +""" + +from morca.civil import Civil, days_from_civil +from morca.day import Day +from morca.key import TimeKey +from morca.month import Month + + +def _iso_weekday(days: Int64) -> Int: + """0 for Monday to 6 for Sunday, of a day `days` after 1970-01-01 (a + Thursday).""" + return Int((days + 3) % 7) + + +def weeks_in(year: Int) -> Int: + """The number of ISO weeks in the ISO year `year`: 52 or 53.""" + # December 28th is always in the year's last week. + return Week.containing(year, 12, 28, 0).week + + +@fieldwise_init +struct Week(TimeKey): + """An ISO week, `week` from 1 to `weeks_in(year)` of ISO year `year`.""" + + var year: Int + var week: Int + + @staticmethod + def parse(s: StringSlice) raises -> Week: + """Reads `YYYY-Www`.""" + var parts = s.split("-W") + if ( + len(parts) != 2 + or parts[0].byte_length() != 4 + or parts[1].byte_length() != 2 + ): + raise Error("not a week (YYYY-Www): '", s, "'") + var w: Week + try: + w = Week(Int(parts[0]), Int(parts[1])) + except: + raise Error("not a week (YYYY-Www): '", s, "'") + if w.year < 0 or w.week < 1 or w.week > weeks_in(w.year): + raise Error("not a week (YYYY-Www): '", s, "'") + return w^ + + def next(self) -> Week: + """The week after this one.""" + if self.week < weeks_in(self.year): + return Week(self.year, self.week + 1) + return Week(self.year + 1, 1) + + def prev(self) -> Week: + """The week before this one.""" + if self.week > 1: + return Week(self.year, self.week - 1) + return Week(self.year - 1, weeks_in(self.year - 1)) + + @staticmethod + def containing(year: Int, month: Int, day: Int, hour: Int) -> Week: + var days = days_from_civil(year, month, day) + # The week's Thursday: its ISO year is the week's. + var thursday = days - Int64(_iso_weekday(days)) + 3 + var y = Civil.of(thursday * 86400).year + var w = Int((thursday - days_from_civil(y, 1, 1)) // 7) + 1 + return Week(y, w) + + def monday(self) -> Day: + """The week's first day.""" + var jan4 = days_from_civil(self.year, 1, 4) + var d = jan4 - Int64(_iso_weekday(jan4)) + Int64(7 * (self.week - 1)) + var c = Civil.of(d * 86400) + return Day(c.year, c.month, c.day) + + def days(self) -> List[Day]: + """The week's seven days, Monday first.""" + var out = List[Day]() + var d = self.monday() + for _ in range(7): + out.append(d.copy()) + d = d.next() + return out^ + + def months(self) -> List[Month]: + """The one or two months the week's days fall in, in order.""" + var out = List[Month]() + for d in self.days(): + var m = d.month_of() + if len(out) == 0 or not (out[len(out) - 1] == m): + out.append(m^) + return out^ + + def __eq__(self, other: Self) -> Bool: + return self.year == other.year and self.week == other.week + + def __lt__(self, other: Self) -> Bool: + if self.year != other.year: + return self.year < other.year + return self.week < other.week + + def write_to(self, mut writer: Some[Writer]): + if self.year < 1000: + writer.write("0") + if self.year < 100: + writer.write("0") + if self.year < 10: + writer.write("0") + writer.write(self.year, "-W") + if self.week < 10: + writer.write("0") + writer.write(self.week) diff --git a/tests/sense_errors.sh b/tests/sense_errors.sh index b1cfdb64..6e2dfc6c 100755 --- a/tests/sense_errors.sh +++ b/tests/sense_errors.sh @@ -2,7 +2,7 @@ # A sensor whose datasets hold something it can't run doesn't stop # (tests/fake_sensor stands in for its binary, so no warehouse is needed): # -# - a snapshot whose morca.partitions isn't a partition key (a week) is +# - a snapshot whose morca.partitions isn't a partition key (a quarter) is # recorded as seen with no run and why, and a snapshot of another dataset # seen in the same tick still starts its run; # - the next ticks start nothing more, and say nothing more of it; @@ -35,16 +35,16 @@ tick() { ./build/morca tick >> "$log" 2>&1 || fail "a tick failed"; } printf 'raw.a 1 other 2024-01\nraw.b -1 other -\n' > "$FAKE_SENSE" ./build/morca register ./tests/fake_sensor >> "$log" 2>&1 || fail "register" -# raw.a gets a snapshot another writer says holds a week, raw.b a month. -printf 'raw.a 2 other 2024-W12\nraw.b 3 other 2024-03\n' > "$FAKE_SENSE" +# raw.a gets a snapshot another writer says holds a quarter (no partition key), raw.b a month. +printf 'raw.a 2 other 2024-Q01\nraw.b 3 other 2024-03\n' > "$FAKE_SENSE" tick [ "$(q "SELECT partitions FROM orca.runs WHERE pipeline = 'fake_sensor'")" = 2024-03 ] \ || fail "raw.b@3 started no run for 2024-03: $(q "SELECT id, partitions FROM orca.runs")" -[ "$(q "SELECT run IS NULL AND error LIKE '%not a partition key: ''2024-W12''%' FROM orca.sensed WHERE dataset = 'raw.a' AND snapshot_id = 2")" = t ] \ +[ "$(q "SELECT run IS NULL AND error LIKE '%not a partition key: ''2024-Q01''%' FROM orca.sensed WHERE dataset = 'raw.a' AND snapshot_id = 2")" = t ] \ || fail "raw.a@2 isn't seen with why: $(q "SELECT * FROM orca.sensed")" [ "$(q "SELECT run IS NOT NULL AND error IS NULL FROM orca.sensed WHERE dataset = 'raw.b' AND snapshot_id = 3")" = t ] \ || fail "raw.b@3 isn't seen with its run" -grep -q "sensed raw.a@2 for fake_sensor and ran nothing: morca: not a partition key: '2024-W12'" "$log" \ +grep -q "sensed raw.a@2 for fake_sensor and ran nothing: morca: not a partition key: '2024-Q01'" "$log" \ || fail "the unusable snapshot isn't said" tick tick diff --git a/tests/test_fill.mojo b/tests/test_fill.mojo index 7a348972..6c68c46c 100644 --- a/tests/test_fill.mojo +++ b/tests/test_fill.mojo @@ -112,6 +112,35 @@ def test_range_hourly() raises: assert_equal(fill_refusal("hourly", "2024-01-01T00", False, "p"), "") +def test_range_weekly() raises: + # Wednesday 2026-11-18 is in 2026-W47: W46 is the last that ended. + var t = parse_time("2026-11-18T10:00Z") + assert_equal(last_ended("weekly", "UTC", t), "2026-W46") + var weeks = range_keys("weekly", "2026-W44", "UTC", t) + assert_equal(len(weeks), 3) + assert_equal(weeks[0], "2026-W44") + assert_equal(weeks[2], "2026-W46") + # Across New Year, through a 53-week year. + var ny = range_keys( + "weekly", "2026-W52", "UTC", parse_time("2027-01-12T00:00Z") + ) + assert_equal(len(ny), 3) + assert_equal(ny[1], "2026-W53") + assert_equal(ny[2], "2027-W01") + # A week ends at Monday 00:00 on the schedule's clock: 03:00 UTC is + # still Sunday in New York. + var mon = parse_time("2026-11-16T03:00Z") + assert_equal(last_ended("weekly", "UTC", mon), "2026-W46") + assert_equal(last_ended("weekly", "America/New_York", mon), "2026-W45") + # A lag of 2 hours: W46 ended an hour ago and waits until 02:00. + var early = parse_time("2026-11-16T01:00Z") + assert_equal(last_ended("weekly", "UTC", early, 7200), "2026-W45") + var w = lag_wait("weekly", "UTC", early, 7200) + assert_equal(w[0], "2026-W46") + assert_equal(w[1], parse_time("2026-11-16T02:00Z")) + assert_equal(fill_refusal("weekly", "2026-W01", False, "p"), "") + + def test_lag_values() raises: assert_equal(parse_fill_lag("0"), 0) assert_equal(parse_fill_lag("2h"), 7200) diff --git a/tests/test_steps.mojo b/tests/test_steps.mojo index b458603a..520fe3f8 100644 --- a/tests/test_steps.mojo +++ b/tests/test_steps.mojo @@ -1,4 +1,4 @@ -"""Months, days, pipelines, describe and the in-memory executor.""" +"""Months, weeks, days, pipelines, describe and the in-memory executor.""" from std.testing import ( TestSuite, @@ -25,11 +25,14 @@ from morca import ( Name, Pipeline, Step, + Week, + Weekly, parse_months, parse_names, parse_partitions, ) from morca.name import path_segment +from morca.week import weeks_in # --- names --- @@ -209,6 +212,103 @@ def test_hour_ranges_and_fills() raises: ) +# --- weeks --- + + +def test_week_parse_and_write() raises: + var w = Week.parse("2026-W46") + assert_equal(w.year, 2026) + assert_equal(w.week, 46) + assert_equal(String(w), "2026-W46") + assert_equal(String(w.monday()), "2026-11-09") + assert_equal(String(w.next()), "2026-W47") + assert_equal(String(w.prev()), "2026-W45") + assert_true(Week.parse("2026-W09") < Week.parse("2026-W10")) + assert_true(Week.parse("2025-W52") < Week.parse("2026-W01")) + for bad in [ + "2026-W00", + "2026-W54", + "2024-W53", # 2024 has 52 weeks + "2026-W1", + "2026W46", + "2026-46", + "2026-11", + "26-W46", + "2026-Wxx", + ]: + with assert_raises(contains="not a week"): + _ = Week.parse(bad) + + +def test_iso_week_year_boundaries() raises: + # 53 weeks in a year that starts on a Thursday (2015, 2026), or a leap + # year that starts on a Wednesday (2020); 52 otherwise. + assert_equal(weeks_in(2015), 53) + assert_equal(weeks_in(2020), 53) + assert_equal(weeks_in(2026), 53) + assert_equal(weeks_in(2024), 52) + assert_equal(weeks_in(2025), 52) + assert_equal(weeks_in(2027), 52) + # Days near New Year in the other ISO year's weeks. + assert_equal(String(Week.containing(2024, 12, 30, 0)), "2025-W01") + assert_equal(String(Week.containing(2024, 12, 29, 23)), "2024-W52") + assert_equal(String(Week.containing(2021, 1, 3, 0)), "2020-W53") + assert_equal(String(Week.containing(2021, 1, 4, 0)), "2021-W01") + assert_equal(String(Week.containing(2026, 1, 1, 0)), "2026-W01") + assert_equal(String(Week.containing(2027, 1, 1, 0)), "2026-W53") + assert_equal(String(Week.containing(2027, 1, 3, 0)), "2026-W53") + assert_equal(String(Week.containing(2027, 1, 4, 0)), "2027-W01") + assert_equal(String(Week.containing(2010, 1, 3, 0)), "2009-W53") + assert_equal(String(Week.containing(2008, 12, 29, 0)), "2009-W01") + assert_equal(String(Week.containing(1970, 1, 1, 0)), "1970-W01") + assert_equal(String(Week.parse("2026-W01").monday()), "2025-12-29") + assert_equal(String(Week.parse("2026-W53").monday()), "2026-12-28") + assert_equal(String(Week.parse("2020-W53").next()), "2021-W01") + assert_equal(String(Week.parse("2024-W52").next()), "2025-W01") + assert_equal(String(Week.parse("2021-W01").prev()), "2020-W53") + assert_equal(String(Week.parse("2025-W01").prev()), "2024-W52") + # Every day of 2000-2040 is in the week its Monday is, and consecutive + # weeks' Mondays are seven days apart. + var d = Day(2000, 1, 1) + var last = Week.containing(2000, 1, 1, 0) + while d.year < 2041: + var wk = Week.containing(d.year, d.month, d.day, 0) + if not (wk == last): + assert_equal(String(last.next()), String(wk)) + assert_equal(String(wk.monday()), String(d)) + last = wk.copy() + d = d.next() + var ms = Week.parse("2026-W53").months() + assert_equal(len(ms), 2) + assert_equal(String(ms[0]), "2026-12") + assert_equal(String(ms[1]), "2027-01") + + +def test_week_ranges_and_fills() raises: + var ws = parse_partitions[Weekly[]]("2020-W52..2021-W02") + assert_equal(len(ws), 4) + assert_equal(String(ws[1]), "2020-W53") + assert_equal(String(ws[2]), "2021-W01") + assert_equal(len(parse_partitions[Weekly[]]("2024-W01..2024-W52")), 52) + assert_equal(len(parse_partitions[Weekly[]]("2015-W01..2015-W53")), 53) + with assert_raises(contains="before the pipeline's first partition"): + _ = parse_partitions[Weekly["2026-W40"]]("2026-W39..2026-W41") + # A fire on Monday 2026-11-16 06:00 fills the week that just ended + # with fills=Previous, the one it falls in with Current. + assert_equal( + String(Weekly[].fill(2026, 11, 16, 6, previous=True)), "2026-W46" + ) + assert_equal( + String(Weekly[].fill(2026, 11, 16, 6, previous=False)), "2026-W47" + ) + assert_equal( + String(Weekly[].fill(2026, 11, 15, 23, previous=False)), "2026-W46" + ) + assert_equal( + String(Weekly[].fill(2027, 1, 4, 0, previous=True)), "2026-W53" + ) + + def test_keys_before_the_start_are_refused() raises: assert_equal( len(parse_partitions[Monthly["2011-01"]]("2011-01..2011-03")), 3 @@ -318,6 +418,27 @@ def test_an_hourly_pipeline() raises: assert_true('"input": "morca.hour.Hour"' in d, d) +def week_of(mut ctx: Ctx, w: Week) raises -> Int: + return w.week + + +comptime WeekOf = Step["week", Week, Int, week_of, Mapped] +comptime Weeks = Pipeline["weeks", WeekOf, partitions=Weekly["2024-W01"]] + + +def test_a_weekly_pipeline() raises: + var res = Weeks.run( + parse_partitions[Weeks.partitions]("2024-W51..2025-W02") + ) + assert_true(res.succeeded()) + assert_equal(len(res.steps), 4) + assert_equal(res.output[Int]("week", "2024-W52"), 52) + assert_equal(res.output[Int]("week", "2025-W01"), 1) + var d = Weeks.describe() + assert_true('"partitions": {"kind": "weekly", "start": "2024-W01"}' in d, d) + assert_true('"input": "morca.week.Week"' in d, d) + + def test_a_daily_pipeline() raises: var ds = parse_partitions[Days.partitions]("2024-02-27..2024-03-02") var res = Days.run(ds) diff --git a/tests/test_warehouse.mojo b/tests/test_warehouse.mojo index 010aa8f5..f64d091a 100644 --- a/tests/test_warehouse.mojo +++ b/tests/test_warehouse.mojo @@ -142,6 +142,35 @@ def test_parse_sense() raises: _ = parse_sense("a.b twelve other -\n") +def test_map_weekly_keys() raises: + assert_equal(key_kind("2026-W46"), "weekly") + assert_equal(key_kind("2026-X46"), "") + assert_equal(join_keys(map_keys(["2024-12-31"], "weekly")), "2025-W01") + assert_equal(join_keys(map_keys(["2026-11-15T23"], "weekly")), "2026-W46") + assert_equal( + join_keys(map_keys(["2026-11-09", "2026-11-15"], "weekly")), + "2026-W46", + ) + # A month is the weeks with a day in it, a week the months it spans. + assert_equal( + join_keys(map_keys(["2026-02"], "weekly")), + "2026-W05,2026-W06,2026-W07,2026-W08,2026-W09", + ) + assert_equal( + join_keys(map_keys(["2026-02"], "weekly", "2026-W07")), + "2026-W07,2026-W08,2026-W09", + ) + assert_equal( + join_keys(map_keys(["2026-W53"], "monthly")), "2026-12,2027-01" + ) + var days = map_keys(["2026-W46"], "daily") + assert_equal(len(days), 7) + assert_equal(days[0], "2026-11-09") + assert_equal(days[6], "2026-11-15") + assert_equal(len(map_keys(["2026-W46"], "hourly")), 7 * 24) + assert_equal(join_keys(map_keys(["2026-W46"], "weekly")), "2026-W46") + + def test_map_hourly_keys() raises: assert_equal(key_kind("2024-02-29T07"), "hourly") assert_equal(key_kind("2024-02-29 07"), "") diff --git a/tests/weekly.mojo b/tests/weekly.mojo new file mode 100644 index 00000000..955c7fc5 --- /dev/null +++ b/tests/weekly.mojo @@ -0,0 +1,35 @@ +"""A weekly pipeline (ISO weeks) run at 06:00 every Monday, filling the week +that just ended (`fills=Previous`), the step returning its week number +(tests/weekly.sh). +""" + +from morca import ( + Cron, + Ctx, + Mapped, + Pipeline, + Previous, + Step, + Week, + Weekly, + morca_main, +) + + +def count(mut ctx: Ctx, w: Week) raises -> Int: + return w.week + + +comptime Count = Step["count", Week, Int, count, Mapped] +comptime Mondays = Pipeline[ + "weekly", + Count, + partitions=Weekly["2026-W40"], + schedule=Cron["0 6 * * 1"], + fills=Previous, + catchup=False, +] + + +def main() raises: + morca_main[Mondays]() diff --git a/tests/weekly.sh b/tests/weekly.sh new file mode 100755 index 00000000..0c832a31 --- /dev/null +++ b/tests/weekly.sh @@ -0,0 +1,82 @@ +#!/bin/sh +# Weekly partitions (morca.week, morca.partitions.Weekly): a Monday 06:00 +# schedule with fills=Previous runs the ISO week that just ended; backfills +# take week ranges; the coverage calendar is an ISO year of weeks. Needs +# MORCA_DATABASE_URL (pixi run test-weekly starts a throwaway server), curl +# and jq. +set -eu +cd "$(dirname "$0")/.." +. "$PWD/tests/port.sh" +mkdir -p build +scripts/pipelines.sh weekly +# shellcheck disable=SC2086 +scripts/mojo-build.sh -I src $DEP_INC bin/scheduler.mojo -o build/morca + +log=build/weekly.log +: > "$log" +ui= +fail() { + echo "FAIL: $*" + tail -30 "$log" + [ -z "$ui" ] || kill "$ui" 2> /dev/null || true + exit 1 +} +q() { psql "$MORCA_DATABASE_URL" -qAtc "$1"; } +settle() { + i=0 + while [ "$(q "SELECT count(*) FROM orca.runs WHERE status IN ('queued', 'running')")" -gt 0 ]; do + MORCA_NOW="$1" ./build/morca tick >> "$log" 2>&1 + i=$((i + 1)); [ "$i" -lt 300 ] || fail "runs didn't finish"; sleep 0.2 + done +} +runs() { q "SELECT string_agg(partitions, ' ' ORDER BY id) FROM orca.runs WHERE pipeline = 'weekly'"; } + +out=$(MORCA_NOW=2026-11-16T05:00Z ./build/morca register ./build/weekly 2>&1) || fail "register: $out" +echo "$out" >> "$log" +[ "$(q "SELECT describe->'partitions'->>'kind' FROM orca.pipelines WHERE name = 'weekly'")" = weekly ] \ + || fail "not registered as weekly" + +# Monday 2026-11-16 06:00 fills the week that just ended, 2026-W46. +MORCA_NOW=2026-11-16T06:00Z ./build/morca tick >> "$log" 2>&1 || fail "tick W47 Monday" +[ "$(runs)" = "2026-W46" ] || fail "the first Monday's run: $(runs)" +settle 2026-11-16T06:00Z +[ "$(q "SELECT status FROM orca.runs WHERE pipeline = 'weekly'")" = succeeded ] || fail "the run didn't succeed" + +# Nothing more that week; the next Monday fills W47. +MORCA_NOW=2026-11-19T12:00Z ./build/morca tick >> "$log" 2>&1 || fail "tick Thursday" +[ "$(runs)" = "2026-W46" ] || fail "a run mid-week: $(runs)" +MORCA_NOW=2026-11-23T06:00Z ./build/morca tick >> "$log" 2>&1 || fail "tick W48 Monday" +settle 2026-11-23T06:00Z +[ "$(runs)" = "2026-W46 2026-W47" ] || fail "runs: $(runs)" + +# A backfill over weeks skips the done ones and refuses weeks before the start. +MORCA_NOW=2026-11-23T07:00Z ./build/morca backfill weekly 2026-W43..2026-W47 >> "$log" 2>&1 \ + || fail "backfill" +[ "$(runs)" = "2026-W46 2026-W47 2026-W43 2026-W44 2026-W45" ] \ + || fail "backfill runs: $(runs)" +settle 2026-11-23T07:00Z +if MORCA_NOW=2026-11-23T07:00Z ./build/morca backfill weekly 2026-W39 >> "$log" 2>&1; then + fail "a week before the start was backfilled" +fi + +# The UI's coverage: the 53 weeks of ISO year 2026. +free_port port +base="http://127.0.0.1:$port" +MORCA_NOW=2026-11-23T07:00Z MORCA_UI_ADDR="127.0.0.1:$port" ./build/morca ui >> "$log" 2>&1 & +ui=$! +i=0 +until curl -fs "$base/" > /dev/null 2>&1; do + i=$((i + 1)); [ "$i" -lt 100 ] || fail "morca ui didn't start"; sleep 0.1 +done +cal=$(curl -fs "$base/pipelines/weekly/calendar?month=2026") || fail "calendar" +echo "$cal" | jq -e '.unit == "week" and .today == "2026-W48" and (.cells | length) == 53 + and .cells[52].key == "2026-W53" and .next == null and .prev == null' > /dev/null || fail "calendar: $cal" +echo "$cal" | jq -e '[.cells[] | select(.state == "done") | .key] + == ["2026-W43", "2026-W44", "2026-W45", "2026-W46", "2026-W47"]' > /dev/null \ + || fail "done weeks: $cal" +page=$(curl -fs "$base/pipelines/weekly?tab=operate") || fail "pipeline page" +echo "$page" | grep -q 'Weeks of weekly' || fail "no weeks on the coverage card" +kill "$ui" +wait "$ui" 2> /dev/null || true +ui= +echo "ok weekly: Monday fires run the week that ended, backfills take week ranges, coverage is a year of weeks"