Skip to content

morca-sdk: whole pipelines in Rust, with no Mojo toolchain (#521) - #533

Merged
winding-lines merged 2 commits into
mainfrom
morca-sdk-521
Nov 17, 2026
Merged

winding-lines merged 2 commits into
mainfrom
morca-sdk-521

Conversation

@winding-lines

Copy link
Copy Markdown
Contributor

Closes #521.

A Rust crate, rust/morca-sdk, that builds a whole pipeline into a pipeline binary the prebuilt morca scheduler registers and runs. No Mojo compiler, no Mojo runtime: examples/toy-rust (examples/toy.mojo rebuilt) is a 3.5 MB binary that links only libSystem on macOS.

What's in it

  • Defining a pipeline: name, steps in order (mapped/gather), partitions (monthly, weekly, daily, hourly, keyed), cron schedule, run retries and backoff, max_runs, keys_per_run, sensor, source. The builder is typed, so each step's input type must be the step before it's output, checked at compile time. build() checks the rest at start-up (names, datasets, start key, sensor, keyed dataset). Edge types are checked at run time: outputs are stored as {"type": <Rust type name>, "value": ...} and decoded only into the same type.
  • Steps take &mut Ctx: log, progress, metric, secret, stopping() (the outdated flag), plus tables. These are the calls morca-worker's RunContext has. Panics fail the attempt. defer(seconds, why) defers it.
  • Run store: Postgres and SQLite, with the statements of db/runs.mojo, db/steps.mojo and db/sqlite_runs.mojo. It covers claim, version, interrupt, prior, ownership checks at every attempt, the commit fence (half the stale limit), owned-run retries for MORCA_DB_RETRY_SECONDS, and finish/defer. The heartbeat runs on its own thread and connection, with the watchdog (timeouts on both stores, due cancels on Postgres, audit row, _exit(1)) and the outdated query. SQLite gets its schema made by the binary, the same migrations and schema_version as Mojo's. On Postgres the binary checks the columns it uses and leaves migrations to the scheduler, as the doc says.
  • Lineage (tables feature): read (pinned, recorded), append (find_commit first, fenced, orca.* summary keys, recorded), ensure_table, arrow_schema, find_commit, plus sense and keys. The catalog is the spike's (Spike: iceberg-rust against morca's catalog, summary keys and pinned reads #520) MorcaCatalog, with its one-readdir and key-prefix listing.
  • Subcommands: describe (compact JSON, pipeline first, "protocol": 1, no morca), resume <run> [--allow-new-code], sense, keys, keep-log (with crash detection for Mojo dumps and Rust panics), put-log, sample (runs morca sample from the PATH). Exit codes are 0/1/2/3 as in the doc.
  • Arrow 58, because iceberg-rust 0.11.0 is built on it and the batches a step hands iceberg must be the same major version. morca-worker's arrow 60 never meets it, since a worker is a cdylib Mojo loads and a morca-sdk binary loads none. The comment is in Cargo.toml and docs/rust-pipelines.md.
  • Docs: docs/rust-pipelines.md (guide and gaps), rust/morca-sdk/README.md, README links, changes/521-morca-sdk.minor.md.

Tests

pixi run -e rust test-rust-sdk (tests/rust_sdk.sh), added to the existing rust-worker CI job (the rust env has conda-forge rust 1.100, Mojo and Postgres):

  1. cargo test, with and without tables.
  2. The Rust toy builds with cargo alone and links no Mojo library.
  3. Conformance: tests/conformance/toy-rust.json for the toy, which gives 49 ok and 1 skip (sqlite/cancel.due, the doc's rule). tests/conformance/clean-rust.json for rust/morca-sdk/examples/clean.rs, a Rust twin of tests/sensors/clean.mojo, which gives 61 ok and the same 1 skip. That covers sense, tables.commits, tables.reads, tables.crash_resume, tables.retry_finds_commit and tables.pinned on both stores. Its setup commits with examples/source.rs, which uses no Mojo.
  4. Side by side: the Mojo toy (local) and the Rust toy (--placement control) are registered with one morca scheduler. Both fire for 2024-01 from the same schedule, then a backfill of 2024-02..04 each runs: 8 runs, all succeeded. The control run's driver.log is kept by the Rust binary's keep-log. morca status, /pipelines/<p> (JSON) and /runs list both.

One suite change: describe.morca now passes a binary that leaves morca out and says protocol, as the doc's Versions section allows ("A binary not built with morca-mojo says protocol and leaves morca out"). Mojo binaries are checked as before.

Mojo registry hashes are unchanged: toy 0xa3bcc571ad7c6939, daily 0x81ead348d0a21c1f, fanout 0x1f0ebf1f2a93d993.

Not in v1 (docs/rust-pipelines.md lists these)

  • MORCA_STATE_URL (worker pools), ec2, pools and Kubernetes. dista tasks are out of scope per the issue (Local mode on macOS: forked workers crash after libcurl initialised (S3 warehouse) #135).
  • replaces=: iceberg-rust 0.11.0 has no overwrite/delete transaction, so steps only append.
  • Object storage for MORCA_LOGS, keep-log and put-log, which take local paths only. Tables on S3 do work, through the spike's catalog, but this PR doesn't test them on MinIO.
  • Postgres over TLS: the client connects with NoTls.
  • run, status and fires, which are for people only.
  • Resuming a Mojo build's run under Rust code, since the outputs are encoded differently.

🤖 Generated with Claude Code

https://claude.ai/code/session_017TycszSbyw92Efu4ZeCYL6

winding-lines and others added 2 commits November 16, 2026 23:09
A Rust crate, rust/morca-sdk, that builds a whole pipeline (steps,
partitions, schedule, retries, Iceberg tables) into a pipeline binary the
prebuilt morca scheduler registers and runs, speaking protocol 1 for
default and control placement: describe, resume, sense, keys, keep-log,
put-log, on Postgres and SQLite, with the heartbeat, the watchdog,
ownership checks, the commit fence, summary keys, find_commit and pinned
reads. The tables feature reuses the iceberg-rust spike's catalog (#520)
on iceberg 0.11.0 and arrow 58.

examples/toy-rust is examples/toy.mojo rebuilt in Rust. tests/rust_sdk.sh
runs the conformance suite on it and on a Rust sensor with tables over
SQLite and Postgres, then runs it beside the Mojo toy under one scheduler;
CI runs it in the rust-worker job. The suite's describe.morca check now
accepts a binary that leaves `morca` out and says `protocol`, as the
protocol doc's Versions section allows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017TycszSbyw92Efu4ZeCYL6
…the bad input of a gather

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017TycszSbyw92Efu4ZeCYL6
@winding-lines
winding-lines merged commit 56c5a00 into main Nov 17, 2026
35 checks passed
@winding-lines
winding-lines deleted the morca-sdk-521 branch November 17, 2026 07:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

morca-sdk: whole pipelines in Rust, with no Mojo toolchain

1 participant