Repository navigation
morca-sdk: whole pipelines in Rust, with no Mojo toolchain (#521) - #533
Merged
Merged
Conversation
A Rust crate, rust/morca-sdk, that builds a whole pipeline (steps, partitions, schedule, retries, Iceberg tables) into a pipeline binary the prebuilt morca scheduler registers and runs, speaking protocol 1 for default and control placement: describe, resume, sense, keys, keep-log, put-log, on Postgres and SQLite, with the heartbeat, the watchdog, ownership checks, the commit fence, summary keys, find_commit and pinned reads. The tables feature reuses the iceberg-rust spike's catalog (#520) on iceberg 0.11.0 and arrow 58. examples/toy-rust is examples/toy.mojo rebuilt in Rust. tests/rust_sdk.sh runs the conformance suite on it and on a Rust sensor with tables over SQLite and Postgres, then runs it beside the Mojo toy under one scheduler; CI runs it in the rust-worker job. The suite's describe.morca check now accepts a binary that leaves `morca` out and says `protocol`, as the protocol doc's Versions section allows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017TycszSbyw92Efu4ZeCYL6
…the bad input of a gather Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017TycszSbyw92Efu4ZeCYL6
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #521.
A Rust crate,
rust/morca-sdk, that builds a whole pipeline into a pipeline binary the prebuiltmorcascheduler registers and runs. No Mojo compiler, no Mojo runtime:examples/toy-rust(examples/toy.mojo rebuilt) is a 3.5 MB binary that links only libSystem on macOS.What's in it
mapped/gather), partitions (monthly, weekly, daily, hourly, keyed), cron schedule, run retries and backoff,max_runs,keys_per_run, sensor, source. The builder is typed, so each step's input type must be the step before it's output, checked at compile time.build()checks the rest at start-up (names, datasets, start key, sensor, keyed dataset). Edge types are checked at run time: outputs are stored as{"type": <Rust type name>, "value": ...}and decoded only into the same type.&mut Ctx: log, progress, metric, secret,stopping()(the outdated flag), plus tables. These are the calls morca-worker'sRunContexthas. Panics fail the attempt.defer(seconds, why)defers it.db/runs.mojo,db/steps.mojoanddb/sqlite_runs.mojo. It covers claim, version, interrupt, prior, ownership checks at every attempt, the commit fence (half the stale limit), owned-run retries forMORCA_DB_RETRY_SECONDS, and finish/defer. The heartbeat runs on its own thread and connection, with the watchdog (timeouts on both stores, due cancels on Postgres, audit row,_exit(1)) and the outdated query. SQLite gets its schema made by the binary, the same migrations andschema_versionas Mojo's. On Postgres the binary checks the columns it uses and leaves migrations to the scheduler, as the doc says.tablesfeature):read(pinned, recorded),append(find_commit first, fenced,orca.*summary keys, recorded),ensure_table,arrow_schema,find_commit, plussenseandkeys. The catalog is the spike's (Spike: iceberg-rust against morca's catalog, summary keys and pinned reads #520)MorcaCatalog, with its one-readdir and key-prefix listing.describe(compact JSON,pipelinefirst,"protocol": 1, nomorca),resume <run> [--allow-new-code],sense,keys,keep-log(with crash detection for Mojo dumps and Rust panics),put-log,sample(runsmorca samplefrom the PATH). Exit codes are 0/1/2/3 as in the doc.changes/521-morca-sdk.minor.md.Tests
pixi run -e rust test-rust-sdk(tests/rust_sdk.sh), added to the existingrust-workerCI job (the rust env has conda-forge rust 1.100, Mojo and Postgres):cargo test, with and withouttables.tests/conformance/toy-rust.jsonfor the toy, which gives 49 ok and 1 skip (sqlite/cancel.due, the doc's rule).tests/conformance/clean-rust.jsonforrust/morca-sdk/examples/clean.rs, a Rust twin of tests/sensors/clean.mojo, which gives 61 ok and the same 1 skip. That coverssense,tables.commits,tables.reads,tables.crash_resume,tables.retry_finds_commitandtables.pinnedon both stores. Its setup commits withexamples/source.rs, which uses no Mojo.local) and the Rust toy (--placement control) are registered with onemorca scheduler. Both fire for 2024-01 from the same schedule, then a backfill of 2024-02..04 each runs: 8 runs, all succeeded. The control run'sdriver.logis kept by the Rust binary'skeep-log.morca status,/pipelines/<p>(JSON) and/runslist both.One suite change:
describe.morcanow passes a binary that leavesmorcaout and saysprotocol, as the doc's Versions section allows ("A binary not built with morca-mojo saysprotocoland leavesmorcaout"). Mojo binaries are checked as before.Mojo registry hashes are unchanged: toy 0xa3bcc571ad7c6939, daily 0x81ead348d0a21c1f, fanout 0x1f0ebf1f2a93d993.
Not in v1 (docs/rust-pipelines.md lists these)
MORCA_STATE_URL(worker pools),ec2, pools and Kubernetes. dista tasks are out of scope per the issue (Local mode on macOS: forked workers crash after libcurl initialised (S3 warehouse) #135).replaces=: iceberg-rust 0.11.0 has no overwrite/delete transaction, so steps only append.MORCA_LOGS,keep-logandput-log, which take local paths only. Tables on S3 do work, through the spike's catalog, but this PR doesn't test them on MinIO.NoTls.run,statusandfires, which are for people only.🤖 Generated with Claude Code
https://claude.ai/code/session_017TycszSbyw92Efu4ZeCYL6