A pipeline step can be written in Rust: a worker library, loaded into the
run's process, that implements the interface below. Each Rust worker comes
with a small proxy pipeline in Mojo, which declares the pipeline, its
partitions, and the dataset the step writes as for any step. The step calls
morca.rust_step, which runs the worker and writes the record batches it
streams back the way morca writes any step's output: Parquet, then one
Iceberg commit that a retry recognises. A worker that writes Parquet
itself hands over files instead, or as well, and they go into the same
commit.
proxy pipeline (Mojo) worker (Rust cdylib)
───────────────────── ────────────────────
Step[..., writes=["demo.numbers"]]
ensure_table(ctx, NUMBERS, Schema.parse(SCHEMA))
rust_step(ctx, "numbers_worker", t) ──▶ morca_worker_run(ctx, NULL, &out)
check the stream's schema ◀── out: ArrowArrayStream
pull, align, write Parquet ◀── get_next, one batch at a time
one commit: append_files
schemas/numbers.json, read by both
examples/rust-worker is a complete one, run in
CI by pixi run -e rust test-rust-worker (tests/rust_worker.sh), and
examples/hindcast a real one: a Rust worker that
fetches and decodes NOAA's GFS forecasts with crates.io crates, then Mojo
steps that score them (pixi run -e rust test-hindcast). The
morca-worker crate, in rust/morca-worker, is the
Rust side: a worker author writes only safe Rust.
This is ABI version 1. What isn't in it yet is at the end.
# Cargo.toml
[lib]
crate-type = ["cdylib", "rlib"] # rlib too, for cargo test
[dependencies]
morca-worker = { git = "https://github.com/morca-dev/morca", rev = "<morca's commit>" }With a schemas/numbers.json of one long column, id:
use std::sync::Arc;
use morca_worker::arrow_array::{Int64Array, RecordBatch};
use morca_worker::{export_worker, schema, Input, Output, Result, RunContext, Worker};
const SCHEMA: &str = include_str!("../schemas/numbers.json");
struct Numbers;
impl Worker for Numbers {
const NAME: &'static str = "numbers";
const VERSION: &'static str = env!("CARGO_PKG_VERSION");
fn run(ctx: &RunContext, _input: Option<Input>) -> Result<Output> {
let schema = Arc::new(schema::to_arrow(SCHEMA)?);
ctx.log(format!("numbers for {}", ctx.partition()));
let (s, c) = (schema.clone(), ctx.clone());
let batches = (0..10).map_while(move |i| {
if c.stopping() { return None; } // end early when asked
let ids = Int64Array::from_iter_values(i * 100..(i + 1) * 100);
Some(RecordBatch::try_new(s.clone(), vec![Arc::new(ids)]).map_err(Into::into))
});
Ok(Output::from_iter(schema, batches))
}
}
export_worker!(Numbers);Worker::runis called once per attempt and returns anOutput: aRecordBatchReadermorca pulls from (Output::from_reader,from_batchesorfrom_iter). Work can happen inrun, in the stream, or both. A stream that makes each batch when it's pulled holds one batch at a time, and produces at the rate morca writes:get_nextis pull-based, so backpressure comes for free.RunContexthas the pipeline, step, partition, run, attempt, the registry hash (code), the step'sversion=(step_version), the parameters the proxy passed (params(),param(name), a JSON object) and the run's secrets (secret(name)).log,progressandmetricwrite to the step's log, andstopping()is the stop flag.- Errors are any
std::error::Error; the message fails the attempt. Panics do too: the crate catches them at the boundary, inrunand in each batch. Don't build a worker withpanic = "abort", which would take the whole process down instead. - Threads are the worker's own business: rayon, a tokio runtime, a few
std::threads, sized to the run's grant (a Rust step is a single-process step, below).RunContextisSend + Sync; a clone in another thread can log and readstopping(). Stop and join every thread before the stream ends.
morca_worker::testing::Run runs a worker through the same C ABI in a
cargo test, and returns its batches, the files it reported, and its log
lines.
A worker that already produces Parquet, or whose output is too large to hand over batch by batch, writes the files itself and reports them:
// The worker's own dependency: parquet = { version = "60", default-features
// = false, features = ["arrow", "snap"] }, arrow-rs's, pure Rust.
use parquet::arrow::ArrowWriter;
fn run(ctx: &RunContext, _input: Option<Input>) -> Result<Output> {
let schema = Arc::new(schema::to_arrow(SCHEMA)?); // field ids included
let c = ctx.clone();
let files = (0..4).map(move |i| {
let path = c.output_dir()?.join(format!("part-{i}.parquet"));
let mut w = ArrowWriter::try_new(File::create(&path)?, schema.clone(), None)?;
w.write(&make_batch(i)?)?;
w.close()?;
Ok(path)
});
Ok(Output::from_files(ctx, files))
}ctx.output_dir()is a local directory, empty and this attempt's own (under$TMPDIR). morca removes it, with whatever's left in it, when the call ends.ctx.add_file(path)reports a finished file (a relative path is in the output directory). Call it inrunand returnOutput::files(), which is a stream of no columns, or returnOutput::from_files(ctx, iter), whose iterator writes one file per item: morca takes each file before it asks for the next, so the worker's disk holds one at a time. From another thread,add_fileis queued and delivered on the run thread, as logs are.- Streams and files mix. A worker may stream some rows and write
others to files in the same run (examples/rust-worker's
mixmode); all of them land in the step's one commit. A stream of no columns says the worker only writes files.
morca takes each reported file the next time it has control: when run
returns, and after each batch it pulls. For each file it:
- Fails the attempt if it isn't there:
the Rust worker reported the file …, which doesn't exist.add_filereturns that error to the worker too, when it's called on the run thread. - Checks its schema against the table's, from the footer, by the rule in
"The schema" below, with two differences, since the file is committed as it is and
read later by field id: every column must carry the table's field id,
and a type must be one iceberg's reader reads as the table's: the same
(a timestamp or time in the same unit), or
intforlong,floatfordouble, a decimal of the same scale and no more precision. The error names the file and the column. - Finds its partition from the footer's statistics: for each field of
the table's partition spec, the source column's minimum and maximum in
every row group must give one value under the transform (under
bucket, one value outright), and a column that's all null is the null partition. A file holding rows of two partitions fails the attempt, naming the column: write a file per partition. arrow-rs writes the statistics by default. A string column the writer truncated statistics for has a minimum and maximum that differ, so partition by a short key. - Uploads it where the table's data files go, with the table's
FileIO(a local warehouse or object storage, S3 included), asdata/<partition>/<uuid>-w<n>.parquet, and deletes the local file if it's in the output directory. The file is read whole once to upload it, so keep files to a few hundred MB. - Keeps its entry for the commit: row count, file size, split offsets
and per-column sizes, value and null counts and bounds, all from the
footer (iceberg's
data_file_from_footer, as for files morca writes).
At the end of the stream the files are committed with those morca wrote
from the stream: one append_files, the same commit path, and the same
recognition of an earlier attempt's commit after a crash. A file of no
rows is dropped. The worker writes local files, not to the warehouse
itself, so it needs no storage credentials and no object store crate,
and a file that doesn't fit never reaches the table's location.
The proxy pipeline is the source of truth, as for any step: it declares
writes=[...] and creates the table with
ensure_table(ctx, dataset, Schema.parse(...), spec). The schema is an
Iceberg schema JSON file, such as schemas/numbers.json:
- Mojo embeds it at build time:
scripts/embed-json.sh schemas/numbers.json NUMBERS_SCHEMA > build/x/numbers_schema.mojowrites a module with the file as a string, and-I build/xmakes it importable, so the binary needs no file next to it. - Rust embeds it with
include_str!, andmorca_worker::schema::to_arrowturns it into the Arrow schema the worker builds its batches with, each field carrying its Iceberg id (PARQUET:field_id). cargo testchecks the worker's schema against the file withschema::checkorschema::assert_fits, so drift fails there first.- morca checks the stream's schema (
get_schema) against the table's before it writes anything, by the same rule, and fails the attempt naming the column.
The rule. Columns are matched by name, at every depth. Every column of the table must be in the worker's schema, and no other; a field id, when the worker gives one, must be the table's. A required column mustn't be nullable. A type fits when it's the table's, or one Iceberg widens to it:
| Table | The worker's Arrow type |
|---|---|
int |
Int8, Int16, Int32 |
long |
Int8, Int16, Int32, Int64 |
double |
Float32, Float64 |
string |
Utf8, LargeUtf8 |
binary |
Binary, LargeBinary |
decimal(P,S) |
Decimal128 of scale S and precision up to P |
timestamp, timestamp_ns |
Timestamp in any unit, without a time zone |
timestamptz, timestamptz_ns |
Timestamp in any unit, with a time zone |
time |
Time64 |
uuid, fixed[N] |
FixedSizeBinary(16), FixedSizeBinary(N) |
| struct, list, map | the same kind, its fields, element or key and value by the same rule |
| anything else | the type to_arrow gives |
morca checks a nested column's kind with the stream's schema, and its
leaves as each batch is aligned to the table (iceberg's align_batch),
which also fails naming the column.
def rust_step(
mut ctx: Ctx,
library: String,
mut table: Table,
params: String = "{}",
step_version: Int = 0,
stop_when: def() thin -> Bool = outdated,
flush_bytes: Int = 64 MiB,
) raises -> RustRun # snapshot_id, rows, files, stopped, resumed- A retry that finds its commit returns it. If an earlier attempt of
the step committed to the table in this run (
find_commit), the worker doesn't run again and nothing is written: a process killed after the commit, before morca's database heard of it, is resumed without duplicate rows. - It loads the library late, when the step runs, never at program
start (see the process model below), and refuses a library whose
morca_worker_abi_version()isn't 1. - It calls
morca_worker_runwith the context, and takes the stream. - It checks the stream's schema against the table's.
- It pulls batches and writes Parquet as they arrive, each aligned to
the table's schema, gathering small batches into files of about
flush_bytes, so a large output never sits in memory whole. - It takes the worker's own Parquet files, if it reported any, when
runreturns and after each batch (above). - At the end of the stream it commits once:
append_files, which isreplace_partition_filesfor a dataset the step declared inreplaces=, with the files from the stream and the worker's own. One commit, not one per batch: morca recognises a step's commits by their number after a crash.
Steps 4 to 7 run inside liblake, the library morca.tables loads, so a
proxy pipeline compiles no iceberg: the stream's arrays move to the library
as they're pulled, without a copy (#474).
Any failure before the commit (the worker's error or panic, a schema or a
file that doesn't fit, a batch that can't be aligned) fails the attempt,
and the run's retries apply. Files already written or uploaded are left
unreferenced in data/, as with an interrupted append.
RustRun has snapshot_id, rows (streamed and in the worker's files),
files (all data files committed), worker_files (of those, the
worker's own), stopped and resumed.
library is a path, a file name (libx.so), or a bare name x, which
is libx.so (libx.dylib on macOS), looked up as morca's other native
libraries are: $CONDA_PREFIX/lib, the executable's directory and its
../lib, then the dynamic linker's path.
export_worker! generates these four functions; a library exports them and
nothing else. A worker in another language implements them itself.
#include <stdint.h>
#include <stddef.h>
#define MORCA_WORKER_ABI_VERSION 1
typedef struct MorcaRunContext {
uint64_t size; /* sizeof this struct as morca built it */
const char *pipeline; /* NUL-terminated UTF-8 */
const char *step;
const char *partition; /* empty for a gather step */
const char *params; /* a JSON object, from the proxy step */
const char *secrets; /* a JSON object: secret name -> value */
const char *code; /* the pipeline's registry hash, hex */
int64_t run; /* 0 for a run in memory */
int64_t attempt; /* from 1 */
int64_t step_version; /* the step's version= */
const int32_t *stop; /* atomic; non-zero: stop early */
void *host; /* passed back to the callbacks */
void (*log)(void *host, const char *msg, size_t len);
void (*progress)(void *host, int64_t done, int64_t total);
void (*metric)(void *host, const char *name, size_t name_len, double value);
/* From size 136: */
const char *output_dir; /* a local directory for the worker's files */
int32_t (*add_file)(void *host, const char *path, size_t len);
} MorcaRunContext; /* 136 bytes on a 64-bit platform */
uint32_t morca_worker_abi_version(void);
const char *morca_worker_describe(void);
int32_t morca_worker_run(const MorcaRunContext *ctx,
struct ArrowArrayStream *input,
struct ArrowArrayStream *out,
char **error);
void morca_worker_free(char *p);morca_worker_abi_versionis 1. morca refuses another, naming both.sizecomes first. A later morca can add fields at the end without a new version: a worker reads the fields it knows. The first morca with Rust workers passed 120 bytes;output_dirandadd_filecame after, still version 1, since a worker built before them reads only the first 120. The crate reads a context of 120 bytes or more, and the two file fields only whensizeis 136 or more; under an older morca,output_dir()andadd_filefail saying so, and streams work as before.add_fileis called on the run thread with a UTF-8 path, absolute or relative tooutput_dir, of a complete Parquet file. It returns 0 when morca has taken note of it, non-zero when it refuses it (it isn't there); morca then fails the attempt with its reason, whatever the worker does.morca_worker_describereturns static JSON the library owns and never frees:{"name", "version", "abi", "morca_worker"}. morca logs it. It doesn't carry the schema; the proxy pipeline does.morca_worker_runreturns 0 having moved a stream into*out(the Arrow C stream interface). morca owns it from then on and releases it once it's done with it, at the end of the stream or on a failure. On failurerunreturns non-zero (1 an error, 2 a panic) and points*errorat a message the library allocated, which morca frees withmorca_worker_free. There is no global "last error", which wouldn't be safe with two calls in one process. An error during the stream is the stream's own:get_nextreturns non-zero andget_last_errorsays why, in the worker's words: the crate drops arrow-rs'sExternal error:from the front, and morca logs it asmorca: the Rust worker <library> failed: <message>.inputis NULL in version 1 (below).- Lifetimes.
ctxand everything it points to, the stop flag included, stay valid until morca has releasedout, which is when the worker's side of the call ends. - Callbacks are called only from the thread morca called
runon; morca callsget_nextandreleaseon that thread too. The crate queues what other threads send and delivers it there, the next time the run thread logs or a batch is pulled. Progress is logged at most every 10 s, and when it reaches its total. - The stop flag is an
int32the worker reads atomically (acquire), from any thread, between units of work (ctx.stopping()). morca sets it whenstop_when()returns true, polling it every 50 ms on a thread of its own; by default that'smorca.outdated(), true once a newer version of the pipeline leads or the run's scheduler drains. A worker that sees it ends its stream early; morca commits what came and setsRustRun.stopped, and the step decides what a stopped run means (the example logs it and returns). A worker that ignores it runs to the end.
A cdylib exports only its #[no_mangle] functions, so with
export_worker! that's the four above (add_file is a callback in the
context, not an export); tests/rust_worker.sh checks it with
nm. Rust's standard library is linked in statically, as it always is for
a cdylib, so the library depends on libc and nothing of Rust's. That
keeps the library's libc use apart from Mojo's external_call
declarations, and morca opens it with RTLD_LOCAL, so its symbols don't
mix with the process's. It's also opened with RTLD_NODELETE and never
unloaded: a Rust library with thread-locals must outlive its threads.
- A Rust step is a single-process step.
rust_stepruns the worker once, in the run's process. The worker can use the run's granted cores with its own threads. Given up, for now: spreading over machines (workers = N), retries per key, and per-task bars on the run's timeline; see below. - Loaded late. The library is opened when the step runs, never at program start: dista forks its workers, and a library that has started threads (tokio) before a fork breaks the child (#135 is the same failure with libcurl). Nothing Rust crosses a fork.
- Isolation. Each run is its own process, so a Rust abort fails only that attempt, and the run's retries apply. The usual failure is a caught panic, an error message.
- Memory. A run's learned grant (
memory_gb = "auto") comes from its cgroup's peak, and Rust's heap is in the same process and cgroup, so a Rust step is sized like any other. morca can't see why; a worker that wants its own peak on record reports it withctx.metric. - Batches cross with one copy. arrow-mlake's stream import copies each batch into Mojo memory and releases the producer's at once, so the worker holds one batch at a time. The copy runs at memory speed.
Cargo is a build-time dependency only. The image the runs use carries the built library, not the Rust toolchain:
- In development and CI,
cargo build --releaseand point the proxy at the library, or install it into the environment'slib/. In this repository therustpixi environment (the data environment plus conda-forge'srust) builds and tests the example. - For an image, make the worker a pixi package whose recipe runs
cargo build --releaseand installslib/lib<name>.sointo the environment, as magmalake's iceberg-rs.mojo does with itsffi/(recipe.yamloverpixi-build-rattler-build,rustin its build requirements). The project image's build has the toolchain;morca-runtimedoesn't need it, since the library is found in$CONDA_PREFIX/liblike morca's other native libraries. - Build the worker against the
morca-workercrate of the morca the pipeline uses: the ABI version is checked when the step runs.
- Rust steps as dista tasks (
rust_stepin aMappedstep over tasks), with each task process loading the library itself. - Several outputs: one stream per declared output, drained in declaration order.
- A step's input as the
inputstream. [steps.<name>] rust = "crates/x"in morca.toml. For now the proxy names the library.