Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changes/537-sdk-replaces.minor.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
- **morca-sdk `replaces`**: a Rust step can replace the partitions it writes, as Mojo's `replaces=` does. (#537)
28 changes: 28 additions & 0 deletions docs/pipeline-protocol.md
Original file line number Diff line number Diff line change
Expand Up @@ -852,6 +852,34 @@ The run token is what keeps two databases over the same tables, or a
database restored from a backup, from taking each other's commits for their
own (#292).

### Replacing partitions

A dataset in a step's `replaces` (describe's `replaces`, also in its
`writes`) is one whose partitions a new run replaces rather than appends to
(`replace_partitions`, `src/morca/_lake/impl.mojo:1051-1325`):

- The run's first commit to a partition is one `overwrite` snapshot that
removes every data and delete file in it that this run didn't commit, and
adds the new files. A file is this run's if its snapshot's summary has
this run's `orca.pipeline` and `orca.run` (`Ctx.same_run`), so later
commits of the run to the partition, from any step or attempt, find only
its own files and append. A commit that removes nothing is an `append`.
- The partitions are those of the new files under the table's current
spec; files under an older spec are left alone, and an unpartitioned
table is one partition.
- What to remove is planned again on every try, against the snapshot the
try commits on top of.
- A file in a partition being replaced that another writer committed after
this run's first commit to the table fails the attempt, committing
nothing.
- The snapshot carries `orca.replaced-files` and `orca.replaced-records`,
and is found again on a retry or resume as any commit is
([Finding an earlier attempt's commit](#finding-an-earlier-attempts-commit)).

**Contract** that a rerun of a key leaves one copy of its rows: `morca
backfill --rerun` relies on it, and refuses a pipeline that appends
instead (`appending_writes`, `src/morca/_internal/queueing.mojo:356-366`).

### Pinned reads

Every attempt of a step reads the snapshots the first attempt read, so a
Expand Down
39 changes: 36 additions & 3 deletions docs/rust-pipelines.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,8 @@ it was stored as, so a step whose types changed fails with what it got
rather than misreading it.

`step("name")` sets a step's `retries`, `backoff_seconds` (doubling),
`timeout_seconds`, data `version`, and its `reads` and `writes`. A plain
`timeout_seconds`, data `version`, and its `reads`, `writes` and
`replaces`. A plain
`"name"` takes the defaults: no retries, no timeout.

A step fails by returning an error: `?` on any error type, or
Expand Down Expand Up @@ -169,8 +170,42 @@ Pipeline::<Month>::new("clean_rust")
writes nothing. `find_commit(dataset)` asks first, for a step that writes
files itself. `arrow_schema(dataset)` is the schema the batches need.
- `ensure_table(dataset, schema, spec)` creates a table that isn't there.
- `write_files(dataset, batches)` writes the Parquet files without
committing them, and `append_files(dataset, files)` commits them as
`append` would; a step that does this calls `find_commit` first.
- A step may read and write only the datasets it declares.

#### Replacing partitions

A dataset in `replaces` instead of `writes` is one whose partitions a new
run replaces, as Mojo's `replaces=`: `append` and `append_files` to it make
the run's first commit to each partition an `overwrite` that removes what
earlier runs left there, so `morca backfill --rerun` leaves one copy of a
key's rows.

```rust
.mapped(step("load").reads(["raw.events"]).replaces(["lake.monthly"]), load)
```

The semantics are the Mojo side's (docs/pipeline-protocol.md, "Replacing
partitions"): the partitions are those the new rows land in, files this run
committed are never removed, a commit that removes nothing appends, the
plan is made again on every retry, another writer's file in the partition
committed after the run's first commit fails the attempt, and the snapshot
carries `orca.replaced-files` and `orca.replaced-records`. A crash between
writing and committing, and a retry after committing, behave as for an
append. `replace_partitions` and `replace_partition_files` replace a
dataset declared in `writes`. A dataset in `replaces` is hashed as a write,
so declaring it doesn't change the registry hash.

iceberg-rust 0.11.0 has no overwrite transaction, so morca-sdk writes that
snapshot itself with iceberg-rust's manifest and manifest-list writers
(src/tables/replace.rs) and commits it through its catalog, fenced like any
commit. [examples/replace.rs](../rust/morca-sdk/examples/replace.rs), the
twin of tests/sensors/replace.mojo, is checked by the conformance suite as
tests/conformance/replace-rust.json, and Mojo's `morca.tables` reads its
table as one copy of each month.

The binary then also answers `sense` (a sensor's datasets) and `keys` (a
keyed pipeline's keys). [rust/morca-sdk/examples/clean.rs](../rust/morca-sdk/examples/clean.rs)
is a sensor, the twin of tests/sensors/clean.mojo; the conformance suite runs
Expand Down Expand Up @@ -204,8 +239,6 @@ empty.
later.
- **dista tasks**: a dista task is Mojo code, and forking a threaded Rust
process is the #135 hazard. A step parallelises inside itself instead.
- **`replaces=`**: iceberg-rust 0.11.0 has no overwrite or delete
transaction, so a step only appends.
- **Object storage for logs**: `MORCA_LOGS`, `keep-log` and `put-log` take
local paths; an `s3://` `MORCA_LOGS` keeps no attempt logs yet.
- **Postgres over TLS**: the client connects without TLS
Expand Down
5 changes: 3 additions & 2 deletions pixi.toml
Original file line number Diff line number Diff line change
Expand Up @@ -487,8 +487,9 @@ test-rust-worker = { cmd = "tests/rust_worker.sh", depends-on = ["build-liblake"
test-hindcast = { cmd = "tests/hindcast.sh", depends-on = ["build-liblake"] }
# morca-sdk (#521): whole pipelines in Rust, with no Mojo toolchain; the
# conformance suite on the Rust toy and a Rust sensor with tables, then the
# Mojo toy and the Rust toy side by side under one scheduler.
test-rust-sdk = { cmd = "scripts/with-postgres.sh tests/rust_sdk.sh" }
# Mojo toy and the Rust toy side by side under one scheduler. liblake is for
# the Mojo reader of the table a Rust step replaces.
test-rust-sdk = { cmd = "scripts/with-postgres.sh tests/rust_sdk.sh", depends-on = ["build-liblake"] }

# MinIO for the tests over object storage (scripts/with-minio.sh). conda-forge
# builds it for these two platforms only.
Expand Down
4 changes: 4 additions & 0 deletions rust/morca-sdk/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -62,5 +62,9 @@ required-features = ["tables"]
name = "source"
required-features = ["tables"]

[[example]]
name = "replace"
required-features = ["tables"]

[profile.release]
strip = true
6 changes: 4 additions & 2 deletions rust/morca-sdk/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,8 +37,10 @@ cargo build --release && morca register target/release/days
Postgres or a SQLite file, and passes the conformance suite
(tests/conformance.py).
- The `tables` feature adds Iceberg reads (pinned on a retry, recorded) and
appends (fenced, with the `orca.*` summary keys, found again on a retry)
through iceberg-rust 0.11.0 and arrow-rs 58, plus `sense` and `keys`.
commits (fenced, with the `orca.*` summary keys, found again on a retry)
through iceberg-rust 0.11.0 and arrow-rs 58, plus `sense` and `keys`. A
commit appends, or with `step(..).replaces([..])` replaces the partitions
it writes in one `overwrite` snapshot, as Mojo's `replaces=` does.
- Rust 1.95 or later.

docs/rust-pipelines.md in the morca repository is the guide, including what
Expand Down
117 changes: 117 additions & 0 deletions rust/morca-sdk/examples/replace.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
//! A sensor that replaces its partitions, the Rust twin of
//! tests/sensors/replace.mojo: reads raw.events and writes as many rows to
//! lake.monthly, partitioned by month, a month at a time, declaring
//! lake.monthly in `replaces` so a new run of a month leaves one copy of
//! its rows. The conformance suite runs it as
//! tests/conformance/replace-rust.json.
//!
//! It writes its files and commits them in two calls (`write_files`, then
//! `append_files`), so that with MORCA_REPLACE_DIE set it can die between
//! the two; MORCA_REPLACE_SLEEP makes each attempt take that many seconds
//! first.
//!
//! cargo build --release --features tables --example replace
//! replace rows <warehouse> each month in lake.monthly and its rows

use std::collections::BTreeMap;
use std::sync::Arc;

use morca_sdk::tables::arrow_array::{ArrayRef, Int64Array, RecordBatch, StringArray};
use morca_sdk::tables::iceberg::arrow::schema_to_arrow_schema;
use morca_sdk::tables::iceberg::spec::{
NestedField, PrimitiveType, Schema, Transform, Type, UnboundPartitionSpec,
};
use morca_sdk::tables::read_as_other;
use morca_sdk::{Ctx, Month, Pipeline, StepResult, step};

const TABLE: &str = "lake.monthly";

fn schema() -> Schema {
Schema::builder()
.with_schema_id(0)
.with_fields(vec![
NestedField::required(1, "id", Type::Primitive(PrimitiveType::Long)).into(),
NestedField::required(2, "month", Type::Primitive(PrimitiveType::String)).into(),
])
.build()
.expect("a valid schema")
}

fn spec() -> UnboundPartitionSpec {
UnboundPartitionSpec::builder()
.with_spec_id(0)
.add_partition_field(2, "month", Transform::Identity)
.expect("a valid field")
.build()
}

fn rows(month: &str, n: usize) -> Result<RecordBatch, String> {
let schema = Arc::new(schema_to_arrow_schema(&schema()).map_err(|e| e.to_string())?);
let ids: ArrayRef = Arc::new(Int64Array::from_iter_values(0..n as i64));
let months: ArrayRef = Arc::new(StringArray::from_iter_values((0..n).map(|_| month)));
RecordBatch::try_new(schema, vec![ids, months]).map_err(|e| e.to_string())
}

fn load(ctx: &mut Ctx, m: Month) -> StepResult<i64> {
if let Some(s) = std::env::var("MORCA_REPLACE_SLEEP")
.ok()
.and_then(|s| s.parse::<f64>().ok())
{
std::thread::sleep(std::time::Duration::from_secs_f64(s));
}
let n: usize = ctx.read("raw.events")?.iter().map(|b| b.num_rows()).sum();
ctx.ensure_table(TABLE, schema(), Some(spec()))?;
let files = if ctx.find_commit(TABLE)?.is_some() {
vec![]
} else {
ctx.write_files(TABLE, vec![rows(&m.to_string(), n)?])?
};
if !files.is_empty() && std::env::var_os("MORCA_REPLACE_DIE").is_some() {
eprintln!("replace: dying between writing {m}'s files and committing them");
// As a kill would: no attempt finished, nothing committed.
std::process::exit(70);
}
let snap = ctx.append_files(TABLE, files)?;
ctx.log(format!("{n} rows for {m} in snapshot {snap}"));
Ok(n as i64)
}

/// `rows <warehouse>`: each month in lake.monthly and how many rows it has.
fn print_rows(wh: &str) -> Result<(), String> {
// SAFETY: single-threaded, before anything reads the environment.
unsafe { std::env::set_var("MORCA_WAREHOUSE", wh) };
let mut by = BTreeMap::<String, usize>::new();
for b in read_as_other(TABLE)? {
let c = b
.column_by_name("month")
.and_then(|c| c.as_any().downcast_ref::<StringArray>())
.ok_or("no month column")?;
for m in c.iter().flatten() {
*by.entry(m.to_string()).or_default() += 1;
}
}
for (m, n) in by {
println!("{m} {n}");
}
Ok(())
}

fn main() {
let args: Vec<String> = std::env::args().collect();
if args.get(1).map(String::as_str) == Some("rows") {
let Some(wh) = args.get(2) else {
eprintln!("usage: replace rows <warehouse>");
std::process::exit(2);
};
if let Err(e) = print_rows(wh) {
eprintln!("replace: {e}");
std::process::exit(1);
}
return;
}
let p = Pipeline::<Month>::new("replace_rust")
.start("2024-01")
.sensor()
.mapped(step("load").reads(["raw.events"]).replaces([TABLE]), load);
morca_sdk::main(p.build());
}
17 changes: 17 additions & 0 deletions rust/morca-sdk/src/ctx.rs
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,10 @@ pub struct Ctx<'a> {
/// The location of each commit this attempt made, in order.
pub(crate) commits: Vec<String>,
pub(crate) declared_reads: Vec<String>,
/// What the step writes, `replaces` among them.
pub(crate) declared_writes: Vec<String>,
/// The datasets whose partitions a new run of the step replaces.
pub(crate) declared_replaces: Vec<String>,
pub(crate) log_path: Option<String>,
pub(crate) store: &'a mut Owned,
}
Expand Down Expand Up @@ -151,6 +154,20 @@ impl Ctx<'_> {
self.commits.iter().filter(|t| *t == location).count()
}

/// Whether the step declared that it replaces `dataset`'s partitions
/// (`replaces`).
pub fn replaces(&self, dataset: &str) -> bool {
self.declared_replaces.iter().any(|d| d == dataset)
}

/// `Ctx.same_run`: whether a snapshot with `summary` was committed by
/// any step of this run (the same pipeline and run).
pub fn same_run(&self, summary: &HashMap<String, String>) -> bool {
self.run > 0
&& summary.get("orca.pipeline") == Some(&self.pipeline)
&& summary.get("orca.run") == Some(&self.run.to_string())
}

/// `Ctx.made`: whether a snapshot with `summary` is this step's next
/// commit to the table at `location`, by any attempt of this run. Every
/// key must be there; a run token, when both have one, must agree.
Expand Down
3 changes: 2 additions & 1 deletion rust/morca-sdk/src/exec.rs
Original file line number Diff line number Diff line change
Expand Up @@ -242,7 +242,8 @@ fn run_step(
reads: vec![],
commits: vec![],
declared_reads: st.spec.reads.clone(),
declared_writes: st.spec.writes.clone(),
declared_writes: st.spec.all_writes(),
declared_replaces: st.spec.replaces.clone(),
log_path: log,
store,
};
Expand Down
Loading
Loading