Skip to content

morca-sdk: replaces= (overwrite a partition) for Rust pipelines - #538

Merged
winding-lines merged 1 commit into
mainfrom
sdk-replaces
Nov 17, 2026
Merged

winding-lines merged 1 commit into
mainfrom
sdk-replaces

Conversation

@winding-lines

@winding-lines winding-lines commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Closes #537.

A Rust step declared with step("load").replaces(["lake.monthly"]) now replaces the partitions it writes, as a Mojo step with replaces= does: append / append_files to that dataset make the run's first commit to each partition one overwrite snapshot that removes what earlier runs left there, so rerunning a key leaves one copy of its rows.

Approach: (a), the snapshot written with iceberg-rust's lower-level APIs

I picked (a), writing the overwrite with iceberg-rust's own lower-level pieces, over (b), the liblake C ABI:

  • iceberg-rust 0.11.0 exports everything the snapshot needs: ManifestReader, ManifestWriterBuilder (add_delete_file, add_existing_file, add_file), ManifestListWriter, Snapshot::builder, SnapshotSummaryCollector and TableMetadataBuilder. The only private piece is TableCommit's builder, so MorcaCatalog gets commit_next(base, metadata). It names the next version, refuses retryably unless base is still current, and shares the claim and write with update_table (write_staged). The new code is about 350 lines (src/tables/replace.rs).
  • The SDK stays free of Mojo: no liblake to ship with a pipeline binary, and no LAKE_LIB at run time. With (b), a Rust binary would also have to load a Mojo runtime into a threaded process (the Local mode on macOS: forked workers crash after libcurl initialised (S3 warehouse) #135 hazard), and every commit would go through a second catalog implementation.

The semantics match replace_partitions in src/morca/_lake/impl.mojo:

  • The partitions replaced are those of the new files under the current spec. An unpartitioned table is one partition.
  • A file is this run's if its snapshot has the run's orca.pipeline and orca.run (Ctx::same_run). The run's own files are never removed, so later commits of the run to a partition append.
  • What to remove is planned again on every try, against the snapshot that try builds on. Manifests whose partition summaries can't hold a replaced partition aren't read.
  • A file another writer committed to the partition after the run's first commit fails the attempt, and nothing is committed.
  • orca.replaced-files and orca.replaced-records are always set. A commit that removes nothing is an append, as in Mojo.
  • The same fence is checked before the commit. find_commit runs before writing anything, and after a retryable failure. total-* summary totals follow iceberg's update_totals.

Also in this PR

  • The step builder's replaces counts as a write, and is hashed as a write (Adopt a previous version's done keys when a pipeline's declarations change #295). describe lists the dataset in writes and adds "replaces" only when non-empty, as the Mojo binary does. Registry hashes are unchanged: the toy, daily and fanout manifests still pin theirs.

  • Ctx::write_files / append_files (Mojo's split, so a step can write files and commit them later), replace_partitions / replace_partition_files, replaces() and same_run(). There is also tables::read_as_other for tests and tools.

  • examples/replace.rs and its Mojo twin tests/sensors/replace.mojo, plus tests/sensors/rows_by.mojo, a Mojo morca.tables reader.

  • Conformance (tests/conformance.py): manifests may now list replaces, die_env and rows. The new checks are:

    • tables.replace: a new run commits an overwrite, its replaced keys agree with deleted-*, and the table holds one copy.
    • tables.replace_rerun: running the same keys again still leaves one copy.
    • tables.replace_die_before_commit: a step that exits between writing and committing leaves no snapshot; the resumed run commits once and leaves one copy.
    • tables.replace_readers: each reader in rows sees one copy of each partition.

    The existing tables.crash_resume, retry_finds_commit and pinned checks now also run on a replacing step. The checks run for replace-rust.json in test-rust-sdk and for replace.json (Mojo) in test-conformance-tables, on SQLite and Postgres. test-rust-sdk now depends on build-liblake, for the Mojo reader.

  • docs/pipeline-protocol.md gets a "Replacing partitions" section. docs/rust-pipelines.md drops the gap and documents replaces, and the crate README is updated.

Checked locally

  • cargo test with and without tables, clippy --all-targets, fmt.
  • Conformance on SQLite for the Rust example: every check passes, including the four new ones (the Mojo reader is built in the full run).
  • I read the replaced table independently with pyiceberg 0.12.0 and DuckDB 2.0.0's iceberg_scan (via uv run; they're not in a pixi env, so CI doesn't run this). Both show the current snapshot as overwrite, total-records 12, and 6 rows each for 2024-03 and 2024-04, which is what the SDK's reader prints too.

🤖 Generated with Claude Code

https://claude.ai/code/session_017TycszSbyw92Efu4ZeCYL6

step(..).replaces([..]) mirrors Mojo's replaces=: append and append_files
to such a dataset commit the run's first write to each partition as one
overwrite snapshot that removes what earlier runs left there, with the
orca.replaced-* summary keys, the same fence, find_commit and planning
on every retry. iceberg-rust 0.11.0 has no overwrite transaction, so the
snapshot is written from iceberg-rust's manifest and manifest-list
writers and committed through MorcaCatalog::commit_next.

Also write_files/append_files and replace_partitions on Ctx, and the
conformance suite's tables.replace* checks, run on the Rust example and
its Mojo twin (tests/sensors/replace.mojo), with Mojo's morca.tables as a
reader of the replaced table.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017TycszSbyw92Efu4ZeCYL6
@winding-lines
winding-lines merged commit cfae1e7 into main Nov 17, 2026
35 checks passed
@winding-lines
winding-lines deleted the sdk-replaces branch November 17, 2026 13:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

morca-sdk: replaces= (overwrite a partition) for Rust pipelines

1 participant