Repository navigation
morca-sdk: replaces= (overwrite a partition) for Rust pipelines - #538
Merged
Merged
Conversation
step(..).replaces([..]) mirrors Mojo's replaces=: append and append_files to such a dataset commit the run's first write to each partition as one overwrite snapshot that removes what earlier runs left there, with the orca.replaced-* summary keys, the same fence, find_commit and planning on every retry. iceberg-rust 0.11.0 has no overwrite transaction, so the snapshot is written from iceberg-rust's manifest and manifest-list writers and committed through MorcaCatalog::commit_next. Also write_files/append_files and replace_partitions on Ctx, and the conformance suite's tables.replace* checks, run on the Rust example and its Mojo twin (tests/sensors/replace.mojo), with Mojo's morca.tables as a reader of the replaced table. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017TycszSbyw92Efu4ZeCYL6
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #537.
A Rust step declared with
step("load").replaces(["lake.monthly"])now replaces the partitions it writes, as a Mojo step withreplaces=does:append/append_filesto that dataset make the run's first commit to each partition oneoverwritesnapshot that removes what earlier runs left there, so rerunning a key leaves one copy of its rows.Approach: (a), the snapshot written with iceberg-rust's lower-level APIs
I picked (a), writing the overwrite with iceberg-rust's own lower-level pieces, over (b), the liblake C ABI:
ManifestReader,ManifestWriterBuilder(add_delete_file,add_existing_file,add_file),ManifestListWriter,Snapshot::builder,SnapshotSummaryCollectorandTableMetadataBuilder. The only private piece isTableCommit's builder, soMorcaCataloggetscommit_next(base, metadata). It names the next version, refuses retryably unlessbaseis still current, and shares the claim and write withupdate_table(write_staged). The new code is about 350 lines (src/tables/replace.rs).LAKE_LIBat run time. With (b), a Rust binary would also have to load a Mojo runtime into a threaded process (the Local mode on macOS: forked workers crash after libcurl initialised (S3 warehouse) #135 hazard), and every commit would go through a second catalog implementation.The semantics match
replace_partitionsin src/morca/_lake/impl.mojo:orca.pipelineandorca.run(Ctx::same_run). The run's own files are never removed, so later commits of the run to a partition append.orca.replaced-filesandorca.replaced-recordsare always set. A commit that removes nothing is anappend, as in Mojo.find_commitruns before writing anything, and after a retryable failure.total-*summary totals follow iceberg'supdate_totals.Also in this PR
The step builder's
replacescounts as a write, and is hashed as a write (Adopt a previous version's done keys when a pipeline's declarations change #295). describe lists the dataset inwritesand adds"replaces"only when non-empty, as the Mojo binary does. Registry hashes are unchanged: the toy, daily and fanout manifests still pin theirs.Ctx::write_files/append_files(Mojo's split, so a step can write files and commit them later),replace_partitions/replace_partition_files,replaces()andsame_run(). There is alsotables::read_as_otherfor tests and tools.examples/replace.rsand its Mojo twintests/sensors/replace.mojo, plustests/sensors/rows_by.mojo, a Mojomorca.tablesreader.Conformance (
tests/conformance.py): manifests may now listreplaces,die_envandrows. The new checks are:tables.replace: a new run commits anoverwrite, its replaced keys agree withdeleted-*, and the table holds one copy.tables.replace_rerun: running the same keys again still leaves one copy.tables.replace_die_before_commit: a step that exits between writing and committing leaves no snapshot; the resumed run commits once and leaves one copy.tables.replace_readers: each reader inrowssees one copy of each partition.The existing
tables.crash_resume,retry_finds_commitandpinnedchecks now also run on a replacing step. The checks run forreplace-rust.jsonintest-rust-sdkand forreplace.json(Mojo) intest-conformance-tables, on SQLite and Postgres.test-rust-sdknow depends onbuild-liblake, for the Mojo reader.docs/pipeline-protocol.md gets a "Replacing partitions" section. docs/rust-pipelines.md drops the gap and documents
replaces, and the crate README is updated.Checked locally
cargo testwith and withouttables,clippy --all-targets,fmt.iceberg_scan(viauv run; they're not in a pixi env, so CI doesn't run this). Both show the current snapshot asoverwrite,total-records12, and 6 rows each for 2024-03 and 2024-04, which is what the SDK's reader prints too.🤖 Generated with Claude Code
https://claude.ai/code/session_017TycszSbyw92Efu4ZeCYL6