Repository navigation
Spike: iceberg-rust against morca's catalog, summary keys and pinned reads (#520) - #529
Merged
Merged
Conversation
…reads (#520) A Rust side (iceberg 0.11.0, arrow-rs 58, object_store 0.14) with a small catalog that commits as iceberg-mojo's filesystem catalog does, a Mojo side over morca.tables, and scripts that check one against the other on local disk and MinIO. Findings in spikes/iceberg-rust/README.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017TycszSbyw92Efu4ZeCYL6
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #520. This is a spike: it only adds
spikes/iceberg-rust/and changes nothing in morca. The full write-up, with the evidence, is inspikes/iceberg-rust/README.md.Answer: yes. A Rust pipeline can commit to and read morca's tables as
morca.tablesdoes, with no Mojo. It uses iceberg 0.11.0, arrow-rs/parquet 58.4.1 and object_store 0.14.2 as they are, plus a catalog of about 500 lines over object_store. iceberg-rust has no catalog that commits the way morca's filesystem catalog does.TableCommit::applychecks a commit's requirements and applies its updates. It also names the next file<V>-<uuid>.metadata.json, as iceberg-mojo does, soMorcaCatalogonly adds iceberg-mojo's protocol on top: the version hint, the claim, the listing of<V:05>-, then the write. Rust and Mojo writers commit to one table at once with no losses: 40 rounds of 8 writers on local disk and 20 on MinIO. A Rust and a Mojo writer creating the same new table at once also end up with one table. Getting there took two listing fixes, and both apply to any port:LocalFileSystemlisting lost 10 commits in 40 rounds. Two writers both won a metadata version, because readdir isn't a snapshot and the listing's slow walk widened the window. One plainreaddir, as iceberg-mojo does, lost none.start-after) lost 8 in 20 rounds. A key-prefix ListObjectsV2, as iceberg-mojo sends, lost none._commit_appendloop.set_snapshot_propertiescarries theorca.*keys. They read back fromSnapshot::summary(). A Rustfind_commit(newest first, matching asCtx.madedoes) recognises Mojo's commits, and Mojo's recognises Rust's. A retry on either side returns the other side's snapshot and commits nothing.scan().snapshot_id(id)reads a snapshot with its own schema. Both sides read each other's snapshots by id, and a Mojo step's pin reads Rust's.month(ts)intodata/ts_month=…/Parquet files with field ids. Mojo reads them, and Rust reads Mojo's. Schema evolution works both ways: Mojo'sensure_tableaddsscore(id 4), and iceberg-rust'sUpdateSchemaActionaddsnote(id 5). Two gaps:replaces=would need work on the Rust side.Ctx.rsidebinary is 31 MB.LicenseRef-Modular-Proprietary. Redistributing it in a Rust crate needs Modular's terms read first. All 401 Rust crates are under permissive licences.Possible morca issues (not filed):
How to run it:
spikes/iceberg-rust/build.sh, thenroundtrip.shandrace.sh(see the README). Tested on macOS arm64 with conda-forgeminio-server. No AWS testing, and no CI job runs these.🤖 Generated with Claude Code
https://claude.ai/code/session_017TycszSbyw92Efu4ZeCYL6