Skip to content

Spike: iceberg-rust against morca's catalog, summary keys and pinned reads (#520) - #529

Merged
winding-lines merged 1 commit into
mainfrom
spike-iceberg-rust-520
Nov 17, 2026
Merged

winding-lines merged 1 commit into
mainfrom
spike-iceberg-rust-520

Conversation

@winding-lines

@winding-lines winding-lines commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Refs #520. This is a spike: it only adds spikes/iceberg-rust/ and changes nothing in morca. The full write-up, with the evidence, is in spikes/iceberg-rust/README.md.

Answer: yes. A Rust pipeline can commit to and read morca's tables as morca.tables does, with no Mojo. It uses iceberg 0.11.0, arrow-rs/parquet 58.4.1 and object_store 0.14.2 as they are, plus a catalog of about 500 lines over object_store. iceberg-rust has no catalog that commits the way morca's filesystem catalog does.

  1. Catalog. iceberg-rust has no filesystem catalog. Its public TableCommit::apply checks a commit's requirements and applies its updates. It also names the next file <V>-<uuid>.metadata.json, as iceberg-mojo does, so MorcaCatalog only adds iceberg-mojo's protocol on top: the version hint, the claim, the listing of <V:05>-, then the write. Rust and Mojo writers commit to one table at once with no losses: 40 rounds of 8 writers on local disk and 20 on MinIO. A Rust and a Mojo writer creating the same new table at once also end up with one table. Getting there took two listing fixes, and both apply to any port:
    • Local disk. object_store's LocalFileSystem listing lost 10 commits in 40 rounds. Two writers both won a metadata version, because readdir isn't a snapshot and the listing's slow walk widened the window. One plain readdir, as iceberg-mojo does, lost none.
    • MinIO. Listing from an offset (start-after) lost 8 in 20 rounds. A key-prefix ListObjectsV2, as iceberg-mojo sends, lost none.
    • Retries. I also wrapped iceberg-rust's retries, which have no jitter, in morca's jittered _commit_append loop.
  2. Summary keys. set_snapshot_properties carries the orca.* keys. They read back from Snapshot::summary(). A Rust find_commit (newest first, matching as Ctx.made does) recognises Mojo's commits, and Mojo's recognises Rust's. A retry on either side returns the other side's snapshot and commits nothing.
  3. Pinned reads. scan().snapshot_id(id) reads a snapshot with its own schema. Both sides read each other's snapshots by id, and a Mojo step's pin reads Rust's.
  4. Writes. iceberg-rust's writers split rows by month(ts) into data/ts_month=…/ Parquet files with field ids. Mojo reads them, and Rust reads Mojo's. Schema evolution works both ways: Mojo's ensure_table adds score (id 4), and iceberg-rust's UpdateSchemaAction adds note (id 5). Two gaps:
    • iceberg-rust 0.11.0 has no overwrite or delete transaction, so replaces= would need work on the Rust side.
    • iceberg is on arrow 58 while morca-worker is on arrow 60. The versions need aligning, or batches need to cross through C Data.
  5. Fallback (not needed).
    • ABI. liblake exports 46 C functions, of which a Rust pipeline would use about 25, plus host callbacks for Ctx.
    • Size. liblake.dylib is 4.3 MB. With the Mojo runtime (libKGENCompilerRTShared, 1.2 MB), four shims and the codec and OpenSSL libraries it loads, the set is about 13 MB on macOS arm64. The stripped rside binary is 31 MB.
    • Licences. The Mojo runtime is LicenseRef-Modular-Proprietary. Redistributing it in a Rust crate needs Modular's terms read first. All 401 Rust crates are under permissive licences.

Possible morca issues (not filed):

  • On MinIO, 8 Mojo writers committing at once ran out of commit retries 58 times in 160 writes. Nothing was lost, and the step's next attempt commits.
  • morca's local-disk protocol relies on readdir the same way the Rust port did, though no losses showed in 40 rounds.

How to run it: spikes/iceberg-rust/build.sh, then roundtrip.sh and race.sh (see the README). Tested on macOS arm64 with conda-forge minio-server. No AWS testing, and no CI job runs these.

🤖 Generated with Claude Code

https://claude.ai/code/session_017TycszSbyw92Efu4ZeCYL6

…reads (#520)

A Rust side (iceberg 0.11.0, arrow-rs 58, object_store 0.14) with a small
catalog that commits as iceberg-mojo's filesystem catalog does, a Mojo
side over morca.tables, and scripts that check one against the other on
local disk and MinIO. Findings in spikes/iceberg-rust/README.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017TycszSbyw92Efu4ZeCYL6
@winding-lines winding-lines added the no-changelog Needs no changelog fragment (CI, docs or tests only) label Nov 17, 2026
@winding-lines
winding-lines merged commit b238369 into main Nov 17, 2026
37 checks passed
@winding-lines
winding-lines deleted the spike-iceberg-rust-520 branch November 17, 2026 05:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

no-changelog Needs no changelog fragment (CI, docs or tests only)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant