Skip to content

Spike: iceberg-rust against morca's catalog, summary keys and pinned reads #520

Closed

Description

@winding-lines

Part of the plan for pure-Rust pipelines.

A Rust pipeline has to commit to and read morca's tables exactly as morca.tables does. Check whether the Rust Iceberg ecosystem can do that without any Mojo.

Questions

  1. Catalog: morca's filesystem catalog commits by writing each metadata version under a unique name and listing the directory for a clash, on local disk and on S3. Can iceberg-rust (or a small Rust catalog on top of object_store) commit the same way, and can both sides commit to one table without corrupting it?
  2. Summary properties: can a commit carry the orca.* snapshot summary keys, and can those be read back when walking snapshots newest first (find_commit)?
  3. Pinned reads: can a scan be pinned to a snapshot id (use_snapshot)?
  4. Writes: Parquet written by arrow-rs/parquet, with the partition spec and the schema evolution morca uses. Can the Mojo reader read it, and the other way round?
  5. Fallback: if any of 1–3 fails, how much of liblake's C ABI (lib-tins: lake, morca.tables over iceberg and parquet behind a C ABI #454) would a Rust pipeline need, and what would shipping liblake plus the Mojo runtime libraries cost (size, licence terms for redistribution)?

Done when

There's a written answer for each question, with a small Rust program that commits to a table the Mojo side then reads, and the reverse, on local disk and on MinIO.

Activity

  1. winding-lines commented on Nov 17, 2026

    @winding-lines
    ContributorAuthor

    Tracked in #522.

  2. winding-lines commented on Nov 17, 2026

    @winding-lines
    ContributorAuthor

    Answered in spikes/iceberg-rust/README.md (PR #529): yes, iceberg-rust 0.11 with a small catalog commits to and reads morca's tables both ways, on disk and on MinIO, with no Mojo. The liblake fallback isn't needed. Gaps for #521: no overwrite/delete transaction in iceberg-rust 0.11 (replaces=), and arrow 58 vs 60 between iceberg and morca-worker.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions