Skip to content
@magmalake

magmalake

magmalake

Data infrastructure for Mojo, version 0.x. A native implementation of Apache Iceberg and everything underneath it — Parquet, Avro, Thrift, compression codecs, Roaring bitmaps, object storage, SQL catalogs and threads — built as small, independently usable tins, each gated against the reference implementation of the format it speaks and published on mojoshelf.org (pixi shelf add iceberg-mojo).

Mojo can open an Iceberg table today without reaching back through Python or the JVM. The stack does link a few very mature C libraries — libzstd, liblz4, libbrotli, libcurl, libpq and pthreads — so the claim is no Python and no JVM in the path, not no C.

➡️ magmalake.org is the canonical, up-to-date source. This README is a pointer, not a copy: it drifts, the site doesn't.

The tins

tin what it is version
iceberg Native Apache Iceberg: metadata v1–v3, scan planning, reads, writes (fast-append, delete, overwrite, expire), REST and SQL catalogs 0.8.2
parquet The first native Parquet reader/writer in Mojo, Arrow C Data Interface out, multi-core reads 0.11.4
avro Object Container Files, read + write, all four codecs, schema-compiled cursor 0.4.3
objectstore FileIO over local, HTTP(S), S3, GCS, Azure; pooled libcurl transport 0.3.3
postgres libpq binding: queries, parameters, COPY, transactions — the deployable SQL catalog 0.3.2
sqlite libsqlite3 binding; the local-development SQL catalog 0.3.4
roaring 32/64-bit Roaring bitmaps, Iceberg deletion-vector framing 0.1.3
thrift Compact + binary protocol runtime and the parquet.thrift structs 0.1.3
threads pthreads for Mojo — spawn/join/pin, atomics, typed parallel_for, worker pools 0.6.1
zstd FFI binding to libzstd 0.1.4
lz4 FFI binding to liblz4 (block, frame, Hadoop framing) 0.1.3
brotli FFI binding to libbrotli 0.1.3
snappy Pure-Mojo Snappy, raw + framing 0.1.4
hashes CRC-32, MurmurHash3 x86-32, XXH64 + Iceberg bucket transform 0.1.3
iceberg-rs Rust cdylib over iceberg-rust, kept as a cross-implementation oracle 0.1.2

All Apache-2.0, CI-tested on stable Mojo 1.2.0 and the current nightly, macOS and Linux.

A few of the numbers

Apple M4, reproducible via each repo's pixi run bench. Medians, with the tails and the full tables on magmalake.org/performance.

  • Parquet read, 1M rows × 4 columns: 4.7 ms on one core, 2.1 ms on eight — faster than pyarrow with its own threads
  • Parquet read, 100k rows of mixed and nested types: 3.3 ms on one core, 0.96 ms on eight
  • Avro decode, manifest-shaped records: 19.2M records/s (fastavro: 1.74M)

Where it loses is on the site too, with the reason: threaded pyarrow is still ahead on nested reads, and the remaining gap is measured rather than estimated.

Agent skills

What building this stack taught us about making data code fast is packaged as an agent skill, mojo-data-performance, in magmalake/skills. It covers a pipeline's per-row loops over record batches and the readers, writers and codecs under them; how to tell decoding time from your own work's time without a second implementation to compare against; and the measurement discipline, the tests that catch a fast path which silently never fires, and the Mojo ownership and threading hazards underneath.

npx skills add magmalake/skills --yes

That installs it for whichever agents are found on the machine. To name the agents, pass them space-separated: the flag is variadic, and a comma-separated list is rejected. In Claude Code you can also install it as a plugin, with /plugin marketplace add magmalake/skills and then /plugin install skills@magmalake.

Full detail: magmalake.org

Performance · Correctness · Status · Writing

The site carries the complete tables, the correctness bar each tin is gated on, the upstream bugs this surfaced in PyIceberg, pyarrow and iceberg-rust, and what is not yet implemented.

Proposals and issues

What is proposed but unbuilt lives in the issue tracker (short link: magmalake.org/issues). Requests for a tin are welcome there.

Popular repositories Loading

  1. avro.mojo avro.mojo Public

    Pure-Mojo Apache Avro: schema, binary encoding, Object Container Files (null/deflate/zstd/snappy)

    Mojo 1

  2. parquet.mojo parquet.mojo Public

    Native Apache Parquet reader and writer for Mojo — all encodings and codecs, Arrow C Data Interface output

    Mojo 1

  3. threads.mojo threads.mojo Public

    Minimal pthreads for Mojo — spawn/join/pin, parallel_for, atomic flags — extracted from flare's runtime

    Mojo 1

  4. postgres.mojo postgres.mojo Public

    Forked from dvirarad/mojo-postgres

    PostgreSQL client for Mojo🔥 — libpq bindings

    Mojo 1

  5. threads.example threads.example Public

    Companion code for the magmalake post on writing multithreaded code in Mojo 1.0 — one file, built against threads-mojo from mojoshelf

    Mojo 1

  6. restate.mojo restate.mojo Public

    Durable execution flows in Mojo via Restate: Rust SDK event loop and journal behind a C-ABI shim, business logic in Mojo, served on one thread or N

    Mojo

Repositories

Showing 10 of 40 repositories

People

This organization has no public members. You must be a member to see who’s a part of this organization.

Top languages

Loading…

Most used topics

Loading…