Data infrastructure for Mojo, version 0.x. A native implementation of Apache Iceberg and everything underneath it — Parquet, Avro, Thrift, compression codecs, Roaring bitmaps, object storage, SQL catalogs and threads — built as small, independently usable tins, each gated against the reference implementation of the format it speaks and published on mojoshelf.org (pixi shelf add iceberg-mojo).
Mojo can open an Iceberg table today without reaching back through Python or the JVM. The stack does link a few very mature C libraries — libzstd, liblz4, libbrotli, libcurl, libpq and pthreads — so the claim is no Python and no JVM in the path, not no C.
➡️ magmalake.org is the canonical, up-to-date source. This README is a pointer, not a copy: it drifts, the site doesn't.
| tin | what it is | version |
|---|---|---|
| iceberg | Native Apache Iceberg: metadata v1–v3, scan planning, reads, writes (fast-append, delete, overwrite, expire), REST and SQL catalogs | 0.8.2 |
| parquet | The first native Parquet reader/writer in Mojo, Arrow C Data Interface out, multi-core reads | 0.11.4 |
| avro | Object Container Files, read + write, all four codecs, schema-compiled cursor | 0.4.3 |
| objectstore | FileIO over local, HTTP(S), S3, GCS, Azure; pooled libcurl transport | 0.3.3 |
| postgres | libpq binding: queries, parameters, COPY, transactions — the deployable SQL catalog | 0.3.2 |
| sqlite | libsqlite3 binding; the local-development SQL catalog | 0.3.4 |
| roaring | 32/64-bit Roaring bitmaps, Iceberg deletion-vector framing | 0.1.3 |
| thrift | Compact + binary protocol runtime and the parquet.thrift structs |
0.1.3 |
| threads | pthreads for Mojo — spawn/join/pin, atomics, typed parallel_for, worker pools |
0.6.1 |
| zstd | FFI binding to libzstd | 0.1.4 |
| lz4 | FFI binding to liblz4 (block, frame, Hadoop framing) | 0.1.3 |
| brotli | FFI binding to libbrotli | 0.1.3 |
| snappy | Pure-Mojo Snappy, raw + framing | 0.1.4 |
| hashes | CRC-32, MurmurHash3 x86-32, XXH64 + Iceberg bucket transform | 0.1.3 |
| iceberg-rs | Rust cdylib over iceberg-rust, kept as a cross-implementation oracle | 0.1.2 |
All Apache-2.0, CI-tested on stable Mojo 1.2.0 and the current nightly, macOS and Linux.
Apple M4, reproducible via each repo's pixi run bench. Medians, with the tails and the full tables on magmalake.org/performance.
- Parquet read, 1M rows × 4 columns: 4.7 ms on one core, 2.1 ms on eight — faster than pyarrow with its own threads
- Parquet read, 100k rows of mixed and nested types: 3.3 ms on one core, 0.96 ms on eight
- Avro decode, manifest-shaped records: 19.2M records/s (fastavro: 1.74M)
Where it loses is on the site too, with the reason: threaded pyarrow is still ahead on nested reads, and the remaining gap is measured rather than estimated.
What building this stack taught us about making data code fast is packaged as an agent skill, mojo-data-performance, in magmalake/skills. It covers a pipeline's per-row loops over record batches and the readers, writers and codecs under them; how to tell decoding time from your own work's time without a second implementation to compare against; and the measurement discipline, the tests that catch a fast path which silently never fires, and the Mojo ownership and threading hazards underneath.
npx skills add magmalake/skills --yesThat installs it for whichever agents are found on the machine. To name the agents, pass them space-separated: the flag is variadic, and a comma-separated list is rejected. In Claude Code you can also install it as a plugin, with /plugin marketplace add magmalake/skills and then /plugin install skills@magmalake.
Full detail: magmalake.org
Performance · Correctness · Status · Writing
The site carries the complete tables, the correctness bar each tin is gated on, the upstream bugs this surfaced in PyIceberg, pyarrow and iceberg-rust, and what is not yet implemented.
What is proposed but unbuilt lives in the issue tracker (short link: magmalake.org/issues). Requests for a tin are welcome there.