Part of magmalake — data lake building blocks in Mojo.
A Mojo binding to libzstd — one-shot and streaming compress/decompress —
via FFI, built the same way zlib.mojo
wraps zlib: a small C shim (shim/zstd_wrapper.c) compiled to
libzstdmojo.so and loaded through an OwnedDLHandle. No link flags for
consumers; the shim is dlopened at runtime.
zstd is Apache Iceberg's default Parquet codec, a Puffin blob codec, and an
Avro codec — that's the headline use this binding targets. It wraps the real
libzstd (conda-forge's zstd package); a pure-Mojo zstd is out of scope.
pixi shelf add zstd-mojoWorking with a coding agent? npx skills add mojoshelf/mojoshelf --skill mojoshelf-consume --yes teaches it to find and install tins itself — it installs the shelf CLI too.
That resolves the tin from mojoshelf and adds it — along with the tins it depends on — as pixi git source dependencies. magmalake tins are not published to a conda channel, so pixi add zstd-mojo will not find them.
- pixi
- Nothing else —
zstd(the C library) and the shim are pulled in and built automatically as pixi dependencies.
from zstd import compress, decompress
var z = compress(raw) # Span[UInt8] -> List[UInt8], level 1..22 (default 3)
var raw2 = decompress(z) # -> List[UInt8], byte-for-byte back
var n = decompress_into(z, dst) # write straight into a caller-owned buffer
# (e.g. a Parquet page's declared size) —
# no growth, no extra allocation
is_zstd_frame(z) # Bool — magic-number sniff, no allocation
frame_content_size(z) # Optional[Int] — decompressed size from
# the frame header, if it carries oneBuild a consumer with this package on the import path:
mojo build your.mojo -I ../zstd.mojo/src -o your-binThe shim is found at run time, with or without CONDA_PREFIX
(src/zstd/_dlopen.mojo): $CONDA_PREFIX/lib/libzstdmojo.so first, then
next to the executable and in its ../lib, then on the binary's rpath,
which mojo build points at the environment it built in. So a binary built
with pixi run mojo build also runs from a plain shell. When none has it,
the first zstd call raises an Error naming the places it looked and saying
to set CONDA_PREFIX. For distribution, put libzstdmojo.so next to the
binary or in ../lib.
For data that doesn't fit in memory at once, Compressor/Decompressor wrap
ZSTD_CCtx/ZSTD_DCtx (the modern API's ZSTD_CStream/ZSTD_DStream):
from zstd import Compressor, Decompressor
var c = Compressor(level=9)
var out = List[UInt8]()
for chunk in chunks:
out.extend(c.update(chunk))
out.extend(c.finish()) # flush the trailing block + frame epilogue
var d = Decompressor()
var back = List[UInt8]()
for chunk in compressed_chunks:
back.extend(d.update(chunk))
back.extend(d.finish()) # no-op — kept for symmetry with Compressordecompress() itself falls back to this same streaming path internally when
a frame's content size is unknown (e.g. streamed with an unset pledged size),
growing its output buffer as it goes.
Errors raise with libzstd's own error name, straight from
ZSTD_getErrorName — e.g. zstd.decompress: Corruption_detected or
zstd.decompress_into: Dst_Too_Small.
pixi run test # builds the shim, then round-trips + a known-frame decodeCovers: round trips at several sizes (0 B, 1 B, 1 KiB, 1 MiB compressible,
1 MiB random) and levels; decoding a frame produced independently by Python
zstandard, baked in as byte constants; streaming matching one-shot output;
and a corrupt frame raising.
pixi run -e bench bench # the table below
pixi run -e bench bench -- --json # every repetition, for tracking
pixi run -e bench bench -- --only bench_compress_random_9One M-series macOS core, 64 MiB per shape, through bench.mojo — mean of three timed repetitions, spread under 1% on every row:
| data | level | compress | decompress | ratio |
|---|---|---|---|---|
| compressible | 1 | 20.5 GB/s | 11.2 GB/s | 10824x |
| compressible | 3 | 13.0 GB/s | 11.3 GB/s | 10824x |
| compressible | 9 | 2.12 GB/s | 12.0 GB/s | 747x |
| compressible | 19 | 2.18 GB/s | 22.3 GB/s | 11792x |
| random | 1 | 9.54 GB/s | 33.3 GB/s | ~1.0x |
| random | 3 | 10.3 GB/s | 33.3 GB/s | ~1.0x |
| random | 9 | 5.79 GB/s | 33.5 GB/s | ~1.0x |
Every push to main re-runs these on a GitHub runner and appends to a history
published at
magmalake.github.io/zstd.mojo/benchmarks.
Those numbers are slower and noisier than the table above, which was taken on
an M4 — each history is keyed by machine, so the two stay separate series and
are never averaged together.
Rates are against the uncompressed size in both directions, so every level shares a scale. Level 19 costs barely more than level 9 on this input, because it compresses about 11,800x and the expensive search finds a match almost immediately — that is a property of the synthetic buffer, not a general claim about level 19.
What these numbers measure. This tin is a binding, not an implementation: the compression is libzstd's, reached through the C shim below. So the table measures libzstd plus the cost of crossing the FFI boundary, which at 64 MiB per call is noise. It is not a measurement of Mojo code — the Mojo here is 394 lines of binding, and the only part of it these numbers can indict is per-call overhead, which is the subject of the next paragraph.
Figures published before 2026-09-01 were single cold passes and ran lower on the fast rows (compressible level 1 read ~10.2 GB/s against 20.5 here); at 3 ms per pass, building and first-touching the buffer dominated them.
The shim is dlopened once per process and never closed, so the one-shot
compress()/decompress() calls are safe in a hot loop. They did not used to
be: through 0.1.0 every call opened and closed the handle, which on macOS costs
around 450 µs whether the library is already resident or not. That fixed cost
swamped the actual work on anything page-sized — a 24 MiB ZSTD Parquet file
(332 pages) spent ~150 ms in dlopen and ~25 ms decompressing. Caching the
handle took the fixed cost of a call from ~415 µs to ~1.4 µs.
shim/ is a pixi-build-cmake package: CMakeLists.txt
finds conda-forge's zstd CMake package config (find_package(zstd CONFIG)
— zstd ships one, unlike zlib's bundled CMake Find module) and links
zstd_wrapper.c into libzstdmojo.so, installed to $CONDA_PREFIX/lib. It's
a pixi source dependency (zstd-shim = { path = "shim" }) of the top-level
workspace, so pixi install/pixi run builds it automatically — no manual
build step.
The shim itself stays thin: it's mostly ABI hygiene (building
ZSTD_inBuffer/ZSTD_outBuffer structs internally so Mojo never replicates
their layout) rather than reimplementing anything — one-shot compress/decompress,
error-code checks, and streaming all forward straight to the matching libzstd
call.
- Targets
osx-arm64,linux-64,linux-aarch64. - One-shot and streaming compress/decompress, frame introspection
(
is_zstd_frame,frame_content_size). No dictionary support, no multi-threaded compression — out of scope for a leaf tin; open an issue if you need either. - Builds on one pinned Mojo nightly,
mojo-compiler ==1.3.0.dev2026111805; a workspace that consumes the package needs that same pin.