Repository navigation
perf(deflate,checksum): table-driven inflate with a fast loop; faster checksums - #167
Merged
Merged
Conversation
Adler-32 now sums 32 lanes per block (the inner loop auto-vectorises) and folds the lane totals back into a and b, reducing once per 128 KiB: ~4.0 -> ~20-27 GB/s on an Apple M5 Max. CRC-32 runs four independent slice-by-8 chains over consecutive 2 KiB stretches and joins them by multiplying with x^(8*2048) mod P (a compile-time constant), so the table loads overlap: ~2.6 -> ~7 GB/s. Inputs under 8 KiB take the plain slice-by-8 loop as before. Both are checked against bit/byte-at-a-time reference implementations over many lengths and split points, including a block of 0xFF bytes that drives the Adler lane sums to their bound. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… into the output The decoder was ~0.25x zlib: a bit-walking canonical decoder behind a 9-bit LUT, one state-machine step per field, every byte written twice (output and ring window), the bit buffer refilled a byte at a time. - New Huffman tables (src/deflate/inftable.rs): 11-bit literal/length and 8-bit distance root tables with subtables for longer codes. Each entry packs the literal byte or the length/distance base, the code length and the extra-bit count, with code+extra bits in the low byte so a whole length or distance field is dropped with one shift. Incomplete codes, reserved symbols (286/287, distance 30/31) and empty codes give the same errors as before. The fixed-code tables are built once per run of fixed blocks; the tables live in the decoder instead of a per-block box. - A fast loop runs while >= 16 input bytes and >= 277 output bytes remain: 64-bit bit buffer refilled 8 bytes at a time, up to two literals per refill, length and distance decoded in one go, the next entry looked up before the match copy so the table load overlaps it, and match copies in 8/16-byte chunks (with pattern expansion for distances under 8). The symbol-by-symbol state machine still handles the edges, so streaming with any buffer sizes, Z_BLOCK stops, exact consumed counts and errors are unchanged. - Matches within the current call's output are copied from the output itself; the window only holds earlier calls' output and is brought up to date once per call (one copy of at most 32 KiB). Stored blocks are bulk copied. Checked against the previous decoder with a differential test (not committed) over thousands of random streams from every encoder level and strategy, corrupted, truncated and random inputs, small windows and preset dictionaries, under 1-byte to whole-buffer input/output chunking and decode_block: identical results, consumed/written counts, statuses, errors, unused_bits and block-boundary state. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Mark subtable pointers with their own bit so a lookup tests one bit, and compare the fast loop's positions against precomputed limits. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 31, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Decoder speed item of #156. The deflate decoder is now 3.5–4.6× faster: 0.8–1.0× zlib on text, source and binaries, and up to 1.34× zlib on repetitive data (before: 0.55–0.75×).
Measured on Apple M5 Max against system zlib 1.2.12; full table in BENCH.md. gzip trails on repetitive data because the system zlib uses the ARM CRC instructions, and calling them needs
unsafe, which the crate forbids.Changes
src/deflate/inftable.rs). Each entry holds the value, the code length and the extra bits.Behavior
No API changes. Outputs, errors, consumed counts, block stops and unused bits are unchanged. This was checked by a differential test against the previous decoder, which was not committed:
Also, 60,000 corrupted zlib and gzip streams decoded with no panics. Rebased on #166, and every test suite passes.
🤖 Generated with Claude Code