Skip to content

feat(deflate,zlib,gzip): exact consumption, need-dictionary signal, Z_BLOCK stops - #165

Merged
MagicalTux merged 1 commit into
masterfrom
deflate-decoder-controls
Oct 31, 2026
Merged

MagicalTux merged 1 commit into
masterfrom
deflate-decoder-controls

Conversation

@MagicalTux

Copy link
Copy Markdown
Member

Decoder part of #156, with zlib's inflate as the reference.

API

  • Error::NeedDictionary, a new variant (Error is #[non_exhaustive]).
  • zlib::Decoder::dictionary_id() -> Option<u32>, needs_dictionary() -> bool, set_dictionary(&[u8]) -> Result<(), Error>. This is zlib's Z_NEED_DICT + inflateSetDictionary. A dictionary whose Adler-32 doesn't match the DICTID gives ChecksumMismatch and the decoder keeps waiting.
  • deflate::Decoder, zlib::Decoder, gzip::Decoder:
    • decode_block(input, output): like inflate(Z_BLOCK), it stops after each non-final block, and after the header for zlib and gzip.
    • at_block_boundary(), in_final_block()
    • unused_bits(): like zlib's data_type & 7.

Behavior changes

  • Exact consumed: when the stream ends, the bytes after it are never counted as consumed, for raw deflate, zlib and gzip (single- and multi-member). zlib and gzip read their trailers straight from the input; the carry-over buffer is gone.
  • FDICT without a configured dictionary now returns NeedDictionary after reading the DICTID. Before, it returned Unsupported. The embed build still returns Unsupported.
  • An empty stored block (as from a sync flush) now counts as progress, so the decoder no longer returns with the next block header unread.

Validation

  • Static fixtures from CPython's zlib cover Z_BLOCK, sync, full and partial flushes, fixed Huffman, and a preset dictionary.
  • Exact consumption is checked across trailing garbage and single/multi-member gzip, feeding input in 1, 2, 7 bytes or all at once, with 1/5/4096-byte outputs.
  • Block stops are compared against an independent bit-level parser.
  • A live cross-check runs 50 fresh Python zlib streams (levels 0–9, every strategy, every flush mode, window bits 9–15, dictionaries). It skips when python3 is missing.
  • Decode speed is unchanged: 189–192 MB/s before and after, run interleaved.
  • Rebased onto feat(deflate,zlib,gzip): level 0, strategies, mem level, window bits, preset dictionary, partial/block flush #164; both test suites pass together.

🤖 Generated with Claude Code

…_BLOCK stops

Decoder-side gaps from #156, with zlib's inflate as the reference:

- Exact consumed counts: the deflate decoder hands back the whole bytes
  still in its bit reader whenever it returns for a reason other than
  waiting on input (stream end, block stop, output full), so `consumed`
  ends at the byte holding the final block's last bit. zlib and gzip
  read their trailers straight from the input and drop the
  drain_trailing_bytes carry-over; bytes after the trailer (or after the
  last gzip member) are never counted. An empty stored block now counts
  as progress, so the decoder no longer returns with a header pending.
- Need dictionary: an FDICT zlib stream without a configured dictionary
  reports its DICTID bytes as consumed, then returns the new
  Error::NeedDictionary (not poisoning) until
  zlib::Decoder::set_dictionary supplies a dictionary whose Adler-32
  matches. New dictionary_id() and needs_dictionary(). The embed build
  still rejects FDICT as Unsupported.
- Block stops: decode_block() on the deflate, zlib and gzip decoders
  returns after each non-final block (and, for the containers, right
  after the header), like inflate(Z_BLOCK); at_block_boundary(),
  in_final_block() and unused_bits() expose zlib's data_type bits.

Tests: static CPython-zlib fixtures plus a cross-check of 50 fresh
python3 streams (all levels, strategies, flush modes, window sizes,
raw/zlib/gzip, preset dictionaries) against an independent block
parser, whole and a byte at a time.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant