Repository navigation
feat(deflate,zlib,gzip): exact consumption, need-dictionary signal, Z_BLOCK stops - #165
Merged
Merged
Conversation
…_BLOCK stops Decoder-side gaps from #156, with zlib's inflate as the reference: - Exact consumed counts: the deflate decoder hands back the whole bytes still in its bit reader whenever it returns for a reason other than waiting on input (stream end, block stop, output full), so `consumed` ends at the byte holding the final block's last bit. zlib and gzip read their trailers straight from the input and drop the drain_trailing_bytes carry-over; bytes after the trailer (or after the last gzip member) are never counted. An empty stored block now counts as progress, so the decoder no longer returns with a header pending. - Need dictionary: an FDICT zlib stream without a configured dictionary reports its DICTID bytes as consumed, then returns the new Error::NeedDictionary (not poisoning) until zlib::Decoder::set_dictionary supplies a dictionary whose Adler-32 matches. New dictionary_id() and needs_dictionary(). The embed build still rejects FDICT as Unsupported. - Block stops: decode_block() on the deflate, zlib and gzip decoders returns after each non-final block (and, for the containers, right after the header), like inflate(Z_BLOCK); at_block_boundary(), in_final_block() and unused_bits() expose zlib's data_type bits. Tests: static CPython-zlib fixtures plus a cross-check of 50 fresh python3 streams (all levels, strategies, flush modes, window sizes, raw/zlib/gzip, preset dictionaries) against an independent block parser, whole and a byte at a time. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 31, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Decoder part of #156, with zlib's inflate as the reference.
API
Error::NeedDictionary, a new variant (Erroris#[non_exhaustive]).zlib::Decoder::dictionary_id() -> Option<u32>,needs_dictionary() -> bool,set_dictionary(&[u8]) -> Result<(), Error>. This is zlib's Z_NEED_DICT + inflateSetDictionary. A dictionary whose Adler-32 doesn't match the DICTID givesChecksumMismatchand the decoder keeps waiting.deflate::Decoder,zlib::Decoder,gzip::Decoder:decode_block(input, output): likeinflate(Z_BLOCK), it stops after each non-final block, and after the header for zlib and gzip.at_block_boundary(),in_final_block()unused_bits(): like zlib'sdata_type & 7.Behavior changes
consumed: when the stream ends, the bytes after it are never counted as consumed, for raw deflate, zlib and gzip (single- and multi-member). zlib and gzip read their trailers straight from the input; the carry-over buffer is gone.NeedDictionaryafter reading the DICTID. Before, it returnedUnsupported. Theembedbuild still returnsUnsupported.Validation
🤖 Generated with Claude Code