Repository navigation
Built-in engines: archive speed, round 2 - #1686
Merged
Merged
Conversation
…ecoding - ReadAheadStream: 7z coders that feed another coder, the folder output and single compressed files decode on a thread of their own while the next stage works - 7z packed streams are read under the archive stream's lock (coders on threads) - LZMA2 segments and xz blocks decoded on workers stream out as they decode - LZMA decoder: match copies deferred and done back to back, 16-byte chunk copies with a guard past the dictionary, tree bits read both children ahead, shorter bit masks Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ector table costs - Binary-tree match finder searches on up to four threads: the trees are split by hash, so the matches (and bytes) are the same whatever the number of threads; walks stop a little short of the dictionary so no thread reads a slot another took over - Fast level: lazy parse with a look at the next position, 256 KiB dictionary (tables stay in cache), 1 MiB blocks; 7z solid blocks stay 64 MiB - Range encoder keeps low and range in locals for trees, literals and direct bits, coded without a branch on the bit - bzip2: blocks on up to eight threads, group costs of all tables in one vector - Tests: search threads give the same bytes, a distance past the dictionary is damage within the window's guard, read-ahead errors, end, stop and cancel, BCJ/BCJ2 folders with coders on threads Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Second round on the built-in archive engines' speed, after #1672: the single-stream decoders and the LZMA / bzip2 encoders.
What changed
Unpacking
ReadAheadStream: a decoder runs on a thread of its own, a few 1 MiB blocks ahead of its reader. In 7z every coder that feeds another (LZMA → BCJ, the three LZMA streams of BCJ2) and the folder's output read ahead, so the chain's stages and the file writing (CRC, disk) run side by side; single compressed files (.xz, .lzma, .bz2, .gz, the outer layer of tar.*) too. The worker asks the decoder for 64 KiB at a time, hands out exactly its bytes and then its exception; disposing stops the worker and waits for it. 7z packed streams are read under the archive stream's lock.Packing
Tests
ArchiveThreadTests: match finder on 0/1/2/4 search threads gives the same bytes (dictionary smaller than the block); a distance past the dictionary but inside the window's guard is damage; read-ahead hands out every byte then the decoder's error, ends and stops with the stream, stops when cancelled; BCJ / BCJ2 folders from 7-Zip unpack the same with read-ahead on and off.dotnet format --verify-no-changes --severity warnclean on both projects.Numbers
Same laptop and harness as #1672 (i7-1185G7, 4 cores / 8 threads, 16 GB, Windows 11, Release, .NET 10; inputs generated in code; unpack through
ArchiveExtractor.ExtractAsync, pack throughArchivePacker.PackAsync; 7-Zip 26.05 x64x/awith its default threads). Per operation: round 1 (#1672's head), round 2 and 7-Zip interleaved run by run, 3 runs, median wall time. Each run is a fresh process, so Filee's times include JIT warm-up (the app ships ReadyToRun). Other work on the machine made single runs noisy (±10–20 %), so read small differences as even; round 1's column is from today's runs, not copied from #1672.Unpack (wall time)
Pack (wall time · archive size)
Ratio at the changed levels (output size vs 7-Zip at the same level): 7z Fast +0.9 %, xz Fast +0.7 % (round 1: +0.6 %, +0.6 %); Normal and Maximum unchanged (the search threads don't change the bytes; the shorter tree reach costs < 0.01 %).
Still slower than 7-Zip, and why
-mx5xz/7z): decoding is serial. Round 2 cut the decoder's CPU per byte by about 18–20 % (profiled: match copies missing the cache, tree bits waiting on loads); 7-Zip's hand-written decoder still needs about 0.75–0.8× of Filee's time. BCJ / BCJ2 conversion now runs on its own thread, so these folders are bound by the LZMA decode alone.Charts (generated inputs only) are in the run notes, not in the repo.
🤖 Generated with Claude Code