Repository navigation
perf(archives): pack and unpack archives on several threads - #1672
Merged
Merged
Conversation
- LZMA2 writer: the input is cut into blocks of one to four dictionaries (by level), each starting with a dictionary reset, coded side by side; the bytes do not depend on the number of threads. The binary-tree match finder searches ahead on a thread of its own when processors are spare. - .xz writer: one block per LZMA2 block, both sizes in every block header, so xz, 7-Zip and Filee can unpack the blocks side by side. - LZMA2 reader: 7z data with dictionary resets and .xz blocks with sizes are decoded on worker threads and handed out in order; a segment that fails is decoded again on the reader's thread, so damaged data gives the same bytes and error as before. Memory per stream stays within an eighth of the machine's. - bzip2 (.bz2, .tar.bz2, 7z): Filee's own decoder with table-driven Huffman codes; blocks are transformed back on worker threads. Randomised blocks from old encoders still go to SharpCompress. - Unpacking: small files are written on worker threads (size, CRC and budget are still checked on the reader's thread), the file time is set through the open handle, and the copy buffer is rented. - Match finder and encoder compare eight bytes at a time. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Data that hardly shrank is stored chunks; handing it to worker threads cost more than copying it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Nov 19, 2026
adnan-bhaldar
pushed a commit
to adnan-bhaldar/Filee
that referenced
this pull request
Nov 19, 2026
… and retries The metadata job hit its 30-minute limit on KnifeLemon#1663, KnifeLemon#1669, KnifeLemon#1671, KnifeLemon#1672 and KnifeLemon#1679: apt took 23-30 minutes to install Ubuntu's FFmpeg. Skip the runner's man-db index rebuild, skip recommends, give apt network timeouts and three attempts, and limit the step to 12 minutes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes the built-in archive engines use the machine's processors, so packing and unpacking keep up with (and mostly beat) 7-Zip.
What changed
xz -Twrites), so xz, 7-Zip and Filee can unpack them side by side.Numbers
Laptop i7-1185G7 (4 cores / 8 threads, 16 GB), Windows 11, Release, .NET 10. Each operation ran 3× per side, interleaved (before, after, 7-Zip 26.05 x64 with its default threads); other work on the machine made single runs noisy (±20 %), so read small differences as even. Inputs are generated in code (Zipf-word text, x86-like code with rel32 calls, random bytes, deflated text, 10,000 small files); archives to unpack were made by 7-Zip with defaults (
-mx5, which uses several threads and so dictionary resets). Unpack goes throughArchiveExtractor.ExtractAsync(tar.* includes the second stage), pack throughArchivePacker.PackAsync; 7-Zipx/a(tar.* as two runs). Every archive Filee wrote was checked with7z tin the tests.Unpack (wall s, median of 3; peak working set MB)
Pack (wall s, median of 3 · peak working set MB · archive size)
The two 7z rows on random / deflated data were measured again after the second commit (mostly stored LZMA2 data is copied on the reader's thread); with the first commit alone they were 0.18 s / 0.38 s.
Still slower than 7-Zip, and why
-mx5: its blocks are 128 MB, so a 256 MB file has only 2 parts to share.Tests
ArchiveThreadTests: 1 vs 4 threads give the same xz bytes (Store/Fast/Normal); match finder on its own thread gives the same bytes (Normal/Maximum); xz/7z damage gives the same error and the same bytes before it on threads; 7z with resets unpacks on threads (and passes7z t); bzip2 stream vs push decoder on damaged and cut data; randomised bzip2 goes to the fallback; cancellation stops xz and bzip2 readers.LzmaTests: the block-header offset helpers now skip the sizes.EntryWriter.AddCopywaits for it; randomised bzip2 → fallback); two timeouts under load passed on rerun.dotnet format --verify-no-changes --severity warnclean for Filee.Engines and Filee.Engines.Tests.🤖 Generated with Claude Code