Skip to content

perf(archives): pack and unpack archives on several threads - #1672

Merged
KnifeLemon merged 3 commits into
builtin-enginesfrom
p4/perf-archives
Nov 19, 2026
Merged

KnifeLemon merged 3 commits into
builtin-enginesfrom
p4/perf-archives

Conversation

@KnifeLemon

@KnifeLemon KnifeLemon commented Nov 18, 2026 •

Copy link
Copy Markdown
Owner

Makes the built-in archive engines use the machine's processors, so packing and unpacking keep up with (and mostly beat) 7-Zip.

What changed

  • LZMA2 writer (7z, .xz, .tar.xz): the input is cut into blocks (1 dictionary at Fast, 2 at Normal, 4 at Maximum), each starting with a dictionary reset, and the blocks are coded side by side. Block ends depend only on the input and level, so the bytes are the same on 1 or 16 threads. With processors to spare, the binary-tree match finder runs on its own thread (same bytes too).
  • .xz writer: one .xz block per LZMA2 block, with both sizes in each block header (as xz -T writes), so xz, 7-Zip and Filee can unpack them side by side.
  • LZMA2 reader: 7z LZMA2 data with dictionary resets (what 7-Zip writes with several threads, and Filee now) and .xz blocks with sizes are decoded on worker threads and handed out in order. A part that fails is decoded again on the reader's thread, so damaged data gives the same bytes and the same error as before. Memory is bounded (an eighth of RAM, parts up to 512 MiB).
  • bzip2 (.bz2, .tar.bz2, 7z): Filee's own decoder replaces SharpCompress; Huffman codes up to 10 bits come from a table, and blocks are transformed back on worker threads. Randomised blocks (bzip2 0.9.0 and some old encoders) still go to SharpCompress, from the start, past what was already handed out.
  • Unpacking many files: small files (≤ 1 MiB) are written on worker threads; the reader's thread still reads each one, counts it against the budget and checks its size and CRC, so every error comes where it did. File times are set through the open handle (no second open), and the copy buffer is rented.
  • Match finder / encoder: compares 8 bytes at a time.
  • Unchanged: every limit (UnpackBudget, dictionary caps, entry limits), cancellation, error keys.

Numbers

Laptop i7-1185G7 (4 cores / 8 threads, 16 GB), Windows 11, Release, .NET 10. Each operation ran 3× per side, interleaved (before, after, 7-Zip 26.05 x64 with its default threads); other work on the machine made single runs noisy (±20 %), so read small differences as even. Inputs are generated in code (Zipf-word text, x86-like code with rel32 calls, random bytes, deflated text, 10,000 small files); archives to unpack were made by 7-Zip with defaults (-mx5, which uses several threads and so dictionary resets). Unpack goes through ArchiveExtractor.ExtractAsync (tar.* includes the second stage), pack through ArchivePacker.PackAsync; 7-Zip x / a (tar.* as two runs). Every archive Filee wrote was checked with 7z t in the tests.

Unpack (wall s, median of 3; peak working set MB)

Operation Filee before Filee after 7-Zip 26.05 after vs before 7-Zip vs after
7z LZMA2, text 256 MB (7-Zip MT: 2 blocks) 7.64 s · 103 MB 4.83 s · 496 MB 4.03 s · 338 MB 1.6× 0.83× (7-Zip faster)
7z LZMA2, text, made by 7-Zip on 1 thread 10.71 s · 103 MB 7.47 s · 103 MB 6.07 s · 69 MB 1.4× 0.81× (7-Zip faster)
7z LZMA, text 256 MB 8.05 s · 102 MB 8.09 s · 103 MB 6.43 s · 41 MB 1.0× 0.79× (7-Zip faster)
7z BCJ + LZMA2, x86-like 64 MB 3.32 s · 103 MB 3.78 s · 103 MB 2.28 s · 99 MB 0.9× 0.60× (7-Zip faster)
7z BCJ2, x86-like 64 MB 3.77 s · 94 MB 2.18 s · 95 MB 0.92 s · 81 MB 1.7× 0.42× (7-Zip faster)
7z solid, 10,000 small files (66 MB) 26.77 s · 111 MB 13.13 s · 113 MB 37.72 s · 93 MB 2.0× 2.87× (Filee faster)
7z non-solid, same files 28.02 s · 68 MB 17.41 s · 71 MB 37.21 s · 11 MB 1.6× 2.14× (Filee faster)
7z, random 32 MB 0.19 s · 101 MB 0.07 s · 101 MB 0.14 s · 72 MB 2.8× 2.01× (Filee faster)
7z, deflated data 64 MB 0.17 s · 101 MB 0.13 s · 101 MB 0.27 s · 136 MB 1.3× 2.03× (Filee faster)
xz, text 256 MB 10.77 s · 103 MB 3.11 s · 495 MB 2.17 s · 338 MB 3.5× 0.70× (7-Zip faster)
ZIP Deflate, 10,000 files + 330 MB 29.69 s · 64 MB 15.00 s · 68 MB 47.41 s · 13 MB 2.0× 3.16× (Filee faster)
ZIP Deflate64, same 37.47 s · 98 MB 20.88 s · 88 MB 45.63 s · 13 MB 1.8× 2.19× (Filee faster)
bz2, text 256 MB 23.27 s · 43 MB 6.71 s · 102 MB 6.32 s · 14 MB 3.5× 0.94× (7-Zip faster)
tar.gz 450 MB 29.06 s · 57 MB 31.20 s · 64 MB 88.10 s · 13 MB 0.9× 2.82× (Filee faster)
tar.xz 450 MB 66.90 s · 112 MB 29.38 s · 928 MB 95.71 s · 650 MB 2.3× 3.26× (Filee faster)
tar.bz2 450 MB 127.69 s · 63 MB 27.45 s · 153 MB 74.29 s · 14 MB 4.7× 2.71× (Filee faster)

Pack (wall s, median of 3 · peak working set MB · archive size)

Operation Filee before Filee after 7-Zip 26.05 after vs before 7-Zip vs after
7z Fast (60 MB folder) 13.7 s · 116 MB · 37.64 MB out 4.6 s · 567 MB · 37.76 MB out 3.2 s · 50 MB · 37.55 MB out 3.0× 0.69× (7-Zip faster)
7z Normal 78.5 s · 231 MB · 33.34 MB out 29.9 s · 722 MB · 33.53 MB out 33.1 s · 386 MB · 33.15 MB out 2.6× 1.11× (Filee faster)
7z Maximum 83.1 s · 706 MB · 33.17 MB out 59.8 s · 760 MB · 33.17 MB out 36.9 s · 783 MB · 33.03 MB out 1.4× 0.62× (7-Zip faster)
xz Fast (52 MB file) 10.2 s · 71 MB · 29.73 MB out 2.9 s · 342 MB · 29.80 MB out 1.3 s · 47 MB · 29.62 MB out 3.4× 0.45× (7-Zip faster)
xz Normal 45.2 s · 223 MB · 26.82 MB out 25.3 s · 469 MB · 26.86 MB out 20.8 s · 384 MB · 26.72 MB out 1.8× 0.82× (7-Zip faster)
xz Maximum 44.9 s · 678 MB · 26.79 MB out 34.3 s · 718 MB · 26.79 MB out 20.7 s · 550 MB · 26.71 MB out 1.3× 0.60× (7-Zip faster)
ZIP Fast 1.4 s · 47 MB · 52.22 MB out 1.5 s · 46 MB · 52.22 MB out 1.7 s · 67 MB · 42.32 MB out 0.9× 1.19× (Filee faster)
ZIP Normal 2.8 s · 48 MB · 40.94 MB out 2.7 s · 47 MB · 40.94 MB out 5.0 s · 62 MB · 40.21 MB out 1.0× 1.84× (Filee faster)
ZIP Maximum 4.0 s · 48 MB · 40.94 MB out 4.2 s · 46 MB · 40.94 MB out 26.2 s · 67 MB · 39.97 MB out 1.0× 6.18× (Filee faster)
bz2 Normal 2.6 s · 164 MB · 25.69 MB out 2.7 s · 157 MB · 25.69 MB out 1.7 s · 87 MB · 25.70 MB out 1.0× 0.62× (7-Zip faster)
tar.gz Normal 2.7 s · 50 MB · 40.23 MB out 2.6 s · 50 MB · 40.23 MB out 8.8 s · 10 MB · 38.75 MB out 1.0× 3.39× (Filee faster)
tar.xz Normal 64.3 s · 236 MB · 33.55 MB out 24.5 s · 633 MB · 33.69 MB out 32.7 s · 384 MB · 33.23 MB out 2.6× 1.34× (Filee faster)
tar.bz2 Normal 4.0 s · 161 MB · 31.75 MB out 3.9 s · 153 MB · 31.75 MB out 3.1 s · 87 MB · 31.67 MB out 1.0× 0.79× (7-Zip faster)

The two 7z rows on random / deflated data were measured again after the second commit (mostly stored LZMA2 data is copied on the reader's thread); with the first commit alone they were 0.18 s / 0.38 s.

Still slower than 7-Zip, and why

  • One LZMA stream with no dictionary reset (7z LZMA, LZMA2 made on one thread, BCJ/BCJ2 folders): only one thread can decode it; Filee's decoder needs about 1.3–1.45× the CPU of 7-Zip's per byte, and 7-Zip also runs BCJ2's four coders on their own threads.
  • xz / 7z LZMA2 from 7-Zip's -mx5: its blocks are 128 MB, so a 256 MB file has only 2 parts to share.
  • Packing at Fast and Maximum: Filee's LZMA encoder needs 1.3–1.8× the CPU of 7-Zip's per byte (match finder and price-based parse); Normal wins on wall time only because it spreads over more threads. At Maximum a 60 MB input is one block (4 dictionaries), so it gets two threads (match finder + parse), like 7-Zip, and the per-byte cost decides. 7-Zip's Fast also uses a 256 KB dictionary and 1 MB blocks.
  • bzip2: unpacking is about even (the Huffman step is serial); packing was already threaded and is unchanged.
  • Peak memory is higher when parts are decoded side by side (≤ 1/8 of RAM by design).

Tests

  • New ArchiveThreadTests: 1 vs 4 threads give the same xz bytes (Store/Fast/Normal); match finder on its own thread gives the same bytes (Normal/Maximum); xz/7z damage gives the same error and the same bytes before it on threads; 7z with resets unpacks on threads (and passes 7z t); bzip2 stream vs push decoder on damaged and cut data; randomised bzip2 goes to the fallback; cancellation stops xz and bzip2 readers.
  • LzmaTests: the block-header offset helpers now skip the sizes.
  • Filtered run (Archive|Lzma|SevenZip|Bzip2|Nsis|DiskImage|Checksum|FolderArchives|FolderIso|GeoZip|Snb, 563 tests), Release, 7-Zip present: two real failures fixed (NSIS reads back a file it just wrote → EntryWriter.AddCopy waits for it; randomised bzip2 → fallback); two timeouts under load passed on rerun.
  • dotnet format --verify-no-changes --severity warn clean for Filee.Engines and Filee.Engines.Tests.

🤖 Generated with Claude Code

- LZMA2 writer: the input is cut into blocks of one to four dictionaries
  (by level), each starting with a dictionary reset, coded side by side;
  the bytes do not depend on the number of threads. The binary-tree match
  finder searches ahead on a thread of its own when processors are spare.
- .xz writer: one block per LZMA2 block, both sizes in every block header,
  so xz, 7-Zip and Filee can unpack the blocks side by side.
- LZMA2 reader: 7z data with dictionary resets and .xz blocks with sizes
  are decoded on worker threads and handed out in order; a segment that
  fails is decoded again on the reader's thread, so damaged data gives the
  same bytes and error as before. Memory per stream stays within an eighth
  of the machine's.
- bzip2 (.bz2, .tar.bz2, 7z): Filee's own decoder with table-driven
  Huffman codes; blocks are transformed back on worker threads.
  Randomised blocks from old encoders still go to SharpCompress.
- Unpacking: small files are written on worker threads (size, CRC and
  budget are still checked on the reader's thread), the file time is set
  through the open handle, and the copy buffer is rented.
- Match finder and encoder compare eight bytes at a time.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@KnifeLemon KnifeLemon added the built-in engines Filee: built-in engines (Project 4), on the builtin-engines branch, not released yet label Nov 18, 2026
KnifeLemon and others added 2 commits November 19, 2026 07:42
Data that hardly shrank is stored chunks; handing it to worker threads cost
more than copying it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@KnifeLemon
KnifeLemon merged commit 7765fd4 into builtin-engines Nov 19, 2026
21 of 22 checks passed
adnan-bhaldar pushed a commit to adnan-bhaldar/Filee that referenced this pull request Nov 19, 2026
… and retries

The metadata job hit its 30-minute limit on KnifeLemon#1663, KnifeLemon#1669, KnifeLemon#1671, KnifeLemon#1672 and KnifeLemon#1679: apt took 23-30 minutes to
install Ubuntu's FFmpeg. Skip the runner's man-db index rebuild, skip recommends, give apt network timeouts and
three attempts, and limit the step to 12 minutes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

built-in engines Filee: built-in engines (Project 4), on the builtin-engines branch, not released yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant