Skip to content

Built-in engines: archive speed, round 2 - #1686

Merged
KnifeLemon merged 3 commits into
builtin-enginesfrom
p4/perf-archives-2
Nov 19, 2026
Merged

KnifeLemon merged 3 commits into
builtin-enginesfrom
p4/perf-archives-2

Conversation

@KnifeLemon

@KnifeLemon KnifeLemon commented Nov 19, 2026 •

Copy link
Copy Markdown
Owner

Second round on the built-in archive engines' speed, after #1672: the single-stream decoders and the LZMA / bzip2 encoders.

What changed

Unpacking

  • ReadAheadStream: a decoder runs on a thread of its own, a few 1 MiB blocks ahead of its reader. In 7z every coder that feeds another (LZMA → BCJ, the three LZMA streams of BCJ2) and the folder's output read ahead, so the chain's stages and the file writing (CRC, disk) run side by side; single compressed files (.xz, .lzma, .bz2, .gz, the outer layer of tar.*) too. The worker asks the decoder for 64 KiB at a time, hands out exactly its bytes and then its exception; disposing stops the worker and waits for it. 7z packed streams are read under the archive stream's lock.
  • LZMA2 segments (7z) and .xz blocks decoded on workers are handed out as they decode, not when the whole segment is done; after a worker failure the reader's own decoder skips what was already handed out (same bytes, same error as before).
  • LZMA decoder: match copies are put off and done back to back (their sources mostly miss the cache; done together the loads overlap), 16-byte chunk copies with a 32-byte guard past the dictionary (distances are checked against the dictionary, not the window), tree bits read both children's probabilities ahead, shorter bit masks. About 20 % less CPU per byte.

Packing

  • Binary-tree match finder (Normal, Maximum): the search runs on up to four threads. A position's tree only holds positions with the same 4-byte hash, so the trees are split between the threads by hash; the last thread to finish a batch adds the 2-/3-byte matches and joins the lists as one search would. Same match lists, so the same bytes on any number of threads (test). Walks stop 64 KiB short of the dictionary so no thread reads a tree slot another one already reused (the same with one thread; ratio change < 0.01 %).
  • Fast level: lazy parse (a look at the next position, a match one shorter but far nearer wins), 256 KiB dictionary (the tables stay in cache), 1 MiB blocks; 7z solid blocks stay 64 MiB.
  • Range encoder: trees, literals and direct bits keep low/range in locals and code each bit without a branch on its value.
  • bzip2: blocks on up to 8 threads (was 4); each group's cost in all tables is one vector sum.
  • Unchanged: every limit (UnpackBudget, dictionary caps, memory caps), cancellation, error messages; output deterministic regardless of thread count.

Tests

  • New in ArchiveThreadTests: match finder on 0/1/2/4 search threads gives the same bytes (dictionary smaller than the block); a distance past the dictionary but inside the window's guard is damage; read-ahead hands out every byte then the decoder's error, ends and stops with the stream, stops when cancelled; BCJ / BCJ2 folders from 7-Zip unpack the same with read-ahead on and off.
  • Archive filter (Archive, Lzma, SevenZip, Zip, Tar, Bzip, Xz, Nsis, Unpack, DiskImage, Checksum, FolderArchives, FolderIso, GeoZip, Snb, Cab, Lha, Alz, Egg, Lzip) with 7-Zip 26.05 present: 804 passed, 0 failed, 37 skipped (platform-only). dotnet format --verify-no-changes --severity warn clean on both projects.

Numbers

Same laptop and harness as #1672 (i7-1185G7, 4 cores / 8 threads, 16 GB, Windows 11, Release, .NET 10; inputs generated in code; unpack through ArchiveExtractor.ExtractAsync, pack through ArchivePacker.PackAsync; 7-Zip 26.05 x64 x / a with its default threads). Per operation: round 1 (#1672's head), round 2 and 7-Zip interleaved run by run, 3 runs, median wall time. Each run is a fresh process, so Filee's times include JIT warm-up (the app ships ReadyToRun). Other work on the machine made single runs noisy (±10–20 %), so read small differences as even; round 1's column is from today's runs, not copied from #1672.

Unpack (wall time)

Operation Round 1 (#1672) Round 2 7-Zip 26.05 round 2 vs 1 7-Zip time / round 2
7z LZMA2, text 256 MB (7-Zip MT: 2 blocks) 3.32 s 2.68 s 2.09 s 1.2× 0.78× (7-Zip faster)
7z LZMA2, text, made by 7-Zip on 1 thread 5.15 s 3.86 s 3.01 s 1.3× 0.78× (7-Zip faster)
7z LZMA, text 256 MB 4.84 s 3.73 s 3.10 s 1.3× 0.83× (7-Zip faster)
7z BCJ + LZMA2, x86-like 64 MB 2.19 s 1.82 s 1.21 s 1.2× 0.67× (7-Zip faster)
7z BCJ2, x86-like 64 MB 2.14 s 1.39 s 0.91 s 1.5× 0.66× (7-Zip faster)
7z solid, 10,000 small files (66 MB) 15.96 s 13.98 s 37.08 s 1.1× 2.65× (Filee faster)
7z non-solid, same files 17.84 s 17.83 s 37.39 s 1.0× 2.10× (Filee faster)
7z, random 32 MB 0.23 s 0.19 s 0.13 s 1.2× 0.68× (7-Zip faster)
7z, deflated data 64 MB 0.35 s 0.28 s 0.20 s 1.3× 0.72× (7-Zip faster)
xz, text 256 MB (7-Zip: 2 blocks) 3.47 s 3.17 s 2.18 s 1.1× 0.69× (7-Zip faster)
ZIP Deflate, 10,000 files + 330 MB 18.93 s 16.63 s 40.77 s 1.1× 2.45× (Filee faster)
ZIP Deflate64, same 22.49 s 21.26 s 41.98 s 1.1× 1.97× (Filee faster)
bz2, text 256 MB 6.04 s 5.69 s 5.44 s 1.1× 0.96× (7-Zip faster)
tar.gz 450 MB 32.04 s 30.84 s 48.09 s 1.0× 1.56× (Filee faster)
tar.xz 450 MB 47.16 s 23.26 s 51.01 s 2.0× 2.19× (Filee faster)
tar.bz2 450 MB 37.90 s 35.92 s 57.32 s 1.1× 1.60× (Filee faster)

Pack (wall time · archive size)

Operation Round 1 (#1672) Round 2 7-Zip 26.05 round 2 vs 1 7-Zip time / round 2
7z Fast (60 MB folder) 5.80 s · 37.76 MB out 5.37 s · 37.88 MB out 3.75 s · 37.55 MB out 1.1× 0.70× (7-Zip faster)
7z Normal 37.04 s · 33.53 MB out 34.04 s · 33.53 MB out 35.37 s · 33.15 MB out 1.1× 1.04× (Filee faster)
7z Maximum 64.64 s · 33.17 MB out 34.96 s · 33.17 MB out 36.83 s · 33.03 MB out 1.8× 1.05× (Filee faster)
xz Fast (52 MB file) 2.76 s · 29.80 MB out 2.26 s · 29.84 MB out 1.50 s · 29.62 MB out 1.2× 0.66× (7-Zip faster)
xz Normal 26.73 s · 26.86 MB out 17.78 s · 26.87 MB out 21.91 s · 26.72 MB out 1.5× 1.23× (Filee faster)
xz Maximum 35.83 s · 26.79 MB out 21.57 s · 26.79 MB out 21.66 s · 26.71 MB out 1.7× 1.00× (even)
ZIP Fast 1.85 s · 52.22 MB out 1.72 s · 52.22 MB out 1.82 s · 42.32 MB out 1.1× 1.06× (Filee faster)
ZIP Normal 2.99 s · 40.94 MB out 3.10 s · 40.94 MB out 5.45 s · 40.21 MB out 1.0× 1.76× (Filee faster)
ZIP Maximum 4.45 s · 40.94 MB out 4.71 s · 40.94 MB out 33.23 s · 39.97 MB out 0.9× 7.06× (Filee faster)
bz2 Normal 3.44 s · 25.69 MB out 2.09 s · 25.69 MB out 1.92 s · 25.70 MB out 1.6× 0.92× (7-Zip faster)
tar.gz Normal 4.43 s · 40.23 MB out 4.04 s · 40.23 MB out 12.73 s · 38.75 MB out 1.1× 3.15× (Filee faster)
tar.xz Normal 29.59 s · 33.69 MB out 29.45 s · 33.69 MB out 36.47 s · 33.23 MB out 1.0× 1.24× (Filee faster)
tar.bz2 Normal 4.98 s · 31.75 MB out 4.06 s · 31.75 MB out 3.79 s · 31.67 MB out 1.2× 0.93× (7-Zip faster)

Ratio at the changed levels (output size vs 7-Zip at the same level): 7z Fast +0.9 %, xz Fast +0.7 % (round 1: +0.6 %, +0.6 %); Normal and Maximum unchanged (the search threads don't change the bytes; the shorter tree reach costs < 0.01 %).

Still slower than 7-Zip, and why

  • One LZMA stream (7z LZMA, LZMA2 without resets, BCJ/BCJ2, and each 128 MB half of 7-Zip's -mx5 xz/7z): decoding is serial. Round 2 cut the decoder's CPU per byte by about 18–20 % (profiled: match copies missing the cache, tree bits waiting on loads); 7-Zip's hand-written decoder still needs about 0.75–0.8× of Filee's time. BCJ / BCJ2 conversion now runs on its own thread, so these folders are bound by the LZMA decode alone.
  • Fast packing (7z, xz): Filee's hash-chain search and lazy parse still take about 1.3–1.6× 7-Zip's CPU per byte (both use every thread); round 2 got there mostly by keeping the tables in cache and the range coder in registers.
  • bzip2 packing is about even now (0.92×); bzip2 unpacking was already even.
  • The tiny 7z cases (random / deflated data, < 0.3 s) are mostly process and thread start-up.

Charts (generated inputs only) are in the run notes, not in the repo.

🤖 Generated with Claude Code

KnifeLemon and others added 3 commits November 19, 2026 10:11
…ecoding

- ReadAheadStream: 7z coders that feed another coder, the folder output and single
  compressed files decode on a thread of their own while the next stage works
- 7z packed streams are read under the archive stream's lock (coders on threads)
- LZMA2 segments and xz blocks decoded on workers stream out as they decode
- LZMA decoder: match copies deferred and done back to back, 16-byte chunk
  copies with a guard past the dictionary, tree bits read both children ahead,
  shorter bit masks

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ector table costs

- Binary-tree match finder searches on up to four threads: the trees are split by
  hash, so the matches (and bytes) are the same whatever the number of threads;
  walks stop a little short of the dictionary so no thread reads a slot another
  took over
- Fast level: lazy parse with a look at the next position, 256 KiB dictionary
  (tables stay in cache), 1 MiB blocks; 7z solid blocks stay 64 MiB
- Range encoder keeps low and range in locals for trees, literals and direct
  bits, coded without a branch on the bit
- bzip2: blocks on up to eight threads, group costs of all tables in one vector
- Tests: search threads give the same bytes, a distance past the dictionary is
  damage within the window's guard, read-ahead errors, end, stop and cancel,
  BCJ/BCJ2 folders with coders on threads

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@KnifeLemon KnifeLemon added the built-in engines Filee: built-in engines (Project 4), on the builtin-engines branch, not released yet label Nov 19, 2026
@KnifeLemon
KnifeLemon merged commit 902b3c0 into builtin-engines Nov 19, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

built-in engines Filee: built-in engines (Project 4), on the builtin-engines branch, not released yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant