Skip to content

Built-in engines: archive speed, round 3, and .zst - #1692

Merged
KnifeLemon merged 6 commits into
builtin-enginesfrom
p4/perf-archives-3
Nov 19, 2026
Merged

KnifeLemon merged 6 commits into
builtin-enginesfrom
p4/perf-archives-3

Conversation

@KnifeLemon

@KnifeLemon KnifeLemon commented Nov 19, 2026 •

Copy link
Copy Markdown
Owner

Third round on the built-in archive engines' speed (after #1672 and #1686), and reading Zstandard files (#1646).

.zst, .tar.zst, .tzst (#1646)

  • New formats zst (ZST, .zst) and tzst (TAR.ZST, .tar.zst, .tzst), read only, registered like XZ / TAR.XZ: extensions, fields and grade (specialist, developer), the extractor's lists (built in; 7-Zip as the fallback reader), TAR.ZST unpacked in two stages, ZST converts to GZ / BZ2 / XZ, every readable archive target. filee formats zst now knows it.
  • ZstdReader puts SharpCompress's decoder (already used for zstd RPM payloads) behind Filee's checks: the first frame's magic (other data is "not a valid archive" instead of an empty file), frames one after another and skippable frames, a cut or trailing data as "damaged", a window over 128 MiB (zstd --long=28 and up) left to 7-Zip, as the zstd tool refuses it without --memory (the decoder would set the window aside before a byte shows the file is real). RPM payloads in zstd use the same reader. Every existing limit (UnpackBudget, the room on the drive) applies.
  • Not written. SharpCompress has a zstd encoder (its multithreaded output is the same for any number of threads ≥ 1), but docs/FORMAT-RULES.md rule 5 keeps a new developer-only output out while the developer field is on by default. Writing is a small change once that is decided.
  • Library step (v104): existing users' Archives donut learns tar.zst, tzst and zst (custom claims stay put); new installs get them already (ExtensionsOf(FormatCategory.Archive)). ZstSettingsTests covers it.
  • Shared files touched: FormatRegistry, FormatFields (+ FormatFieldsTests pinned lists, specialist count 131 → 133), README.md (one line), docs/ENGINES.md (archive table rows). No i18n, no CHANGELOG.

Speed

Unpacking

  • LZMA decoder: when a match is decoded, its source is asked into the cache (Sse.Prefetch0, x86; nothing elsewhere), so it arrives while the next symbols are decoded and before the deferred copy runs (both hinted addresses stay inside the window). The three older repeat distances moved to fields, leaving registers to the coder. Same bytes. This is the only reason Filee.Engines now allows unsafe code: the hint takes a pointer, nothing reads through one.
  • bzip2 decoder: the move-to-front shift moves 16 bytes at a time instead of calling a memmove per symbol; one thread decodes every block's symbols while others undo the transform, so this step set the pace. Same bytes.
  • BCJ2: the main stream's bytes up to each possible jump are copied in one go instead of byte by byte (CPU only; the LZMA streams set the pace).

Packing

  • Hash-chain search (Fast): a search of its own with its values in registers and one two-byte check per candidate (same matches as before), a 10-bit hash for the two-byte table (4 KB instead of 256 KB per thread, so the tables of eight threads fit the cache better; output 0.001 % smaller), the next position's table slots prefetched. Still the same bytes for any number of threads (Lzma2_blocks_give_the_same_bytes_on_one_thread_or_four). Normal and Maximum: unchanged bytes.
  • bzip2 encoder: the same 16-byte move-to-front shift, and the transform writes single rows without a span fill. Same bytes.
  • Tried and dropped (measured): conditional-move forms of the direct bits (the JIT keeps a branch in loops), decoding the literal after a match inside the match loop, AggressiveOptimization on the decoder loops (−1 % cold, +3 % warm), a smaller four-byte hash at Fast (slower and larger), keeping bzip2's bit buffer in locals (slower).

Tests

  • New: .tar.zst and .zst unpack in-process, RPM payload in zstd, frames + skippable frames, "not an archive" / damaged / wide window → 7-Zip refusal, names recognised and read only, zstd fuzz seeds, every readable archive (incl. ZST / TAR.ZST) converts in one step, ZST → XZ but nothing → ZST, the 16-byte move-to-front shift against a byte loop for every count.
  • Archive filter of Built-in engines: archive speed, round 2 #1686 (+ Zstd), Release, 7-Zip 26.05 present: 834 passed, 0 failed, 18 skipped; then SevenZipReaderTests, ArchiveThreadTests and ArchiveFuzzTests again after the BCJ2 change: 81 passed; after bounding the prefetch hint to the window, the LZMA / 7z / thread / fuzz / in-process classes: 152 passed. Filee.Core.Tests: 1769 passed. dotnet format --verify-no-changes --severity warn clean on Filee.Engines, Filee.Engines.Tests, Filee.Core, Filee.Core.Tests.
  • Bench checks: decoded CRCs of ten archives equal round 1's; Fast output round-trips; Normal output byte-identical to round 2.

Numbers

Same laptop and harness as #1672 / #1686 (i7-1185G7, 4 cores / 8 threads, 16 GB, Windows 11, Release, .NET 10; inputs generated in code; unpack through ArchiveExtractor.ExtractAsync, pack through ArchivePacker.PackAsync; 7-Zip 26.05 x64 x / a with its default threads). Per operation: round 2 (builtin-engines at #1686's merge), round 3 and 7-Zip interleaved run by run, 3 runs, median wall time; each run a fresh process, so Filee's times include JIT warm-up. Other agents' builds and tests kept the machine busy the whole time, so this round the bench runs every measured process (Filee and 7-Zip alike) at high priority and waits up to 20 s for a quieter moment before each run; even so, single runs moved by ±10–20 %, and rows where nothing changed (tar.gz, ZIP Deflate64, the random / deflated 7z) show that noise. Absolute times are not comparable with #1686's table (other machine state); the round 2 column is from today's runs. The zstd inputs (level 3, one frame) come from SharpCompress's encoder, since 7-Zip only reads zstd.

Unpack (wall time)

Operation Round 2 (#1686) Round 3 7-Zip 26.05 round 3 vs 2 7-Zip time / round 3
7z LZMA2, text 256 MB (7-Zip MT: 2 blocks) 3.17 s 3.35 s 2.68 s 0.95× 0.80× (7-Zip faster)
7z LZMA2, text, made by 7-Zip on 1 thread 4.02 s 3.88 s 3.47 s 1.04× 0.89× (7-Zip faster)
7z LZMA, text 256 MB 3.93 s 3.66 s 3.10 s 1.07× 0.85× (7-Zip faster)
7z BCJ + LZMA2, x86-like 64 MB 2.01 s 1.88 s 1.17 s 1.06× 0.62× (7-Zip faster)
7z BCJ2, x86-like 64 MB 1.71 s 1.68 s 0.91 s 1.02× 0.54× (7-Zip faster)
7z solid, 10,000 small files (66 MB) 12.03 s 11.52 s 30.81 s 1.04× 2.67× (Filee faster)
7z non-solid, same files 15.45 s 14.58 s 31.10 s 1.06× 2.13× (Filee faster)
7z, random 32 MB 0.14 s 0.13 s 0.10 s 1.08× 0.71× (7-Zip faster)
7z, deflated data 64 MB 0.20 s 0.22 s 0.16 s 0.90× 0.72× (7-Zip faster)
xz, text 256 MB (7-Zip: 2 blocks) 2.54 s 2.56 s 2.21 s 0.99× 0.86× (7-Zip faster)
ZIP Deflate, 10,000 files + 330 MB 13.87 s 13.79 s 33.51 s 1.01× 2.43× (Filee faster)
ZIP Deflate64, same 22.98 s 19.84 s 46.17 s 1.16× 2.33× (Filee faster)
bz2, text 256 MB 5.80 s 5.43 s 6.65 s 1.07× 1.23× (Filee faster)
tar.gz 450 MB 27.45 s 18.56 s 40.97 s 1.48× 2.21× (Filee faster)
tar.xz 450 MB 22.81 s 19.93 s 43.01 s 1.14× 2.16× (Filee faster)
tar.bz2 450 MB 29.73 s 26.23 s 48.21 s 1.13× 1.84× (Filee faster)
zst, text 256 MB (level 3) 0.78 s 0.74 s 0.61 s 1.05× 0.83× (7-Zip faster)
tar.zst 450 MB (level 3, new) — 28.05 s 47.51 s — 1.69× (Filee faster)

Pack (wall time · archive size)

Operation Round 2 (#1686) Round 3 7-Zip 26.05 round 3 vs 2 7-Zip time / round 3
7z Fast (60 MB folder) 5.07 s · 37.88 MB 3.89 s · 37.88 MB 2.92 s · 37.55 MB 1.30× 0.75× (7-Zip faster)
7z Normal 31.25 s · 33.53 MB 29.97 s · 33.53 MB 31.30 s · 33.15 MB 1.04× 1.04× (Filee faster)
7z Maximum 31.56 s · 33.17 MB 31.42 s · 33.17 MB 33.84 s · 33.03 MB 1.00× 1.08× (Filee faster)
xz Fast (52 MB file) 1.71 s · 29.84 MB 1.56 s · 29.84 MB 1.12 s · 29.62 MB 1.10× 0.72× (7-Zip faster)
xz Normal 15.49 s · 26.87 MB 15.46 s · 26.87 MB 19.25 s · 26.72 MB 1.00× 1.24× (Filee faster)
xz Maximum 19.40 s · 26.79 MB 19.16 s · 26.79 MB 19.64 s · 26.71 MB 1.01× 1.03× (Filee faster)
ZIP Fast 1.75 s · 52.22 MB 1.60 s · 52.22 MB 1.97 s · 42.32 MB 1.09× 1.23× (Filee faster)
ZIP Normal 3.03 s · 40.94 MB 3.08 s · 40.94 MB 5.04 s · 40.21 MB 0.98× 1.63× (Filee faster)
ZIP Maximum 4.10 s · 40.94 MB 4.12 s · 40.94 MB 23.98 s · 39.97 MB 1.00× 5.82× (Filee faster)
bz2 Normal 1.73 s · 25.69 MB 1.60 s · 25.69 MB 1.33 s · 25.70 MB 1.08× 0.83× (7-Zip faster)
tar.gz Normal 2.85 s · 40.23 MB 2.88 s · 40.23 MB 8.71 s · 38.75 MB 0.99× 3.03× (Filee faster)
tar.xz Normal 23.41 s · 33.69 MB 23.59 s · 33.69 MB 30.44 s · 33.23 MB 0.99× 1.29× (Filee faster)
tar.bz2 Normal 3.31 s · 31.75 MB 3.11 s · 31.75 MB 2.87 s · 31.67 MB 1.07× 0.92× (7-Zip faster)

Ratio at the changed levels (output size vs 7-Zip at the same level): 7z Fast +0.9 %, xz Fast +0.7 % (as in round 2; the new two-byte hash makes Fast's output 0.001 % smaller). Normal and Maximum: the same bytes as round 2.

JIT warm-up. The shipped app is ReadyToRun; this bench is not, so a short single-stream job spends its first few hundred milliseconds in unoptimized code (a profile of the BCJ2 run had about as many samples in Tier0 decoder code as in optimized code). The same operations with the bench published ReadyToRun (3 runs each, interleaved, same conditions):

Operation Round 3, JIT (as above) Round 3, ReadyToRun 7-Zip 26.05 7-Zip / ReadyToRun
7z LZMA, text 256 MB 3.66 s 3.54 s 2.92 s 0.82×
7z LZMA2, text 256 MB (2 blocks) 2.30 s 2.05 s 1.79 s 0.87×
7z BCJ + LZMA2, x86-like 64 MB 1.97 s 1.45 s 1.13 s 0.78×
7z BCJ2, x86-like 64 MB 1.70 s 1.08 s 0.81 s 0.75×
xz, text 256 MB 2.28 s 2.23 s 1.74 s 0.78×
7z Fast 4.12 s 3.45 s 2.75 s 0.80×
xz Fast 2.21 s 1.48 s 1.18 s 0.80×
bz2 Normal 1.70 s 2.26 s 1.43 s 0.63×

(bzip2 packing is the exception: its precompiled sort is slower until it tiers up.)

Still slower than 7-Zip, and why

  • One LZMA stream (7z LZMA, LZMA2 without resets, BCJ / BCJ2, and each half of 7-Zip's -mx5 xz / 7z): decoding is serial and bound by the range coder's dependency chain (shift, multiply, compare, per bit) plus cache misses. The prefetch took about 5–6 % off the decoder (paired runs side by side); with ReadyToRun the gap is 0.75–0.87×. What is left is code generation: the JIT keeps a branch where 7-Zip's assembly uses conditional moves (tried, it can't be had in a loop today), and the decoder loops still spill a few values. x86-like data (BCJ, BCJ2) is the worst case because most of a cold run's 1–2 s is warm-up.
  • Fast packing (7z, xz): 25–30 % less CPU than round 2 at 7z Fast, about 8 % at xz Fast, but still 1.25× (7z) to 1.5× (xz) 7-Zip's CPU; the hash-chain walk waits on cache misses (eight threads' tables don't fit the 12 MB cache), and the lazy parse searches the next position as 7-Zip does.
  • bzip2 packing (0.83×; TAR.BZ2 0.92×): the suffix sort is most of it and wasn't touched this round.
  • zst (0.83×, 0.6–0.8 s): SharpCompress's decoder; TAR.ZST as a whole is 1.7× faster than 7-Zip's two stages.
  • Tiny 7z cases (random / deflated data, < 0.3 s): process and thread start.

Charts (generated inputs only) are in the run notes, not in the repo.

🤖 Generated with Claude Code

KnifeLemon and others added 2 commits November 19, 2026 13:40
ZST and TAR.ZST become archive formats like XZ and TAR.XZ: recognised by
extension, unpacked in-process (the TAR in a second stage), converted to
every archive target, and ZST to the other single-file formats. ZstdReader
puts SharpCompress's decoder behind Filee's checks: the first frame's magic
(other data is "not an archive", not an empty file), frames one after
another and skippable frames, damage and early ends as "damaged", windows
over 128 MiB left to 7-Zip. RPM payloads in zstd use the same reader.

Read only: writing would add a developer-only output, which FORMAT-RULES
rule 5 keeps out while the developer field is on by default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- LZMA decoder: a match's source is asked into the cache (prefetch hint,
  x86) when the match is decoded, so it arrives while the next symbols are
  decoded and before the deferred copy; the three older repeat distances
  live in fields, which leaves registers to the coder. Same bytes.
- Hash-chain search (Fast): a search of its own with its values in
  registers and one two-byte check per candidate (same matches), a 10-bit
  hash for the two-byte table (4 KB instead of 256 KB per thread), and the
  next position's table slots prefetched. Output stays deterministic for
  any number of threads; Normal and Maximum are unchanged.
- bzip2: the move-to-front shift moves 16 bytes at a time (decoder and
  encoder; the decoder's symbol thread sets the pace), and the transform
  writes single rows without a span fill. Same bytes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@KnifeLemon KnifeLemon added the built-in engines Filee: built-in engines (Project 4), on the builtin-engines branch, not released yet label Nov 19, 2026
KnifeLemon and others added 4 commits November 19, 2026 16:07
The converter took the main stream byte by byte through a buffered reader;
it now scans its own buffer for the next E8 / E9 / 0F 8x opcode and copies
the run before it at once. Same bytes, same checks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A source that wraps around the window's end is hinted only up to it, so no
reference past the array is ever formed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…chives-3

# Conflicts:
#	tests/Filee.Core.Tests/FormatFieldsTests.cs
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@KnifeLemon
KnifeLemon merged commit d0016e7 into builtin-engines Nov 19, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

built-in engines Filee: built-in engines (Project 4), on the builtin-engines branch, not released yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant