Repository navigation
Built-in engines: archive speed, round 3, and .zst - #1692
Merged
Merged
Conversation
ZST and TAR.ZST become archive formats like XZ and TAR.XZ: recognised by extension, unpacked in-process (the TAR in a second stage), converted to every archive target, and ZST to the other single-file formats. ZstdReader puts SharpCompress's decoder behind Filee's checks: the first frame's magic (other data is "not an archive", not an empty file), frames one after another and skippable frames, damage and early ends as "damaged", windows over 128 MiB left to 7-Zip. RPM payloads in zstd use the same reader. Read only: writing would add a developer-only output, which FORMAT-RULES rule 5 keeps out while the developer field is on by default. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- LZMA decoder: a match's source is asked into the cache (prefetch hint, x86) when the match is decoded, so it arrives while the next symbols are decoded and before the deferred copy; the three older repeat distances live in fields, which leaves registers to the coder. Same bytes. - Hash-chain search (Fast): a search of its own with its values in registers and one two-byte check per candidate (same matches), a 10-bit hash for the two-byte table (4 KB instead of 256 KB per thread), and the next position's table slots prefetched. Output stays deterministic for any number of threads; Normal and Maximum are unchanged. - bzip2: the move-to-front shift moves 16 bytes at a time (decoder and encoder; the decoder's symbol thread sets the pace), and the transform writes single rows without a span fill. Same bytes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The converter took the main stream byte by byte through a buffered reader; it now scans its own buffer for the next E8 / E9 / 0F 8x opcode and copies the run before it at once. Same bytes, same checks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A source that wraps around the window's end is hinted only up to it, so no reference past the array is ever formed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…chives-3 # Conflicts: # tests/Filee.Core.Tests/FormatFieldsTests.cs
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Third round on the built-in archive engines' speed (after #1672 and #1686), and reading Zstandard files (#1646).
.zst, .tar.zst, .tzst (#1646)
zst(ZST,.zst) andtzst(TAR.ZST,.tar.zst,.tzst), read only, registered like XZ / TAR.XZ: extensions, fields and grade (specialist, developer), the extractor's lists (built in; 7-Zip as the fallback reader), TAR.ZST unpacked in two stages, ZST converts to GZ / BZ2 / XZ, every readable archive target.filee formats zstnow knows it.ZstdReaderputs SharpCompress's decoder (already used for zstd RPM payloads) behind Filee's checks: the first frame's magic (other data is "not a valid archive" instead of an empty file), frames one after another and skippable frames, a cut or trailing data as "damaged", a window over 128 MiB (zstd --long=28and up) left to 7-Zip, as the zstd tool refuses it without--memory(the decoder would set the window aside before a byte shows the file is real). RPM payloads in zstd use the same reader. Every existing limit (UnpackBudget, the room on the drive) applies.docs/FORMAT-RULES.mdrule 5 keeps a new developer-only output out while the developer field is on by default. Writing is a small change once that is decided.tar.zst,tzstandzst(custom claims stay put); new installs get them already (ExtensionsOf(FormatCategory.Archive)).ZstSettingsTestscovers it.FormatRegistry,FormatFields(+FormatFieldsTestspinned lists, specialist count 131 → 133),README.md(one line),docs/ENGINES.md(archive table rows). No i18n, no CHANGELOG.Speed
Unpacking
Sse.Prefetch0, x86; nothing elsewhere), so it arrives while the next symbols are decoded and before the deferred copy runs (both hinted addresses stay inside the window). The three older repeat distances moved to fields, leaving registers to the coder. Same bytes. This is the only reasonFilee.Enginesnow allows unsafe code: the hint takes a pointer, nothing reads through one.Packing
Lzma2_blocks_give_the_same_bytes_on_one_thread_or_four). Normal and Maximum: unchanged bytes.AggressiveOptimizationon the decoder loops (−1 % cold, +3 % warm), a smaller four-byte hash at Fast (slower and larger), keeping bzip2's bit buffer in locals (slower).Tests
.tar.zstand.zstunpack in-process, RPM payload in zstd, frames + skippable frames, "not an archive" / damaged / wide window → 7-Zip refusal, names recognised and read only, zstd fuzz seeds, every readable archive (incl. ZST / TAR.ZST) converts in one step, ZST → XZ but nothing → ZST, the 16-byte move-to-front shift against a byte loop for every count.Zstd), Release, 7-Zip 26.05 present: 834 passed, 0 failed, 18 skipped; thenSevenZipReaderTests,ArchiveThreadTestsandArchiveFuzzTestsagain after the BCJ2 change: 81 passed; after bounding the prefetch hint to the window, the LZMA / 7z / thread / fuzz / in-process classes: 152 passed.Filee.Core.Tests: 1769 passed.dotnet format --verify-no-changes --severity warnclean on Filee.Engines, Filee.Engines.Tests, Filee.Core, Filee.Core.Tests.Numbers
Same laptop and harness as #1672 / #1686 (i7-1185G7, 4 cores / 8 threads, 16 GB, Windows 11, Release, .NET 10; inputs generated in code; unpack through
ArchiveExtractor.ExtractAsync, pack throughArchivePacker.PackAsync; 7-Zip 26.05 x64x/awith its default threads). Per operation: round 2 (builtin-enginesat #1686's merge), round 3 and 7-Zip interleaved run by run, 3 runs, median wall time; each run a fresh process, so Filee's times include JIT warm-up. Other agents' builds and tests kept the machine busy the whole time, so this round the bench runs every measured process (Filee and 7-Zip alike) at high priority and waits up to 20 s for a quieter moment before each run; even so, single runs moved by ±10–20 %, and rows where nothing changed (tar.gz, ZIP Deflate64, the random / deflated 7z) show that noise. Absolute times are not comparable with #1686's table (other machine state); the round 2 column is from today's runs. The zstd inputs (level 3, one frame) come from SharpCompress's encoder, since 7-Zip only reads zstd.Unpack (wall time)
Pack (wall time · archive size)
Ratio at the changed levels (output size vs 7-Zip at the same level): 7z Fast +0.9 %, xz Fast +0.7 % (as in round 2; the new two-byte hash makes Fast's output 0.001 % smaller). Normal and Maximum: the same bytes as round 2.
JIT warm-up. The shipped app is ReadyToRun; this bench is not, so a short single-stream job spends its first few hundred milliseconds in unoptimized code (a profile of the BCJ2 run had about as many samples in Tier0 decoder code as in optimized code). The same operations with the bench published ReadyToRun (3 runs each, interleaved, same conditions):
(bzip2 packing is the exception: its precompiled sort is slower until it tiers up.)
Still slower than 7-Zip, and why
-mx5xz / 7z): decoding is serial and bound by the range coder's dependency chain (shift, multiply, compare, per bit) plus cache misses. The prefetch took about 5–6 % off the decoder (paired runs side by side); with ReadyToRun the gap is 0.75–0.87×. What is left is code generation: the JIT keeps a branch where 7-Zip's assembly uses conditional moves (tried, it can't be had in a loop today), and the decoder loops still spill a few values. x86-like data (BCJ, BCJ2) is the worst case because most of a cold run's 1–2 s is warm-up.Charts (generated inputs only) are in the run notes, not in the repo.
🤖 Generated with Claude Code