Repository navigation
6.1: open workbooks from your own sources and streams, managed inflate, leaner typed parse - #140
Open
GabrielMarquezMatte wants to merge 45 commits into
Open
GabrielMarquezMatte wants to merge 45 commits into
GabrielMarquezMatte wants to merge 45 commits into
Conversation
…-api-v6.0.0 chore: promote public API for v6.0.0
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… 6.0 options layout Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ard stream reads Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Python callbacks now catch BaseException so Ctrl-C fails the read instead of truncating a CSV stream. C++ open_source rejects a size above INT64_MAX before releasing the source, and the C++ tests cover destruction on failure, parallel reads and stream errors. The smoke test pins the 88-byte 6.0 options struct. Source error wording no longer claims 0 is valid, StreamBuffer clamps its limit to Array.MaxLength, and the header, docstring and READMEs are tightened. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lay functionality
Bumps Meziantou.Analyzer from 3.0.290 to 3.0.294 --- updated-dependencies: - dependency-name: Meziantou.Analyzer dependency-version: 3.0.294 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: analyzers ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps Microsoft.Testing.Extensions.CodeCoverage from 18.11.2 to 18.12.0 Bumps Microsoft.Testing.Extensions.TrxReport from 2.4.1 to 2.5.1 --- updated-dependencies: - dependency-name: Microsoft.Testing.Extensions.CodeCoverage dependency-version: 18.12.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: test - dependency-name: Microsoft.Testing.Extensions.TrxReport dependency-version: 2.5.1 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: test ... Signed-off-by: dependabot[bot] <support@github.com>
…elop/analyzers-55e46a7a66 Bump the analyzers group with 1 update
…ts/ExcelReader.Tests/develop/test-4352e14058 Bump the test group with 2 updates
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replaces DeflateStream at all four sites in ZipIndex. A deflate entry that ends before its final block now throws InvalidDataException; DeflateStream returned the bytes decoded so far and reported end of stream. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…oded bytes When input had ended and the window filled while the fast loop was decoding the zero padding, the decoder reported a full window instead of truncation, so up to a few hundred invented bytes reached the caller before the exception. Also pins matches at the full 32,768-byte distance. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both threw InvalidOperationException, which FuzzOracle sanctions, so Guard swallowed them. Divergence now throws OracleDivergenceException and the canary UnreachableException; a smoke self-check proves Guard lets a divergence through. When InflateStream rejects input, the bytes it delivered first must be a prefix of DeflateStream's output. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Keeps the litlen and offset tables in one array so the offset table is a constant displacement, which removes a register spill on every match. Tracks bitsleft as a long whose low byte alone is meaningful, subtracting whole entries and refilling with 7 - (bitsleft >> 3) as libdeflate does, and drops a redundant mask before the extra-bits shift. Inflate only, interleaved on the i7-1365U: 1.10x on the XLSX sheet, 1.07x on the XLSB sheet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t-loop change FastLoop lost a parameter, so the previous profile no longer matched it by signature. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… without a copy ColumnBuilder wrote every value into 32 KB managed chunks and then copied them all into the native block it returned. It now writes into a native block that doubles as it fills; a column built in one part hands that block over directly, and only the parallel CSV path, or a test tracking allocations, still copies. A SafeHandle owns each block so a builder dropped on an error path still frees it. On the 65K fixture the peak during to_record_batch falls from 26.0 to 18.1 MiB (10.7 MiB from 16.7 when streamed); typed parsing is 3-5% faster on the i7-1365U. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mn buffers ColumnBuilder.Append and Concat changed signature, so the previous profile no longer matched them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #140 +/- ##
==========================================
+ Coverage 90.00% 90.02% +0.02%
==========================================
Files 190 204 +14
Lines 13963 14849 +886
Branches 2558 2711 +153
==========================================
+ Hits 12567 13368 +801
- Misses 936 1004 +68
- Partials 460 477 +17 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Benchmark ResultsBaseline: master Moved by at least 10%
104 within ±10%, 38 with no baseline entry (new, renamed, or reparameterized). Full results per groupExcelReader.Benchmarks.ArrowConversionBenchmark
ExcelReader.Benchmarks.ChunkedParseBenchmark
ExcelReader.Benchmarks.ColdStartBenchmark
ExcelReader.Benchmarks.CsvParallelParseBenchmark
ExcelReader.Benchmarks.CsvParallelVsSepBenchmark
ExcelReader.Benchmarks.CsvParseBenchmark
ExcelReader.Benchmarks.CsvReadBenchmark
ExcelReader.Benchmarks.CsvWriteBenchmark
ExcelReader.Benchmarks.DataReaderBenchmark
ExcelReader.Benchmarks.EncryptedWorkbookBenchmark
ExcelReader.Benchmarks.InflateBenchmark
ExcelReader.Benchmarks.NativeOpenBenchmark
ExcelReader.Benchmarks.NativeRowReadBenchmark
ExcelReader.Benchmarks.NativeTypedParseBenchmark
ExcelReader.Benchmarks.ParseBenchmark
ExcelReader.Benchmarks.ReadBenchmark
ExcelReader.Benchmarks.RealDataReadBenchmark
ExcelReader.Benchmarks.RealDataTypedParseBenchmark
ExcelReader.Benchmarks.RecordWriteBenchmark
ExcelReader.Benchmarks.StringHeavyReadBenchmark
ExcelReader.Benchmarks.WriteBenchmark
ExcelReader.Benchmarks.WritePathBenchmark
ExcelReader.Benchmarks.XlsReadBenchmark
ExcelReader.Benchmarks.XlsWriteBenchmark
ExcelReader.Benchmarks.XlsxSharedStringHotPathBenchmark
|
The generated factory was `new T()`, which fails with CS9035 when T or a base type declares a required member. It now sets every required member to default in the object initializer and lets the parse overwrite the ones it maps, matching FromAttributes, whose Activator.CreateInstance the runtime never checks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… benchmarks for native open operations
… across C++, Python, and Rust implementations
…rmats to reflect improved performance metrics
…merator overhead based on recent measurements
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this change?
Everything on
developsincev6.0.0, as the candidate for 6.1.0. Two features and two performancechanges, with no change to the .NET public API and only additions to the C ABI (
XL_ABI_VERSIONstays 6).Open a workbook from your own bytes
fetched in blocks and each block fetched once (
57dc08c,fcb2605).xl_open_source(random access through aread_atcallback) andxl_open_stream(a forward-only stream that cannot seek), plus
xl_set_source_errorso a callback can report whyit failed.
xl_open_optionsgains source and buffering fields and still accepts the 6.0 layoutthrough
struct_size.Sourceor any readable stream, C++ anxl::Sourceorxl::InputStream, Rust aSourceor anyRead. Documented in each binding's README.(
ReplayStream).blocks other threads while a batch parses.
Managed inflate decoder
System.IO.Compression.DeflateStreamis replaced at all four sites inZipIndexbyInflateStream,a managed, resumable port of libdeflate's decoder design (MIT notice added to
THIRD-PARTY-NOTICES.md, which now ships in the NuGet package).DeflateStream: about 1.7x on the XLSX sheet and 1.4x on the XLSB sheet,after a fast-loop pass guided by the JIT disassembly (one table array instead of two, removing a
register spill; libdeflate-style bit accounting).
fast-loop pass; the gain with prefetch is smaller because the parse becomes the bottleneck).
inflatefuzz target that compares output withDeflateStream, and an oracle fix so divergencesand canary trips are no longer swallowed by
FuzzOracle.Guard.Typed parse builds columns in native memory
ColumnBuilderused to write into 32 KB managed chunks and then copy everything into the nativeblock it returns. It now writes into a native block that grows by doubling, and a column built in one
part hands that block over without a copy. A
SafeHandleowns each block, so a builder dropped on anerror path still frees it.
to_record_batchon the 65K fixture: 26.0 → 18.1 MiB whole sheet,16.7 → 10.7 MiB streamed.
Also
for the decode loop and the column builders.
docs/performance/reader-performance.md: inflate measurements, and a correction of the earlier"the inflate floor does not move" claim.
Checklist
dotnet build ExcelReader.slnx --configuration Releasebuilds clean (warnings are errors)dotnet test tests/ExcelReader.Tests/ExcelReader.Tests.csproj --configuration Releasepasses (2,990 tests)PublicAPI.Unshipped.txtupdated (no .NET public API change since v6.0.0)developTest plan
1 pre-existing xfail.
the full 32,768-byte distance, cuts just before the window fills), differential tests against
DeflateStreamover generated shapes at every compression level and over every deflated entry inthe test workbooks, and fuzz
checkwith 156,780 cases and 0 failures.GrowableNativeBufferTests, plus the existing allocation-tracking tests, which stillpass through the copy path.
ratios are meaningful.
Before tagging 6.1.0:
get slower (each ZIP entry now rents buffers and builds Huffman tables).
xl_parse_typedbefore and after.inflatetarget.benchmarks.mdand the Python README figures from the Ryzen run.