Repository navigation
perf: decode large documents with a fraction of the memory, with a benchmark - #18
Merged
Merged
Conversation
Replace alabaster with the michel-albert-sphinx extension (v0.3.1), which runs on sphinx-immaterial, and drop the unused sphinx-rtd-theme. Build the docs with --keep-going so every warning shows up in one run. sphinx-immaterial links TypeVars in signatures to type parameters; the module-level TypeVars in x690.types have no such target, so that warning category is suppressed. Co-Authored-By: Claude <noreply@anthropic.com>
Releases 1.1.0 and 1.0.0post1 used a different underline than the older releases, so everything from 1.0.0 down nested under 1.0.0post1. Co-Authored-By: Claude <noreply@anthropic.com>
The start page included the README below a table of contents, and several of its examples no longer ran (Sequence takes a list, Type is now X690Type). The new structure: - start page: what x690 is, install, an example, features, scope and project status - usage: encoding, decoding single values and streams, enforce_type, strict and pretty() - custom types: reusing an existing type and writing decode_raw and encode_raw, built on an SNMP GetRequest - supported types: which universal types convert to which Python value - API reference: one hand-written page per module instead of sphinx-apidoc output, so the CI no longer generates it The module docstrings of x690 and x690.types shrink to short summaries that point to these pages, and the glossary drops stale puresnmp references. Co-Authored-By: Claude <noreply@anthropic.com>
The detailed sections moved to the documentation. The Sequence examples in the old README raised TypeError. Co-Authored-By: Claude <noreply@anthropic.com>
pytest now collects the .rst pages under docs/ and README.rst. A fixture restores the type registry after each page, so the custom types a page defines do not leak into other tests. Co-Authored-By: Claude <noreply@anthropic.com>
Break imports, constructors and byte strings over several lines so the examples stay readable without scrolling, and note that pythonize() converts the items of a sequence while value keeps x690 objects. Co-Authored-By: Claude <noreply@anthropic.com>
The new version colours code with the design system's code tokens. Co-Authored-By: Claude <noreply@anthropic.com>
x690 is no longer in maintenance mode: context-specific tags and Python values for more universal types are planned. The README, contributing guide and changelog say so, and the package is classified as Production/Stable instead of Mature. Co-Authored-By: Claude <noreply@anthropic.com>
Decode the same SNMP request with pyasn1, asn1tools, asn1crypto, pycrate and x690, and compare where each takes the structure from, what it returns and what a type checker sees. Every example was run against pyasn1 0.6.4, asn1tools 0.169.0, asn1crypto 1.5.1 and pycrate 0.8.1. The page explains the design decisions behind x690 and lists its current limitations with their issues (#6 to #11). The start page links to it and its status section reflects active development. Co-Authored-By: Claude <noreply@anthropic.com>
benchmarks/ compares decode time and peak memory of x690 with pyasn1, asn1tools, asn1crypto and pycrate on synthetic CDR-like data, in four workloads: a stream of records, a whole document read by iteration, only its last record, and a whole document read by index. The last one shows the quadratic indexing of #16. Run it with 'uv run --group bench python benchmarks/run_all.py'. It writes a JSON report with the machine, library versions and x690 commit; benchmarks/report.py renders one or more reports as tables. Memory is measured on top of the loaded input, exactly on Linux and from ru_maxrss elsewhere. Co-Authored-By: Claude <noreply@anthropic.com>
report.py now starts with the system, CPU, memory, Python, library versions, x690 commit and memory limit of every report it renders. The JSON report already held these; the tables did not show them. Co-Authored-By: Claude <noreply@anthropic.com>
A decoded Sequence re-decoded its whole item list on every len(), index or iteration, so reading every item by index was quadratic: 200 items took 40,200 header reads. The sequence now records the start of each item once, in a compact array, and decodes items on demand. Iterating no longer builds the list of all items first, so the items read can be freed as the loop goes on. decode_raw no longer copies the whole value to check whether it is empty. Closes #16 Co-Authored-By: Claude <noreply@anthropic.com>
X690Type declared __slots__, but its subclasses did not, so every value still carried a __dict__, and bounds was assigned per instance outside the slots. A decoded Integer took 472 bytes; it now takes 120. bounds is a slot initialised in __init__. Custom types that declare no slots keep working with a __dict__ of their own. Closes #17 Co-Authored-By: Claude <noreply@anthropic.com>
Same machine and limits as the before report, at commit 62cddca. At 1,000,000 records x690 reads a whole document in 22 s with 8 MB on top of the input (was 32 s and 558 MB), the last record in 1.6 s (was 7.2 s), and every record by index in 22 s (10,000 records took 409 s). Co-Authored-By: Claude <noreply@anthropic.com>
The choosing page gains a performance section: decode time and peak memory of all five libraries at 1,000,000 records, on the constrained machine of the committed reports. The design notes explain that a sequence records where its items start and creates items on access. The changelog lists the sequence and slots changes and the benchmark. Co-Authored-By: Claude <noreply@anthropic.com>
The performance section now leads with a run on a 6-core desktop with 31 GB of memory, where every library finishes: x690 reads a whole document of 1,000,000 records in 10 s with 8 MB on top of the input, against 13 s and 1.4 to 2 GB for asn1tools and asn1crypto, and 70 s and 146 s with 5.5 and 9.6 GB for pycrate and pyasn1. The small machine remains as the memory-limited case. Co-Authored-By: Claude <noreply@anthropic.com>
The start page and README now say that x690 decodes documents of a million values in a few megabytes, and the 'No copies' feature becomes 'Large documents' with the benchmark numbers and a link to them. The choosing page no longer says speed is not a selling point: x690 is the fastest for whole documents, asn1crypto for record streams. It also states the memory growth per item and lists limited memory as a reason to choose x690. Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes large documents cheap to decode and adds a benchmark against pyasn1, asn1tools, asn1crypto and pycrate.
Stacked on #15 (it extends the "Choosing x690" page from that PR). Merge #15 first, then retarget this PR to
main.Changes
Sequencerecords the start of each item once, in a compactarray, and decodes items on demand. Indexing was quadratic: reading 200 items by index took 40,200 header reads, now at most 600. Iteration no longer builds the full item list first, anddecode_rawno longer copies the value to test for emptiness.__slots__: every type declares__slots__, andboundsis a slot. A decodedIntegertakes 120 bytes instead of 472. Custom types without slots keep working.benchmarks/): four workloads (stream, full, partial, indexed) on synthetic CDR-like data, one process per case, memory measured on top of the input.uv run --group bench python benchmarks/run_all.pywrites a JSON report with machine details;benchmarks/report.pyrenders and compares reports.benchmarks/results/: before and after on a small machine (2 cores, 3.7 GB, 2 GB limit per case) and on a desktop (12 threads, 31 GB, no limit).Results at 1,000,000 records (80 MB)
Desktop (Ryzen 5 5600X, 31 GB, CPython 3.12, no limit;
user-desktop-*.json):For a whole document, x690 is now the fastest of the five: asn1crypto 12.9 s / 1,957 MB, asn1tools 13.2 s / 1,436 MB, pycrate 70 s / 5,494 MB, pyasn1 146 s / 9,569 MB. On streams asn1crypto stays fastest (6.3 s).
Small machine (2 cores, 3.7 GB, CPython 3.13, 2 GB limit;
constrained-*.json):pycrate and pyasn1 run out of memory on whole documents there. The other libraries' numbers stayed within a few percent between before and after on both machines.
Verification
pytest: 459 passed, coverage 99.69 %. New tests for indexing, slicing, offsets inside a larger buffer, a linear bound on header reads (fails on the old code with 40,200 reads), and the absence of instance dictionaries.mypy --strict,pre-commit run --all-files,sphinx-build -W --keep-going: clean.Reproduce
Closes #16
Closes #17
🤖 Generated with Claude Code