Repository navigation
feat(xml): Parsec had no XML parser, and XML can't go through HTML's - #198
Open
jamesconwaydev wants to merge 5 commits into
Open
jamesconwaydev wants to merge 5 commits into
jamesconwaydev wants to merge 5 commits into
Conversation
Parsec's parsers each carry their own spans, line counting and errors, and there was nothing for JavaScript at all. crates/parsec-base holds what the parsers will share: Span, LineIndex (line and column, in characters or UTF-16, and UTF-16 offsets), Diagnostic with Severity and labels, and Limits with a step Budget. It has no dependencies. The HTML and CSS code synced from Netweir doesn't use it. crates/parsec-js lexes ECMAScript 2025's lexical grammar: identifiers with ID_Start and ID_Continue from the Unicode 17.0 data crates/parsec's tables use, and escapes; every numeric literal form, BigInt and separators included; strings; template parts; regular expression literals with their flags checked; punctuators; comments; a hashbang; and Annex B's HTML-like comments, in scripts only. The caller passes the goal, so the lexer never guesses what `/` or `}` is. Legacy octal literals and escapes are flagged on the token, a warning in a script and an error in a module. A token is 12 bytes with nothing behind it on the heap. A fuzz target, lex, joins the nightly fuzz run.
# Conflicts: # ROADMAP.md
A sitemap, a feed or an SVG could only be read by the HTML parser, which recovers from every error, folds names to lowercase and knows no namespaces. XML is a trust boundary too: ten nested entities expand a 1 KB document to 3 GB. crates/parsec-xml is XML 1.0 (Fifth Edition) and Namespaces in XML 1.0, not validating, with its own tokenizer and tree over parsec-base: names resolved to namespaces, a span on every node and attribute, the internal subset's entities and attribute defaults, and nothing ever fetched. Entity expansion is capped in depth and size, elements nest without recursion, every well-formedness error is fatal with its code, line and column, bytes come in through the BOM or the declaration, and the serializer writes prefixes and declarations back as they were. A fuzz target joins the nightly run, and release.yml publishes the crate after parsec-base.
This was referenced Nov 21, 2026
Brings the branch onto its new base after the pull request below it merged.
jamesconwaydev
enabled auto-merge
November 22, 2026 01:42
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #169.
Problem. Parsec had no XML parser. A sitemap, an RSS or Atom feed or an SVG could only go through the HTML parser, which recovers from every error, folds names to lowercase and knows nothing of namespaces, so
<atom:link>and<link>in a feed looked the same and a broken feed parsed as if it weren't. XML is also a trust boundary in a way HTML isn't: a 1 KB document with ten nested entities ("billion laughs") expands to 3 GB, and one 100 KB entity referenced 200,000 times is 20 GB of text from a 1 MB file.Change.
crates/parsec-xml(parsec_xml), XML 1.0 Fifth Edition and Namespaces in XML 1.0 (third edition), not validating. Its own tokenizer and tree; it shares onlyparsec-basewith the other parsers, as the wiki's Architecture page says.parse,parse_withandparse_bytesgive aDocument: an arena of nodes in document order, names interned once and resolved toName { qname, prefix, local, namespace }, aSpanon every node and attribute, attributes in source order,lookup_namespace,xml_langandpreserves_space.ATTLISTdefaults applied (a defaultedxmlns:pbinds its prefix), tokenized types' values collapsed, element and notation declarations checked. The external subset and external entities are never fetched or opened.Document::warningssays what wasn't read, a reference to an unread entity stays in the tree asNodeKind::EntityRef, and entity and attribute-list declarations after an unread parameter entity are skipped (XML 1.0, 5.1).parsec_base::LimitsandBudget: entity expansion stops withentity-expansion-limitpast 1 MiB plus 8 bytes per input byte, and withentity-too-deeppastmax_entity_depth(16, never pastENTITY_DEPTH_CEILING, 64). Elements nest at most 2,048 deep by default. Open elements live on the parser's own stack, so nothing recurses on nesting; only entity expansion does, and its depth is capped. A step limit and an input limit are there too.Errorwith a stable code (mismatched-end-tag,undeclared-prefix,recursive-entityand 35 others), its span, and the line and column fromLineIndex.invalid-byte-sequencewith its position, and a declaration that contradicts the BOM isencoding-mismatch. UTF-8 input isn't copied.to_xmlandwrite_xmlwrite the document or a subtree back, every prefix and declaration as written; a subtree written alone takes along the declarations it's in the scope of.xmlin the nightly run: any bytes, namespaces on and off, spans on character boundaries, and serialized output that parses back to itself.parsec-xmlgoes inrelease.yml's publish list afterparsec-base.parsec-html-encoding, which depends on the HTML parser; its BOM step isEncoding::for_bom, which this calls, and XML's declaration sniffing is different from HTML's prescan.Tests.
tests/hostile.rs: billion laughs, in content and in an attribute, stops in milliseconds; the 20 GB quadratic blowup stops; a 100 KB default attribute on 100,000 elements is shared, not copied; entity depth stops at 16 and at the ceiling whatever the option says, for parameter entities too; a million nested elements istoo-deepin under a second, and with no limit parses, walks and serializes without recursion; many attributes, many namespace declarations (on one element and nested), many entities, long names and long text all take ten times as long for ten times the input. Each fails on a parser that recurses per element or compares attributes pairwise.tests/parse.rs: the tree and document order; namespaces, scoping and twelve namespace errors;xml:langandxml:space; spans and line and column; line-end normalization; entities in content and attributes; parameter entities; what isn't read and the warnings for it; attribute defaults and types; 50 not-well-formed snippets with their codes; bytes in UTF-16 with and without a BOM, Shift_JIS, bad bytes and mismatches; serializer round trips.cargo fmt --check,cargo clippy --workspace --all-targets -- -D warnings,cargo test --workspace, the docs with-D warnings,cargo deny checkandscripts/check-headers.shpass locally.