Skip to content

feat(xml): Parsec had no XML parser, and XML can't go through HTML's - #198

Open
jamesconwaydev wants to merge 5 commits into
mainfrom
xml
Open

jamesconwaydev wants to merge 5 commits into
mainfrom
xml

Conversation

@jamesconwaydev

Copy link
Copy Markdown
Collaborator

Fixes #169.

Problem. Parsec had no XML parser. A sitemap, an RSS or Atom feed or an SVG could only go through the HTML parser, which recovers from every error, folds names to lowercase and knows nothing of namespaces, so <atom:link> and <link> in a feed looked the same and a broken feed parsed as if it weren't. XML is also a trust boundary in a way HTML isn't: a 1 KB document with ten nested entities ("billion laughs") expands to 3 GB, and one 100 KB entity referenced 200,000 times is 20 GB of text from a 1 MB file.

Change.

  • New crate crates/parsec-xml (parsec_xml), XML 1.0 Fifth Edition and Namespaces in XML 1.0 (third edition), not validating. Its own tokenizer and tree; it shares only parsec-base with the other parsers, as the wiki's Architecture page says.
  • parse, parse_with and parse_bytes give a Document: an arena of nodes in document order, names interned once and resolved to Name { qname, prefix, local, namespace }, a Span on every node and attribute, attributes in source order, lookup_namespace, xml_lang and preserves_space.
  • The internal subset is read as the spec asks of a processor that doesn't validate: general and parameter entities expanded, ATTLIST defaults applied (a defaulted xmlns:p binds its prefix), tokenized types' values collapsed, element and notation declarations checked. The external subset and external entities are never fetched or opened. Document::warnings says what wasn't read, a reference to an unread entity stays in the tree as NodeKind::EntityRef, and entity and attribute-list declarations after an unread parameter entity are skipped (XML 1.0, 5.1).
  • Limits through parsec_base::Limits and Budget: entity expansion stops with entity-expansion-limit past 1 MiB plus 8 bytes per input byte, and with entity-too-deep past max_entity_depth (16, never past ENTITY_DEPTH_CEILING, 64). Elements nest at most 2,048 deep by default. Open elements live on the parser's own stack, so nothing recurses on nesting; only entity expansion does, and its depth is capped. A step limit and an input limit are there too.
  • Every well-formedness error is fatal: an Error with a stable code (mismatched-end-tag, undeclared-prefix, recursive-entity and 35 others), its span, and the line and column from LineIndex.
  • Bytes in: the byte order mark, then the first bytes' pattern (UTF-16 without a BOM), then the declaration's encoding, decoded strictly with encoding_rs; a byte that isn't the encoding is invalid-byte-sequence with its position, and a declaration that contradicts the BOM is encoding-mismatch. UTF-8 input isn't copied.
  • to_xml and write_xml write the document or a subtree back, every prefix and declaration as written; a subtree written alone takes along the declarations it's in the scope of.
  • Fuzz target xml in the nightly run: any bytes, namespaces on and off, spans on character boundaries, and serialized output that parses back to itself.
  • parsec-xml goes in release.yml's publish list after parsec-base.
  • I took encoding_rs directly rather than through parsec-html-encoding, which depends on the HTML parser; its BOM step is Encoding::for_bom, which this calls, and XML's declaration sniffing is different from HTML's prescan.

Tests.

  • tests/hostile.rs: billion laughs, in content and in an attribute, stops in milliseconds; the 20 GB quadratic blowup stops; a 100 KB default attribute on 100,000 elements is shared, not copied; entity depth stops at 16 and at the ceiling whatever the option says, for parameter entities too; a million nested elements is too-deep in under a second, and with no limit parses, walks and serializes without recursion; many attributes, many namespace declarations (on one element and nested), many entities, long names and long text all take ten times as long for ten times the input. Each fails on a parser that recurses per element or compares attributes pairwise.
  • tests/parse.rs: the tree and document order; namespaces, scoping and twelve namespace errors; xml:lang and xml:space; spans and line and column; line-end normalization; entities in content and attributes; parameter entities; what isn't read and the warnings for it; attribute defaults and types; 50 not-well-formed snippets with their codes; bytes in UTF-16 with and without a BOM, Shift_JIS, bad bytes and mismatches; serializer round trips.
  • I ran the W3C XML conformance suite against this locally (it comes in the next pull request): 1,915 cases pass, none fail, 223 of them matching their canonical output exactly, and every parsed case serializes and parses back the same.
  • The fuzz target ran 5 minutes locally from an xmlconf corpus without a finding.

cargo fmt --check, cargo clippy --workspace --all-targets -- -D warnings, cargo test --workspace, the docs with -D warnings, cargo deny check and scripts/check-headers.sh pass locally.

Parsec's parsers each carry their own spans, line counting and errors,
and there was nothing for JavaScript at all.

crates/parsec-base holds what the parsers will share: Span, LineIndex
(line and column, in characters or UTF-16, and UTF-16 offsets),
Diagnostic with Severity and labels, and Limits with a step Budget. It
has no dependencies. The HTML and CSS code synced from Netweir doesn't
use it.

crates/parsec-js lexes ECMAScript 2025's lexical grammar: identifiers
with ID_Start and ID_Continue from the Unicode 17.0 data
crates/parsec's tables use, and escapes; every numeric literal form,
BigInt and separators included; strings; template parts; regular
expression literals with their flags checked; punctuators; comments; a
hashbang; and Annex B's HTML-like comments, in scripts only. The caller
passes the goal, so the lexer never guesses what `/` or `}` is. Legacy
octal literals and escapes are flagged on the token, a warning in a
script and an error in a module. A token is 12 bytes with nothing
behind it on the heap.

A fuzz target, lex, joins the nightly fuzz run.
A sitemap, a feed or an SVG could only be read by the HTML parser,
which recovers from every error, folds names to lowercase and knows no
namespaces. XML is a trust boundary too: ten nested entities expand a
1 KB document to 3 GB.

crates/parsec-xml is XML 1.0 (Fifth Edition) and Namespaces in XML 1.0,
not validating, with its own tokenizer and tree over parsec-base: names
resolved to namespaces, a span on every node and attribute, the
internal subset's entities and attribute defaults, and nothing ever
fetched. Entity expansion is capped in depth and size, elements nest
without recursion, every well-formedness error is fatal with its code,
line and column, bytes come in through the BOM or the declaration, and
the serializer writes prefixes and declarations back as they were.

A fuzz target joins the nightly run, and release.yml publishes the
crate after parsec-base.
Brings the branch onto its new base after the pull request below it merged.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

xml The XML parser, parsec-xml

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant