Skip to content

feat(xpath): XPath could only query netweir-html's tree - #369

Merged
jamesconwaydev merged 1 commit into
mainfrom
xpath-tree
Nov 22, 2026
Merged

jamesconwaydev merged 1 commit into
mainfrom
xpath-tree

Conversation

@jamesconwaydev

Copy link
Copy Markdown
Collaborator

Fixes #368.

Problem. netweir-xpath could only read a netweir_html::Document, with a Namespaces object alongside for an XML document built into that tree. Parsec's XML parser has a tree of its own (parsec-html/parsec#198), and parsec-xpath is synced from this crate, so XPath over that tree had no way in short of copying every document into the HTML tree. The crate also depended on netweir-html unconditionally, which parsec-xml can't take.

Change.

  • A new Tree trait in src/tree.rs, as proposed on Serialize XML, and XPath over it parsec-html/parsec#171: kind, parent/first_child/next_sibling/prev_sibling, qualified_name, local_name, namespace_uri, tag_key (TagKey::Known(u16) or TagKey::Name), attribute_count and attribute(n, i) returning an Attr (qualified name, local name, namespace URI, value; xmlns declarations left out), data, pi_target, is_xml, id and lang. Node ids stay u32.
  • Doc holds a &dyn Tree; Doc::new(tree, index) takes any tree, and Doc::html still takes a netweir_html::Document. Index::build and Index::covers take a tree too. Index::build is generic, because building reads every node: a first version with dyn there too, and with the attribute count walked, ran about 4% slower on the 3 MB page. The index now gives elements one tag id per lowercase name whatever the tree calls them, so it no longer needs netweir_html::tag_number.
  • impl Tree for netweir_html::Document and XmlTree (an XML document built into that tree, reading namespace URIs from the host's Namespaces) live in src/html.rs, behind a new html feature that's on by default. With default-features = false the crate has no netweir-html dependency.
  • Namespace tests compare URIs instead of the host's ids, so Namespaces::id_of and Item::Attr's ns field go. Item shrinks from 16 bytes to 12.
  • Attribute access stays O(1) per attribute: XmlTree and the HTML impl read through Document::attr_at, and a new Document::attr_count gives the count without walking the attributes.
  • netweir-dom builds an XmlTree from XmlNames for an XML document. Work limits, step counting and the stacker stack growth are unchanged.
  • local-name() of a processing instruction is its whole target now, as libxml2 gives it. Before, an XML target with a colon lost its prefix.
  • CI's lint job runs clippy and the tests for netweir-xpath with --no-default-features.

Tests.

  • New crates/netweir-xpath/tests/tree.rs queries a tree written by hand, with no netweir-html in it: prefixed and unprefixed names, element and attribute namespaces, xmlns left out of @*, text, comments, a processing instruction, id() through xml:id and lang() through xml:lang. It runs with and without the html feature. Breaking the tree's namespace_uri or lang fails all four tests.

  • tests/xpath.rs and tests/libxml2.rs pass with their queries and answers unchanged. namespaces_come_from_the_host changes two lines to build an XmlTree, as Doc::xml is gone, and its Namespaces impl drops id_of. Both files now declare required-features = ["html"].

  • bench/query.py, netweir's XPath time (parse and queries together), best of 15, two runs each at a load average under 4 on 12 cores:

    Page Before After
    1 MB 6.46 ms, 6.36 ms 6.57 ms, 6.47 ms
    3 MB 19.17 ms, 19.14 ms 19.22 ms, 19.31 ms

    The same six queries alone on a parsed page with its index built, three interleaved runs each: 1 MB 3.91 to 3.97 ms before and 3.91 to 4.02 ms after; 3 MB 11.65 to 11.81 ms before and 11.70 to 12.10 ms after. That's within 2%, so queries keep dyn.

Passed locally: cargo fmt --all --check, cargo clippy --workspace --all-targets -- -D warnings, the same for netweir-xpath with --no-default-features, cargo test -p netweir-xpath -p netweir-dom, cargo test -p netweir-xpath --no-default-features, and uv run pytest (1,478 passed, 11 skipped).

netweir-xpath read every document as a netweir_html::Document plus a
Namespaces object, so a crate with a tree of its own, such as Parsec's
XML parser, couldn't run XPath over it without copying the document
into the HTML tree.

A query now reads its document through a Tree trait: node kind and
links, qualified and local names, namespace URIs, a tag key for the
index, attributes by count and place (namespace declarations left out),
text, a processing instruction's target, and the tree's own rules for
id() and lang(). Doc holds a &dyn Tree, and Index::build is generic, as
building reads every node and the calls through dyn showed there.

netweir-html's Document implements Tree behind a default html feature,
and XmlTree reads an XML document built into that tree with the host's
Namespaces, as netweir-dom builds one. Without the feature the crate
doesn't depend on netweir-html. Namespace tests compare URIs, so
Namespaces loses id_of and Item::Attr its namespace id.

tests/tree.rs queries a tree written by hand, and CI builds and tests
the crate without the html feature.
@jamesconwaydev
jamesconwaydev added this pull request to the merge queue Nov 21, 2026
Merged via the queue into main with commit 0196e9e Nov 22, 2026
38 of 39 checks passed
@jamesconwaydev
jamesconwaydev deleted the xpath-tree branch November 22, 2026 00:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

XPath can only read the HTML tree, so Parsec's XML parser can't use it

1 participant