Repository navigation
feat(xpath): XPath could only query netweir-html's tree - #369
Merged
Merged
Conversation
netweir-xpath read every document as a netweir_html::Document plus a Namespaces object, so a crate with a tree of its own, such as Parsec's XML parser, couldn't run XPath over it without copying the document into the HTML tree. A query now reads its document through a Tree trait: node kind and links, qualified and local names, namespace URIs, a tag key for the index, attributes by count and place (namespace declarations left out), text, a processing instruction's target, and the tree's own rules for id() and lang(). Doc holds a &dyn Tree, and Index::build is generic, as building reads every node and the calls through dyn showed there. netweir-html's Document implements Tree behind a default html feature, and XmlTree reads an XML document built into that tree with the host's Namespaces, as netweir-dom builds one. Without the feature the crate doesn't depend on netweir-html. Namespace tests compare URIs, so Namespaces loses id_of and Item::Attr its namespace id. tests/tree.rs queries a tree written by hand, and CI builds and tests the crate without the html feature.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #368.
Problem.
netweir-xpathcould only read anetweir_html::Document, with aNamespacesobject alongside for an XML document built into that tree. Parsec's XML parser has a tree of its own (parsec-html/parsec#198), andparsec-xpathis synced from this crate, so XPath over that tree had no way in short of copying every document into the HTML tree. The crate also depended onnetweir-htmlunconditionally, whichparsec-xmlcan't take.Change.
Treetrait insrc/tree.rs, as proposed on Serialize XML, and XPath over it parsec-html/parsec#171:kind,parent/first_child/next_sibling/prev_sibling,qualified_name,local_name,namespace_uri,tag_key(TagKey::Known(u16)orTagKey::Name),attribute_countandattribute(n, i)returning anAttr(qualified name, local name, namespace URI, value;xmlnsdeclarations left out),data,pi_target,is_xml,idandlang. Node ids stayu32.Docholds a&dyn Tree;Doc::new(tree, index)takes any tree, andDoc::htmlstill takes anetweir_html::Document.Index::buildandIndex::coverstake a tree too.Index::buildis generic, because building reads every node: a first version withdynthere too, and with the attribute count walked, ran about 4% slower on the 3 MB page. The index now gives elements one tag id per lowercase name whatever the tree calls them, so it no longer needsnetweir_html::tag_number.impl Tree for netweir_html::DocumentandXmlTree(an XML document built into that tree, reading namespace URIs from the host'sNamespaces) live insrc/html.rs, behind a newhtmlfeature that's on by default. Withdefault-features = falsethe crate has nonetweir-htmldependency.Namespaces::id_ofandItem::Attr'snsfield go.Itemshrinks from 16 bytes to 12.XmlTreeand the HTML impl read throughDocument::attr_at, and a newDocument::attr_countgives the count without walking the attributes.netweir-dombuilds anXmlTreefromXmlNamesfor an XML document. Work limits, step counting and thestackerstack growth are unchanged.local-name()of a processing instruction is its whole target now, as libxml2 gives it. Before, an XML target with a colon lost its prefix.netweir-xpathwith--no-default-features.Tests.
New
crates/netweir-xpath/tests/tree.rsqueries a tree written by hand, with nonetweir-htmlin it: prefixed and unprefixed names, element and attribute namespaces,xmlnsleft out of@*, text, comments, a processing instruction,id()throughxml:idandlang()throughxml:lang. It runs with and without thehtmlfeature. Breaking the tree'snamespace_uriorlangfails all four tests.tests/xpath.rsandtests/libxml2.rspass with their queries and answers unchanged.namespaces_come_from_the_hostchanges two lines to build anXmlTree, asDoc::xmlis gone, and itsNamespacesimpl dropsid_of. Both files now declarerequired-features = ["html"].bench/query.py, netweir's XPath time (parse and queries together), best of 15, two runs each at a load average under 4 on 12 cores:The same six queries alone on a parsed page with its index built, three interleaved runs each: 1 MB 3.91 to 3.97 ms before and 3.91 to 4.02 ms after; 3 MB 11.65 to 11.81 ms before and 11.70 to 12.10 ms after. That's within 2%, so queries keep
dyn.Passed locally:
cargo fmt --all --check,cargo clippy --workspace --all-targets -- -D warnings, the same fornetweir-xpathwith--no-default-features,cargo test -p netweir-xpath -p netweir-dom,cargo test -p netweir-xpath --no-default-features, anduv run pytest(1,478 passed, 11 skipped).