Skip to content

test(xml): the W3C XML conformance suite never ran against parsec-xml - #199

Open
jamesconwaydev wants to merge 2 commits into
xmlfrom
xml-conformance
Open

jamesconwaydev wants to merge 2 commits into
xmlfrom
xml-conformance

Conversation

@jamesconwaydev

Copy link
Copy Markdown
Collaborator

Fixes #170.

Stacked on #198.

Problem. parsec-xml had only its own tests. Nothing showed it handled the 2,585 cases of the W3C XML Conformance Test Suite, which is where XML parsers prove they reject what they must and accept what they must, and CI's summary had no XML row.

Change.

  • crates/parsec-xml/scripts/fetch-xmlconf.sh downloads the suite's last release, xmlts20130923.tar.gz from https://www.w3.org/XML/Test/, checks its SHA-256 (9b61db9f…fc1f) and unpacks it into tests/xmlconf, with COMMIT naming the release. The suite has no git home to pin as a submodule, so the address and hash are the pin. Its files stay out of the repository and the package.
  • tests/xmlconf.rs runs every case as a processor that doesn't validate: valid and invalid documents must parse, not-wf ones must be errors, and a valid document with canonical output must produce exactly that output (James Clark's canonical form, and the second form with notations). Every document that parses is also written out with to_xml and must parse back to the same thing.
  • The suite's index is XML whose parts are external entities, which Parsec doesn't read; the index parses with them left as EntityRef nodes, and the driver reads each part itself.
  • Counts go to $CONFORMANCE_DIR/xmlconf.json for the CI summary. The test and coverage jobs fetch the suite first. Without it, a local cargo test skips the suite and says so; under CI it fails instead.

Counts. 1,915 pass (223 matching their canonical output byte for byte), 0 fail, 670 skipped, each for its reason:

  • 310 are for XML 1.0 before the Fifth Edition (EDITION="1 2 3 4").
  • 274 are XML 1.1 or Namespaces 1.1.
  • 27 are error cases, which the spec lets a processor report or not.
  • 59 are not-wf cases whose fault is in an external entity or external DTD, which Parsec never reads. I checked a sample by hand: not-wf/ext-sa/*, not-wf/not-sa/*, oasis p09fail1 and p30fail1, ibm P62–P79. Each puts the error in a .dtd or .ent file.

One valid case, ibm-valid-P29-ibm29v01.xml, parses and round-trips but isn't compared with its canonical output: that output puts a processing instruction from the internal subset before the <!DOCTYPE>, which no canonical form does. It's listed in BAD_OUTPUT with that reason.

Tests. The suite is the test. To check it can fail, I removed the -- check from comments and the space collapsing of tokenized attributes: 6 cases failed. Running it found two driver bugs, both fixed here: cases with no ENTITIES attribute were taken as using external entities, and hst-lhs-007 and -008 (a declaration that contradicts the BOM) then showed the parser accepted them, which #198 fixes with encoding-mismatch.

cargo fmt --check, cargo clippy --workspace --all-targets -- -D warnings, cargo test --workspace with CONFORMANCE_DIR set, cargo deny check and scripts/check-headers.sh pass locally.

crates/parsec-xml/scripts/fetch-xmlconf.sh downloads the suite's last
release, pinned by address and SHA-256, and tests/xmlconf.rs runs all
2,585 cases as a processor that doesn't validate: valid and invalid
documents parse, not-wf ones are errors, canonical output matches, and
each parsed document round-trips through the serializer. 1,915 pass,
none fail, and the 670 skipped each have a reason in the summary.

CI's test and coverage jobs fetch the suite first; locally, cargo test
skips it with a note when it isn't there.
Brings the branch onto its new base after the pull request below it merged.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

xml The XML parser, parsec-xml

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant