Skip to content

feat(reduce): a page that broke Parsec couldn't become a test - #211

Open
jamesconwaydev wants to merge 3 commits into
mainfrom
reduce
Open

jamesconwaydev wants to merge 3 commits into
mainfrom
reduce

Conversation

@jamesconwaydev

@jamesconwaydev jamesconwaydev commented Nov 21, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #203.

Problem. When a real page breaks Parsec, the page belongs to the site and can't go in the repository. Getting from a 1 MB page to the few bytes that matter was done by hand, and nothing held a found bug down afterwards: there was no place for real-web regressions and no test that ran them.

Change.

  • crates/parsec-reduce, a workspace binary with publish = false: cargo run --release -p parsec-reduce -- page.html --check <panic|hang:MS|differs-from-html5ever|cmd:COMMAND> [-o OUT]. It runs ddmin over tags and text runs first, then attributes, then words of text, then characters, and repeats the four until a round removes nothing. It's deterministic, prints the size after each pass, and writes page.reduced.html (or -o).
    • panic holds every candidate to the page's own panic location (or a stack overflow), so the reduction can't slide onto a different bug.
    • differs-from-html5ever compares html5lib-tests tree dumps. Parsec's comes from SerializeOptions::html5test; html5ever's (0.40, the version bench/ uses) is dumped in src/tree.rs. Both parse with scripting off, and Parsec with declarative shadow roots off, since html5ever has none.
    • cmd:COMMAND runs through sh -c with the candidate's path as $1, and counts as failing while it exits non-zero.
    • Every candidate runs in a child process (the binary calls itself with --child), so an abort or a hang is killed and can't take the reducer down with it.
  • crates/parsec-html/tests/regressions.rs parses every tests/regressions/*.html on its own thread with a 10 s budget. A panic or a slow parse fails it. So does a .tree beside the fixture that doesn't match, a file over 64 KB, or a missing <!-- site: URL; date: YYYY-MM-DD; failed: ...; issue: #N --> first line. BLESS_REGRESSIONS=1 writes the .tree files. It runs in CI with cargo test --workspace. The directory has no fixtures yet: the first real one comes with the first crawl failure (Netweir crawls production sites through Parsec and reports what breaks #204 onward).
  • CONTRIBUTING.md has a new "When a real page breaks Parsec" section: how to reduce a page, where the fixture goes, and the policy (reduced inputs only, never whole pages, always with the site's URL and the date).
  • scripts/package-twice.sh and CI's reproducible job pass --exclude parsec-reduce, so a release never builds or attaches a package of it. Its dependency on parsec-html-core carries version = "0.2.0", which cargo package --list (the test-data check) needs. That makes it one more crates/*/Cargo.toml requirement for the version bump, which the Releasing page's glob already covers.
  • crates/parsec-html now excludes tests/ from its package, like the other crates, so CI's "No crate ships test data" check still passes.

Trade-off. The page is read as UTF-8, and anything invalid is replaced. A bug in byte decoding can't be reduced with the built-in checks. cmd: with your own decoding check can, and the limit is noted in main.rs. html5ever is now in the workspace's Cargo.lock, but only parsec-reduce uses it.

Seeded bug. A generated shop page of 1,000,087 bytes (cards, inline style and script, a header nav), with one <aside><nav><ul><li class=deals><table> buried at card 3,141. The "bug" is a Python html.parser script that exits 1 when a <table> starts inside an <li> inside a <nav>:

$ parsec-reduce shop.html --check 'cmd:python3 -I nested.py "$1"'
shop.html: fails
  tags         1000087 bytes
  tags          492741 bytes
  tags             135 bytes
  tags              89 bytes
  tags              54 bytes
  tags              50 bytes
  ...
1000087 bytes to 16 in 64 runs and 1.6 s: shop.reduced.html
$ cat shop.reduced.html
<nav><li><table>

1 MB down to 16 bytes, which is 1-minimal: no single character more can go. I also ran --child differs over every seventh page of the final boss batch (the UTF-8 ones) and over the shop page, and Parsec's and html5ever's dumps matched on all of them. So differs-from-html5ever doesn't fire on pages where the two agree.

Tests.

  • reduce::tests: tags_cover_the_input, attributes_of_start_tags_only, words_of_text_only and chars_are_whole_characters cover each split (they fail if a split drops or overlaps bytes). ddmin_keeps_only_what_fails checks ddmin keeps exactly ch of abcdefghij in under 40 tests, and ddmin_with_nothing_to_remove covers the empty and already-minimal cases. reduce_finds_the_nested_structure reduces a 300-row page to <li<table twice, with the same result both times.
  • tree::tests: the_dumps_agree_where_the_parsers_do covers doctypes, templates, SVG and MathML with xlink attributes, comments and noscript, and fails if the html5ever dump drifts from html5lib's format. the_dump_is_html5lib_s pins the exact text.
  • main tests: check parsing (bad hang:/cmd: rejected), panic signatures, the output path.
  • tests/cli.rs: the --child modes, a_page_that_doesnt_fail_is_refused (exit 1 under panic, hang:5000 and differs-from-html5ever), bad_arguments (exit 2), and a_command_check_reduces_a_page, a 128 KB page reduced through cmd: to <li<table (Unix only, as it needs sh).
  • regressions.rs: every_fixture, headers (seven malformed headers, each with its reason), and a_fixture_is_held_to_its_tree (bless, match, mismatch, no header, over 64 KB).

Passed locally: cargo test --workspace, cargo clippy --workspace --all-targets -- -D warnings, cargo fmt --all --check, cargo doc for both crates with -D warnings, cargo deny check licenses bans sources and scripts/check-headers.sh.

When a real site's page breaks Parsec, the page is the site's and can't
go in the repository, and finding the few bytes that matter by hand
takes an afternoon.

crates/parsec-reduce is a binary, not published, that shrinks a page by
delta debugging: ddmin over tags and text runs, then attributes, then
words, then characters, round after round until nothing more goes.
`--check` says what failing is: `panic` (at the page's own panic
location), `hang:MS`, `differs-from-html5ever` (html5lib tree dumps,
scripting off) or `cmd:COMMAND`. Each candidate runs in a child
process, so an abort or a hang can't stop the reducer.

Fixtures go in crates/parsec-html/tests/regressions/, each starting
with a site, date, failure and issue comment. tests/regressions.rs
parses every one inside 10 s without a panic, and matches a `.tree`
beside it when there is one; it refuses files over 64 KB or without the
header. CONTRIBUTING.md has the policy: reduced inputs only, never
whole pages.
@jamesconwaydev jamesconwaydev added the testing Trying it out and reporting back label Nov 21, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

testing Trying it out and reporting back

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Shrink a failing page to a regression fixture

1 participant