Skip to content

feat: JavaScript couldn't be parsed from Python, WebAssembly or C - #219

Open
jamesconwaydev wants to merge 5 commits into
js-reparsefrom
js-bindings
Open

jamesconwaydev wants to merge 5 commits into
js-reparsefrom
js-bindings

Conversation

@jamesconwaydev

Copy link
Copy Markdown
Collaborator

Part of #174, #175, #176.

Stacked on #217; the diff against js-reparse is this change alone. The XML halves of these issues wait for the XML parser, so they stay open after this.

Problem. parsec-js parses JavaScript and reads page state, but only from Rust. A Python crawler that wants window.__NUXT__ off a page it already parsed with parsec_html, a Node tool that wants ESTree, or a C program had to shell out or reach for another parser.

Change.

  • Python (bindings/python/src/js.rs): parsec_html.js.parse(source, *, module=False, max_depth=None, max_steps=None) gives the ESTree as dicts and lists, with start and end as str indexes. A syntax error raises parsec_html.js.JSSyntaxError, a ValueError with code, message, line, column, start, end and frame; past max_steps it raises TimeoutError, as the selector limits do. parsec_html.js.page_state(source) reads one script, HTMLParser.page_state() every inline script a browser would run. Both build JSON with the GIL released and hand it to json.loads. import parsec_html.js works; the stub types it as parsec_html.js.
  • WebAssembly (bindings/wasm/src/js.rs): parseJs(source, { module, maxDepth, maxSteps }), pageState(source) and Document.pageState(), as plain objects through JSON.parse. A syntax error is a SyntaxError with code, line, column, start, end (UTF-16, as JavaScript indexes) and frame; past maxSteps, an Error naming maxSteps.
  • C: parsec_js_parse(src, len, opts, &json, &len, &err) gives ESTree JSON with byte offsets, freed with parsec_string_free like every owned string. A syntax error or too-deep nesting is the new PARSEC_ERR_JS, past max_steps PARSEC_ERR_TIMEOUT, each with a parsec_js_error (code, message, offsets, line, column, frame) freed by parsec_js_error_free. parsec_js_page_state reads one script, parsec_js_document_state every script in a parsed parsec_document. The source must be UTF-8, since the offsets count the bytes passed.
  • max_depth is at most 256, the default, in every binding: deeper needs more stack than a thread is sure to have, so a larger value is refused rather than risking an overflow.
  • A page-state piece is {"name", "value"}, or {"name", "error": {"code", "source"}} when the value would need code to run; source is the first 80 characters of what stopped it. The page versions take scripts and modules, and pass over src=, data blocks like application/json, <template> contents, and scripts that don't parse.
  • parsec-js gains what the bindings share: EstreeOptions::offsets (UTF-16 units by default, characters, or bytes), data::script_type (the HTML Standard's rule for which type and language run as JavaScript), and Found::write_json. parsec-base gains LineIndex::char_offset.
  • js is a default feature in the Python and WASM crates; --no-default-features builds the HTML parser alone. The C library always has it.

Trade-off. The JavaScript parser adds 197,554 bytes to the package's .wasm, 1,128,220 to 1,325,774 (80,437 gzipped, 530,941 to 611,378). I kept it in the default package, since it's the product, and added a CI step that builds --no-default-features and fails if that .wasm passes 1,180,000 bytes, so the HTML-only build can't grow by accident. A strict json() isn't exposed: Python, JavaScript and C users have their own JSON parsers for __NEXT_DATA__.

Tests.

  • bindings/python/tests/test_js.py, 8 tests: a script and a module (start/end slice the str across an emoji), a syntax error's code, place, message and frame, ((((, [[[[, {{{{ and ---- 100,000 deep raising too-deep, max_depth=3, max_steps raising TimeoutError, max_depth over 256 refused, page state of a script, of all six real-page fixtures in crates/parsec-js/tests/pages, and of a page mixing them with src=, JSON, a broken script, a module and a <template>, and 8 threads parsing at once.
  • bindings/wasm/tests/api.rs, 7 new tests, the same cases (UTF-16 offsets, SyntaxError and its fields, depth and step limits, bad maxDepth/maxSteps), run in Node and Firefox; tests/package.test.mjs checks the exports from the built package.
  • crates/parsec-capi/tests/capi.c, javascript(): ESTree JSON for a script and a module with byte offsets, the error struct's fields, a 100,000-deep (, max_depth, max_steps, bad flags, non-UTF-8, NULLs, page state of a script and of a parsed document; with JS_PAGES defined (as tests/c.rs and the sanitizer job now do), javascript_pages() reads each real-page fixture. every_status_has_a_message covers PARSEC_ERR_JS.
  • crates/parsec-js/tests/estree.rs, offsets_count_the_units_asked_for; tests/data.rs, a_long_refusal_is_cut_to_80_characters_in_json; doctests for script_type and Found::write_json.

Passed locally: cargo test --workspace, cargo clippy --workspace --all-targets -- -D warnings, cargo fmt --all --check, cargo doc for parsec-js and parsec-base with -D warnings, scripts/check-headers.sh, the cbindgen header check and C++ compile, cargo deny; maturin develop --release and pytest on 3.13 (51 passed, 1 skipped) and free-threaded 3.14t with the GIL off (52 passed), clippy with and without abi3t and with --no-default-features; wasm-pack test --node and --headless --firefox --test browser (19 each), clippy for wasm32 with and without the feature, ./build.sh and node --test tests/package.test.mjs (15); CMake and ctest against package.sh's package (2 of 2), and tests/capi.c under ASan and UBSan, with leaks --atExit reporting 0 leaks on macOS.

A Python caller slices a str by characters and a C caller by bytes,
but Ast::to_estree counted start and end in UTF-16 units, so an emoji
before a node put every offset after it off by one for them.

EstreeOptions::offsets now picks UTF-16 units (the default, as acorn
counts), characters or bytes, with LineIndex::char_offset in
parsec-base for the characters.

The bindings also need to know which <script> elements a browser runs
as JavaScript, and to hand page state on as JSON. data::script_type
decides the first from type and language as the HTML Standard does;
Found::write_json writes a piece of page state, cutting a refusal's
source to 80 characters.
parsec_html.js.parse() gives a script's or a module's ESTree as dicts
and lists, start and end as str indexes, and raises JSSyntaxError (a
ValueError with code, message, line, column, start, end and frame), or
TimeoutError past max_steps=. max_depth= is at most 256, the default,
since deeper needs more stack than a thread is sure to have.

parsec_html.js.page_state() reads one script's state, and
HTMLParser.page_state() every inline script a browser would run. Both
build JSON with the GIL released and hand it to json.loads.

The js feature is on by default; --no-default-features builds the HTML
parser alone.
parseJs(source, { module, maxDepth, maxSteps }) gives ESTree as plain
objects, and a syntax error throws a SyntaxError with code, line,
column, start, end and frame. pageState(source) and
Document.pageState() read the state a page keeps in its scripts. Each
builds JSON in Rust and hands it to JSON.parse.

The JavaScript parser adds 197,554 bytes to the .wasm (80,437
gzipped), so it sits behind a default js feature. CI now builds with
--no-default-features too and fails if that HTML-only .wasm grows past
1,180,000 bytes; today it's 1,128,220.
parsec_js_parse gives ESTree JSON with byte offsets, from UTF-8 only
(anything else is PARSEC_ERR_INVALID, since offsets count the bytes
passed). A syntax error or nesting past max_depth is the new
PARSEC_ERR_JS, past max_steps PARSEC_ERR_TIMEOUT; either fills a
parsec_js_error with the code, message, offsets, line, column and code
frame, freed by parsec_js_error_free.

parsec_js_page_state reads one script's state as JSON, and
parsec_js_document_state every inline script of a parsed document.

tests/capi.c covers each, and with JS_PAGES defined reads the page
fixtures in crates/parsec-js/tests/pages, as tests/c.rs and the
sanitizer job now do.
@jamesconwaydev jamesconwaydev added c-api The C ABI and header python The Python package wasm WebAssembly and npm javascript The JavaScript parser, parsec-js labels Nov 22, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

c-api The C ABI and header javascript The JavaScript parser, parsec-js python The Python package wasm WebAssembly and npm

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant