Repository navigation
feat: JavaScript couldn't be parsed from Python, WebAssembly or C - #219
Open
jamesconwaydev wants to merge 5 commits into
Open
jamesconwaydev wants to merge 5 commits into
jamesconwaydev wants to merge 5 commits into
Conversation
A Python caller slices a str by characters and a C caller by bytes, but Ast::to_estree counted start and end in UTF-16 units, so an emoji before a node put every offset after it off by one for them. EstreeOptions::offsets now picks UTF-16 units (the default, as acorn counts), characters or bytes, with LineIndex::char_offset in parsec-base for the characters. The bindings also need to know which <script> elements a browser runs as JavaScript, and to hand page state on as JSON. data::script_type decides the first from type and language as the HTML Standard does; Found::write_json writes a piece of page state, cutting a refusal's source to 80 characters.
parsec_html.js.parse() gives a script's or a module's ESTree as dicts and lists, start and end as str indexes, and raises JSSyntaxError (a ValueError with code, message, line, column, start, end and frame), or TimeoutError past max_steps=. max_depth= is at most 256, the default, since deeper needs more stack than a thread is sure to have. parsec_html.js.page_state() reads one script's state, and HTMLParser.page_state() every inline script a browser would run. Both build JSON with the GIL released and hand it to json.loads. The js feature is on by default; --no-default-features builds the HTML parser alone.
parseJs(source, { module, maxDepth, maxSteps }) gives ESTree as plain
objects, and a syntax error throws a SyntaxError with code, line,
column, start, end and frame. pageState(source) and
Document.pageState() read the state a page keeps in its scripts. Each
builds JSON in Rust and hands it to JSON.parse.
The JavaScript parser adds 197,554 bytes to the .wasm (80,437
gzipped), so it sits behind a default js feature. CI now builds with
--no-default-features too and fails if that HTML-only .wasm grows past
1,180,000 bytes; today it's 1,128,220.
parsec_js_parse gives ESTree JSON with byte offsets, from UTF-8 only (anything else is PARSEC_ERR_INVALID, since offsets count the bytes passed). A syntax error or nesting past max_depth is the new PARSEC_ERR_JS, past max_steps PARSEC_ERR_TIMEOUT; either fills a parsec_js_error with the code, message, offsets, line, column and code frame, freed by parsec_js_error_free. parsec_js_page_state reads one script's state as JSON, and parsec_js_document_state every inline script of a parsed document. tests/capi.c covers each, and with JS_PAGES defined reads the page fixtures in crates/parsec-js/tests/pages, as tests/c.rs and the sanitizer job now do.
This was referenced Nov 22, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #174, #175, #176.
Stacked on #217; the diff against
js-reparseis this change alone. The XML halves of these issues wait for the XML parser, so they stay open after this.Problem.
parsec-jsparses JavaScript and reads page state, but only from Rust. A Python crawler that wantswindow.__NUXT__off a page it already parsed withparsec_html, a Node tool that wants ESTree, or a C program had to shell out or reach for another parser.Change.
bindings/python/src/js.rs):parsec_html.js.parse(source, *, module=False, max_depth=None, max_steps=None)gives the ESTree as dicts and lists, withstartandendasstrindexes. A syntax error raisesparsec_html.js.JSSyntaxError, aValueErrorwithcode,message,line,column,start,endandframe; pastmax_stepsit raisesTimeoutError, as the selector limits do.parsec_html.js.page_state(source)reads one script,HTMLParser.page_state()every inline script a browser would run. Both build JSON with the GIL released and hand it tojson.loads.import parsec_html.jsworks; the stub types it asparsec_html.js.bindings/wasm/src/js.rs):parseJs(source, { module, maxDepth, maxSteps }),pageState(source)andDocument.pageState(), as plain objects throughJSON.parse. A syntax error is aSyntaxErrorwithcode,line,column,start,end(UTF-16, as JavaScript indexes) andframe; pastmaxSteps, an Error namingmaxSteps.parsec_js_parse(src, len, opts, &json, &len, &err)gives ESTree JSON with byte offsets, freed withparsec_string_freelike every owned string. A syntax error or too-deep nesting is the newPARSEC_ERR_JS, pastmax_stepsPARSEC_ERR_TIMEOUT, each with aparsec_js_error(code, message, offsets, line, column, frame) freed byparsec_js_error_free.parsec_js_page_statereads one script,parsec_js_document_stateevery script in a parsedparsec_document. The source must be UTF-8, since the offsets count the bytes passed.max_depthis at most 256, the default, in every binding: deeper needs more stack than a thread is sure to have, so a larger value is refused rather than risking an overflow.{"name", "value"}, or{"name", "error": {"code", "source"}}when the value would need code to run;sourceis the first 80 characters of what stopped it. The page versions take scripts and modules, and pass oversrc=, data blocks likeapplication/json,<template>contents, and scripts that don't parse.parsec-jsgains what the bindings share:EstreeOptions::offsets(UTF-16 units by default, characters, or bytes),data::script_type(the HTML Standard's rule for whichtypeandlanguagerun as JavaScript), andFound::write_json.parsec-basegainsLineIndex::char_offset.jsis a default feature in the Python and WASM crates;--no-default-featuresbuilds the HTML parser alone. The C library always has it.Trade-off. The JavaScript parser adds 197,554 bytes to the package's
.wasm, 1,128,220 to 1,325,774 (80,437 gzipped, 530,941 to 611,378). I kept it in the default package, since it's the product, and added a CI step that builds--no-default-featuresand fails if that.wasmpasses 1,180,000 bytes, so the HTML-only build can't grow by accident. A strictjson()isn't exposed: Python, JavaScript and C users have their own JSON parsers for__NEXT_DATA__.Tests.
bindings/python/tests/test_js.py, 8 tests: a script and a module (start/endslice thestracross an emoji), a syntax error's code, place, message and frame,((((,[[[[,{{{{and----100,000 deep raisingtoo-deep,max_depth=3,max_stepsraisingTimeoutError,max_depthover 256 refused, page state of a script, of all six real-page fixtures incrates/parsec-js/tests/pages, and of a page mixing them withsrc=, JSON, a broken script, a module and a<template>, and 8 threads parsing at once.bindings/wasm/tests/api.rs, 7 new tests, the same cases (UTF-16 offsets,SyntaxErrorand its fields, depth and step limits, badmaxDepth/maxSteps), run in Node and Firefox;tests/package.test.mjschecks the exports from the built package.crates/parsec-capi/tests/capi.c,javascript(): ESTree JSON for a script and a module with byte offsets, the error struct's fields, a 100,000-deep(,max_depth,max_steps, bad flags, non-UTF-8, NULLs, page state of a script and of a parsed document; withJS_PAGESdefined (astests/c.rsand the sanitizer job now do),javascript_pages()reads each real-page fixture.every_status_has_a_messagecoversPARSEC_ERR_JS.crates/parsec-js/tests/estree.rs,offsets_count_the_units_asked_for;tests/data.rs,a_long_refusal_is_cut_to_80_characters_in_json; doctests forscript_typeandFound::write_json.Passed locally:
cargo test --workspace,cargo clippy --workspace --all-targets -- -D warnings,cargo fmt --all --check,cargo docforparsec-jsandparsec-basewith-D warnings,scripts/check-headers.sh, the cbindgen header check and C++ compile,cargo deny;maturin develop --releaseand pytest on 3.13 (51 passed, 1 skipped) and free-threaded 3.14t with the GIL off (52 passed), clippy with and withoutabi3tand with--no-default-features;wasm-pack test --nodeand--headless --firefox --test browser(19 each), clippy for wasm32 with and without the feature,./build.shandnode --test tests/package.test.mjs(15); CMake andctestagainstpackage.sh's package (2 of 2), andtests/capi.cunder ASan and UBSan, withleaks --atExitreporting 0 leaks on macOS.