Repository navigation
feat(js): parsec-js could lex JavaScript but not parse it - #187
Open
jamesconwaydev wants to merge 3 commits into
Open
jamesconwaydev wants to merge 3 commits into
jamesconwaydev wants to merge 3 commits into
Conversation
parsec-js had a lexer and nothing to drive it, so test262 ran through a driver that guessed each `/` and `}` from the token before. - An arena AST (`ast.rs`): typed node kinds that follow ESTree's, a span on every node, lists as ranges of one shared vector, comments beside the tree. - A recursive-descent parser for ECMAScript 2025 scripts with Annex B: every statement and declaration, the expression grammar, destructuring, classes with private names and static blocks, async and generators, optional chaining and ASI. It asks the lexer for every token with `Goal::Div` and reads a `/` or `}` again with the goal it turns out to need. Cover grammars are parsed once and turned into patterns in place. The first error ends the parse, as a `Diagnostic`. - Recursion counts against a depth limit, 256 by default, and tokens against the step budget. - `Ast::to_estree` writes ESTree JSON in acorn's shape, offsets in UTF-16, with its own stack, so a deep tree never touches the thread's. - test262 runs through the parser. Early errors that need scope analysis or the regular expression grammar are skipped with the reason and counted. - ESTree fixtures from acorn 8.19.0 in tests/estree, the script that writes them, and one that compares any files with acorn.
early_error matches paths with `/`, and on Windows each test's name had `\`, so the 535 early errors it names failed there instead of being skipped. The name now uses `/` on every OS.
This was referenced Nov 21, 2026
Brings the branch onto its new base after the pull request below it merged.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #153. Part of #152, which closes with the bench in #156: its check is on the bench corpus.
Builds on #186 (and so #185), which this is based on; once those merge, this is rebased onto
main.Problem.
parsec-jscould lex JavaScript and do nothing else with it. test262 ran through a driver that guessed each/and}from the token before, so a regular expression after a block's}lexed as division, and the 4,174 parse negatives whose error is in the syntax, not the tokens, went unchecked.Change.
src/ast.rs). Every node in one arena, addressed byNodeId, with itsSpanbeside it.Nodeis a typed enum whose variants follow ESTree's node types, plusParen, which keeps the parentheses the early errors need. Lists (a block's statements, a call's arguments) are ranges of one shared vector, so a parse allocates a few vectors, not one per node. Comments go inAst::comments, and sloppy-mode legacy octal warnings inAst::warnings.src/parser/). Recursive descent over clauses 13 to 15: every statement and declaration, the expression grammar by precedence climbing, destructuring, classes with private names (checked against the class bodies around them), static blocks, async functions and generators, optional chaining, automatic semicolon insertion and the restricted productions, and Annex B's additions for sloppy code (if (a) function f() {}, labelled functions,for (var a = 1 in b),f() = 1as a runtime error).Goal::Div. Where an expression starts, a/or/=is read again withGoal::RegExp, and after a substitution, the}withGoal::TemplateTail.Lexer::rescandoes it from the token's start; nothing else is ever read twice.(a, b)becomes arrow parameters in place when=>follows,[a, b]a pattern when=does. Errors only the expression reading has ({a = 1}, a second__proto__) wait on the object until it's known which reading it is.yieldandawaitby context, strict mode from directive prologues (with the retroactive checks on a function's name and parameters), reserved words,break/continuetargets,new.targetandsuperwhere they're allowed,argumentsin class initializers, duplicate parameters where they're errors,??mixed with||, unary before**.parsec_base::Diagnosticwith a stable code (unexpected-token,invalid-target,reserved-word, ...).Limits::max_depth, 256 when it's unset (DEFAULT_MAX_DEPTH), and tokens againstmax_steps. typescript.js, 9 MB, needs a depth of 53. At 256 the deepest-nesting input I found uses about 230 KB of stack in a release build and 700 KB in a debug one. The 1,000,000-deep test and theparsefuzz target come with Limits: depth, step budget, no stack overflow #155.src/estree.rs).Ast::to_estreewrites acorn's shape forecmaVersion: 2025: the same types and fields,startandendin UTF-16 code units,locas an option, numbers asNumber::toStringwrites them, lone surrogates as\udxxx, and BigInt and regex values asnullwithbigintandregexbeside them, as acorn's are afterJSON.stringify. It keeps its own stack, so a chain of a million.xs writes without touching the thread's.src/text.rsreads what a token means: an identifier's name with escapes decoded, a string's or template's value in UTF-16 code units, a number's value, a BigInt's digits.tests/test262.rsdrives the parser. A file runs sloppy and strict unless its flags say one, and passes only if every run does what it should. Parse negatives the parser accepts are skipped and counted only whenearly_errornames them: redeclarations, which need scope analysis, and regular expression patterns, which the lexer leaves unchecked. Both are M11. Modules are skipped until Modules: import, export and import attributes #154. The counts go to CI's summary astest262-parse.examples/estree.rsprints a file's ESTree JSON.scripts/gen-estree-fixtures.shwrites the fixtures intests/estreewith acorn 8.19.0, andscripts/compare-acorn.shcompares any files, or all of test262, with acorn..gitattributeskeeps the fixtures byte for byte, since their offsets point into the sources beside them.shortcut:comment updated.Tests.
2a1ade303f72): 51,534 passed, 0 failed, 2,561 skipped. All 47,895 script files that should parse do. Of the 4,174 parse negatives, 3,639 fail to parse; the other 535 are the early errors above (358 regular expression patterns, 177 redeclarations). 1,007 module files and 1,019 past ES2025 or not UTF-8 are skipped. 7 s in a debug build.tests/estree.rs: 11 sources covering every node type, Annex B, ASI, Unicode identifiers and astral characters (UTF-16 offsets), comments andloc, against acorn 8.19.0's trees.scripts/compare-acorn.sh, run locally: all 47,542 test262 scripts that acorn and parsec-js both parse give the same tree, and so do typescript.js 5.9.3, react-dom 19.2.0 (production and development) and react 19.2.0.tests/parser.rs: 25 tests, one per group of productions, each with what it accepts and the code for what it rejects. Without the matching change,annex_b_function_declarationsfails onwhile (a) b: function f() {},conditional_and_assignmentonf() = 1,classesonclass A { get\n*a() {} }, andblocks_and_control_flowon a 10,000-longelse ifchain.text.rsunit tests: numbers print as JavaScript prints them (1e+21,5e-7), hex and BigInt values past 64 bits, string and template escapes, lone surrogates.cargo test --workspace,cargo clippy --workspace --all-targets -- -D warnings,cargo fmt --all --check,cargo docwith-D warningsforparsec-jsandparsec-base,cargo deny check licenses bans sourcesandscripts/check-headers.shpass locally.