Skip to content

feat(js): parsec-js could lex JavaScript but not parse it - #187

Open
jamesconwaydev wants to merge 3 commits into
js-test262from
js-parser
Open

jamesconwaydev wants to merge 3 commits into
js-test262from
js-parser

Conversation

@jamesconwaydev

Copy link
Copy Markdown
Collaborator

Fixes #153. Part of #152, which closes with the bench in #156: its check is on the bench corpus.

Builds on #186 (and so #185), which this is based on; once those merge, this is rebased onto main.

Problem. parsec-js could lex JavaScript and do nothing else with it. test262 ran through a driver that guessed each / and } from the token before, so a regular expression after a block's } lexed as division, and the 4,174 parse negatives whose error is in the syntax, not the tokens, went unchecked.

Change.

  • The tree (src/ast.rs). Every node in one arena, addressed by NodeId, with its Span beside it. Node is a typed enum whose variants follow ESTree's node types, plus Paren, which keeps the parentheses the early errors need. Lists (a block's statements, a call's arguments) are ranges of one shared vector, so a parse allocates a few vectors, not one per node. Comments go in Ast::comments, and sloppy-mode legacy octal warnings in Ast::warnings.
  • The parser (src/parser/). Recursive descent over clauses 13 to 15: every statement and declaration, the expression grammar by precedence climbing, destructuring, classes with private names (checked against the class bodies around them), static blocks, async functions and generators, optional chaining, automatic semicolon insertion and the restricted productions, and Annex B's additions for sloppy code (if (a) function f() {}, labelled functions, for (var a = 1 in b), f() = 1 as a runtime error).
    • Every token is read with Goal::Div. Where an expression starts, a / or /= is read again with Goal::RegExp, and after a substitution, the } with Goal::TemplateTail. Lexer::rescan does it from the token's start; nothing else is ever read twice.
    • Cover grammars are parsed once as what they cover: (a, b) becomes arrow parameters in place when => follows, [a, b] a pattern when = does. Errors only the expression reading has ({a = 1}, a second __proto__) wait on the object until it's known which reading it is.
    • Grammar parameters and the early errors that need no scope: yield and await by context, strict mode from directive prologues (with the retroactive checks on a function's name and parameters), reserved words, break/continue targets, new.target and super where they're allowed, arguments in class initializers, duplicate parameters where they're errors, ?? mixed with ||, unary before **.
    • The first error ends the parse, as a parsec_base::Diagnostic with a stable code (unexpected-token, invalid-target, reserved-word, ...).
  • Limits. Recursion counts against Limits::max_depth, 256 when it's unset (DEFAULT_MAX_DEPTH), and tokens against max_steps. typescript.js, 9 MB, needs a depth of 53. At 256 the deepest-nesting input I found uses about 230 KB of stack in a release build and 700 KB in a debug one. The 1,000,000-deep test and the parse fuzz target come with Limits: depth, step budget, no stack overflow #155.
  • ESTree (src/estree.rs). Ast::to_estree writes acorn's shape for ecmaVersion: 2025: the same types and fields, start and end in UTF-16 code units, loc as an option, numbers as Number::toString writes them, lone surrogates as \udxxx, and BigInt and regex values as null with bigint and regex beside them, as acorn's are after JSON.stringify. It keeps its own stack, so a chain of a million .xs writes without touching the thread's.
  • src/text.rs reads what a token means: an identifier's name with escapes decoded, a string's or template's value in UTF-16 code units, a number's value, a BigInt's digits.
  • tests/test262.rs drives the parser. A file runs sloppy and strict unless its flags say one, and passes only if every run does what it should. Parse negatives the parser accepts are skipped and counted only when early_error names them: redeclarations, which need scope analysis, and regular expression patterns, which the lexer leaves unchecked. Both are M11. Modules are skipped until Modules: import, export and import attributes #154. The counts go to CI's summary as test262-parse.
  • examples/estree.rs prints a file's ESTree JSON. scripts/gen-estree-fixtures.sh writes the fixtures in tests/estree with acorn 8.19.0, and scripts/compare-acorn.sh compares any files, or all of test262, with acorn.
  • .gitattributes keeps the fixtures byte for byte, since their offsets point into the sources beside them.
  • README's JavaScript row, ROADMAP's new M10 section and the changelog say what's done. The test driver for the lexer's own tests keeps its guess, with the shortcut: comment updated.

Tests.

  • test262 (2a1ade303f72): 51,534 passed, 0 failed, 2,561 skipped. All 47,895 script files that should parse do. Of the 4,174 parse negatives, 3,639 fail to parse; the other 535 are the early errors above (358 regular expression patterns, 177 redeclarations). 1,007 module files and 1,019 past ES2025 or not UTF-8 are skipped. 7 s in a debug build.
  • tests/estree.rs: 11 sources covering every node type, Annex B, ASI, Unicode identifiers and astral characters (UTF-16 offsets), comments and loc, against acorn 8.19.0's trees.
  • scripts/compare-acorn.sh, run locally: all 47,542 test262 scripts that acorn and parsec-js both parse give the same tree, and so do typescript.js 5.9.3, react-dom 19.2.0 (production and development) and react 19.2.0.
  • tests/parser.rs: 25 tests, one per group of productions, each with what it accepts and the code for what it rejects. Without the matching change, annex_b_function_declarations fails on while (a) b: function f() {}, conditional_and_assignment on f() = 1, classes on class A { get\n*a() {} }, and blocks_and_control_flow on a 10,000-long else if chain.
  • text.rs unit tests: numbers print as JavaScript prints them (1e+21, 5e-7), hex and BigInt values past 64 bits, string and template escapes, lone surrogates.

cargo test --workspace, cargo clippy --workspace --all-targets -- -D warnings, cargo fmt --all --check, cargo doc with -D warnings for parsec-js and parsec-base, cargo deny check licenses bans sources and scripts/check-headers.sh pass locally.

parsec-js had a lexer and nothing to drive it, so test262 ran through a
driver that guessed each `/` and `}` from the token before.

- An arena AST (`ast.rs`): typed node kinds that follow ESTree's, a span
  on every node, lists as ranges of one shared vector, comments beside
  the tree.
- A recursive-descent parser for ECMAScript 2025 scripts with Annex B:
  every statement and declaration, the expression grammar, destructuring,
  classes with private names and static blocks, async and generators,
  optional chaining and ASI. It asks the lexer for every token with
  `Goal::Div` and reads a `/` or `}` again with the goal it turns out to
  need. Cover grammars are parsed once and turned into patterns in place.
  The first error ends the parse, as a `Diagnostic`.
- Recursion counts against a depth limit, 256 by default, and tokens
  against the step budget.
- `Ast::to_estree` writes ESTree JSON in acorn's shape, offsets in UTF-16,
  with its own stack, so a deep tree never touches the thread's.
- test262 runs through the parser. Early errors that need scope analysis
  or the regular expression grammar are skipped with the reason and
  counted.
- ESTree fixtures from acorn 8.19.0 in tests/estree, the script that
  writes them, and one that compares any files with acorn.
early_error matches paths with `/`, and on Windows each test's name had
`\`, so the 535 early errors it names failed there instead of being
skipped. The name now uses `/` on every OS.
Brings the branch onto its new base after the pull request below it merged.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

javascript The JavaScript parser, parsec-js

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant