Skip to content

feat(js): parsec-js couldn't tell which declaration a name refers to - #193

Open
jamesconwaydev wants to merge 2 commits into
js-datafrom
js-scope
Open

jamesconwaydev wants to merge 2 commits into
js-datafrom
js-scope

Conversation

@jamesconwaydev

Copy link
Copy Markdown
Collaborator

Fixes #158.

Builds on #192, which this is based on; once the stack below it merges, this is rebased onto its base.

Problem. Since #187 parsec-js gives a tree, but nothing says what a name in it means. A tool that renames a variable, lists a script's globals or bundles modules has to know that the x in { let x; x } is the block's and the one after the block isn't, that f() before function f() {} calls it, and that arguments in an arrow belongs to the function around it. The same gap left 190 of test262's parse negatives accepted: let a; var a; and try {} catch (e) { let e; } parse fine, and only the scopes show the name declared twice.

Change. A new module, parsec_js::scope. Nothing runs.

  • Scopes::analyze(&ast, src) -> Scopes, one walk over the tree on its own stack, so a 200,000-deep member chain analyzes on a 256 KB thread.
  • Scopes (ScopeKind): Global, Module, Function, FunctionBody (a function's body when a parameter has a default or computed key, so function f(a = x) { var x } reads the outer x, as the spec's separate environment has it), Block (blocks, switch cases, for heads with let or const, and the block B.3.3 puts around if (a) function f() {}), Catch, With, Class, ClassField (each field initializer's implicit function) and StaticBlock. Each has its node, parent, strict, and dynamic.
  • Bindings (BindingKind): var and function declarations hoisted to their function; let, const, class in their block; parameters; catch parameters; imports; a function expression's and a class's own name inside it; and arguments, made in a non-arrow function only when something refers to it.
  • Annex B: in sloppy code, a plain function declared in a block also gets a var binding in its function (B.3.2), unless a var of its name would be an early error there or it's a parameter's name; references outside the block resolve to that. A var in a catch block may share a simple parameter's name (B.3.4), for (var e of …) included, as test262 now has it.
  • References: every IdentifierReference, with binding (None when free), access (Read, Write, ReadWrite), and tdz when it comes before its let, const or class in the same function (let x = x;, class C extends C {}, for (let x of x)). Property names, labels, new.target, private names and re-exported names aren't references.
  • dynamic: a direct eval(…) (or (eval)(…), but not eval?.() or (0, eval)()) marks its scope and every scope around it; a with body is a dynamic scope of its own.
  • Lookups: scope_of(node), reference(node), declaration(node), lookup(scope, name), free_names() (each once, in order of first use), and scopes(), bindings(), references().
  • diagnostics(): duplicate-declaration for a lexical name declared twice in a scope, or as a parameter or catch parameter too, and lexical-var-conflict when a var and a lexical name meet, with a label on the first declaration. Sloppy blocks may declare a plain function twice; generators and async functions may not, and nor may strict code or a module's top level.
  • tests/test262.rs runs every file through the analysis as well as the parser. The redeclaration path fragments leave its early_error list; the regular expression ones and the two undeclared-export ones stay until the next PRs in this stack.
  • The parse fuzz target now also checks each reference is an identifier whose binding has its name, or a free name, and each diagnostic's span.
  • parsec-html-bench js-scope times the analysis on the bench corpus and prints every source's free names.
  • README gets a row, ROADMAP an M11 section (the same heading and intro as docs: the README still called Parsec only an HTML parser #178's, so whichever merges second keeps both), and the changelog an entry.

Tests. tests/scope.rs, 22 tests. A helper writes each reference as name#n, the nth declaration of that name it resolved to, so a test reads as script("let x; { let x; x } x", &["x#2", "x#1"]).

  • Resolution: references_resolve_to_the_nearest_declaration, var_and_function_declarations_hoist_to_their_function, let_const_and_class_are_block_scoped, parameters_and_the_body (with and without the separate body scope), arguments_is_implicit_in_functions_but_not_arrows, a_function_expression_sees_its_own_name, catch_parameters_are_scoped_to_the_clause, class_fields_and_static_blocks_get_scopes, modules_bind_imports_and_refer_to_exports, escaped_names_are_the_same_name, names_that_are_not_references_are_left_alone.
  • the_temporal_dead_zone_is_flagged: 13 cases, including the ones a nested function or a static field makes safe.
  • annex_b_hoists_sloppy_functions_in_blocks: 12 cases, each of the conditions that stop the hoisting.
  • writes_are_told_from_reads, eval_and_with_make_scopes_dynamic, lookups_by_node, free_names_are_listed_once_in_order, deep_trees_are_analyzed_without_recursion.
  • Early errors: lexical_names_declared_twice (21 sources), lexical_and_var_names_that_meet (15), redeclarations_the_spec_allows (19, which fail if the analysis is too eager), a_diagnostic_points_at_the_second_and_labels_the_first.

test262 goes from 3,824 to 4,014 of the 4,374 parse negatives caught, 360 still skipped (358 regular expression patterns, 2 undeclared exports), 0 failures. None of the 48,408 files that should parse gets a diagnostic, which is the check on false positives.

On the bench corpus, the analysis runs at 239 MB/s on typescript.js (45,147 scopes, 245,302 references), 273 MB/s on react-dom, 364 MB/s on three.js. The free names it lists are the globals each bundle uses; typescript.js's 63 include BreakpointResolver and the like, which its esbuild module.exports annotation names and never declares. The analysis is opt-in and the parser is unchanged.

cargo +nightly fuzz run parse -- -max_total_time=120 -max_len=4096: 1,331,398 runs, no crash. cargo test --workspace, cargo clippy --workspace --all-targets -- -D warnings and the bench's, cargo fmt --check for both, cargo doc for each crate with -D warnings, cargo deny check licenses bans sources for both, the fuzz manifests' cargo metadata and scripts/check-headers.sh pass locally.

The parser gives a tree, but a tool that renames a variable, finds a
global or bundles modules needs to know what each name means, and
test262's 190 redeclaration tests still parsed, since only the scopes
can tell that `let a; var a;` declares a name twice.

parsec_js::scope adds Scopes::analyze, one pass over the tree with its
own stack:

- scopes for the global or module level, functions (with a separate
  body scope when a parameter has a default or computed key), blocks,
  switches, for heads with let or const, catch clauses, with bodies,
  classes, class fields' initializers and static blocks;
- bindings with var and function declarations hoisted, Annex B's
  function-in-block var for sloppy code where a var of the name would
  be no error and it isn't a parameter, B.3.4's var in a catch block,
  a function expression's and a class's own names, imports, and an
  implicit arguments in each non-arrow function that uses it;
- every IdentifierReference resolved or listed in free_names, with its
  access (read, write or both) and a TDZ flag when it comes before its
  let, const or class in the same function;
- dynamic on the scopes a direct eval reaches and on with bodies;
- lookups by node: scope_of, reference, declaration, and lookup by
  name;
- diagnostics for duplicate-declaration and lexical-var-conflict, with
  the first declaration labelled.

test262 now runs every file through the analysis too: the 190
redeclaration negatives fail through it, and none of the 48,408 files
that should parse gets a diagnostic, so the scope fragments leave the
early_error list. The parse fuzz target checks the analysis's
references and bindings, and parsec-html-bench js-scope times it on
the bench corpus and lists every free name.
Brings the branch onto its new base after the pull request below it merged.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

javascript The JavaScript parser, parsec-js

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant