Skip to content

feat(js): one keystroke in a large file meant parsing all of it again - #217

Open
jamesconwaydev wants to merge 3 commits into
js-diagfrom
js-reparse
Open

jamesconwaydev wants to merge 3 commits into
js-diagfrom
js-reparse

Conversation

@jamesconwaydev

Copy link
Copy Markdown
Collaborator

Fixes #167.

Stacked on #216; the diff against js-diag is this change alone.

Problem. An editor or a language server parses again on every keystroke. With only parse, changing one letter in typescript.js (9 MB) cost a full parse, 33 ms, though the statement it touched was a hundred-odd bytes.

Change.

  • Ast::reparse(src, &Edit { range, text }, &options) makes the edit and parses again only the smallest function body or statement list item that holds it, keeping its first and last characters. It returns the new source and the span it parsed again (None when it parsed the whole source).
  • What the parser knew when it reached that piece comes from the nodes above it: strict code (the module, a "use strict" prologue, a class), the function kind for yield, await, return, new.target and super, the loops, switches and labels around it, the class bodies whose private names it may use, and a bound on the depth so max_depth still holds. parse_piece in the parser starts there with that context.
  • The piece's nodes and lists go in place of the old ones in the arena, and what follows is renumbered and its spans, comments and warnings moved. The tree is the one parse gives for the new source, node for node, so Ast::ids() stays in order with nothing left over.
  • It parses the whole source instead when a piece can't be sure: the piece doesn't end where it did (an edit opened a /*, a string, a template, or added a }), its first token changed kind, the edit is in a directive prologue or a parameter list, the statement starts with import or export (the module checks its exports as a whole), a private name the piece uses isn't declared around it, or max_steps is set, since a step budget counts the whole source. Up to three pieces, smallest first, are tried before that.
  • An edit inside a comment that stays one comment, with a line break only if it had one, changes no token: only spans move.
  • edit-out-of-range is the error for a range past the end, backwards, or splitting a character, and is in the parsec_js::diagnostics table.

Trade-off. Moving the spans and ids after the piece is linear in the tree, so on typescript.js that, not the piece, is most of the 2.2 ms. Relative spans would make it constant, at the cost of every span lookup.

Tests.

  • tests/reparse.rs, reparse_matches_a_full_parse_on_test262: seeded random edits, twelve in a row per file, to test262's harness and every 40th test in test/language, mostly the letter, digit and space changes typing makes and the rest brackets, quotes, comment and template delimiters, /, line breaks and keywords. After each, the ESTree, comments and warnings must equal a full parse's, or the error must be the same and the tree unchanged. 6,204 edits a run: 5,306 parsed in part, 119 whole, 779 syntax errors. Twenty more seeds locally, about 124,000 edits, found nothing.
  • reparse_matches_a_full_parse_on_bundles: the same on react-dom, three and typescript.js when bench/fetch-pages.sh has fetched them: 660 edits a run in release, about 470 in part. Five seeds locally, nothing.
  • an_ast_from_reparse_reparses_again: 1,000 edits in a row to one tree.
  • reparse::tests::random_edits_keep_the_arena_exact: the arena itself, nodes, lists, comments and root, equals a full parse's, over 3,000 random edits (60,000 locally).
  • what_was_known_there_is_known_again: super(), a private name, a labelled continue and a sloppy legacy octal, each in a piece parsed again; edits_that_reach_outside_fall_back: /*, `, ', a prologue becoming "use strict", and two statements joined; a_bad_range_is_refused.
  • The new reparse fuzz target compares reparse with parse on any source and edit, and joins the nightly matrix: 3,383,800 runs in 3 minutes, nothing found.

cargo run --release -- js-reparse in bench/, 400 single-character edits in a row to typescript.js: Ast::reparse 2.2 ms median, parse 33.1 ms, 15 times as fast; none parsed whole, and the median piece was 152 bytes. Parse speed on the js bench, two runs each against js-perf: typescript.js 278 and 273 MB/s before, 274 and 271 after; all sources 287 and 283 before, 284 and 281 after.

Passed locally: cargo test --workspace, cargo clippy --workspace --all-targets -- -D warnings (and in bench/), cargo fmt --all --check, RUSTDOCFLAGS=-D warnings cargo doc, scripts/check-headers.sh.

An editor or a language server reparses on every keystroke. With only
`parse`, a one-letter change to typescript.js cost a full 33 ms parse.

- `Ast::reparse(src, &Edit { range, text }, &options)` makes the edit
  and parses again only the smallest function body or statement around
  it, from the new source, with what the parser knew there worked out
  from the nodes above: strict code, the function kind, `super`, loops,
  labels, private names in scope and a bound on the depth. The new
  nodes and lists go in the old ones' place in the arena, the rest are
  renumbered and their spans moved, so the tree is the one `parse`
  gives, node for node.
- A piece is only used if it starts at the same token kind and ends
  where it did. An edit that opens a comment, string or template past
  it, one in a directive prologue or a parameter list, a statement
  starting with `import` or `export`, or a step limit sends it to a
  full parse. An edit inside a comment that stays one comment moves
  spans and nothing else.
- tests/reparse.rs makes seeded random edits in chains to test262's
  harness and a sample of its tests, and to the bench bundles when
  they're fetched, and checks each against `parse`: the ESTree,
  comments and warnings, or the same error with the tree unchanged. A
  unit test checks the arena itself is identical. A `reparse` fuzz
  target does the same and joins the nightly run.
- `bench js-reparse` times single-character edits to typescript.js:
  2.2 ms median against 33 ms for `parse`.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

javascript The JavaScript parser, parsec-js

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant