Repository navigation
feat(js): one keystroke in a large file meant parsing all of it again - #217
Open
jamesconwaydev wants to merge 3 commits into
Open
jamesconwaydev wants to merge 3 commits into
jamesconwaydev wants to merge 3 commits into
Conversation
An editor or a language server reparses on every keystroke. With only
`parse`, a one-letter change to typescript.js cost a full 33 ms parse.
- `Ast::reparse(src, &Edit { range, text }, &options)` makes the edit
and parses again only the smallest function body or statement around
it, from the new source, with what the parser knew there worked out
from the nodes above: strict code, the function kind, `super`, loops,
labels, private names in scope and a bound on the depth. The new
nodes and lists go in the old ones' place in the arena, the rest are
renumbered and their spans moved, so the tree is the one `parse`
gives, node for node.
- A piece is only used if it starts at the same token kind and ends
where it did. An edit that opens a comment, string or template past
it, one in a directive prologue or a parameter list, a statement
starting with `import` or `export`, or a step limit sends it to a
full parse. An edit inside a comment that stays one comment moves
spans and nothing else.
- tests/reparse.rs makes seeded random edits in chains to test262's
harness and a sample of its tests, and to the bench bundles when
they're fetched, and checks each against `parse`: the ESTree,
comments and warnings, or the same error with the tree unchanged. A
unit test checks the arena itself is identical. A `reparse` fuzz
target does the same and joins the nightly run.
- `bench js-reparse` times single-character edits to typescript.js:
2.2 ms median against 33 ms for `parse`.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #167.
Stacked on #216; the diff against
js-diagis this change alone.Problem. An editor or a language server parses again on every keystroke. With only
parse, changing one letter in typescript.js (9 MB) cost a full parse, 33 ms, though the statement it touched was a hundred-odd bytes.Change.
Ast::reparse(src, &Edit { range, text }, &options)makes the edit and parses again only the smallest function body or statement list item that holds it, keeping its first and last characters. It returns the new source and the span it parsed again (Nonewhen it parsed the whole source)."use strict"prologue, a class), the function kind foryield,await,return,new.targetandsuper, the loops, switches and labels around it, the class bodies whose private names it may use, and a bound on the depth somax_depthstill holds.parse_piecein the parser starts there with that context.parsegives for the new source, node for node, soAst::ids()stays in order with nothing left over./*, a string, a template, or added a}), its first token changed kind, the edit is in a directive prologue or a parameter list, the statement starts withimportorexport(the module checks its exports as a whole), a private name the piece uses isn't declared around it, ormax_stepsis set, since a step budget counts the whole source. Up to three pieces, smallest first, are tried before that.edit-out-of-rangeis the error for a range past the end, backwards, or splitting a character, and is in theparsec_js::diagnosticstable.Trade-off. Moving the spans and ids after the piece is linear in the tree, so on typescript.js that, not the piece, is most of the 2.2 ms. Relative spans would make it constant, at the cost of every span lookup.
Tests.
tests/reparse.rs,reparse_matches_a_full_parse_on_test262: seeded random edits, twelve in a row per file, to test262's harness and every 40th test intest/language, mostly the letter, digit and space changes typing makes and the rest brackets, quotes, comment and template delimiters,/, line breaks and keywords. After each, the ESTree, comments and warnings must equal a full parse's, or the error must be the same and the tree unchanged. 6,204 edits a run: 5,306 parsed in part, 119 whole, 779 syntax errors. Twenty more seeds locally, about 124,000 edits, found nothing.reparse_matches_a_full_parse_on_bundles: the same on react-dom, three and typescript.js whenbench/fetch-pages.shhas fetched them: 660 edits a run in release, about 470 in part. Five seeds locally, nothing.an_ast_from_reparse_reparses_again: 1,000 edits in a row to one tree.reparse::tests::random_edits_keep_the_arena_exact: the arena itself, nodes, lists, comments and root, equals a full parse's, over 3,000 random edits (60,000 locally).what_was_known_there_is_known_again:super(), a private name, a labelledcontinueand a sloppy legacy octal, each in a piece parsed again;edits_that_reach_outside_fall_back:/*,`,', a prologue becoming"use strict", and two statements joined;a_bad_range_is_refused.reparsefuzz target comparesreparsewithparseon any source and edit, and joins the nightly matrix: 3,383,800 runs in 3 minutes, nothing found.cargo run --release -- js-reparseinbench/, 400 single-character edits in a row to typescript.js:Ast::reparse2.2 ms median,parse33.1 ms, 15 times as fast; none parsed whole, and the median piece was 152 bytes. Parse speed on thejsbench, two runs each againstjs-perf: typescript.js 278 and 273 MB/s before, 274 and 271 after; all sources 287 and 283 before, 284 and 281 after.Passed locally:
cargo test --workspace,cargo clippy --workspace --all-targets -- -D warnings(and inbench/),cargo fmt --all --check,RUSTDOCFLAGS=-D warnings cargo doc,scripts/check-headers.sh.