Repository navigation
feat(js): Parsec had no JavaScript lexer, nor a base parsers could share - #185
Conversation
Parsec's parsers each carry their own spans, line counting and errors, and there was nothing for JavaScript at all. crates/parsec-base holds what the parsers will share: Span, LineIndex (line and column, in characters or UTF-16, and UTF-16 offsets), Diagnostic with Severity and labels, and Limits with a step Budget. It has no dependencies. The HTML and CSS code synced from Netweir doesn't use it. crates/parsec-js lexes ECMAScript 2025's lexical grammar: identifiers with ID_Start and ID_Continue from the Unicode 17.0 data crates/parsec's tables use, and escapes; every numeric literal form, BigInt and separators included; strings; template parts; regular expression literals with their flags checked; punctuators; comments; a hashbang; and Annex B's HTML-like comments, in scripts only. The caller passes the goal, so the lexer never guesses what `/` or `}` is. Legacy octal literals and escapes are flagged on the token, a warning in a script and an error in a module. A token is 12 bytes with nothing behind it on the heap. A fuzz target, lex, joins the nightly fuzz run.
|
@foothillsdigital, there's a queue for you whenever you have time. Here's the order that makes it easiest, since the JavaScript work is a stack:
I'll rebase each one onto main as the one below it merges, so you won't need to. #178 and #185 both touch ROADMAP's M9 section; I'll resolve that once the first is in. Thanks! |
foothillsdigital
left a comment
There was a problem hiding this comment.
Approved. cargo test -p parsec-base -p parsec-js passes, with the 34 lexer tests and the base's tests in the output, clippy with -D warnings is clean, and CI is green.
I ran 60 edge cases through tests/driver to see where a hand-written lexer usually slips. All of them match the spec:
- Numbers:
08.5is a decimal with a leading-zero warning, while08n,08_1,01n,0.0n,1e3n,0_1and0x_1are errors.5..toString,1n.xand1.e3split right, anda?.5:1gives?then.5. - Strings:
"\08"is a legacy octal escape,"\0"isn't, and"\8"in a module is an error.\u{0000000041}is accepted and\u{110000}and\u{}are refused, and U+2028 in a string is fine. - Templates:
`\08`,`\u{`and`\u{110000}`getINVALID_ESCAPEand no diagnostic. Nested templates pair up. - Identifiers:
awaitisIdent+ESCAPED.0a,a band\u{1F600}are refused, while ZWJ inside a name and the Other_ID_Start characters (℘, ゛) are accepted.#!past the start is an error. - Whitespace: U+FEFF, U+00A0, U+1680 and U+3000 are whitespace, U+180E isn't.
- Regular expressions:
/[/]/,/[\]/]/and/a/dgimsylex whole, while/a/gg,/a/uv,/a/xand an escaped flag are refused. A newline or U+2028 ends an unterminated one at the line. - HTML-like comments:
-->is a comment after a newline or a/*\n*/, but not after/**/or mid-line, and in a module it's always-->.<!--in a module is<!--.
One thing to decide, not to change here: release.yml now publishes parsec-base and parsec-js with every release. If 0.2.1 (#135) is cut after this merges, a lexer-only parsec-js 0.2.1 goes to crates.io, while #178's ROADMAP says 0.3.0 is the first release with it. Either cut 0.2.1 before merging this, or reword the ROADMAP line. Both names are already reserved, so it isn't a problem if one goes out early.
# Conflicts: # ROADMAP.md
|
@foothillsdigital, #178 landed first, so this needed main merged in. I kept #178's M9 intro in ROADMAP and put this PR's Done bullets under it. The push dismissed your approval, so it needs one more look; nothing else changed. Auto-merge is on. |
Fixes #143. Fixes #147. Part of #150.
Problem. Parsec had nothing for JavaScript, and each of its parsers carries its own spans, line counting and error type. A JavaScript parser needs a lexer first, and the hard part of one is that
/divides after an expression and starts a regular expression anywhere else, and}either closes a block or carries a template on. A lexer that guesses getsa / b / cor`${x}`wrong.Change.
crates/parsec-base(parsec_base), with no dependencies:Span(twou32s),LineIndex(line and column in characters or UTF-16 code units, UTF-16 offsets, and ECMAScript's U+2028/U+2029 line ends as an option),DiagnosticwithSeverityand labels, andLimitswith a stepBudget. Column lookups stay cheap on a minified one-line script: checkpoints every 1 KiB. Nothing synced from Netweir uses it.crates/parsec-js(parsec_js):Lexer::next_token(goal), where the goal isDiv,RegExp,TemplateTailorRegExpOrTemplateTail, the spec's input element goals, so the parser decides and the lexer never guesses. It covers identifiers withID_Start/ID_Continueand\uescapes (an escaped keyword staysIdent, flaggedESCAPED), private names, every numeric literal form (hex, octal, binary, BigInt, separators only between digits, nothing straight after a number), strings with every escape and line continuations, template parts, regular expression literals with their flags checked (the body is left for the pattern grammar), every punctuator (?.not before a digit), comments, a hashbang, and Annex B's<!--and line-start-->in scripts only.017,08) and escapes ("\01","\8") carryLEGACY_OCTALand a diagnostic: a warning in a script, since only the parser knows whether the code is strict, and an error in a module. A template's NotEscapeSequence getsINVALID_ESCAPEand no diagnostic, since a tagged template allows it.Invalid, with a diagnostic.scripts/gen-unicode.pywritessrc/unicode.rsfrom Unicode 17.0.0'sDerivedCoreProperties.txt, the versioncrates/parsec's\p{}tables and test262 use, the way crates/parsec and parsec-url generate theirs.fuzz/with alextarget, infuzz.yml's nightly matrix: both source types, the test driver's goals and goals the input picks, checking that every token is non-empty, on character boundaries and in order. Its corpus comes from test262 in the next pull request, which is why Fuzz the JavaScript lexer #150 stays open.crates/parsecandcrates/parsec-xpatharen't touched.release.ymlpublishesparsec-baseandparsec-jstoo, first in the list since nothing else depends on them yet. Both names are reserved on crates.io with trusted publishing set up for this workflow.Tests.
crates/parsec-base: spans;line_col,line_col_utf16andutf16_offsetchecked at every character against a count from the start, over LF, CR, CRLF, U+2028/U+2029 and characters of every width, on a line longer than a block; line starts; diagnostics; limits and the budget.crates/parsec-js/tests/lexer.rs, 34 tests, one or more per production: whitespace, line terminators andnewline_before, comments and the newline a block comment can hold, the hashbang, identifiers and escapes, keywords, private names, every punctuator, the goals, decimal, non-decimal, BigInt, separators, legacy octal, strings and their escapes, legacy octal escapes, regexps and flags, templates and NotEscapeSequences, both HTML-like comments, spans tiling the source, and twelve 1 MB hostile inputs that a rescanning lexer would take hours on.cargo +nightly fuzz run lexfor 90 seconds: 1.86 million runs, nothing found.cargo fmt --all --check,cargo clippy --workspace --all-targets -- -D warnings,cargo test --workspace(393 passed),cargo docfor both crates with-D warnings,cargo deny check licenses bans sources,scripts/check-headers.sh.