Skip to content

feat(js): Parsec had no JavaScript lexer, nor a base parsers could share - #185

Merged
jamesconwaydev merged 3 commits into
mainfrom
js-lexer
Nov 22, 2026
Merged

jamesconwaydev merged 3 commits into
mainfrom
js-lexer

Conversation

@jamesconwaydev

@jamesconwaydev jamesconwaydev commented Nov 21, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #143. Fixes #147. Part of #150.

Problem. Parsec had nothing for JavaScript, and each of its parsers carries its own spans, line counting and error type. A JavaScript parser needs a lexer first, and the hard part of one is that / divides after an expression and starts a regular expression anywhere else, and } either closes a block or carries a template on. A lexer that guesses gets a / b / c or `${x}` wrong.

Change.

  • crates/parsec-base (parsec_base), with no dependencies: Span (two u32s), LineIndex (line and column in characters or UTF-16 code units, UTF-16 offsets, and ECMAScript's U+2028/U+2029 line ends as an option), Diagnostic with Severity and labels, and Limits with a step Budget. Column lookups stay cheap on a minified one-line script: checkpoints every 1 KiB. Nothing synced from Netweir uses it.
  • crates/parsec-js (parsec_js): Lexer::next_token(goal), where the goal is Div, RegExp, TemplateTail or RegExpOrTemplateTail, the spec's input element goals, so the parser decides and the lexer never guesses. It covers identifiers with ID_Start/ID_Continue and \u escapes (an escaped keyword stays Ident, flagged ESCAPED), private names, every numeric literal form (hex, octal, binary, BigInt, separators only between digits, nothing straight after a number), strings with every escape and line continuations, template parts, regular expression literals with their flags checked (the body is left for the pattern grammar), every punctuator (?. not before a digit), comments, a hashbang, and Annex B's <!-- and line-start --> in scripts only.
  • Legacy octal literals (017, 08) and escapes ("\01", "\8") carry LEGACY_OCTAL and a diagnostic: a warning in a script, since only the parser knows whether the code is strict, and an error in a module. A template's NotEscapeSequence gets INVALID_ESCAPE and no diagnostic, since a tagged template allows it.
  • A token is 12 bytes, kind, flags and span, with nothing on the heap; only diagnostics allocate. The lexer never fails: what it can't read comes back as far as it goes, or as Invalid, with a diagnostic.
  • scripts/gen-unicode.py writes src/unicode.rs from Unicode 17.0.0's DerivedCoreProperties.txt, the version crates/parsec's \p{} tables and test262 use, the way crates/parsec and parsec-url generate theirs.
  • fuzz/ with a lex target, in fuzz.yml's nightly matrix: both source types, the test driver's goals and goals the input picks, checking that every token is non-empty, on character boundaries and in order. Its corpus comes from test262 in the next pull request, which is why Fuzz the JavaScript lexer #150 stays open.
  • README gets one row, ROADMAP an M9 section, and CHANGELOG, CONTRIBUTING and NOTICE a line each.

crates/parsec and crates/parsec-xpath aren't touched.

  • release.yml publishes parsec-base and parsec-js too, first in the list since nothing else depends on them yet. Both names are reserved on crates.io with trusted publishing set up for this workflow.

Tests.

  • crates/parsec-base: spans; line_col, line_col_utf16 and utf16_offset checked at every character against a count from the start, over LF, CR, CRLF, U+2028/U+2029 and characters of every width, on a line longer than a block; line starts; diagnostics; limits and the budget.
  • crates/parsec-js/tests/lexer.rs, 34 tests, one or more per production: whitespace, line terminators and newline_before, comments and the newline a block comment can hold, the hashbang, identifiers and escapes, keywords, private names, every punctuator, the goals, decimal, non-decimal, BigInt, separators, legacy octal, strings and their escapes, legacy octal escapes, regexps and flags, templates and NotEscapeSequences, both HTML-like comments, spans tiling the source, and twelve 1 MB hostile inputs that a rescanning lexer would take hours on.
  • On real code (axe-core, its minified build, lodash): no diagnostics, 170 to 440 MB/s in release.
  • cargo +nightly fuzz run lex for 90 seconds: 1.86 million runs, nothing found.
  • Locally: cargo fmt --all --check, cargo clippy --workspace --all-targets -- -D warnings, cargo test --workspace (393 passed), cargo doc for both crates with -D warnings, cargo deny check licenses bans sources, scripts/check-headers.sh.

Parsec's parsers each carry their own spans, line counting and errors,
and there was nothing for JavaScript at all.

crates/parsec-base holds what the parsers will share: Span, LineIndex
(line and column, in characters or UTF-16, and UTF-16 offsets),
Diagnostic with Severity and labels, and Limits with a step Budget. It
has no dependencies. The HTML and CSS code synced from Netweir doesn't
use it.

crates/parsec-js lexes ECMAScript 2025's lexical grammar: identifiers
with ID_Start and ID_Continue from the Unicode 17.0 data
crates/parsec's tables use, and escapes; every numeric literal form,
BigInt and separators included; strings; template parts; regular
expression literals with their flags checked; punctuators; comments; a
hashbang; and Annex B's HTML-like comments, in scripts only. The caller
passes the goal, so the lexer never guesses what `/` or `}` is. Legacy
octal literals and escapes are flagged on the token, a warning in a
script and an error in a module. A token is 12 bytes with nothing
behind it on the heap.

A fuzz target, lex, joins the nightly fuzz run.
@jamesconwaydev

Copy link
Copy Markdown
Collaborator Author

@foothillsdigital, there's a queue for you whenever you have time. Here's the order that makes it easiest, since the JavaScript work is a stack:

  1. docs: the README still called Parsec only an HTML parser #178: README, ROADMAP and the new header (docs only)
  2. feat: newcomers found no way in from the flat issues list #182: the pinned dashboards
  3. feat(js): Parsec had no JavaScript lexer, nor a base parsers could share #185 (this one): parsec-base and the JavaScript lexer
  4. test(js): test262 never ran, and the lexer's fuzzing had no corpus #186: test262 in CI
  5. feat(js): parsec-js could lex JavaScript but not parse it #187: the parser for scripts, with ESTree output (the big one; it matches acorn on all 48,051 test262 files both parse)
  6. feat(js): modules parsed only as far as their first import or export #188: modules
  7. test(js): the parser's limits were untested, and it had no fuzz target #189: limits and the parse fuzz target
  8. feat(bench): parsec-js had no numbers against oxc or swc #191: the bench against oxc and swc

I'll rebase each one onto main as the one below it merges, so you won't need to. #178 and #185 both touch ROADMAP's M9 section; I'll resolve that once the first is in. Thanks!

@jamesconwaydev

Copy link
Copy Markdown
Collaborator Author

One more at the end of the stack: #192, page data read out of scripts as JSON (Next.js, Nuxt, Redux stores), on top of #191.

@foothillsdigital foothillsdigital left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. cargo test -p parsec-base -p parsec-js passes, with the 34 lexer tests and the base's tests in the output, clippy with -D warnings is clean, and CI is green.

I ran 60 edge cases through tests/driver to see where a hand-written lexer usually slips. All of them match the spec:

  • Numbers: 08.5 is a decimal with a leading-zero warning, while 08n, 08_1, 01n, 0.0n, 1e3n, 0_1 and 0x_1 are errors. 5..toString, 1n.x and 1.e3 split right, and a?.5:1 gives ? then .5.
  • Strings: "\08" is a legacy octal escape, "\0" isn't, and "\8" in a module is an error. \u{0000000041} is accepted and \u{110000} and \u{} are refused, and U+2028 in a string is fine.
  • Templates: `\08`, `\u{` and `\u{110000}` get INVALID_ESCAPE and no diagnostic. Nested templates pair up.
  • Identifiers: await is Ident + ESCAPED. 0a, a b and \u{1F600} are refused, while ZWJ inside a name and the Other_ID_Start characters (℘, ゛) are accepted. #! past the start is an error.
  • Whitespace: U+FEFF, U+00A0, U+1680 and U+3000 are whitespace, U+180E isn't.
  • Regular expressions: /[/]/, /[\]/]/ and /a/dgimsy lex whole, while /a/gg, /a/uv, /a/x and an escaped flag are refused. A newline or U+2028 ends an unterminated one at the line.
  • HTML-like comments: --> is a comment after a newline or a /*\n*/, but not after /**/ or mid-line, and in a module it's always -- >. <!-- in a module is < ! --.

One thing to decide, not to change here: release.yml now publishes parsec-base and parsec-js with every release. If 0.2.1 (#135) is cut after this merges, a lexer-only parsec-js 0.2.1 goes to crates.io, while #178's ROADMAP says 0.3.0 is the first release with it. Either cut 0.2.1 before merging this, or reword the ROADMAP line. Both names are already reserved, so it isn't a problem if one goes out early.

@jamesconwaydev
jamesconwaydev added this pull request to the merge queue Nov 21, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Nov 21, 2026
@jamesconwaydev

Copy link
Copy Markdown
Collaborator Author

@foothillsdigital, #178 landed first, so this needed main merged in. I kept #178's M9 intro in ROADMAP and put this PR's Done bullets under it. The push dismissed your approval, so it needs one more look; nothing else changed. Auto-merge is on.

@foothillsdigital foothillsdigital left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-approved. The merge only brings in #178: the PR still changes the same 27 files, and the M9 section keeps #178's intro with this PR's Done bullets under it. CI is green.

@jamesconwaydev
jamesconwaydev added this pull request to the merge queue Nov 22, 2026
Merged via the queue into main with commit 710d2ab Nov 22, 2026
33 checks passed
@jamesconwaydev
jamesconwaydev deleted the js-lexer branch November 22, 2026 01:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

parsec-js: the ECMAScript lexical grammar A shared foundation crate: input, positions, diagnostics, limits

2 participants