Repository navigation
test(js): test262 never ran, and the lexer's fuzzing had no corpus - #186
Open
jamesconwaydev wants to merge 3 commits into
Open
jamesconwaydev wants to merge 3 commits into
jamesconwaydev wants to merge 3 commits into
Conversation
Parsec's parsers each carry their own spans, line counting and errors, and there was nothing for JavaScript at all. crates/parsec-base holds what the parsers will share: Span, LineIndex (line and column, in characters or UTF-16, and UTF-16 offsets), Diagnostic with Severity and labels, and Limits with a step Budget. It has no dependencies. The HTML and CSS code synced from Netweir doesn't use it. crates/parsec-js lexes ECMAScript 2025's lexical grammar: identifiers with ID_Start and ID_Continue from the Unicode 17.0 data crates/parsec's tables use, and escapes; every numeric literal form, BigInt and separators included; strings; template parts; regular expression literals with their flags checked; punctuators; comments; a hashbang; and Annex B's HTML-like comments, in scripts only. The caller passes the goal, so the lexer never guesses what `/` or `}` is. Legacy octal literals and escapes are flagged on the token, a warning in a script and an error in a module. A token is 12 bytes with nothing behind it on the heap. A fuzz target, lex, joins the nightly fuzz run.
tc39/test262 is now a pinned, shallow submodule in crates/parsec-js/tests/test262, and tests/test262.rs runs every file through the lexer, reading each one's frontmatter for module, onlyStrict and negative.phase. A file that should parse has to tokenize with no errors; one that shouldn't passes when the lexer reports one, and is skipped otherwise, since its error is in the syntax and the parser will check it. The counts go to $CONFORMANCE_DIR like the other suites', so CI's summary has them: 49,655 passed, none failed, 4,440 skipped. test262 found that `-->` can start a comment on the first line, as InputElementHashbangOrRegExp has HTMLCloseComment; the lexer now allows it. The test driver reads a `/` after the `)` of if, while, for or with as a regex, and `of` as an identifier. The lex fuzz target's corpus is 480 test262 files, minimized with cargo fuzz cmin, with test262's licence beside them.
This was referenced Nov 21, 2026
Brings the branch onto its new base after the pull request below it merged.
jamesconwaydev
enabled auto-merge
November 22, 2026 01:41
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #148. Fixes #150.
Builds on #185, which this is based on; once that merges, this is rebased onto
main.Problem. The lexer in #185 had only its own unit tests. test262, the suite every JavaScript engine runs, never touched it, so a production the tests missed would go unnoticed, and the
lexfuzz target started from nothing each night.Change.
2a1ade303f72is a submodule incrates/parsec-js/tests/test262, markedshallowin.gitmodules, since its history is large and only the files are needed.tests/test262.rsreads each file's frontmatter (flags: [module],onlyStrict,negative.phase: parse,features, as[a, b]or- alists) and tokenizes it as a script or module. A file that should parse passes if the lexer reports no errors, with the legacy octal warnings counted as errors in strict files. A parse negative passes if the lexer reports an error, and is skipped if it doesn't: its error is in the syntax, which the parser (M10) checks. Decorator tests, whose@isn't an ES2025 token, and files that aren't UTF-8 are skipped too. The counts go to$CONFORMANCE_DIR/test262-tokens.json, so CI's summary table picks them up with the other suites.-->can start a comment on the first line of a script (annexB/language/comments/single-line-html-close-first-line-*.js), since the starting goal, InputElementHashbangOrRegExp, has HTMLCloseComment. The lexer now allows it./after the)ofif,while,fororwithas a regex, and treatsofas an identifier (test262'sdivision/no-magic-asi.jshasinstance/of/g). Its known limit stays in ashortcut:comment: a regex starting a statement after a block's}lexes as division, until the parser passes real goals.lexfuzz target's corpus,crates/parsec-js/fuzz/corpus/lex, is 480 test262 files from the lexical directories, minimized withcargo fuzz cmin, with test262'sLICENSEbeside them and a-textrule in.gitattributes.Tests.
literals,comments,line-terminators,template-literal,identifiers): each is a syntax or early error, such as an escaped keyword, a"use strict"directive, a NotEscapeSequence in an untagged template, or a regex pattern error.frontmatterchecks the YAML reader on inline and block lists and on a runtime negative.tests/lexer.rs:-->on the first line, after whitespace and a delimited comment;if (a) /b/as a regex,f(a) / 2andof / 2as division.cargo +nightly fuzz run lexfrom the new corpus for 120 seconds: 1.02 million runs, nothing found.cargo fmt --all --check,cargo clippy --workspace --all-targets -- -D warnings,cargo test --workspace(395 passed).