Skip to content

test(js): test262 never ran, and the lexer's fuzzing had no corpus - #186

Open
jamesconwaydev wants to merge 3 commits into
mainfrom
js-test262
Open

jamesconwaydev wants to merge 3 commits into
mainfrom
js-test262

Conversation

@jamesconwaydev

Copy link
Copy Markdown
Collaborator

Fixes #148. Fixes #150.

Builds on #185, which this is based on; once that merges, this is rebased onto main.

Problem. The lexer in #185 had only its own unit tests. test262, the suite every JavaScript engine runs, never touched it, so a production the tests missed would go unnoticed, and the lex fuzz target started from nothing each night.

Change.

  • tc39/test262 at 2a1ade303f72 is a submodule in crates/parsec-js/tests/test262, marked shallow in .gitmodules, since its history is large and only the files are needed.
  • tests/test262.rs reads each file's frontmatter (flags: [module], onlyStrict, negative.phase: parse, features, as [a, b] or - a lists) and tokenizes it as a script or module. A file that should parse passes if the lexer reports no errors, with the legacy octal warnings counted as errors in strict files. A parse negative passes if the lexer reports an error, and is skipped if it doesn't: its error is in the syntax, which the parser (M10) checks. Decorator tests, whose @ isn't an ES2025 token, and files that aren't UTF-8 are skipped too. The counts go to $CONFORMANCE_DIR/test262-tokens.json, so CI's summary table picks them up with the other suites.
  • test262 found one bug: --> can start a comment on the first line of a script (annexB/language/comments/single-line-html-close-first-line-*.js), since the starting goal, InputElementHashbangOrRegExp, has HTMLCloseComment. The lexer now allows it.
  • The test driver, which picks goals from the token before because there's no parser yet, now reads a / after the ) of if, while, for or with as a regex, and treats of as an identifier (test262's division/no-magic-asi.js has instance/of/g). Its known limit stays in a shortcut: comment: a regex starting a statement after a block's } lexes as division, until the parser passes real goals.
  • The lex fuzz target's corpus, crates/parsec-js/fuzz/corpus/lex, is 480 test262 files from the lexical directories, minimized with cargo fuzz cmin, with test262's LICENSE beside them and a -text rule in .gitattributes.
  • NOTICE names test262. README's parsec-js row quotes the count, and ROADMAP's M9 moves test262 in CI #148 and Fuzz the JavaScript lexer #150 to done.

Tests.

  • test262: 49,655 passed, 0 failed, 4,440 skipped (4,409 parse negatives left for the parser, 31 decorator or non-UTF-8 files). Of the 4,676 parse negatives, the lexer catches 267. I read through the uncaught ones in the lexical directories (literals, comments, line-terminators, template-literal, identifiers): each is a syntax or early error, such as an escaped keyword, a "use strict" directive, a NotEscapeSequence in an untagged template, or a regex pattern error.
  • frontmatter checks the YAML reader on inline and block lists and on a runtime negative.
  • tests/lexer.rs: --> on the first line, after whitespace and a delimited comment; if (a) /b/ as a regex, f(a) / 2 and of / 2 as division.
  • cargo +nightly fuzz run lex from the new corpus for 120 seconds: 1.02 million runs, nothing found.
  • Locally: cargo fmt --all --check, cargo clippy --workspace --all-targets -- -D warnings, cargo test --workspace (395 passed).

Parsec's parsers each carry their own spans, line counting and errors,
and there was nothing for JavaScript at all.

crates/parsec-base holds what the parsers will share: Span, LineIndex
(line and column, in characters or UTF-16, and UTF-16 offsets),
Diagnostic with Severity and labels, and Limits with a step Budget. It
has no dependencies. The HTML and CSS code synced from Netweir doesn't
use it.

crates/parsec-js lexes ECMAScript 2025's lexical grammar: identifiers
with ID_Start and ID_Continue from the Unicode 17.0 data
crates/parsec's tables use, and escapes; every numeric literal form,
BigInt and separators included; strings; template parts; regular
expression literals with their flags checked; punctuators; comments; a
hashbang; and Annex B's HTML-like comments, in scripts only. The caller
passes the goal, so the lexer never guesses what `/` or `}` is. Legacy
octal literals and escapes are flagged on the token, a warning in a
script and an error in a module. A token is 12 bytes with nothing
behind it on the heap.

A fuzz target, lex, joins the nightly fuzz run.
tc39/test262 is now a pinned, shallow submodule in
crates/parsec-js/tests/test262, and tests/test262.rs runs every file
through the lexer, reading each one's frontmatter for module, onlyStrict
and negative.phase. A file that should parse has to tokenize with no
errors; one that shouldn't passes when the lexer reports one, and is
skipped otherwise, since its error is in the syntax and the parser will
check it. The counts go to $CONFORMANCE_DIR like the other suites', so
CI's summary has them: 49,655 passed, none failed, 4,440 skipped.

test262 found that `-->` can start a comment on the first line, as
InputElementHashbangOrRegExp has HTMLCloseComment; the lexer now allows
it. The test driver reads a `/` after the `)` of if, while, for or with
as a regex, and `of` as an identifier.

The lex fuzz target's corpus is 480 test262 files, minimized with
cargo fuzz cmin, with test262's licence beside them.
Brings the branch onto its new base after the pull request below it merged.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant