Repository navigation
fix(js): invalid regular expression patterns parsed as if valid - #194
Open
jamesconwaydev wants to merge 2 commits into
Open
jamesconwaydev wants to merge 2 commits into
jamesconwaydev wants to merge 2 commits into
Conversation
The lexer finds where a regular expression literal ends and checks its
flags, but nothing read the pattern, so /[z-a]/, /(?<a>x)(?<a>y)/ and
/\p{Lettr}/u parsed, where an engine throws a SyntaxError before
running anything. 358 of test262's parse negatives were skipped for
it.
parsec_js::regexp::validate reads a pattern by ECMA-262's pattern
grammar and its early errors, in the three modes the flags choose:
without u, with Annex B's B.1.2 (legacy octal and identity escapes,
\c, a lookahead quantified, braces as characters, [\d-a] as a union,
code units rather than code points); with u; and with v's nested
classes, && and --, \q{...} and properties of strings, with what may
match a string kept from being negated. Named groups, ES2025's
duplicate names in separate alternatives, (?ims-ims:...) modifiers,
backreferences, quantifier ranges and every \p{...} name and value in
Unicode 17.0 are checked. Each error has a stable code, listed in the
module docs, and a span in the pattern.
The names come from scripts/gen-regexp-names.py, which reads the UCD's
PropertyValueAliases.txt and copies ECMA-262's tables of binary
properties. Nesting stops at DEFAULT_MAX_DEPTH with too-deep, and fits
a 1 MB thread in a debug build; duplicate group names are counted per
alternative, so checking them is linear.
test262 now validates every literal's pattern: all 358 negatives fail
and every file that should parse still does. A regexp fuzz target
joins the nightly run, and the parse target validates each literal.
Brings the branch onto its new base after the pull request below it merged.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #159: the regular expression grammar. The rest of #159 is the next PR in this stack.
Builds on #193, which this is based on; once the stack below it merges, this is rebased onto its base.
Problem. The lexer finds where a regular expression literal ends and checks its flags, but nothing read the pattern between the slashes.
/[z-a]/,/(?<a>x)(?<a>y)/,/a**/,/\p{Lettr}/uand/[a-z&&b]/vall parsed, where an engine throws a SyntaxError before any code runs. test262 skipped 358 parse negatives for it.Change. A new module,
parsec_js::regexp, with one function:validate(pattern, flags) -> Result<(), Diagnostic>. It reads the pattern by ECMA-262's grammar (22.2.1) and its early errors and gives the error aRegExpbuilt from it would throw. It builds nothing and matches nothing.uorv: Annex B's B.1.2. Legacy octal and identity escapes,\8and\9,\cwithout a letter as a backslash,\kaskuntil a named group appears, a lookahead quantified,{and}as characters unless they form a quantifier with nothing before it (/{1}/throws),[\d-a]as a union, and the pattern read as UTF-16 code units, so/[💩-💫]/throws as it does in V8.u: the strict grammar. Every escape has to be one,],{and}have to be escaped,\1has to name a group.v: nested classes,&&and--(never mixed, never with a range as an operand),\q{…}, properties of strings, the escapable and double punctuators, and the MayContainStrings rule that keeps[^\p{RGI_Emoji}]and\P{RGI_Emoji}errors.\uescapes and astral names; ES2025's duplicate names, allowed only in separate alternatives;\k<name>checked against every group, before or after it; modifiers(?i-ms:…)with no flag twice and at least one; lookbehinds never quantified.\p{…}and\P{…}: General_Category, Script and Script_Extensions values and binary properties by exact name, fromsrc/regexp/names.rs.scripts/gen-regexp-names.pywrites it from Unicode 17.0'sPropertyValueAliases.txt, the versionsrc/unicode.rsand test262 use, and ECMA-262's tables of binary properties and properties of strings; the binary list matchescrates/parsec's.regexp-flags,regexp-nothing-to-repeat,regexp-quantifier-range,regexp-lone-bracket,regexp-unterminated,regexp-group,regexp-modifiers,regexp-group-name,regexp-duplicate-group-name,regexp-unknown-group-name,regexp-backreference,regexp-escape,regexp-range-order,regexp-range-class,regexp-property,regexp-negated-strings,regexp-class-syntax,too-deep. Spans are byte offsets in the pattern.vclasses nest toDEFAULT_MAX_DEPTH, thentoo-deep; only three small functions sit on the stack per level, so 256 levels fit a 1 MB thread in a debug build. Duplicate group names are counted per alternative and disjunction, so 100,000 alternatives of one name check in linear time, not quadratic.tests/test262.rsvalidates every regular expression literal's pattern. The path fragments for patterns leaveearly_error; the two undeclared-export ones stay for the next PR.regexpfuzz target (each input as a pattern with no flag,u,vandgimsyd) joins the nightly matrix infuzz.yml, and theparsetarget validates every literal it parses. CONTRIBUTING lists the new target.Tests.
tests/regexp.rs, 15 tests, each with sources that must pass and sources that must fail with a given code, in each mode the rule touches:flags,atoms_and_alternatives,quantifiers(including B.1.2's quantified lookahead and lone braces),brackets_need_escaping_in_unicode_mode,groups,modifiers,group_names(escaped and astral names, duplicates in and out of separate alternatives),backreferences(forward ones, and B.1.2's octal fallback),character_escapes,classes(ranges, B.1.2's[\d-a]and[\c1], the surrogate-pair range),property_escapes(39 names and values, right and wrong, and properties of strings),v_flag_classes(15 valid set expressions and 43 invalid ones, among them one for each reserved double punctuator).spans_point_at_the_trouble, with multi-byte text before the error.deep_nesting_is_an_error_not_a_crash: 100,000-deep groups andvclasses aretoo-deepon a 1 MB thread; 200 deep is fine.wide_patterns_take_linear_time: 100,000 same-name groups, classes and backreferences.test262 goes from 4,014 to 4,372 of the 4,374 parse negatives caught: all 358 pattern tests now fail through
validate, with 0 failures. Every regular expression in the 48,408 files that should parse validates, includingbuilt-ins/RegExp/property-escapes' 442 generated files, which use every property value; that's the check on false positives.cargo +nightly fuzz run regexp -- -max_total_time=240 -max_len=256: 4,852,130 runs, no crash.cargo +nightly fuzz run parse -- -max_total_time=150 -max_len=4096: 1,510,068 runs, no crash.cargo test --workspace,cargo clippy --workspace --all-targets -- -D warningsand the bench's,cargo fmt --checkfor both,cargo docfor each crate with-D warnings,cargo deny check licenses bans sourcesfor both, the fuzz manifests'cargo metadataandscripts/check-headers.shpass locally.