Skip to content

feat(js): page state in scripts could only be read by running them - #192

Open
jamesconwaydev wants to merge 3 commits into
js-benchfrom
js-data
Open

jamesconwaydev wants to merge 3 commits into
js-benchfrom
js-data

Conversation

@jamesconwaydev

Copy link
Copy Markdown
Collaborator

Fixes #162. Fixes #163.

Builds on #191, which this is based on; once the stack below it merges, this is rebased onto its base.

Problem. Pages keep their data in scripts as JavaScript literals: Nuxt 2's window.__NUXT__=(function(a,b,…){return {…}}(null,"",…)), Next.js App Router's self.__next_f.push([1,"…"]), a Redux store in window['__PRELOADED_STATE__'] = {…}, window.__FLAGS__ = JSON.parse("…"). Since #187 parsec-js has the tree for all of them, but nothing turned it into data, so a scraper still needed a browser or a regex that breaks on the first nested brace.

Change. A new module, parsec_js::data. Nothing in it runs code, reads a property or calls a function.

  • literal(ast, src, expr) -> Result<Json, Refused>. Json is Null, Bool, Number(f64), String, Array and Object(Vec<(String, Json)>) in source order, and to_json_string writes it the way JSON.stringify does.
    • Accepts: objects with name, string and number keys; arrays, with a hole as null; strings with every escape and line continuation decoded; numbers in every form; true, false, null; templates without substitutions; - and + on a number; ! on a number or boolean, for minifiers' !0 and !1; undefined and void on a literal, which leave a key out and are null in an array, as in JSON.stringify; parentheses. A key given twice keeps its first place and its last value.
    • Refuses, with the span that stopped it and a reason: names and anything that would run (not-a-literal), ${…} (substitution), BigInt (bigint, since JSON can't write one), regular expressions, computed keys, methods and accessors, spread, __proto__: (it sets the prototype, not a key), - or ! on a non-number, a value that's undefined as a whole, and a lone surrogate, which a Rust String can't hold. The table is on Refused.
  • json(text): a strict JSON parser, as JSON.parse reads it, for __NEXT_DATA__ and Nuxt 3's __NUXT_DATA__, whose text callers pass straight in. Nesting past DEFAULT_MAX_DEPTH is too-deep. No new dependency: serde_json is only a dev-dependency here, and the parser is 250 lines.
  • page_state(src) parses a script and page_state_in(ast, src) takes a tree; each gives a Found { name, span, value } for every top-level:
    • = assignment to a property chain from window, self or globalThis (window.__NUXT__.config, window['__PRELOADED_STATE__']), alone or in a comma list;
    • var, let or const with a value;
    • self.__next_f.push(…), which next_flight(&found) joins into the Flight payload; the doc comment says what [0], [1, …], [2, …] and [3, …] are.
      The value may also be JSON.parse of one string literal, or a function called at once with literal arguments, Nuxt 2's form: each parameter takes its argument's value. The function must be plain (not async or a generator), with plain parameters and a body that only returns. Substitution copies at most 16 bytes per byte of source, so one large argument named 10,000 times is too-large rather than a gigabyte.
  • fuzz/fuzz_targets/data.rs: any input through page_state and json. Spans stay inside the input, and every value writes out as JSON that reads back as the same text. It joins the nightly matrix in fuzz.yml.
  • tests/pages: seven excerpts of real pages, fetched on 21 November 2026, with a README giving each source. None of the pages states a licence, so each keeps only a few real keys and values. The Nuxt 2 one is Back Market as the Wayback Machine archived it in January 2022, since current sites have moved to Nuxt 3.
  • README gets one row, ROADMAP an M12 section, plus CONTRIBUTING's fuzz command and the changelog.

The bindings don't expose parsec-js yet, so this waits for M16 (#174 to #176) there too.

Tests. tests/data.rs, 16 tests:

  • json_values_round_trip_through_literals: 17 JSON texts through both json and literal, written out and read back the same.
  • json_agrees_with_serde_json on numbers at the edges (5e-324, 2^53 + 1, -0.0).
  • json_refuses_what_json_parse_would: 17 inputs, each with its reason and offset.
  • literal_accepts_what_minified_javascript_writes: 28 accepted forms. literal_refuses_everything_else_where_it_stops: 34 refused forms, each with its reason and the text at its span.
  • page_state_*: each shape, plus 8 sources that look close and aren't page state (window.X += 1, window[k] = 1, an assignment inside an if or a function), the JSON.parse forms, and 18 Nuxt function cases, 12 of them refused.
  • real_pages: every file in tests/pages.
  • deep_nesting_is_an_error_not_a_crash: 100,000-deep [ is too-deep from json and from the parser, on a 1 MB thread.
  • wide_input_takes_linear_time: arrays, objects, a key given 80,000 times, JSON.parse and Nuxt substitution all grow linearly.
  • substitution_cannot_blow_up: the 100 KB argument named 10,000 times.

Run over all the inline scripts of 88 public pages, outside the repository: no panic. Every __next_f push on nextjs.org, vercel.com, moonpay.com, douyu.com and vinted.fr reads, and so does the full 43 KB Back Market __NUXT__. What's refused is code: window.dataLayer = window.dataLayer || [], and new Date(…) inside Nike's state.

cargo +nightly fuzz run data -- -max_total_time=180 -max_len=4096: 5,463,493 runs, no crash. cargo test --workspace, cargo clippy --workspace --all-targets -- -D warnings, cargo fmt --all --check, cargo doc -p parsec-js with -D warnings, cargo deny check licenses bans sources, the fuzz manifests' cargo metadata and scripts/check-headers.sh pass locally.

Nuxt's window.__NUXT__, Next.js's self.__next_f pushes and Redux stores
in a variable hold a page's data as JavaScript literals. With parsec-js
we have the tree, but nothing turned it into data, so a scraper still
needed a browser or a regex.

parsec_js::data adds:

- literal: an expression of literals to a Json value, or a Refused with
  the span that stopped it and a reason. It takes objects with plain
  keys, arrays, strings, numbers, booleans, null, templates without
  substitutions, - and + on numbers, and the !0, !1 and void 0 that
  minifiers write; undefined drops a key and is null in an array, as
  in JSON.stringify. BigInt, computed keys, methods, spread and
  __proto__ are refused.
- json: a strict JSON parser, for __NEXT_DATA__, depth-limited at
  DEFAULT_MAX_DEPTH.
- page_state and page_state_in: top-level assignments to window, self
  and globalThis, var, let and const, JSON.parse of a string, Nuxt 2's
  function form with its literal arguments put in place of its
  parameters (copies capped at 16 bytes a source byte), and Next.js's
  self.__next_f pushes, which next_flight joins.

Each shape is tested on an excerpt of a real page in tests/pages, and
a data fuzz target joins the nightly run.
Brings the branch onto its new base after the pull request below it merged.
# Conflicts:
#	.github/workflows/fuzz.yml
#	README.md

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

javascript The JavaScript parser, parsec-js

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant