Repository navigation
feat(js): page state in scripts could only be read by running them - #192
Open
jamesconwaydev wants to merge 3 commits into
Open
jamesconwaydev wants to merge 3 commits into
jamesconwaydev wants to merge 3 commits into
Conversation
Nuxt's window.__NUXT__, Next.js's self.__next_f pushes and Redux stores in a variable hold a page's data as JavaScript literals. With parsec-js we have the tree, but nothing turned it into data, so a scraper still needed a browser or a regex. parsec_js::data adds: - literal: an expression of literals to a Json value, or a Refused with the span that stopped it and a reason. It takes objects with plain keys, arrays, strings, numbers, booleans, null, templates without substitutions, - and + on numbers, and the !0, !1 and void 0 that minifiers write; undefined drops a key and is null in an array, as in JSON.stringify. BigInt, computed keys, methods, spread and __proto__ are refused. - json: a strict JSON parser, for __NEXT_DATA__, depth-limited at DEFAULT_MAX_DEPTH. - page_state and page_state_in: top-level assignments to window, self and globalThis, var, let and const, JSON.parse of a string, Nuxt 2's function form with its literal arguments put in place of its parameters (copies capped at 16 bytes a source byte), and Next.js's self.__next_f pushes, which next_flight joins. Each shape is tested on an excerpt of a real page in tests/pages, and a data fuzz target joins the nightly run.
This was referenced Nov 21, 2026
Brings the branch onto its new base after the pull request below it merged.
# Conflicts: # .github/workflows/fuzz.yml # README.md
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #162. Fixes #163.
Builds on #191, which this is based on; once the stack below it merges, this is rebased onto its base.
Problem. Pages keep their data in scripts as JavaScript literals: Nuxt 2's
window.__NUXT__=(function(a,b,…){return {…}}(null,"",…)), Next.js App Router'sself.__next_f.push([1,"…"]), a Redux store inwindow['__PRELOADED_STATE__'] = {…},window.__FLAGS__ = JSON.parse("…"). Since #187 parsec-js has the tree for all of them, but nothing turned it into data, so a scraper still needed a browser or a regex that breaks on the first nested brace.Change. A new module,
parsec_js::data. Nothing in it runs code, reads a property or calls a function.literal(ast, src, expr) -> Result<Json, Refused>.JsonisNull,Bool,Number(f64),String,ArrayandObject(Vec<(String, Json)>)in source order, andto_json_stringwrites it the wayJSON.stringifydoes.null; strings with every escape and line continuation decoded; numbers in every form;true,false,null; templates without substitutions;-and+on a number;!on a number or boolean, for minifiers'!0and!1;undefinedandvoidon a literal, which leave a key out and arenullin an array, as inJSON.stringify; parentheses. A key given twice keeps its first place and its last value.not-a-literal),${…}(substitution), BigInt (bigint, since JSON can't write one), regular expressions, computed keys, methods and accessors, spread,__proto__:(it sets the prototype, not a key),-or!on a non-number, a value that'sundefinedas a whole, and a lone surrogate, which a RustStringcan't hold. The table is onRefused.json(text): a strict JSON parser, asJSON.parsereads it, for__NEXT_DATA__and Nuxt 3's__NUXT_DATA__, whose text callers pass straight in. Nesting pastDEFAULT_MAX_DEPTHistoo-deep. No new dependency: serde_json is only a dev-dependency here, and the parser is 250 lines.page_state(src)parses a script andpage_state_in(ast, src)takes a tree; each gives aFound { name, span, value }for every top-level:=assignment to a property chain fromwindow,selforglobalThis(window.__NUXT__.config,window['__PRELOADED_STATE__']), alone or in a comma list;var,letorconstwith a value;self.__next_f.push(…), whichnext_flight(&found)joins into the Flight payload; the doc comment says what[0],[1, …],[2, …]and[3, …]are.The value may also be
JSON.parseof one string literal, or a function called at once with literal arguments, Nuxt 2's form: each parameter takes its argument's value. The function must be plain (not async or a generator), with plain parameters and a body that only returns. Substitution copies at most 16 bytes per byte of source, so one large argument named 10,000 times istoo-largerather than a gigabyte.fuzz/fuzz_targets/data.rs: any input throughpage_stateandjson. Spans stay inside the input, and every value writes out as JSON that reads back as the same text. It joins the nightly matrix infuzz.yml.tests/pages: seven excerpts of real pages, fetched on 21 November 2026, with a README giving each source. None of the pages states a licence, so each keeps only a few real keys and values. The Nuxt 2 one is Back Market as the Wayback Machine archived it in January 2022, since current sites have moved to Nuxt 3.The bindings don't expose parsec-js yet, so this waits for M16 (#174 to #176) there too.
Tests.
tests/data.rs, 16 tests:json_values_round_trip_through_literals: 17 JSON texts through bothjsonandliteral, written out and read back the same.json_agrees_with_serde_jsonon numbers at the edges (5e-324,2^53 + 1,-0.0).json_refuses_what_json_parse_would: 17 inputs, each with its reason and offset.literal_accepts_what_minified_javascript_writes: 28 accepted forms.literal_refuses_everything_else_where_it_stops: 34 refused forms, each with its reason and the text at its span.page_state_*: each shape, plus 8 sources that look close and aren't page state (window.X += 1,window[k] = 1, an assignment inside anifor a function), the JSON.parse forms, and 18 Nuxt function cases, 12 of them refused.real_pages: every file intests/pages.deep_nesting_is_an_error_not_a_crash: 100,000-deep[istoo-deepfromjsonand from the parser, on a 1 MB thread.wide_input_takes_linear_time: arrays, objects, a key given 80,000 times,JSON.parseand Nuxt substitution all grow linearly.substitution_cannot_blow_up: the 100 KB argument named 10,000 times.Run over all the inline scripts of 88 public pages, outside the repository: no panic. Every
__next_fpush on nextjs.org, vercel.com, moonpay.com, douyu.com and vinted.fr reads, and so does the full 43 KB Back Market__NUXT__. What's refused is code:window.dataLayer = window.dataLayer || [], andnew Date(…)inside Nike's state.cargo +nightly fuzz run data -- -max_total_time=180 -max_len=4096: 5,463,493 runs, no crash.cargo test --workspace,cargo clippy --workspace --all-targets -- -D warnings,cargo fmt --all --check,cargo doc -p parsec-jswith-D warnings,cargo deny check licenses bans sources, the fuzz manifests'cargo metadataandscripts/check-headers.shpass locally.