Pre-registered methodology and dataset comparing what 8 request profiles receive from the same URLs:
When websites serve different content to AI crawlers, Googlebot, and normal browsers, what changes, how often does it happen, and does it matter?
Status: Draft methodology (v0.4). Not yet locked, no fetches run, no findings.
The full methodology is in methodology.md. Read it first.
| Path | What |
|---|---|
methodology.md |
Pre-registration document. Versioned; v1 is the lock. |
src/profiles.js |
The 8 request profiles (UAs, headers, render mode). |
src/extract.js |
Metric extractors per methodology §6. |
src/fetch-one.js |
Single (URL × profile) fetcher. CLI entry point. |
src/smoke-test.js |
Heavier smoke test: 5 URLs × 8 profiles. |
src/run-all.js |
Full run orchestrator (TODO). |
These are also enumerated in §10 of the methodology and are non-negotiable:
- URL list is published verbatim before any fetch runs
- Full raw dataset is published (CSV + JSON, all 100×8 records)
- Fetch code is published (this repo, MIT license)
- Negative results are published as the headline if the data supports it
- Methodology is date-locked; deviations are dated addenda
- No paid amplification with framing that diverges from findings
- Conflicts of interest disclosed (Axion Deep Digital sells SEO services)
npm install
npx playwright install chromium
node src/fetch-one.js https://example.com gptbotThe fetcher runs on a single static-IP cloud instance for the locked run window. Setup, IP, and operational notes live in the project's private memory; the IP itself is recorded in the methodology at lock time.
- Methodology and dataset: CC BY 4.0
- Fetch code: MIT