Skip to content

About

Pre-registered methodology and dataset: when websites serve different content to AI crawlers, Googlebot, and normal browsers, what changes, how often, and does it matter?

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

3 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

AI Crawler Diff

Pre-registered methodology and dataset comparing what 8 request profiles receive from the same URLs:

When websites serve different content to AI crawlers, Googlebot, and normal browsers, what changes, how often does it happen, and does it matter?

Status: Draft methodology (v0.4). Not yet locked, no fetches run, no findings.

The full methodology is in methodology.md. Read it first.

Repository contents

Path What
methodology.md Pre-registration document. Versioned; v1 is the lock.
src/profiles.js The 8 request profiles (UAs, headers, render mode).
src/extract.js Metric extractors per methodology §6.
src/fetch-one.js Single (URL × profile) fetcher. CLI entry point.
src/smoke-test.js Heavier smoke test: 5 URLs × 8 profiles.
src/run-all.js Full run orchestrator (TODO).

Pre-registration commitments

These are also enumerated in §10 of the methodology and are non-negotiable:

  1. URL list is published verbatim before any fetch runs
  2. Full raw dataset is published (CSV + JSON, all 100×8 records)
  3. Fetch code is published (this repo, MIT license)
  4. Negative results are published as the headline if the data supports it
  5. Methodology is date-locked; deviations are dated addenda
  6. No paid amplification with framing that diverges from findings
  7. Conflicts of interest disclosed (Axion Deep Digital sells SEO services)

Usage (local)

npm install
npx playwright install chromium
node src/fetch-one.js https://example.com gptbot

Usage (Oracle Cloud fetcher box)

The fetcher runs on a single static-IP cloud instance for the locked run window. Setup, IP, and operational notes live in the project's private memory; the IP itself is recorded in the methodology at lock time.

License

  • Methodology and dataset: CC BY 4.0
  • Fetch code: MIT

About

Pre-registered methodology and dataset: when websites serve different content to AI crawlers, Googlebot, and normal browsers, what changes, how often, and does it matter?

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages