Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .github/workflows/action-self-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,3 +21,10 @@ jobs:
https://www.axiondeepdigital.com/research/ai-visibility-study
check: both
fail-on: critical
- uses: ./
with:
urls: https://www.axiondeepdigital.com/las-cruces/seo
check: gap
viewport: mobile
headers: |
x-rendergap-self-test: 1
24 changes: 24 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# Changelog

## 0.2.0

Hardening pass: every protection below has a hostile-page test, and each was verified by breaking it on purpose and watching a test fail.

- Bot challenge detection names vendors (Cloudflare, DataDome, Imperva, PerimeterX, AWS WAF, SiteGround, DDoS-Guard, Akamai). reCAPTCHA on a contact form is no longer a false challenge.
- Responses are decoded by their declared charset; bodies over 15 MB are capped.
- When the crawler is blocked, challenged, sent non-HTML or caught in a redirect loop, that is reported on its own instead of a noisy content diff.
- Cookie banners, consent tools, dialogs and chat widgets are excluded from the comparison.
- Text is normalized (quotes, dashes, zero-width characters, Unicode compatibility forms).
- Reads open shadow roots and scrolls to trigger lazy-loaded sections.
- Extraction runs in an isolated JavaScript world.
- Dialogs are dismissed, `debugger;` statements skipped, infinite loops stopped, and every step has a deadline. The CLI has a hard ceiling.
- JavaScript and meta refresh redirects are flagged.
- New: `--viewport mobile`, repeatable `--header`, basic auth from URL credentials (never printed).
- Crawler check: robots.txt fetched as each crawler, crawler-specific `X-Robots-Tag`, one retry on HTTP 429, `rate limited` and `inconclusive` statuses, `--fail-on blocked` counts only real blocks.
- robots.txt: exact product-token matching, percent-encoding normalization, BOM handling, robots.txt always allowed.
- Input validation with clear errors and exit code 2 for usage mistakes.
- Action: `viewport` and `headers` inputs, safe URL trimming, collapsible job summary.

## 0.1.0

First release: render gap and crawler access checks, CLI and GitHub Action.
23 changes: 23 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,8 @@ Options:

- `--agent <name>` the crawler whose raw fetch to compare. Default `gptbot`.
- `--fail-on <level>` exit 1 when a finding is at least `critical` (default), `warning` or `info`. Use `never` to always exit 0.
- `--viewport mobile` render the page as a phone instead of a desktop browser.
- `--header "Name: value"` send an extra header on every request, repeatable. Use it for preview deployments behind a bypass token. Credentials in the URL (`https://user:pass@staging.example.com/`) are sent as basic auth and never printed.
- `--json` machine-readable output.
- `--timeout <ms>` page load timeout.

Expand All @@ -74,6 +76,8 @@ CCBot sent a bot challenge (HTTP 200, 5210 bytes)
Google-Extended allowed (robots.txt token only, not fetched)
```

robots.txt is also fetched as each crawler, because some servers hand AI crawlers a stricter file than browsers get; when that happens the crawler is judged by its own copy and the report says so. Other statuses: `rate limited` (HTTP 429 even after one polite retry), `inconclusive` (a normal browser request was blocked too, so the crawler block proves nothing), and `served with X-Robots-Tag noindex`, which honors crawler-specific headers such as `X-Robots-Tag: GPTBot: noindex`.

Crawlers checked: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, CCBot, Googlebot, Bingbot, plus the robots.txt-only tokens Google-Extended and Applebot-Extended. Use `--only gptbot,claudebot` to narrow it, and `--fail-on blocked` to exit 1 when any of them is blocked.

What it can and cannot tell you: requests come from your machine with each crawler's user agent. Some firewalls verify real crawlers by IP address or reverse DNS, so the real crawler can be treated differently, in either direction. A block here is worth investigating; an "allowed" here is not proof the real crawler gets through. Run it against sites you own or have permission to test.
Expand All @@ -91,6 +95,9 @@ Check preview deployments before they ship:
check: both # gap, crawlers, or both
fail-on: critical # gap check threshold
fail-on-blocked: false
viewport: desktop # or mobile
headers: | # optional, one per line
x-vercel-protection-bypass: ${{ secrets.VERCEL_BYPASS }}
```

Results are printed to the log and to the job summary.
Expand All @@ -101,6 +108,22 @@ Results are printed to the log and to the job summary.
- **Rendered side:** a normal page load with a desktop browser user agent, waiting for the network to go quiet, then one more second.
- **Extraction:** one function reads both sides: title, meta tags, canonical, headings, JSON-LD types, phone numbers (tel links and visible text), email links, prices, FAQ items, forms, word count and text blocks.

## What it is built to survive

Real pages fight tools. Each of these has a test in `test/hostile.test.js`, and each protection is checked by deliberately breaking it and confirming a test fails.

- Bot challenge pages (Cloudflare, DataDome, Imperva, PerimeterX, AWS WAF and others) are reported as blocks. A contact form that merely loads reCAPTCHA is not.
- Pages in legacy encodings such as windows-1252 are decoded from the header or the meta tag, so accented text does not show up as a false gap.
- Cookie banners, consent tools, modal dialogs and chat widgets are left out of the comparison, including banners built only from utility classes. A bakery's "cookies" section still counts.
- Content inside web components (open shadow roots) and sections that load on scroll are read.
- Curly quotes, dashes, zero-width characters and odd whitespace compare equal to their plain forms.
- The extractor runs in an isolated JavaScript world, so a page that overwrites `JSON`, `Set`, `Array.from` or `querySelectorAll` cannot break or fool it.
- `alert()` and `confirm()` are dismissed, `debugger;` statements are skipped, and a page stuck in an infinite loop is stopped and read as it stood.
- Pages that never go network-idle are read as loaded. Every in-page step has a deadline and the CLI has a hard ceiling, so a CI job cannot hang.
- When the crawler is blocked, served a challenge, sent a PDF or caught in a redirect loop, that is the finding. You do not get a page of false "missing content" noise.
- JavaScript and meta refresh redirects are flagged, since the crawler and the browser are then reading different pages.
- robots.txt follows RFC 9309: exact product-token matching, longest match wins, `*` and `$` patterns, percent-encoding, byte order marks, and "server error means disallow all".

Findings are about what is present, not whether a crawler ranks or cites you. A clean report means your content is in the first HTML response. It does not mean an AI system will recommend you.

## Contributing
Expand Down
30 changes: 23 additions & 7 deletions action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,14 @@ inputs:
description: "Crawler check: fail when any crawler is blocked (true or false)"
required: false
default: "false"
viewport:
description: "desktop or mobile for the rendered side of the gap check"
required: false
default: "desktop"
headers:
description: "Extra request headers, one 'Name: value' per line (for example a preview bypass token from secrets)"
required: false
default: ""

runs:
using: "composite"
Expand All @@ -46,24 +54,32 @@ runs:
RG_AGENT: ${{ inputs.agent }}
RG_FAIL_ON: ${{ inputs.fail-on }}
RG_FAIL_BLOCKED: ${{ inputs.fail-on-blocked }}
RG_VIEWPORT: ${{ inputs.viewport }}
RG_HEADERS: ${{ inputs.headers }}
RG_BIN: ${{ github.action_path }}/bin/rendergap.js
run: |
if [ -x /usr/bin/google-chrome ]; then export CHROME_PATH=/usr/bin/google-chrome; fi
export NO_COLOR=1
trim() { local s="$1"; s="${s#"${s%%[![:space:]]*}"}"; s="${s%"${s##*[![:space:]]}"}"; printf '%s' "$s"; }
hdr=()
while IFS= read -r h; do
h="$(trim "$h")"; [ -n "$h" ] && hdr+=(--header "$h")
done <<< "$RG_HEADERS"
summary() { { echo "<details open><summary>$1</summary>"; echo; echo '````text'; echo "$2"; echo '````'; echo "</details>"; } >> "$GITHUB_STEP_SUMMARY"; }
status=0
while IFS= read -r url; do
url="$(echo "$url" | xargs)"
url="$(trim "$url")"
[ -z "$url" ] && continue
if [ "$RG_CHECK" = "gap" ] || [ "$RG_CHECK" = "both" ]; then
out="$(node "$RG_BIN" "$url" --agent "$RG_AGENT" --fail-on "$RG_FAIL_ON")" || status=1
echo "$out"
{ echo '```'; echo "$out"; echo '```'; } >> "$GITHUB_STEP_SUMMARY"
code=0; out="$(node "$RG_BIN" "$url" --agent "$RG_AGENT" --fail-on "$RG_FAIL_ON" --viewport "$RG_VIEWPORT" "${hdr[@]}" 2>&1)" || code=$?
echo "$out"; summary "render gap: $url" "$out"
if [ "$code" -ne 0 ]; then status=1; fi
fi
if [ "$RG_CHECK" = "crawlers" ] || [ "$RG_CHECK" = "both" ]; then
flag="never"; [ "$RG_FAIL_BLOCKED" = "true" ] && flag="blocked"
out="$(node "$RG_BIN" crawlers "$url" --fail-on "$flag")" || status=1
echo "$out"
{ echo '```'; echo "$out"; echo '```'; } >> "$GITHUB_STEP_SUMMARY"
code=0; out="$(node "$RG_BIN" crawlers "$url" --fail-on "$flag" "${hdr[@]}" 2>&1)" || code=$?
echo "$out"; summary "crawler access: $url" "$out"
if [ "$code" -ne 0 ]; then status=1; fi
fi
done <<< "$RG_URLS"
exit $status
61 changes: 48 additions & 13 deletions bin/rendergap.js
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
import { parseArgs } from "node:util";
import { readFileSync } from "node:fs";
import { AGENTS, findAgent } from "../src/agents.js";
import { checkCrawlers } from "../src/crawlers.js";
import { checkCrawlers, BLOCKED } from "../src/crawlers.js";
import { meetsThreshold } from "../src/diff.js";
import { formatGap, formatCrawlers } from "../src/report.js";

Expand All @@ -17,32 +17,47 @@ Render gap options
--fail-on <level> exit 1 if a finding is at least: critical | warning | info | never
(default: critical)
--timeout <ms> page load timeout (default: 30000)
--viewport <v> desktop or mobile for the rendered side (default: desktop)

Crawler check options
--only <a,b> check only these crawlers (ids or names)
--fail-on <level> exit 1 when any checked crawler is blocked: blocked | never
(default: never)

Both
--header "K: V" extra request header, repeatable (e.g. a preview bypass token)
--json print JSON instead of text
-h, --help show this help
-v, --version show the version

Crawlers: ${AGENTS.map((a) => a.id).join(", ")}
Set CHROME_PATH to use an installed Chrome instead of the bundled one.`;

const { values, positionals } = parseArgs({
function fail(msg) {
console.error(`rendergap: ${msg}`);
process.exit(2);
}

let parsed;
try {
parsed = parseArgs({
allowPositionals: true,
options: {
agent: { type: "string", default: "gptbot" },
"fail-on": { type: "string" },
timeout: { type: "string", default: "30000" },
viewport: { type: "string", default: "desktop" },
header: { type: "string", multiple: true, default: [] },
only: { type: "string" },
json: { type: "boolean", default: false },
help: { type: "boolean", short: "h", default: false },
version: { type: "boolean", short: "v", default: false },
},
});
});
} catch (err) {
fail(`${err.message}\nRun rendergap --help for usage.`);
}
const { values, positionals } = parsed;

if (values.version) {
const pkg = JSON.parse(readFileSync(new URL("../package.json", import.meta.url)));
Expand All @@ -59,13 +74,30 @@ if (values.help || !target) {

let url;
try {
url = new URL(/^https?:\/\//i.test(target) ? target : `https://${target}`).href;
const u = new URL(/^[a-z][a-z0-9+.-]*:\/\//i.test(target) ? target : `https://${target}`);
if (!/^https?:$/.test(u.protocol)) throw new Error();
url = u.href;
} catch {
console.error(`Not a valid URL: ${target}`);
process.exit(2);
fail(`not a valid http(s) URL: ${target}`);
}

const timeoutMs = Number(values.timeout) || 30000;
const timeoutMs = Number(values.timeout);
if (!Number.isFinite(timeoutMs) || timeoutMs < 1000) fail(`--timeout must be a number of milliseconds, at least 1000`);
if (!["desktop", "mobile"].includes(values.viewport)) fail(`--viewport must be desktop or mobile`);

const headers = {};
for (const h of values.header) {
const i = h.indexOf(":");
if (i < 1) fail(`--header must look like "Name: value", got: ${h}`);
headers[h.slice(0, i).trim()] = h.slice(i + 1).trim();
}

// A hard ceiling so a stuck page or browser can never hang a CI job.
const ceiling = Math.max(timeoutMs * 6, 180000);
setTimeout(() => {
console.error(`rendergap: gave up after ${Math.round(ceiling / 1000)} s`);
process.exit(2);
}, ceiling).unref();

try {
if (isCrawlers) {
Expand All @@ -77,17 +109,20 @@ try {
return a;
});
}
const result = await checkCrawlers(url, { agents, timeoutMs });
console.log(values.json ? JSON.stringify(result, null, 2) : formatCrawlers(result));
const failOn = values["fail-on"] || "never";
if (failOn === "blocked" && result.results.some((r) => r.status !== "allowed")) process.exit(1);
if (!["blocked", "never"].includes(failOn)) fail(`for crawlers, --fail-on must be blocked or never`);
const result = await checkCrawlers(url, { agents, timeoutMs, headers });
console.log(values.json ? JSON.stringify(result, null, 2) : formatCrawlers(result));
if (failOn === "blocked" && result.results.some((r) => BLOCKED.has(r.status))) process.exit(1);
} else {
const failOn = values["fail-on"] || "critical";
if (!["critical", "warning", "info", "never"].includes(failOn)) fail(`--fail-on must be critical, warning, info or never`);
const agent = findAgent(values.agent);
if (!agent || agent.fetchable === false) throw new Error(`Not a fetchable crawler: ${values.agent}`);
if (!agent || agent.fetchable === false) fail(`not a fetchable crawler: ${values.agent}`);
const { renderGap } = await import("../src/render.js");
const result = await renderGap(url, { agent, timeoutMs });
const result = await renderGap(url, { agent, timeoutMs, viewport: values.viewport, headers });
console.log(values.json ? JSON.stringify(result, null, 2) : formatGap(result));
if (meetsThreshold(result.findings, values["fail-on"] || "critical")) process.exit(1);
if (meetsThreshold(result.findings, failOn)) process.exit(1);
}
} catch (err) {
console.error(`rendergap: ${err.message}`);
Expand Down
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "rendergap",
"version": "0.1.0",
"version": "0.2.0",
"description": "See what AI crawlers miss on your page: compare raw HTML with the rendered page, and check which crawlers can reach it at all.",
"type": "module",
"bin": {
Expand Down
5 changes: 4 additions & 1 deletion src/agents.js
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,9 @@
export const BROWSER_UA =
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36";

export const MOBILE_UA =
"Mozilla/5.0 (Linux; Android 14; Pixel 8) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Mobile Safari/537.36";

export const AGENTS = [
{
id: "gptbot",
Expand Down Expand Up @@ -92,6 +95,6 @@ export const AGENTS = [
];

export function findAgent(idOrToken) {
const k = String(idOrToken).toLowerCase();
const k = String(idOrToken || "").trim().toLowerCase();
return AGENTS.find((a) => a.id === k || a.token.toLowerCase() === k);
}
Loading
Loading