Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 5 additions & 4 deletions .github/workflows/vouch.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,14 +27,15 @@ jobs:
run: |
author_lc="$(printf '%s' "$PR_AUTHOR" | tr '[:upper:]' '[:lower:]')"
echo "Checking contributor: ${author_lc}"
if ! grep -v '^[[:space:]]*#' .github/VOUCHED.td | sed 's/[[:space:]]//g' | grep -Fxq "$author_lc"; then
echo "::error::GitHub user '${PR_AUTHOR}' is not yet listed in .github/VOUCHED.td. A maintainer will review your PR, verify robots.txt/test compliance, and vouch your handle."
if ! git show "${BASE_SHA}:.github/VOUCHED.td" | grep -v '^[[:space:]]*#' | sed 's/[[:space:]]//g' | grep -Fxq "$author_lc"; then
echo "::error::GitHub user '${PR_AUTHOR}' is not yet listed in .github/VOUCHED.td on the base branch. A maintainer will review your PR, verify robots.txt/test compliance, and vouch your handle."
exit 1
fi
if [ "$author_lc" != "bartrumb" ] && [ "$author_lc" != "dependabot[bot]" ] && [ "$author_lc" != "github-actions[bot]" ]; then
missing_dco="$(git log --no-merges --format='%H %B' "${BASE_SHA}..${HEAD_SHA}" | awk '/^[0-9a-f]{40} /{sha=$1; ok=0} /^Signed-off-by: .+ <.+@.+>/{ok=1} END{if (sha != "" && !ok) print sha}')"
missing_dco="$(git log --no-merges --format='%H %B' "${BASE_SHA}..${HEAD_SHA}" | awk '/^[0-9a-f]{40} /{if (sha != "" && !ok) print sha; sha=$1; ok=0} /^Signed-off-by: .+ <.+@.+>/{ok=1} END{if (sha != "" && !ok) print sha}')"
if [ -n "$missing_dco" ]; then
echo "::error::External contributor commits must include a DCO sign-off ('git commit -s'). Missing on: ${missing_dco}"
missing_flat="$(printf '%s' "$missing_dco" | tr '\n' ' ' | sed 's/[[:space:]]*$//')"
echo "::error::External contributor commits must include a DCO sign-off ('git commit -s'). Missing on: ${missing_flat}"
exit 1
fi
fi
Expand Down
34 changes: 34 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,40 @@

All notable changes to EdgeFetch are documented in this file. This project adheres to [Semantic Versioning](https://semver.org/).

## [2.6.1] - 2026-11-21

### Security
- **Eliminated residual superlinear scanner & regex paths (`R4-H1`, `R4-H2`, `R4-H3`):**
- Cached monotonic `nextNameBoundary` offsets inside `tryParseTagAt` (`hub/src/safe-input.ts`) so repeated `<x` pseudo-tags finish in linear $O(N)$ time (`< 5 ms` at 16K chars).
- Anchored HTML attribute scanning with a `64 KiB` tag cap (`parseTagAttributes`), eliminated V8 ConsString rope flattening in `sanitizeAndResolveMarkdownProseLinksLinear`, cached `>` offsets per line in `extractMarkdownLinksAndImages`, and fixed an unanchored `.myshopify.com` detector pattern (`hub/src/scrape.ts`, `hub/src/extractors/providers/commerce-social/shopify.ts`).
- Hardened `compileSafeUserRegex` (`hub/src/safe-input.ts`) so non-terminal optional/branch groups containing unbounded quantifiers (e.g. `a*(b)?aa*(b)?aa*$`) cannot reset ambiguity tracking unless followed by a disjoint mandatory barrier or true pattern end.
- **Markdown sanitization & inline-code stash preservation (`R4-M6`, `R4-M7`):**
- Honored CommonMark backslash-escaped backticks (`\``) during inline code-span scanning so `` \`<img src=x onerror=...>` `` cannot smuggle raw HTML past tag stripping.
- Hardened `hasDangerousUriScheme` against entity-encoded schemes (`javascript&colon;...`, numeric entities, and undecoded `&...;` before `/`/`?`/`#`) and stripped dangerous URI schemes on multi-line `[label\nlabel2](...)` links (`hub/src/scrape.ts`).
- Preserved internal `\u0000EF...\u0000` inline-code stash sentinels across NUL-stripping passes (`hub/src/safe-input.ts`) so inline backtick spans survive `<svg>` removal (`SCRAPE_PIPELINE_VERSION = 28`).
- **Go worker, Chromium header scope, Portable Hub & Vouch Gate hardening (`R4-M*`, `D-L1`..`D-L4`):**
- Scoped CDP `Fetch.requestPaused` caller headers to exact `(scheme, canonicalHost, effectivePort)`, restricted Linux `--no-sandbox` fallback strictly to Chromium sandbox-init diagnostics (`isSandboxStartupFailure`), bounded detached `/robots.txt` contexts + partial `1 MiB` trailing lines, and drained `runSession` worker goroutines unconditionally (`internal/browser`, `internal/crawl`, `internal/agent`, `internal/politeness`).
- Hardened Portable Node/Deno Hub against `Host` header overrides in `safe-fetch.ts`, `1xx` response loops, unbounded origin streams (`32 MiB`), slowloris `POST` drips (`requestTimeout = 30s`), unmasked RFC 6455 client WebSocket frames (`1002`), and unvalidated `X-Forwarded-Host` issuer spoofing (`hub/portable/*`).
- Fixed `.github/workflows/vouch.yml` to read `.github/VOUCHED.td` from `${BASE_SHA}` (preventing self-vouching in PR branches) and check DCO `Signed-off-by:` trailers across every non-merge commit in a multi-commit PR.
- Removed destructive global `deduplicateClonedProseParagraphs` (`D-M1`, keeping scoped DOM carousel-clone class stripping), capped `salvageAshbyBoardFromTruncatedJson` at `5,000` jobs + verify concurrency `2` (`D-L1`), required `status === 200` on truncated ATS formatters (`D-L2`), restored `1h` OAuth access TTL alongside `90d` refresh TTL (`D-L3`), aligned MCP `capabilities.tools` (`D-L4`), and escaped `/admin` inline `<script>` backslashes/newlines.

### Fixed
- **ATS `jobs` search precision, Workday domain discovery & `similarOpenRoles` ranking (`hub/src/jobs.ts`, `hub/src/mcp.ts`):**
- Replaced any-token `query` matching with phrase-first (`matchedOn: "phrase"`) and all-tokens (`matchedOn: "all_tokens"`) relevance filtering, ranking exact phrase matches first.
- Added `minBaseMode: "floor" | "ceiling"` (default `"floor"`, checking `pay.min >= minBase`) and `strictPay` (default `false`, surfacing unlisted-pay roles with `payUnknown: true` and `payUnknownCount`).
- Separated geographic `locations` filters from `"Remote"` workplace eligibility so `locations: ["United States", "Remote"]` enforces US geography instead of matching worldwide remote postings.
- Added Workday CXS tenant/site probes (`wd5`/`wd1`) and search-assisted ATS discovery to `domains`, reported explicit `boardStatus: "not_found"` + `tried` when a domain has no ATS board, prevented cross-company bleed when `domains` or `company` is set, made `includePublicFeeds` strictly opt-in (`default: false`) with `sourcesUsed` telemetry, and restricted `verify` `similarOpenRoles` to distinctive title-token matches.

## [2.6.0] - 2026-11-20

### Added
- **Full-store `catalog` sync (`POST /v1/catalog` & MCP `catalog` tool):** Polite (`>= 2s` + jitter) whole-catalog pagination for Shopify (`/products.json` + custom-domain `.js` inventory enrichment) and WooCommerce Store API (`/wp-json/wc/store/v1/products`), NDJSON R2 snapshot streaming, `shrink` projection, `sinceArtifactId` diffing, and UCP `robots.txt` discovery (`SCRAPE_PIPELINE_VERSION = 26`..`27`).
- **ATS `jobs` reliability, `verify` mode & 16 MiB board support:** Added batch job-liveness `mode: "verify"` (`live`, `closed` + `similarOpenRoles`, `board_gone`, `unknown`), domain-to-ATS auto-discovery, `iCIMS` (`?ss=1&in_iframe=1`) and Workday (`onsite` + `?offset=` paging) enhancements, `16 MiB` Ashby board support with linear truncated-JSON salvaging, and Go `KindText` `Accept` allowlist alignment.
- **RFC 9309 full compliance, 3-strikes `429` backoff & 64 KiB linear framework detection:** Added 5-hop `/robots.txt` redirect handling, `Cache-Control: max-age` clamping (`[60s, 24h]`), `robots_unreachable` classification, `ROBOTS_BYPASS_HOSTS` operator override, 3-attempt adaptive `429` backoff, `detectSiteFrameworkLinear`, Inertia `data-page` SSR hydration unlock, fault-isolated `runAdapterSyncInBulkhead`, Chromium native OS sandbox probing, 4-way multi-domain agent concurrency, and `.github/VOUCHED.td` contributor gate.

### Security
- **Round 3 security re-audit fixes (`R3-H1`..`R3-H3`, `R3-M1`..`R3-M10`):** Replaced quadratic HTML/Markdown/provider regexes with linear scanners, prevented split-tag `<im<form></form>g>` reassembly, rejected stacked quantifiers in `compileSafeUserRegex`, pinned dial-time DNS resolutions in `safe-fetch.ts`, sanitized `origin-markdown` HTML/WAF/prompt-injection vectors, scoped CDP custom headers via `Fetch.requestPaused` + `taintedByHeaders` browser recycling, pinned `O_NOFOLLOW` descriptor `fchmod`/`fchown` in `internal/priv`, and tag-locked GitHub `release` environment secrets (`v*`).

## [2.5.5] - 2026-11-20

### Added
Expand Down
2 changes: 1 addition & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
# and one-shot CLI (`docker run --rm ghcr.io/bartrumb/edgefetch-agent:lite scrape https://example.com`).
FROM --platform=$BUILDPLATFORM golang:1.27-alpine@sha256:70bd7ddea8b6cbe5d6943cde279d6b67feb10cc924fb9100d1d8757c1575b280 AS build
ARG TARGETARCH
ARG BUILD_VERSION=2.6.0
ARG BUILD_VERSION=2.6.1
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download && go mod verify
Expand Down
2 changes: 1 addition & 1 deletion Dockerfile.agent
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
# Assistant OS required. Bundles static Go HTTP/2 binary + SSRF-proxied Headless Chromium + SwiftShader + Poppler PDF.
FROM --platform=$BUILDPLATFORM golang:1.27-alpine@sha256:70bd7ddea8b6cbe5d6943cde279d6b67feb10cc924fb9100d1d8757c1575b280 AS build
ARG TARGETARCH
ARG BUILD_VERSION=2.6.0
ARG BUILD_VERSION=2.6.1
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download && go mod verify
Expand Down
2 changes: 1 addition & 1 deletion Dockerfile.all-in-one
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
# together with a local Go `edgefetch` worker + Headless Chromium + Poppler PDF — zero Postgres, zero Redis!
FROM --platform=$BUILDPLATFORM golang:1.27-alpine@sha256:70bd7ddea8b6cbe5d6943cde279d6b67feb10cc924fb9100d1d8757c1575b280 AS go-build
ARG TARGETARCH
ARG BUILD_VERSION=2.6.0
ARG BUILD_VERSION=2.6.1
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download && go mod verify
Expand Down
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
.PHONY: build build-wasm test race fuzz bench bench-extract doctor hub-test up down fmt vet

VERSION ?= 2.6.0
VERSION ?= 2.6.1
LDFLAGS := -s -w -X github.com/bartrumb/edgefetch/v2/internal/agent.Version=$(VERSION)

build:
Expand Down
2 changes: 1 addition & 1 deletion deno.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@bartrumb/edgefetch-deno-hub",
"version": "2.6.0",
"version": "2.6.1",
"tasks": {
"start": "deno run --allow-net --allow-env hub/portable/deno-server.ts"
},
Expand Down
2 changes: 1 addition & 1 deletion edgefetch_fat/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
# Every Chromium subresource socket is forced through EdgeFetch's in-process loopback `SSRFProxy` (`MAP * ~NOTFOUND`).
FROM --platform=$BUILDPLATFORM golang:1.27-alpine@sha256:70bd7ddea8b6cbe5d6943cde279d6b67feb10cc924fb9100d1d8757c1575b280 AS build
ARG TARGETARCH
ARG BUILD_VERSION=2.6.0
ARG BUILD_VERSION=2.6.1
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download && go mod verify
Expand Down
2 changes: 1 addition & 1 deletion edgefetch_fat/config.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
name: "EdgeFetch"
version: "2.6.0"
version: "2.6.1"
slug: "edgefetch_fat"
image: "ghcr.io/bartrumb/edgefetch-fat-addon-{arch}"
description: "Outbound-only Go HTTP/2 + on-demand Headless Chromium JS rendering + streaming Poppler PDF worker for Cloudflare Hibernating WebSocket Durable Objects (zero inbound ports, loopback SSRF CONNECT proxy, AppArmor confined)."
Expand Down
2 changes: 1 addition & 1 deletion extensions/browser-clipper/manifest.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"manifest_version": 3,
"name": "EdgeFetch Tab Clipper & Browser Node",
"version": "2.6.0",
"version": "2.6.1",
"description": "Clip your active browser tab as clean GFM Markdown into your private EdgeFetch FTS5 search index, or optionally contribute idle egress as an EdgeFetch node.",
"permissions": [
"activeTab",
Expand Down
2 changes: 1 addition & 1 deletion extensions/browser-clipper/offscreen.js
Original file line number Diff line number Diff line change
Expand Up @@ -261,7 +261,7 @@ function connect(cfg) {
ws.send(
JSON.stringify({
type: 'hello',
version: '2.6.0',
version: '2.6.1',
arch: 'browser-mv3',
goVersion: 'mv3-extension',
capabilities: [],
Expand Down
2 changes: 1 addition & 1 deletion hub/package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "edgefetch-hub",
"version": "2.6.0",
"version": "2.6.1",
"private": true,
"description": "Cloudflare Hibernating WebSocket Durable Object coordinator, /admin Control Center, and Self-Hosted OAuth 2.1 /mcp server for EdgeFetch.",
"scripts": {
Expand Down
93 changes: 70 additions & 23 deletions hub/portable/deno-server.ts
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,7 @@ function createDenoSafeFetch(selfHost?: string): typeof fetch {
await assertDenoResolvedPublicIp(check.url.hostname);

const hopHeaders = new Headers(baseHeaders);
hopHeaders.delete('host');
if (check.url.origin !== initialOrigin) {
hopHeaders.delete('authorization');
hopHeaders.delete('cookie');
Expand All @@ -134,8 +135,9 @@ function createDenoSafeFetch(selfHost?: string): typeof fetch {
...init,
method,
headers: hopHeaders,
body: method === 'GET' || method === 'HEAD' ? undefined : body,
body: method === 'GET' || method === 'HEAD' || method === 'OPTIONS' ? undefined : body,
redirect: 'manual',
signal: init?.signal ? AbortSignal.any([init.signal, AbortSignal.timeout(30_000)]) : AbortSignal.timeout(30_000),
});

if (res.status >= 300 && res.status < 400) {
Expand Down Expand Up @@ -207,35 +209,79 @@ function resolveDenoTrustedClientIp(
return sockCandidate && net.isIP(sockCandidate) !== 0 ? sockCandidate : '127.0.0.1';
}

const STRICT_SINGLE_HOST_RE =
/^[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?(?:\.[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?)*(?::([1-9]\d{0,4}))?$/;

export function parseStrictSingleHost(raw: string | undefined | null): string | null {
if (typeof raw !== 'string') return null;
if (raw.includes(',') || raw.includes('..') || raw.includes('[') || raw.includes(']')) return null;
const trimmed = raw.trim();
if (!trimmed || trimmed.length > 255 || /\s/.test(trimmed)) return null;
const lower = trimmed.toLowerCase();
const m = STRICT_SINGLE_HOST_RE.exec(lower);
if (!m) return null;
if (m[1] !== undefined) {
const portNum = Number(m[1]);
if (!Number.isInteger(portNum) || portNum < 1 || portNum > 65535) return null;
}
return lower;
}

export function resolvePublicOrigin(params: {
canonicalOrigin: string;
trustProxy: boolean;
hasExplicitPublicOrigin: boolean;
hostHeader?: string | null;
forwardedHostHeader?: string | null;
forwardedProtoHeader?: string | null;
}): { origin: string; trustedForwardedHost: string | null } {
const baseOriginUrl = new URL(params.canonicalOrigin);
if (params.hasExplicitPublicOrigin || !params.trustProxy) {
return { origin: baseOriginUrl.origin, trustedForwardedHost: null };
}

let proto = baseOriginUrl.protocol.replace(/:$/, '');
const protoTokens = (params.forwardedProtoHeader ?? '')
.split(',')
.map((p) => p.trim().toLowerCase())
.filter(Boolean);
const rightmostProto = protoTokens.length > 0 ? protoTokens[protoTokens.length - 1] : undefined;
if (rightmostProto === 'https' || rightmostProto === 'http') {
proto = rightmostProto;
}

const validHost = parseStrictSingleHost(params.hostHeader);
const validFwdHost = parseStrictSingleHost(params.forwardedHostHeader);
const authority = validHost ?? baseOriginUrl.host;
const trustedForwardedHost = validFwdHost !== null && validHost !== null && validFwdHost === validHost ? validFwdHost : null;

return {
origin: new URL(`${proto}://${authority}`).origin,
trustedForwardedHost,
};
}

function sanitizeDenoInboundRequest(
req: Request,
info: { remoteAddr?: { hostname?: string } } | undefined,
trustProxy: boolean,
canonicalOrigin: string,
hasExplicitPublicOrigin: boolean,
): Request {
const baseOriginUrl = new URL(canonicalOrigin);
let targetOrigin = baseOriginUrl.origin;

if (trustProxy && !hasExplicitPublicOrigin) {
let proto = baseOriginUrl.protocol.replace(/:$/, '');
let authority = baseOriginUrl.host;
const fwdProto = req.headers.get('x-forwarded-proto')?.split(',')[0]?.trim().toLowerCase();
if (fwdProto === 'https' || fwdProto === 'http') proto = fwdProto;
const fwdHost = req.headers.get('x-forwarded-host')?.split(',')[0]?.trim();
if (fwdHost && /^[A-Za-z0-9](?:[A-Za-z0-9._-]*[A-Za-z0-9])?(?::\d{1,5})?$/.test(fwdHost) && !fwdHost.includes('..')) {
try {
const parsedFwd = new URL(`http://${fwdHost}/`);
const pNum = parsedFwd.port ? Number(parsedFwd.port) : 80;
if (parsedFwd.hostname && pNum >= 1 && pNum <= 65535 && !fwdHost.endsWith(':0')) {
authority = parsedFwd.host;
}
} catch {
// ignore invalid X-Forwarded-Host
}
}
targetOrigin = `${proto}://${authority}`;
let fallbackReqHost: string | null = null;
try {
fallbackReqHost = new URL(req.url).host;
} catch {
fallbackReqHost = null;
}
const { origin: targetOrigin, trustedForwardedHost } = resolvePublicOrigin({
canonicalOrigin,
trustProxy,
hasExplicitPublicOrigin,
hostHeader: req.headers.get('host') ?? fallbackReqHost,
forwardedHostHeader: req.headers.get('x-forwarded-host'),
forwardedProtoHeader: req.headers.get('x-forwarded-proto'),
});

// Extract raw path + query from `req.url` without re-decoding malformed `%` sequences before `hubWorker` handles them
const rawUrl = req.url;
Expand All @@ -251,13 +297,14 @@ function sanitizeDenoInboundRequest(
// `R3-M9`: Strip client-supplied `cf-connecting-ip` / `x-real-ip`, and strip `x-forwarded-*` unless `EDGEFETCH_TRUST_PROXY=1`.
if (lower === 'cf-connecting-ip' || lower === 'x-real-ip') return;
if (!trustProxy && lower.startsWith('x-forwarded-')) return;
if (lower === 'x-forwarded-host' && trustedForwardedHost === null) return;
headers.append(key, val);
});
headers.set('cf-connecting-ip', trustedIp);
headers.set('x-real-ip', trustedIp);

const method = req.method.toUpperCase();
const hasBody = method !== 'GET' && method !== 'HEAD' && req.body !== null;
const hasBody = method !== 'GET' && method !== 'HEAD' && method !== 'OPTIONS' && req.body !== null;
return new Request(cleanUrl, {
method,
headers,
Expand Down
4 changes: 2 additions & 2 deletions hub/portable/node-adapters.ts
Original file line number Diff line number Diff line change
Expand Up @@ -460,8 +460,8 @@ export class RawServerWebSocket implements HubSocketLike {
this.close(1002, 'rsv_bits_not_allowed');
return;
}
// RFC 6455 §5.1: Server MUST close with 1002 if a client sends an unmasked frame
if (!masked) {
// RFC 6455 §5.1: Client-to-server frames MUST have MASK bit set (`(b1 & 0x80) !== 0`); immediately close/destroy with 1002 if unset!
if ((b1 & 0x80) === 0 || !masked) {
this.close(1002, 'unmasked_client_frame');
return;
}
Expand Down
Loading