Skip to content

Security: bartrumb/edgefetch

SECURITY.md

Security Policy & Threat Model

EdgeFetch is designed to run inside private networks (such as Home Assistant OS servers, homelabs, or branch offices) with zero inbound ports and layered SSRF/privilege controls. Because browser-enabled (browser-v1) and PDF-enabled (pdf-v1) worker nodes execute untrusted web JavaScript and C/C++ document parsers on your hardware, operators running browser/PDF nodes on home networks should also place those nodes on an isolated VLAN or guest subnet (and a dedicated standard non-admin account on Windows/macOS) so a hypothetical zero-day browser renderer escape cannot reach LAN devices.

Threat Model & Defense-in-Depth Controls

1. Zero Inbound Attack Surface

  • Outbound-only sockets in agent mode: edgefetch --mode=agent never calls net.Listen. Router port-forwarding, UPnP, and VPN/tunnel daemons are unnecessary.
  • No generic proxy protocol: EdgeFetch is not an HTTP CONNECT or SOCKS5 relay. It accepts structured JSON FetchCall dispatches over a single authenticated TLS 1.2/1.3 WebSocket and executes bounded HTTP GET (or bounded UTF-8 JSON/form POST up to 64 KiB for public APIs such as Workday CXS job boards) against robots.txt-permitted public https://:443 URLs only.
  • Tokenless Local Loopback Hardening (desktop / serve): When bound to 127.0.0.1 / ::1 without a caller token, EdgeFetch enforces strict DNS-rebinding and browser CSRF defenses: Host must resolve to 127.0.0.1, [::1], or localhost on the active port, Sec-Fetch-Site rejects cross-site, Origin (when present) must match loopback, and state-changing POST endpoints require Content-Type: application/json.

2. Least-Privilege Token Separation (CALLER_TOKEN, AGENT_TOKENS, & EDGEFETCH_SERVER_TOKEN)

  • Caller Token (CALLER_TOKEN): Authorizes upstream workflows to submit POST /fetch batches and view GET /agents on the Hub. Never place CALLER_TOKEN on standby nodes.
  • Standalone Server Token (EDGEFETCH_SERVER_TOKEN): In --mode=server, edgefetch refuses to bind non-loopback addresses (:8080, 0.0.0.0:8080) unless EDGEFETCH_SERVER_TOKEN (>= 24 random chars, verified in constant time via SHA-256 + subtle.ConstantTimeCompare on POST /fetch) is configured, or --allow-unauthenticated (EDGEFETCH_ALLOW_UNAUTHENTICATED=1) is explicitly passed inside an isolated container bridge.
  • Per-Site Agent Tokens (AGENT_TOKENS): Standby nodes hold site-scoped tokens ({"site-1":"...","site-2":"..."}) accepted only on GET /agent/ws and POST /agent/reply.
    • Comparisons use length-independent constant-time byte XOR (sameBytes), failing closed whenever a token is unset, < 24 characters, or malformed JSON.
    • siteId is bound to the matched token key on the server so site-1 cannot spoof site-2, and any fallback AGENT_TOKEN cannot claim a siteId present in AGENT_TOKENS.
    • Every POST /agent/reply must present both the server-generated X-Job-Id and constant-time verified one-time X-Job-Nonce actively pending for that exact siteId (verifyPendingJob checked before decompressing request streams; 413 replies fail the job immediately without peer re-crawls).
  • Mandatory TLS for Non-Loopback Hubs & Client SDKs: internal/agent/config.go refuses cleartext http:// / ws:// hub URLs on any non-loopback address (127.0.0.1 / ::1), even if EDGEFETCH_ALLOW_INSECURE_HUB=1 is set in the environment. Docker bridge containers may opt in via EDGEFETCH_ALLOW_INSECURE_DOCKER_HUB=1 strictly for explicit container service names (hub, edgefetch-hub, host.docker.internal, or an operator-pinned EDGEFETCH_INSECURE_HUB_HOST). Both the TypeScript (sdk/ts) and Python (sdk/py) client SDKs likewise reject cleartext http:// Hub URLs unless targeting loopback or explicitly opted in (allowInsecureHttp / allow_insecure_http).
  • Origin-Locked Operator LLM Keys (hub/src/llm.ts): Server-side LLM environment credentials (OPENAI_API_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY, OPENROUTER_API_KEY, EDGEFETCH_LLM_API_KEY) are bound strictly to their canonical provider origin (or EDGEFETCH_LLM_BASE_URL origin). Any caller-supplied custom llm.baseUrl requires an explicit caller-supplied llm.apiKey, refuses HTTP 3xx redirects (redirect: "manual"), and enforces dial-time public-IP verification.

3. Multi-Layer SSRF, Redirect & Robots Protection

Even if a caller token were compromised, an agent cannot be used to probe local LAN devices, Home Assistant APIs, Tailscale peers, cloud metadata endpoints, or robots-disallowed paths:

  1. Hop-by-Hop URL, RFC 9309 /robots.txt & Content Signals Validation (crawl.PublicHTTPS + WithBlockedHost + WithAllowedHosts):
    • Automatic HTTP client redirects are disabled in both Go and Cloudflare overflow. Every hop (up to 5) re-evaluates URL rules (https://:443 multi-label DNS host, no IP literals, no localhost, no user:pass@), per-host locks/pacing, and 100% RFC 9309 /robots.txt (1 MiB parse cap, UTF-8 BOM stripping, §2.2.1 group combining, §2.2.2 percent-encoding normalization, §2.3.1.2 5-redirect handling, §2.4 Cache-Control: max-age up to 24h, 60s robots_unreachable negative cache, and Content-Signal: ai-input=no enforcement).
    • Callers cannot disable robots.txt (400 robots_required) unless the hub and agent owner explicitly configure ROBOTS_BYPASS_HOSTS + EDGEFETCH_ROBOTS_BYPASS_HOSTS (robots_bypass_hosts in Home Assistant OS add-on options) for owned or explicitly authorized partner domains (empty by default, audited in hub_events).
  2. Dial-Time Socket Peer Inspection (politeness.RejectNonPublicPeer in Go & net.BlockList in Portable Node Hub):
    • In Go, RejectNonPublicPeer runs inside net.Dialer.Control after DNS resolution creates the OS socket and immediately before connect() fires — eliminating DNS-rebinding TOCTOU races, and permanent SSRF refusals (ErrNonPublicPeer) are never retried. Cross-origin redirects automatically strip Authorization, Cookie, and Proxy-Authorization headers.
    • In the self-hosted Node.js Portable Hub (hub/portable/safe-fetch.ts), outbound webhooks, SPA overflow adapters, and public LLM calls resolve DNS through a node:net BlockList custom lookup hook and re-verify socket.remoteAddress before transmitting bytes.
    • Permits IPv6 only inside 2000::/3 global unicast before applying exclusions, blocking 0.0.0.0/8, loopback (127.0.0.0/8, ::1), RFC 1918 LANs (10/8, 172.16/12 including Home Assistant's 172.30.32.0/23 bridge, 192.168/16), RFC 6598 CGNAT / Tailscale (100.64.0.0/10), link-local / cloud metadata (169.254.0.0/16, fe80::/10), 6to4 relay (192.88.99.0/24), ULA (fc00::/7), site-local (fec0::/10), ORCHID/ORCHIDv2 (2001:10::/28, 2001:20::/28), documentation (2001:db8::/32, 3fff::/20), SRv6 (5f00::/16), and IPv6-to-IPv4 transition prefixes (::/96, 0:0:0:0:ffff:0::/96, 64:ff9b::/96, 64:ff9b:1::/48, 2002::/16, 2001::/32, 100::/64).
  3. Headless Browser Subresource & Secret Isolation (internal/browser/proxy.go, renderer.go, internal/priv/priv_linux.go + edgefetch_fat/apparmor.txt):
    • /robots.txt (including Content-Signal: ai-input=no) is verified before launching or navigating Chromium (including every interact navigate step and follow-up resumed session turn) and FinalURL is re-verified after load before returning rendered DOM.
    • Chromium OS Process Sandbox & Transparent Reporting: On Windows and macOS, Chromium launches with its native OS process sandbox enabled (sandbox: "on"). On Linux, EdgeFetch probes Chromium's namespace sandbox at startup and falls back to --no-sandbox only when container restrictions (NO_NEW_PRIVS / unprivileged user namespaces disabled) block sandbox initialization, reporting browserSandbox: "on" | "off" and browserVersion in list_agents and gating unsandboxed rendering when EDGEFETCH_REQUIRE_BROWSER_SANDBOX=1 is set.
    • Subresource Proxy, Origin-Scoped Headers & Process-Group Cleanup: Chromium runs in its own OS process group with --proxy-server=http://127.0.0.1:<port>, --proxy-bypass-list=<-loopback>, --host-resolver-rules="MAP * ~NOTFOUND , EXCLUDE 127.0.0.1", --webrtc-ip-handling-policy=disable_non_proxied_udp, --force-webrtc-ip-handling-policy=disable_non_proxied_udp, --enforce-webrtc-ip-permission-check, and --disable-quic, forcing well-behaved browser TCP subresources through SSRFProxy (CONNECT :443 + RejectNonPublicPeer) while blocking non-proxied WebRTC STUN UDP. Note that proxy flags constrain Chromium's networking stack — a native V8/renderer exploit on an unsandboxed container could still open direct sockets unless confined by OS sandboxing or a network VLAN. Caller extra HTTP headers are attached per-request via CDP Fetch.enable strictly to the initial target hostname and browser instances are recycled after custom-header jobs.
    • Secret & /proc Isolation: Child processes (chromium and Poppler pdftotext) are spawned with a minimal scrubbed cmd.Env (PATH, HOME, TMPDIR, LANG, TZ, FONTCONFIG_PATH — never inheriting EDGEFETCH_* or operator tokens). On Home Assistant OS (edgefetch_fat), priv.LockDownSecretPaths strictly locks /data (0700) and /data/options.json (0600) to 0:0 (root:root, failing closed on symlinks or permission errors) before priv.DropRoot() permanently drops all OS threads to 65532:65532 with PR_SET_NO_NEW_PRIVS=1 and PR_SET_DUMPABLE=0, paired with the edgefetch_fat AppArmor profile (deny capability dac_override / dac_read_search, deny ptrace, deny /proc/*/mem, deny /proc/*/environ, deny /proc/*/task/*/environ, and deny /proc/*/task/*/mem).
    • If a rendered page presents a CAPTCHA or anti-bot challenge (DetectRenderedChallenge), EdgeFetch halts immediately (outcome: "challenge") and never attempts to solve, click through, or wait out challenges.
    • Search (POST /v1/search) uses Serper / Brave / Tavily / SearXNG JSON / local SEARCH D1 FTS5 only — search-engine HTML scraping is never routed through residential HAOS nodes.
  4. Untrusted Agent Artifact Verification & Safe Serving (hub/src/storage.ts, hub/src/ingest.ts):
    • Agent R2 uploads (PUT /agent/artifacts/<blobSha256>) enforce per-job nonce/site/requestId/quota authorization, Content-Length caps, and R2 sha256 + If-None-Match: * create-once checks.
    • IngestWorkflow streams stored gzip blobs through DecompressionStream('gzip') with a hard 16 MiB decompression-bomb cutoff and verifies SHA-256(uncompressed) before setting artifacts.verified_at; poisoned or bombed blobs are deleted from R2 and scrubbed from META D1, and unverified blobs are never derived into documents or indexed in SEARCH D1 FTS5.
    • GET /v1/artifacts/<blobSha256> serves raw content strictly as text/plain; charset=utf-8 with X-Content-Type-Options: nosniff, Content-Security-Policy: sandbox, and Content-Disposition: attachment (never active HTML).

4. Container Hardening, Reproducible Supply Chain & Memory Bounds

  • Distroless Runtime & Multi-UID All-in-One Separation: Production Lite images use gcr.io/distroless/static-debian12 (~9 MB total, no /bin/sh, no package manager). In Dockerfile.all-in-one, /app is read-only (root:root, a-w), the Node.js Hub runs as node:node (1000:1000) owning 0700 /data/hub, and the local Go worker + Chromium run as isolated UID 65532:65532 via su-exec + env -i so worker/browser processes cannot read /proc/<hub_pid>/environ or /data/hub.
  • Linux PR_SET_NO_NEW_PRIVS + PR_SET_DUMPABLE=0 + AllThreadsSyscall6 (internal/priv/priv_linux.go): Reads /data/options.json, locks /data (0700) + /data/options.json (0600) to 0:0, performs best-effort zeroing of the raw read buffer, applies prctl(PR_SET_NO_NEW_PRIVS, 1) across all Go runtime OS threads via syscall.AllThreadsSyscall6, clears supplementary groups (Setgroups([])), drops all threads to UID/GID 65532:65532, applies prctl(PR_SET_DUMPABLE, 0), and verifies uid == 65532 before opening any socket.
  • AppArmor Profile (edgefetch_fat/apparmor.txt): Denies raw/packet sockets, mount, ptrace, dac_override, dac_read_search, sys_admin, net_admin, /proc/*/mem, /proc/*/environ, and /proc/kcore / /sys/firmware, paired with hassio_api: false, homeassistant_api: false, and host_network: false.
  • Bounded Memory, Server Concurrency & SQLite State: Enforces MaxResponseHeaderBytes = 64 KiB, io.LimitReader(maxBytes + 1) per-URL and streaming PDF caps, MaxTotalBytes = 16 MiB default per-batch body budget, MaxConcurrentServerCalls = 4 (503 server_busy), ReadTimeout = 30s / IdleTimeout = 120s, MaxTrackedHosts = 512 LRU host eviction, 1 MiB UTF-8 WebSocket frame caps, 20 MiB streaming gunzip caps, and daily Durable Object alarm() SQLite cleanup.

5. Self-Hosted OAuth 2.1, /mcp & /admin Hardening

  • Loopback-Only Default Redirects: Dynamic Client Registration (POST /oauth/register) is rate-limited per IP, omits ring-buffer noise, evicts only ungranted clients, and rejects non-loopback redirect URIs unless the host matches OAUTH_REDIRECT_ALLOWLIST (requiring explicit CALLER_TOKEN entry on consent even when signed into /admin, and labeling client_name as unverified self-reported).
  • Atomic Refresh Rotation & Replay Revocation: Refresh tokens rotate inside a single synchronous Durable Object SQLite method (oauthRotateRefreshToken) requiring client_id, keeping consumed token hashes until grant expiry, and revoking the entire grant immediately if a rotated refresh token is replayed.
  • Pre-Check Lockout & Query-String Redaction: Brute-force lockouts on /admin/login and /oauth/authorize are checked before evaluating tokens (auth_failures bounded against IPv6 sprays), and URL query strings, #fragments, and user:pass@ credentials are scrubbed from crawl logs, activity events, and edgefetch bug-report error strings.

6. Cryptographic Release Signing, SLSA Provenance & Contributor Secret Isolation

  • Owner-Only Ed25519 Release Signatures (checksums.txt.sig): Official v* GitHub Releases sign checksums.txt using an owner-only Ed25519 private key stored exclusively in GitHub Actions Sealed Secrets (RELEASE_SIGNING_PRIVATE_KEY) and verified against keys/release-signing-pub.pem. Fork pull requests never receive repository secrets, and GitHub Repository Rulesets restrict v* tag creation/mutation and main force-pushes to the repository owner.
  • GitHub SLSA Build Provenance Attestations: Every release archive (*.tar.gz, *.zip, checksums.txt) and multi-arch GHCR container digest (ghcr.io/bartrumb/edgefetch-agent, ghcr.io/bartrumb/edgefetch) publishes cryptographic SLSA provenance attestations verifiable via gh attestation verify --owner bartrumb.
  • Operator-Level Web Bot Auth Isolation (BOT_SIGNING_PRIVATE_KEY): Open-source builds never embed a shared bot-signing private key. Each self-hosted operator mints a dedicated Ed25519 keypair (edgefetch keygen-bot-auth) for their own Hub domain (/.well-known/http-message-signatures-directory), preventing third-party key abuse.

Safe Deployment Checklist

  1. Generate tokens with at least 32 bytes of cryptographic randomness (openssl rand -hex 32).
  2. Use per-site AGENT_TOKENS JSON so a decommissioned device can be revoked individually without rotating other sites.
  3. Set allowed_hosts in your Home Assistant Add-on configuration whenever your workload targets a known set of domains.

Reporting a Vulnerability

Please report security issues privately via GitHub Security Advisories rather than opening a public issue. Reports are acknowledged within 7 days.

There aren't any published security advisories