EdgeFetch is designed to run inside private networks (such as Home Assistant OS servers, homelabs, or branch offices) with zero inbound ports and layered SSRF/privilege controls. Because browser-enabled (browser-v1) and PDF-enabled (pdf-v1) worker nodes execute untrusted web JavaScript and C/C++ document parsers on your hardware, operators running browser/PDF nodes on home networks should also place those nodes on an isolated VLAN or guest subnet (and a dedicated standard non-admin account on Windows/macOS) so a hypothetical zero-day browser renderer escape cannot reach LAN devices.
- Outbound-only sockets in
agentmode:edgefetch --mode=agentnever callsnet.Listen. Router port-forwarding, UPnP, and VPN/tunnel daemons are unnecessary. - No generic proxy protocol: EdgeFetch is not an HTTP
CONNECTor SOCKS5 relay. It accepts structured JSONFetchCalldispatches over a single authenticated TLS 1.2/1.3 WebSocket and executes bounded HTTPGET(or bounded UTF-8 JSON/formPOSTup to64 KiBfor public APIs such as Workday CXS job boards) againstrobots.txt-permitted publichttps://:443URLs only. - Tokenless Local Loopback Hardening (
desktop/serve): When bound to127.0.0.1/::1without a caller token, EdgeFetch enforces strict DNS-rebinding and browser CSRF defenses:Hostmust resolve to127.0.0.1,[::1], orlocalhoston the active port,Sec-Fetch-Siterejectscross-site,Origin(when present) must match loopback, and state-changingPOSTendpoints requireContent-Type: application/json.
- Caller Token (
CALLER_TOKEN): Authorizes upstream workflows to submitPOST /fetchbatches and viewGET /agentson the Hub. Never placeCALLER_TOKENon standby nodes. - Standalone Server Token (
EDGEFETCH_SERVER_TOKEN): In--mode=server,edgefetchrefuses to bind non-loopback addresses (:8080,0.0.0.0:8080) unlessEDGEFETCH_SERVER_TOKEN(>= 24random chars, verified in constant time via SHA-256 +subtle.ConstantTimeCompareonPOST /fetch) is configured, or--allow-unauthenticated(EDGEFETCH_ALLOW_UNAUTHENTICATED=1) is explicitly passed inside an isolated container bridge. - Per-Site Agent Tokens (
AGENT_TOKENS): Standby nodes hold site-scoped tokens ({"site-1":"...","site-2":"..."}) accepted only onGET /agent/wsandPOST /agent/reply.- Comparisons use length-independent constant-time byte XOR (
sameBytes), failing closed whenever a token is unset,< 24characters, or malformed JSON. siteIdis bound to the matched token key on the server sosite-1cannot spoofsite-2, and any fallbackAGENT_TOKENcannot claim asiteIdpresent inAGENT_TOKENS.- Every
POST /agent/replymust present both the server-generatedX-Job-Idand constant-time verified one-timeX-Job-Nonceactively pending for that exactsiteId(verifyPendingJobchecked before decompressing request streams;413replies fail the job immediately without peer re-crawls).
- Comparisons use length-independent constant-time byte XOR (
- Mandatory TLS for Non-Loopback Hubs & Client SDKs:
internal/agent/config.gorefuses cleartexthttp:///ws://hub URLs on any non-loopback address (127.0.0.1/::1), even ifEDGEFETCH_ALLOW_INSECURE_HUB=1is set in the environment. Docker bridge containers may opt in viaEDGEFETCH_ALLOW_INSECURE_DOCKER_HUB=1strictly for explicit container service names (hub,edgefetch-hub,host.docker.internal, or an operator-pinnedEDGEFETCH_INSECURE_HUB_HOST). Both the TypeScript (sdk/ts) and Python (sdk/py) client SDKs likewise reject cleartexthttp://Hub URLs unless targeting loopback or explicitly opted in (allowInsecureHttp/allow_insecure_http). - Origin-Locked Operator LLM Keys (
hub/src/llm.ts): Server-side LLM environment credentials (OPENAI_API_KEY,ANTHROPIC_API_KEY,GEMINI_API_KEY,OPENROUTER_API_KEY,EDGEFETCH_LLM_API_KEY) are bound strictly to their canonical provider origin (orEDGEFETCH_LLM_BASE_URLorigin). Any caller-supplied customllm.baseUrlrequires an explicit caller-suppliedllm.apiKey, refuses HTTP3xxredirects (redirect: "manual"), and enforces dial-time public-IP verification.
Even if a caller token were compromised, an agent cannot be used to probe local LAN devices, Home Assistant APIs, Tailscale peers, cloud metadata endpoints, or robots-disallowed paths:
- Hop-by-Hop URL, RFC 9309
/robots.txt& Content Signals Validation (crawl.PublicHTTPS+WithBlockedHost+WithAllowedHosts):- Automatic HTTP client redirects are disabled in both Go and Cloudflare overflow. Every hop (up to 5) re-evaluates URL rules (
https://:443multi-label DNS host, no IP literals, nolocalhost, nouser:pass@), per-host locks/pacing, and 100% RFC 9309/robots.txt(1 MiBparse cap, UTF-8 BOM stripping, §2.2.1 group combining, §2.2.2 percent-encoding normalization, §2.3.1.2 5-redirect handling, §2.4Cache-Control: max-ageup to 24h,60srobots_unreachablenegative cache, andContent-Signal: ai-input=noenforcement). - Callers cannot disable
robots.txt(400 robots_required) unless the hub and agent owner explicitly configureROBOTS_BYPASS_HOSTS+EDGEFETCH_ROBOTS_BYPASS_HOSTS(robots_bypass_hostsin Home Assistant OS add-on options) for owned or explicitly authorized partner domains (empty by default, audited inhub_events).
- Automatic HTTP client redirects are disabled in both Go and Cloudflare overflow. Every hop (up to 5) re-evaluates URL rules (
- Dial-Time Socket Peer Inspection (
politeness.RejectNonPublicPeerin Go &net.BlockListin Portable Node Hub):- In Go,
RejectNonPublicPeerruns insidenet.Dialer.Controlafter DNS resolution creates the OS socket and immediately beforeconnect()fires — eliminating DNS-rebinding TOCTOU races, and permanent SSRF refusals (ErrNonPublicPeer) are never retried. Cross-origin redirects automatically stripAuthorization,Cookie, andProxy-Authorizationheaders. - In the self-hosted Node.js Portable Hub (
hub/portable/safe-fetch.ts), outbound webhooks, SPA overflow adapters, and public LLM calls resolve DNS through anode:netBlockListcustomlookuphook and re-verifysocket.remoteAddressbefore transmitting bytes. - Permits IPv6 only inside
2000::/3global unicast before applying exclusions, blocking0.0.0.0/8, loopback (127.0.0.0/8,::1), RFC 1918 LANs (10/8,172.16/12including Home Assistant's172.30.32.0/23bridge,192.168/16), RFC 6598 CGNAT / Tailscale (100.64.0.0/10), link-local / cloud metadata (169.254.0.0/16,fe80::/10), 6to4 relay (192.88.99.0/24), ULA (fc00::/7), site-local (fec0::/10), ORCHID/ORCHIDv2 (2001:10::/28,2001:20::/28), documentation (2001:db8::/32,3fff::/20), SRv6 (5f00::/16), and IPv6-to-IPv4 transition prefixes (::/96,0:0:0:0:ffff:0::/96,64:ff9b::/96,64:ff9b:1::/48,2002::/16,2001::/32,100::/64).
- In Go,
- Headless Browser Subresource & Secret Isolation (
internal/browser/proxy.go,renderer.go,internal/priv/priv_linux.go+edgefetch_fat/apparmor.txt):/robots.txt(includingContent-Signal: ai-input=no) is verified before launching or navigating Chromium (including everyinteractnavigatestep and follow-up resumed session turn) andFinalURLis re-verified after load before returning rendered DOM.- Chromium OS Process Sandbox & Transparent Reporting: On Windows and macOS, Chromium launches with its native OS process sandbox enabled (
sandbox: "on"). On Linux, EdgeFetch probes Chromium's namespace sandbox at startup and falls back to--no-sandboxonly when container restrictions (NO_NEW_PRIVS/ unprivileged user namespaces disabled) block sandbox initialization, reportingbrowserSandbox: "on" | "off"andbrowserVersioninlist_agentsand gating unsandboxed rendering whenEDGEFETCH_REQUIRE_BROWSER_SANDBOX=1is set. - Subresource Proxy, Origin-Scoped Headers & Process-Group Cleanup: Chromium runs in its own OS process group with
--proxy-server=http://127.0.0.1:<port>,--proxy-bypass-list=<-loopback>,--host-resolver-rules="MAP * ~NOTFOUND , EXCLUDE 127.0.0.1",--webrtc-ip-handling-policy=disable_non_proxied_udp,--force-webrtc-ip-handling-policy=disable_non_proxied_udp,--enforce-webrtc-ip-permission-check, and--disable-quic, forcing well-behaved browser TCP subresources throughSSRFProxy(CONNECT :443+RejectNonPublicPeer) while blocking non-proxied WebRTC STUN UDP. Note that proxy flags constrain Chromium's networking stack — a native V8/renderer exploit on an unsandboxed container could still open direct sockets unless confined by OS sandboxing or a network VLAN. Caller extra HTTP headers are attached per-request via CDPFetch.enablestrictly to the initial target hostname and browser instances are recycled after custom-header jobs. - Secret &
/procIsolation: Child processes (chromiumand Popplerpdftotext) are spawned with a minimal scrubbedcmd.Env(PATH,HOME,TMPDIR,LANG,TZ,FONTCONFIG_PATH— never inheritingEDGEFETCH_*or operator tokens). On Home Assistant OS (edgefetch_fat),priv.LockDownSecretPathsstrictly locks/data(0700) and/data/options.json(0600) to0:0(root:root, failing closed on symlinks or permission errors) beforepriv.DropRoot()permanently drops all OS threads to65532:65532withPR_SET_NO_NEW_PRIVS=1andPR_SET_DUMPABLE=0, paired with theedgefetch_fatAppArmor profile (deny capability dac_override/dac_read_search,deny ptrace,deny /proc/*/mem,deny /proc/*/environ,deny /proc/*/task/*/environ, anddeny /proc/*/task/*/mem). - If a rendered page presents a CAPTCHA or anti-bot challenge (
DetectRenderedChallenge), EdgeFetch halts immediately (outcome: "challenge") and never attempts to solve, click through, or wait out challenges. - Search (
POST /v1/search) uses Serper / Brave / Tavily / SearXNG JSON / localSEARCHD1 FTS5 only — search-engine HTML scraping is never routed through residential HAOS nodes.
- Untrusted Agent Artifact Verification & Safe Serving (
hub/src/storage.ts,hub/src/ingest.ts):- Agent R2 uploads (
PUT /agent/artifacts/<blobSha256>) enforce per-job nonce/site/requestId/quota authorization,Content-Lengthcaps, and R2sha256+If-None-Match: *create-once checks. IngestWorkflowstreams stored gzip blobs throughDecompressionStream('gzip')with a hard16 MiBdecompression-bomb cutoff and verifiesSHA-256(uncompressed)before settingartifacts.verified_at; poisoned or bombed blobs are deleted from R2 and scrubbed fromMETAD1, and unverified blobs are never derived intodocumentsor indexed inSEARCHD1 FTS5.GET /v1/artifacts/<blobSha256>serves raw content strictly astext/plain; charset=utf-8withX-Content-Type-Options: nosniff,Content-Security-Policy: sandbox, andContent-Disposition: attachment(never active HTML).
- Agent R2 uploads (
- Distroless Runtime & Multi-UID All-in-One Separation: Production Lite images use
gcr.io/distroless/static-debian12(~9 MB total, no/bin/sh, no package manager). InDockerfile.all-in-one,/appis read-only (root:root,a-w), the Node.js Hub runs asnode:node(1000:1000) owning0700/data/hub, and the local Go worker + Chromium run as isolated UID65532:65532viasu-exec+env -iso worker/browser processes cannot read/proc/<hub_pid>/environor/data/hub. - Linux
PR_SET_NO_NEW_PRIVS+PR_SET_DUMPABLE=0+AllThreadsSyscall6(internal/priv/priv_linux.go): Reads/data/options.json, locks/data(0700) +/data/options.json(0600) to0:0, performs best-effort zeroing of the raw read buffer, appliesprctl(PR_SET_NO_NEW_PRIVS, 1)across all Go runtime OS threads viasyscall.AllThreadsSyscall6, clears supplementary groups (Setgroups([])), drops all threads to UID/GID65532:65532, appliesprctl(PR_SET_DUMPABLE, 0), and verifiesuid == 65532before opening any socket. - AppArmor Profile (
edgefetch_fat/apparmor.txt): Denies raw/packet sockets,mount,ptrace,dac_override,dac_read_search,sys_admin,net_admin,/proc/*/mem,/proc/*/environ, and/proc/kcore//sys/firmware, paired withhassio_api: false,homeassistant_api: false, andhost_network: false. - Bounded Memory, Server Concurrency & SQLite State: Enforces
MaxResponseHeaderBytes = 64 KiB,io.LimitReader(maxBytes + 1)per-URL and streaming PDF caps,MaxTotalBytes = 16 MiBdefault per-batch body budget,MaxConcurrentServerCalls = 4(503 server_busy),ReadTimeout = 30s/IdleTimeout = 120s,MaxTrackedHosts = 512LRU host eviction,1 MiBUTF-8 WebSocket frame caps,20 MiBstreaming gunzip caps, and daily Durable Objectalarm()SQLite cleanup.
- Loopback-Only Default Redirects: Dynamic Client Registration (
POST /oauth/register) is rate-limited per IP, omits ring-buffer noise, evicts only ungranted clients, and rejects non-loopback redirect URIs unless the host matchesOAUTH_REDIRECT_ALLOWLIST(requiring explicitCALLER_TOKENentry on consent even when signed into/admin, and labelingclient_nameas unverified self-reported). - Atomic Refresh Rotation & Replay Revocation: Refresh tokens rotate inside a single synchronous Durable Object SQLite method (
oauthRotateRefreshToken) requiringclient_id, keeping consumed token hashes until grant expiry, and revoking the entire grant immediately if a rotated refresh token is replayed. - Pre-Check Lockout & Query-String Redaction: Brute-force lockouts on
/admin/loginand/oauth/authorizeare checked before evaluating tokens (auth_failuresbounded against IPv6 sprays), and URL query strings,#fragments, anduser:pass@credentials are scrubbed from crawl logs, activity events, andedgefetch bug-reporterror strings.
- Owner-Only Ed25519 Release Signatures (
checksums.txt.sig): Officialv*GitHub Releases signchecksums.txtusing an owner-only Ed25519 private key stored exclusively in GitHub Actions Sealed Secrets (RELEASE_SIGNING_PRIVATE_KEY) and verified againstkeys/release-signing-pub.pem. Fork pull requests never receive repository secrets, and GitHub Repository Rulesets restrictv*tag creation/mutation andmainforce-pushes to the repository owner. - GitHub SLSA Build Provenance Attestations: Every release archive (
*.tar.gz,*.zip,checksums.txt) and multi-arch GHCR container digest (ghcr.io/bartrumb/edgefetch-agent,ghcr.io/bartrumb/edgefetch) publishes cryptographic SLSA provenance attestations verifiable viagh attestation verify --owner bartrumb. - Operator-Level Web Bot Auth Isolation (
BOT_SIGNING_PRIVATE_KEY): Open-source builds never embed a shared bot-signing private key. Each self-hosted operator mints a dedicated Ed25519 keypair (edgefetch keygen-bot-auth) for their own Hub domain (/.well-known/http-message-signatures-directory), preventing third-party key abuse.
- Generate tokens with at least 32 bytes of cryptographic randomness (
openssl rand -hex 32). - Use per-site
AGENT_TOKENSJSON so a decommissioned device can be revoked individually without rotating other sites. - Set
allowed_hostsin your Home Assistant Add-on configuration whenever your workload targets a known set of domains.
Please report security issues privately via GitHub Security Advisories rather than opening a public issue. Reports are acknowledged within 7 days.