EdgeFetch is a self-hosted, MIT-licensed web scraping and search server for AI agents. You run one ~8 MB Go binary on any box you own (a Raspberry Pi, your Home Assistant, a spare laptop), and your agents get clean Markdown and structured data through MCP — with zero open ports, zero Redis or Postgres, and no per-page bill.
Built for the agent web: user-triggered, never trains, signs its requests with Web Bot Auth, asks for Markdown first, honours
robots.txtand Content Signals, and treats402as a price, not an error.
# Option 1 — Full 15-Tool Hub + BYO-LLM /v1/agent + 50+ Site/ATS/Commerce Adapters + /admin UI + FTS5 in Docker (zero cloud account):
git clone https://github.com/bartrumb/edgefetch.git && cd edgefetch
export CALLER_TOKEN="$(openssl rand -hex 24)" AGENT_TOKEN="$(openssl rand -hex 24)"
docker compose up --build -d
# Open http://127.0.0.1:8787/admin and sign in with $CALLER_TOKEN
# Option 2 — Standalone ~8 MB Local Go Binary for Claude Code, Claude Desktop, Cursor, Windsurf, Gemini CLI & Codex (zero containers/ports):
go install github.com/bartrumb/edgefetch/v2/cmd/edgefetch@latest
edgefetch init && edgefetch doctor- Outbound-only workers with zero open ports: Workers on a Raspberry Pi, NAS, or Home Assistant box behind CGNAT/Starlink dial out over TLS WebSockets (
WSS /agent/ws) to your Hub. Nothing at home is exposed to the internet. - Dial-time SSRF protection (including the browser):
politeness.RejectNonPublicPeerchecks resolved peer IPs inside Go'snet.Dialer.Controlright beforeconnect()(blocking DNS-rebinding TOCTOU), and headless Chromium subresources are forced through an in-process loopbackCONNECTproxy (--host-resolver-rules=MAP * ~NOTFOUND). - Polite by design — never bypasses
robots.txt, Content Signals, or CAPTCHAs:robots.txtandContent-Signal: ai-inputare enforced on every hop (including30xredirects),Retry-Afteris honored per host, HTTP402(x402) halts immediately aspayment_requiredwithout auto-paying, anti-bot challenges (Cloudflare, Akamai, DataDome, PerimeterX, AWS WAF, Imperva) halt on the first attempt, and IETF RFC 9421 Web Bot Auth (Ed25519HTTP Message Signatures) is built in. - Clean Two-Plane Architecture (Pure Go Edge Engine + Zero-Dependency TypeScript Hub):
- Go Data Plane (
cmd/,internal/— 1 external Go modulecoder/websocket,0transitive deps,~8 MBstatic binary): Runs 100% standalone (edgefetch scrape,edgefetch mcpoverstdio) or as an outbound worker daemon, handling every untrusted socket, dial-time SSRF check, 100% RFC 9309robots.txtcheck, streaming PDF parse, and CDP Chromium session. - TypeScript Coordinator & Adapter Plane (
hub/—0npm runtimedependencies): Optional fleet coordinator running natively on Cloudflare Hibernating WebSocket Durable Objects + D1 FTS5 + R2 ($0idle) or self-hostednode:sqlite. Becausehub/enforces zero npm runtime dependencies, the HTML→Markdown converter, linear security scanners, OAuth 2.1 PKCE server, and 50+ hot-deployable site/ATS/commerce adapters live openly inhub/src/rather thannode_modules.
- Go Data Plane (
- 50+ keyless site, commerce, open-data & ATS adapters + Store
catalog+ Autonomous BYO-LLM/v1/agent: 15 MCP tools (agent,find_extractors,scrape,extract,map,crawl,search,jobs,catalog,search_collection,interact,read_artifact,fetch_url,fetch_batch,list_agents), drop-in Firecrawl/v2/*compatibility, linear64 KiBframework detection, polite full-store Shopify (custom domains +/products.json) & WooCommerce inventory catalogs with variant diffs, and Claude-style dynamic adapter routing across Workable, Greenhouse (US/EU), Ashby, Lever (US/EU), Workday CXS, iCIMS, Rippling, Gem, Pinpoint, Teamtailor, BambooHR, Oracle HCM, SmartRecruiters, SEC EDGAR, Y Combinator, Socrata / CKAN / ArcGIS / Legistar Open Data, DOI/Crossref, OpenAlex, arXiv, ClinicalTrials.gov, PubMed, NIST NVD, GitHub, Stack Overflow, npm, crates.io, Hugging Face, OpenRouter, and 256 MiB streaming PDFs — supporting offline local LLMs (Ollama, LM Studio, vLLM) or online BYO-LLM (OpenAI, Anthropic, Gemini, OpenRouter, Workers AI) with live SSE streaming and$0Zero Data Retention. Add a new adapter in one file vianpm run adapter:new(docs/ADAPTERS.md).
Project status: Young personal open-source project (started November 2026), running daily as the sole web/search layer on the author's 3-node fleet (Home Assistant OS
x86_64, Windows VM, andarm64). MIT licensed; issues and PRs are triaged weekly. See docs/LAUNCH-FAQ.md for honest trade-offs.
Full architectural & trade-off breakdown: docs/COMPARISON.md · Common questions: docs/LAUNCH-FAQ.md
| Dimension | EdgeFetch (v2.6) |
Firecrawl (Self-Hosted) | Firecrawl / Tavily (Cloud) | Crawl4AI | Jina Reader |
|---|---|---|---|---|---|
| License | MIT | AGPL-3.0 (core) | Proprietary SaaS | Apache-2.0 (+ attribution) | Apache-2.0 |
| Minimum Self-Hosted Stack | 1 ~8 MB Go binary or 1 container (node:sqlite + /data, 0 Redis, 0 Postgres) |
Docker Compose + Redis + PostgreSQL + RabbitMQ + Playwright (2–4 GB+ RAM) |
Cloud API key | Python 3.10+ + Playwright (600 MB–1.5 GB RAM) |
Node.js + Puppeteer |
| Worker Supply Chain | 1 Go module (coder/websocket, 0 transitive deps) |
Dozens of npm + OS deps | Vendor SaaS | 40+ PyPI packages | Dozens of npm deps |
Zero-Port Outbound Home/Edge Fleet (WSS) |
Yes — Home Assistant OS, NAS, RPi, Mac/PC behind CGNAT/Starlink | No (needs open ports, VPN, or paid residential proxies) | Paid proxy credits | No (single-host or DIY queue) | No |
Desktop stdio MCP (edgefetch mcp) |
Yes — ~8 MB binary uses pre-installed OS Edge/Chrome/Brave (0 ports, 0 300 MB browser download) |
Requires cloud key or full Docker stack | Cloud API key required | Python + Playwright setup | Cloud URL or Node server |
| DNS-Rebinding SSRF + Browser Subresource Guard | Dial-time RejectNonPublicPeer + loopback CONNECT SSRFProxy (MAP * ~NOTFOUND) |
Application URL blocklist | Vendor-managed | Optional / app-level | App-level |
Web Search (search tool) |
Bring your own Serper/Brave/Tavily keys, or SearXNG + edge-map + D1 FTS5 for $0 |
Proprietary / SearXNG | Separate SaaS plans | Bring-your-own | s.jina.ai |
OpenTelemetry (OTLP/HTTP + W3C traceparent) |
Stitched Hub → Go httptrace (dnsMs, connectMs, tlsMs, ttfbMs) to any collector |
Internal logs / BullMQ | Vendor dashboard only | Python logs | Internal logs |
| Typical Cost (50k scrapes + 5k searches / mo)¹ | $0 – $5 / mo (Cloudflare Hibernating DO + homelab + SearXNG) |
$25 – $80+ / mo (VPS + proxy GBs) |
$49 – $130+ / mo |
$10 – $40+ / mo (VPS + proxies) |
Credit-metered |
¹ Cost assumptions checked Nov 20, 2026: compares Cloudflare Workers Free ($0) or Paid ($5/mo) within R2 (10 GB) and D1 (5 GB) monthly free tiers plus self-hosted SearXNG ($0, using existing home/VPS compute) against published Firecrawl Standard ($19–$99/mo) + Tavily Researcher ($30+/mo) plans.
flowchart TD
Admin["Browser (/admin Control Center)"] -->|"HMAC Session + CSRF"| HubW["Primary Hub (Cloudflare DO / Fly / Railway / Render)"]
Claude["Claude Code / Cursor (/mcp)"] -->|"OAuth 2.1 PKCE or Bearer"| HubW
Claude -.->|"Automatic HA Failover (baseUrls)"| GoHub["Standby Go Hub (--mode=hub,agent on VPS / Tailscale)"]
HubW -->|"WSS /agent/ws (W3C traceparent)"| S1["Docker / VPS Fat Node (amd64)"]
HubW -->|"WSS /agent/ws (W3C traceparent)"| S2["Mac Mini / RPi5 Node (arm64)"]
HubW -->|"WSS /agent/ws (W3C traceparent)"| S3["Home Assistant OS Add-on"]
GoHub -->|"Active-Active EDGEFETCH_HUB_URLS"| S1
GoHub -->|"Active-Active EDGEFETCH_HUB_URLS"| S2
HubW -.->|"Async OTLP/HTTP JSON"| OTel["Any OTel Collector (Tempo / Jaeger / Honeycomb / Axiom)"]
S1 -.->|"Child httptrace Spans"| OTel
Multi-arch OCI manifests (linux/amd64 + linux/arm64) run as non-root 65532:65532:
# Full "Fat" Agent (Go HTTP/2 + SSRF-proxied Headless Chromium + SwiftShader + Poppler PDF):
docker run -d --name edgefetch-agent --restart=unless-stopped \
--security-opt no-new-privileges:true \
-e EDGEFETCH_HUB_URL=https://<your-hub-domain> \
-e EDGEFETCH_AGENT_TOKEN="<YOUR_AGENT_TOKEN>" \
-e EDGEFETCH_SITE_ID="node-1" \
ghcr.io/bartrumb/edgefetch-agent:latest
# Or Ultra-Light ~9 MB Distroless Agent (HTTP/2 + Markdown + pure-Go PDF, ~15 MB RAM):
docker run -d --name edgefetch-lite --restart=unless-stopped --read-only \
-e EDGEFETCH_HUB_URL=https://<your-hub-domain> \
-e EDGEFETCH_AGENT_TOKEN="<YOUR_AGENT_TOKEN>" \
-e EDGEFETCH_SITE_ID="lite-1" \
ghcr.io/bartrumb/edgefetch-agent:lite --mode=agentDownload a release archive from GitHub Releases or install with go install:
go install github.com/bartrumb/edgefetch/v2/cmd/edgefetch@latest
# Auto-configure zero-daemon, zero-port stdio MCP into Claude Desktop & Cursor:
edgefetch install-mcp
# Verify SSRF guard, loopback CONNECT proxy, OS Chrome/Edge & pdftotext, or scrape from terminal:
edgefetch doctor
edgefetch scrape https://example.com
edgefetch scrape --json --query="installation" https://developers.cloudflare.com/workers/
# Optional: mint an IETF RFC 9421 Ed25519 Web Bot Auth signing keypair:
edgefetch keygen-bot-auth
# Or run a 100% self-hosted Go WebSocket Hub + /v1/scrape + /mcp controller on a small VPS:
EDGEFETCH_CALLER_TOKEN="<32-char-secret>" EDGEFETCH_AGENT_TOKEN="<32-char-secret>" \
edgefetch --mode=hub --addr=0.0.0.0:8080- Cloudflare Workers (
$0idle): Click Deploy to Cloudflare above (orcd hub && npm ci && npx wrangler deploy). - Render / Railway / Fly.io (Single All-in-One Container + Persistent
/dataVolume): Userender.yaml,railway.json, orfly launch --copy-config(fly.toml). - Home Assistant OS (
amd64&aarch64): Click Add Repository to Home Assistant above, install EdgeFetch, sethub_url,agent_token, andsite_id, and click Start.
- 25+ Site & ATS Job-Board Extractors (
hub/src/extractors/registry.ts,hub/src/jobs.ts):- ATS Job Boards & Structured
jobsTool: One-call job search & extraction across Greenhouse, Ashby (?includeCompensation=true), Lever, Workday CXS (GET&POST), and SmartRecruiters / Schema.orgJobPostingJSON-LD, normalizing pay ({ min, max, currency, period, basis: "base"|"ote"|"total" }) and filtering out closed postings (contentState: "closed_posting"). - Developer & Research Sites: 1-hop JSON/raw/patch upgrades for GitHub (raw,
.patch, Issues/PRs, Gist), GitLab, Codeberg, Stack Overflow (✅ Accepted Answer), npm, crates.io, Hugging Face (YAML model cards → GFM tables), Discourse, Lobsters, Reddit, Bluesky (ATProto), Mastodon, DEV.to, pkg.go.dev, PyPI, Wikipedia, IETF RFCs, ArXiv, PubMed/PMC, Hacker News, and YouTube transcripts.
- ATS Job Boards & Structured
- Data-Gathering & SPA Reliability:
- Native CDP SPA Engine (
internal/browser/renderer.go): Tracks real CDPNetwork.requestWillBeSent/loadingFinishedXHR/Fetchnetworkidle, triggersIntersectionObserverlazy-load scroll passes, waits on aMutationObserverDOM quiescence check, and flattens open Web ComponentshadowRoottrees. - SSR Hydration Recovery (Next.js App Router
self.__next_f.pushRSC,__NEXT_DATA__, Nuxt 3__NUXT_DATA__, Remix, Apollo): Recovers full prose from thin SPA shells in 1 HTTP hop before launching Chromium. - Optional Ethical Heavy-SPA Fallback (
Crawl4AI&Firecrawl— Cloud or Self-Hosted): ConfigureCRAWL4AI_API_URLand/orFIRECRAWL_API_URL+FIRECRAWL_API_KEYso heavy SPAs (or Lite-only fleets without local Chromium) fall back automatically after verifyingrobots.txt— while never delegatingrobots.txtblocks,401/403/429, or anti-bot challenges. - Conditional
304 Not ModifiedRevalidation: Sends storedETag/Last-Modifiedvalidators fromMETAD1 and serves verified R2 Markdown immediately on304. - 7-Vendor WAF Catalog + 4 Anti-False-Positive Layers: Accurately classifies Cloudflare, Akamai, DataDome, PerimeterX, AWS WAF, Imperva, and Vercel/Fastly/Kasada challenges while stripping
<pre><code>/<blockquote>and vetoing200 OKarticles (>= 250words, JSON-LD,<nav>) so security/Turnstile docs never false-positive. - Opt-in
waybackFallback: Recovers5xx/ DNS-dead link rot from the Internet Archive (never invoked onrobots.txt,403/429, or challenges). - Token Efficiency:
-91.6%LLM token reduction on our offline 8-fixture synthetic regression corpus (bench/RESULTS.md), preserving<T>generics (Promise<void>), GFM tables, KaTeX math,.ipynboutputs, and OpenAPI 3.x schemas.
- Native CDP SPA Engine (
Status legend:
- ✅ Daily-Driver + CI Verified: Running daily on the author's 3-node personal fleet (
arm64, Home Assistant OSx86_64, and Windows VM) since Nov 18, 2026 and gated by automated E2E / smoke suites on every PR. - ✅ Live-Tested + CI Verified: Executed end-to-end on real host OS/runtime during verification and covered by automated CI tests.
- 🔧 Build & Manifest Verified: Static
CGO_ENABLED=0cross-compilation (go build/ GoReleaser) and deployment manifests verified cleanly; community hardware reports welcome!
| Platform / Target | Role(s) | Binary / Image Footprint | Capabilities | Verification Status |
|---|---|---|---|---|
| Cloudflare Workers + Hibernating DO + D1 (FTS5) + R2 + Workflows | Cloud Hub (/mcp, /admin, OAuth 2.1, HUB_SHARDS) |
Serverless bundle (0 npm runtime deps) |
12-tool MCP, Search, FTS5, R2 Artifacts, RFC 9421, OTLP | ✅ Daily-Driver + CI Verified (workerd H7 & S1 suites) |
Home Assistant OS (amd64 x86_64 & aarch64 RPi / NUC / VM) |
Outbound Home Worker (edgefetch_fat) |
Multi-arch Alpine + Chromium + SwiftShader + Poppler | HTTP/2, artifacts-v1, browser-v1, interact-v1, 256 MiB PDF |
✅ Daily-Driver + CI Verified (x86_64 & arm64 home nodes) |
Windows 11 (windows/amd64 Desktop & VM) |
Local stdio MCP, desktop Launchpad, CLI & Fleet Worker |
~8 MB (edgefetch.exe, 0 DLL deps) |
Auto-detects native Windows Chrome / Edge / Brave for CDP JS render | ✅ Daily-Driver + CI Verified (Windows VM node + CLI E2E) |
Linux Docker Worker — Fat (linux/amd64, linux/arm64) |
Standalone Worker (Dockerfile.agent) |
Non-root 65532:65532 + AppArmor + Chromium WebGL |
Full HTTP/2 + SSRF-proxied Headless Chromium + Poppler PDF | ✅ Live-Tested + CI Verified (live AppArmor + WebGL CI smoke) |
Linux Distroless Worker/Server — Lite (linux/amd64, linux/arm64) |
Ultra-light Worker / Server / Go Hub (Dockerfile) |
~8 MB static binary (~15 MiB RAM) |
HTTP/2, Markdown, pure-Go PDF, --mode=hub, RFC 9421, OTLP |
✅ Live-Tested + CI Verified (buildx multi-arch CI) |
All-in-One Container (Dockerfile.all-in-one / Docker Compose) |
Full TypeScript Hub + node:sqlite FTS5 + Go Chromium Worker |
Node 26 Alpine + Go + Chromium + Poppler (0 Redis, 0 Postgres) |
/admin, OAuth 2.1, /mcp, SQLite FTS5, /data artifacts, JS render |
✅ Live-Tested + CI Verified (V2-9 E2E + buildx CI) |
Linux (ubuntu/wsl2 linux/amd64) |
--mode=hub, --mode=hub,agent HA Mesh, CLI (scrape, doctor) |
~8 MB static ELF |
Active-active EDGEFETCH_HUB_URLS, Go Hub /mcp, Fuzz*, -race |
✅ Live-Tested + CI Verified (Go 1.27 -race + govulncheck) |
WebAssembly / WASI (wasip1/wasm) |
Sandboxed Offline HTML → Markdown / JSON Extractor | cmd/edgefetch-wasm (make build-wasm) |
Runs inside wasmtime, wazero, or Wasm sandboxes |
✅ Live-Tested + CI Verified (go build + unit test) |
| Chrome / Firefox Manifest V3 Extension | Active-Tab FTS5 Clipper (POST /v1/ingest-tab) |
extensions/browser-clipper/ |
Clips authenticated tabs ethically into private FTS5 collections | ✅ Live-Tested + CI Verified (V2-11 E2E) |
| Deno Deploy + Turso HTTP + S3 SigV4 | Serverless Hub Adapter | hub/portable/deno-server.ts, s3-bucket.ts, turso-d1.ts |
Zero-dep WebCrypto SigV4 (S3/R2/Tigris/MinIO/B2) + Turso /v2/pipeline |
✅ Live-Tested + CI Verified (V2-11 unit + tsc) |
macOS Apple Silicon & Intel (darwin/arm64, darwin/amd64) |
Local stdio MCP, desktop Launchpad, CLI & Worker |
~7.5–8 MB static binary |
Auto-detects /Applications/Google Chrome.app, Edge & Brave |
🔧 Build & Manifest Verified (.goreleaser.yaml) |
Windows on ARM (windows/arm64) |
Desktop CLI, stdio MCP & Worker |
~7.5 MB (edgefetch.exe) |
HTTP/2, Markdown, pure-Go PDF, OS Edge/Chrome detection | 🔧 Build & Manifest Verified (.goreleaser.yaml) |
| Fly.io / Railway / Render / Coolify | 1-Click Cloud Self-Hosted All-in-One Hub + Worker | fly.toml, railway.json, render.yaml |
Persistent /data SQLite FTS5 + artifacts + Headless Chromium |
🔧 Build & Manifest Verified (runs Dockerfile.all-in-one) |
Kubernetes (amd64 / arm64 Clusters) |
Zero-Trust Fleet Worker Pod | deploy/k8s/agent-deployment.yaml |
Read-only rootfs, seccomp: RuntimeDefault, NetworkPolicy (ingress: []) |
🔧 Build & Manifest Verified |
| Unraid, CasaOS, Portainer, Synology, Proxmox LXC & Termux | Homelab / NAS / LXC / Spare-Phone Worker | deploy/, compose.agent.yaml |
Outbound WSS node (zero port forwarding) |
🔧 Build & Manifest Verified |
OpenWrt (linux/mipsle, armv7) & FreeBSD (freebsd/amd64, arm64) |
Router / Firewall Appliance Lite Worker | ~7.5–8.5 MB static binary |
HTTP/2, Markdown, pure-Go PDF (CGO_ENABLED=0) |
🔧 Build & Manifest Verified (.goreleaser.yaml) |
Run go test -bench=. -benchmem ./... locally:
Go Benchmark (go test -bench=. -benchmem ./...) |
Latency | Heap Allocations |
|---|---|---|
BenchmarkParseRetryAfter |
4.45 ns/op |
0 B/op (0 allocs/op) |
BenchmarkRobotsMatch (* wildcards + $ anchor) |
52.17 ns/op |
0 B/op (0 allocs/op) |
BenchmarkIsPublicIP (IPv4 + NAT64 IPv6 check) |
155.50 ns/op |
0 B/op (0 allocs/op) |
BenchmarkClassifyFeedJSON |
1,229 ns/op |
1,416 B/op (28 allocs/op) |
- Launch & Architecture FAQ: docs/LAUNCH-FAQ.md
- Head-to-Head Comparison & Trade-Offs: docs/COMPARISON.md
- Synthetic Extraction Benchmark (
-91.6%tokens on 8 fixtures): bench/RESULTS.md - Operations, OTLP Tracing, HA Mesh & MCP Guide: docs/USAGE.md
- Hub ↔ Go Agent Wire Protocol Spec: docs/PROTOCOL.md
- Systems & Go Concurrency Internals: docs/INTERNALS.md
- Security Threat Model, Signed Releases & SSRF Controls: SECURITY.md
- Bot, WAF & MCP Registry Certification Guide: docs/REGISTRATION.md
- Contributing Rules & Code of Conduct: CONTRIBUTING.md · CODE_OF_CONDUCT.md
