Skip to content
bartrumbPublic

About

Self-hosted, MIT web scraping + search MCP server for AI agents: one Go binary, zero open ports, $0 idle on Cloudflare.

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

1 star

Watchers

0 watching

Forks

Latest commit

 

History

62 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

EdgeFetch

CI Go 1.26+ Go Reference License: MIT

EdgeFetch is a self-hosted, MIT-licensed web scraping and search server for AI agents. You run one ~8 MB Go binary on any box you own (a Raspberry Pi, your Home Assistant, a spare laptop), and your agents get clean Markdown and structured data through MCP — with zero open ports, zero Redis or Postgres, and no per-page bill.

Built for the agent web: user-triggered, never trains, signs its requests with Web Bot Auth, asks for Markdown first, honours robots.txt and Content Signals, and treats 402 as a price, not an error.

EdgeFetch Built-in /admin Web Control Center — Live Crawl Log, Multi-Arch Fleet, Interactive Waterfall Tester, 429 Cooldowns & OAuth 2.1 MCP Grants

Try It in 60 Seconds

# Option 1 — Full 15-Tool Hub + BYO-LLM /v1/agent + 50+ Site/ATS/Commerce Adapters + /admin UI + FTS5 in Docker (zero cloud account):
git clone https://github.com/bartrumb/edgefetch.git && cd edgefetch
export CALLER_TOKEN="$(openssl rand -hex 24)" AGENT_TOKEN="$(openssl rand -hex 24)"
docker compose up --build -d
# Open http://127.0.0.1:8787/admin and sign in with $CALLER_TOKEN

# Option 2 — Standalone ~8 MB Local Go Binary for Claude Code, Claude Desktop, Cursor, Windsurf, Gemini CLI & Codex (zero containers/ports):
go install github.com/bartrumb/edgefetch/v2/cmd/edgefetch@latest
edgefetch init && edgefetch doctor

Deploy to Cloudflare   Deploy to Render   Deploy on Railway   Deploy on Deno   Add EdgeFetch Repository to Home Assistant

What Stands Out

  • Outbound-only workers with zero open ports: Workers on a Raspberry Pi, NAS, or Home Assistant box behind CGNAT/Starlink dial out over TLS WebSockets (WSS /agent/ws) to your Hub. Nothing at home is exposed to the internet.
  • Dial-time SSRF protection (including the browser): politeness.RejectNonPublicPeer checks resolved peer IPs inside Go's net.Dialer.Control right before connect() (blocking DNS-rebinding TOCTOU), and headless Chromium subresources are forced through an in-process loopback CONNECT proxy (--host-resolver-rules=MAP * ~NOTFOUND).
  • Polite by design — never bypasses robots.txt, Content Signals, or CAPTCHAs: robots.txt and Content-Signal: ai-input are enforced on every hop (including 30x redirects), Retry-After is honored per host, HTTP 402 (x402) halts immediately as payment_required without auto-paying, anti-bot challenges (Cloudflare, Akamai, DataDome, PerimeterX, AWS WAF, Imperva) halt on the first attempt, and IETF RFC 9421 Web Bot Auth (Ed25519 HTTP Message Signatures) is built in.
  • Clean Two-Plane Architecture (Pure Go Edge Engine + Zero-Dependency TypeScript Hub):
    • Go Data Plane (cmd/, internal/ — 1 external Go module coder/websocket, 0 transitive deps, ~8 MB static binary): Runs 100% standalone (edgefetch scrape, edgefetch mcp over stdio) or as an outbound worker daemon, handling every untrusted socket, dial-time SSRF check, 100% RFC 9309 robots.txt check, streaming PDF parse, and CDP Chromium session.
    • TypeScript Coordinator & Adapter Plane (hub/ — 0 npm runtime dependencies): Optional fleet coordinator running natively on Cloudflare Hibernating WebSocket Durable Objects + D1 FTS5 + R2 ($0 idle) or self-hosted node:sqlite. Because hub/ enforces zero npm runtime dependencies, the HTML→Markdown converter, linear security scanners, OAuth 2.1 PKCE server, and 50+ hot-deployable site/ATS/commerce adapters live openly in hub/src/ rather than node_modules.
  • 50+ keyless site, commerce, open-data & ATS adapters + Store catalog + Autonomous BYO-LLM /v1/agent: 15 MCP tools (agent, find_extractors, scrape, extract, map, crawl, search, jobs, catalog, search_collection, interact, read_artifact, fetch_url, fetch_batch, list_agents), drop-in Firecrawl /v2/* compatibility, linear 64 KiB framework detection, polite full-store Shopify (custom domains + /products.json) & WooCommerce inventory catalogs with variant diffs, and Claude-style dynamic adapter routing across Workable, Greenhouse (US/EU), Ashby, Lever (US/EU), Workday CXS, iCIMS, Rippling, Gem, Pinpoint, Teamtailor, BambooHR, Oracle HCM, SmartRecruiters, SEC EDGAR, Y Combinator, Socrata / CKAN / ArcGIS / Legistar Open Data, DOI/Crossref, OpenAlex, arXiv, ClinicalTrials.gov, PubMed, NIST NVD, GitHub, Stack Overflow, npm, crates.io, Hugging Face, OpenRouter, and 256 MiB streaming PDFs — supporting offline local LLMs (Ollama, LM Studio, vLLM) or online BYO-LLM (OpenAI, Anthropic, Gemini, OpenRouter, Workers AI) with live SSE streaming and $0 Zero Data Retention. Add a new adapter in one file via npm run adapter:new (docs/ADAPTERS.md).

Project status: Young personal open-source project (started November 2026), running daily as the sole web/search layer on the author's 3-node fleet (Home Assistant OS x86_64, Windows VM, and arm64). MIT licensed; issues and PRs are triaged weekly. See docs/LAUNCH-FAQ.md for honest trade-offs.


Head-to-Head vs. Other AI Crawlers & MCP Servers

Full architectural & trade-off breakdown: docs/COMPARISON.md · Common questions: docs/LAUNCH-FAQ.md

Dimension EdgeFetch (v2.6) Firecrawl (Self-Hosted) Firecrawl / Tavily (Cloud) Crawl4AI Jina Reader
License MIT AGPL-3.0 (core) Proprietary SaaS Apache-2.0 (+ attribution) Apache-2.0
Minimum Self-Hosted Stack 1 ~8 MB Go binary or 1 container (node:sqlite + /data, 0 Redis, 0 Postgres) Docker Compose + Redis + PostgreSQL + RabbitMQ + Playwright (2–4 GB+ RAM) Cloud API key Python 3.10+ + Playwright (600 MB–1.5 GB RAM) Node.js + Puppeteer
Worker Supply Chain 1 Go module (coder/websocket, 0 transitive deps) Dozens of npm + OS deps Vendor SaaS 40+ PyPI packages Dozens of npm deps
Zero-Port Outbound Home/Edge Fleet (WSS) Yes — Home Assistant OS, NAS, RPi, Mac/PC behind CGNAT/Starlink No (needs open ports, VPN, or paid residential proxies) Paid proxy credits No (single-host or DIY queue) No
Desktop stdio MCP (edgefetch mcp) Yes — ~8 MB binary uses pre-installed OS Edge/Chrome/Brave (0 ports, 0 300 MB browser download) Requires cloud key or full Docker stack Cloud API key required Python + Playwright setup Cloud URL or Node server
DNS-Rebinding SSRF + Browser Subresource Guard Dial-time RejectNonPublicPeer + loopback CONNECT SSRFProxy (MAP * ~NOTFOUND) Application URL blocklist Vendor-managed Optional / app-level App-level
Web Search (search tool) Bring your own Serper/Brave/Tavily keys, or SearXNG + edge-map + D1 FTS5 for $0 Proprietary / SearXNG Separate SaaS plans Bring-your-own s.jina.ai
OpenTelemetry (OTLP/HTTP + W3C traceparent) Stitched Hub → Go httptrace (dnsMs, connectMs, tlsMs, ttfbMs) to any collector Internal logs / BullMQ Vendor dashboard only Python logs Internal logs
Typical Cost (50k scrapes + 5k searches / mo)¹ $0 – $5 / mo (Cloudflare Hibernating DO + homelab + SearXNG) $25 – $80+ / mo (VPS + proxy GBs) $49 – $130+ / mo $10 – $40+ / mo (VPS + proxies) Credit-metered

¹ Cost assumptions checked Nov 20, 2026: compares Cloudflare Workers Free ($0) or Paid ($5/mo) within R2 (10 GB) and D1 (5 GB) monthly free tiers plus self-hosted SearXNG ($0, using existing home/VPS compute) against published Firecrawl Standard ($19–$99/mo) + Tavily Researcher ($30+/mo) plans.


Architecture & High-Availability Mesh

flowchart TD
    Admin["Browser (/admin Control Center)"] -->|"HMAC Session + CSRF"| HubW["Primary Hub (Cloudflare DO / Fly / Railway / Render)"]
    Claude["Claude Code / Cursor (/mcp)"] -->|"OAuth 2.1 PKCE or Bearer"| HubW
    Claude -.->|"Automatic HA Failover (baseUrls)"| GoHub["Standby Go Hub (--mode=hub,agent on VPS / Tailscale)"]
    HubW -->|"WSS /agent/ws (W3C traceparent)"| S1["Docker / VPS Fat Node (amd64)"]
    HubW -->|"WSS /agent/ws (W3C traceparent)"| S2["Mac Mini / RPi5 Node (arm64)"]
    HubW -->|"WSS /agent/ws (W3C traceparent)"| S3["Home Assistant OS Add-on"]
    GoHub -->|"Active-Active EDGEFETCH_HUB_URLS"| S1
    GoHub -->|"Active-Active EDGEFETCH_HUB_URLS"| S2
    HubW -.->|"Async OTLP/HTTP JSON"| OTel["Any OTel Collector (Tempo / Jaeger / Honeycomb / Axiom)"]
    S1 -.->|"Child httptrace Spans"| OTel
Loading

Deployment Modes in Detail

Option A: Standalone Worker Container (Linux, macOS, Windows, NAS, RPi, or VPS)

Multi-arch OCI manifests (linux/amd64 + linux/arm64) run as non-root 65532:65532:

# Full "Fat" Agent (Go HTTP/2 + SSRF-proxied Headless Chromium + SwiftShader + Poppler PDF):
docker run -d --name edgefetch-agent --restart=unless-stopped \
  --security-opt no-new-privileges:true \
  -e EDGEFETCH_HUB_URL=https://<your-hub-domain> \
  -e EDGEFETCH_AGENT_TOKEN="<YOUR_AGENT_TOKEN>" \
  -e EDGEFETCH_SITE_ID="node-1" \
  ghcr.io/bartrumb/edgefetch-agent:latest

# Or Ultra-Light ~9 MB Distroless Agent (HTTP/2 + Markdown + pure-Go PDF, ~15 MB RAM):
docker run -d --name edgefetch-lite --restart=unless-stopped --read-only \
  -e EDGEFETCH_HUB_URL=https://<your-hub-domain> \
  -e EDGEFETCH_AGENT_TOKEN="<YOUR_AGENT_TOKEN>" \
  -e EDGEFETCH_SITE_ID="lite-1" \
  ghcr.io/bartrumb/edgefetch-agent:lite --mode=agent

Option B: Desktop CLI, stdio MCP, Self-Hosted Go Hub & Bot Auth Keygen

Download a release archive from GitHub Releases or install with go install:

go install github.com/bartrumb/edgefetch/v2/cmd/edgefetch@latest

# Auto-configure zero-daemon, zero-port stdio MCP into Claude Desktop & Cursor:
edgefetch install-mcp

# Verify SSRF guard, loopback CONNECT proxy, OS Chrome/Edge & pdftotext, or scrape from terminal:
edgefetch doctor
edgefetch scrape https://example.com
edgefetch scrape --json --query="installation" https://developers.cloudflare.com/workers/

# Optional: mint an IETF RFC 9421 Ed25519 Web Bot Auth signing keypair:
edgefetch keygen-bot-auth

# Or run a 100% self-hosted Go WebSocket Hub + /v1/scrape + /mcp controller on a small VPS:
EDGEFETCH_CALLER_TOKEN="<32-char-secret>" EDGEFETCH_AGENT_TOKEN="<32-char-secret>" \
  edgefetch --mode=hub --addr=0.0.0.0:8080

Option C: Cloud Hub (Cloudflare Workers, Render, Railway, Fly.io) & Home Assistant OS

  1. Cloudflare Workers ($0 idle): Click Deploy to Cloudflare above (or cd hub && npm ci && npx wrangler deploy).
  2. Render / Railway / Fly.io (Single All-in-One Container + Persistent /data Volume): Use render.yaml, railway.json, or fly launch --copy-config (fly.toml).
  3. Home Assistant OS (amd64 & aarch64): Click Add Repository to Home Assistant above, install EdgeFetch, set hub_url, agent_token, and site_id, and click Start.

Feature & Extraction Highlights (v2.6, SCRAPE_PIPELINE_VERSION = 26)

  • 25+ Site & ATS Job-Board Extractors (hub/src/extractors/registry.ts, hub/src/jobs.ts):
    • ATS Job Boards & Structured jobs Tool: One-call job search & extraction across Greenhouse, Ashby (?includeCompensation=true), Lever, Workday CXS (GET & POST), and SmartRecruiters / Schema.org JobPosting JSON-LD, normalizing pay ({ min, max, currency, period, basis: "base"|"ote"|"total" }) and filtering out closed postings (contentState: "closed_posting").
    • Developer & Research Sites: 1-hop JSON/raw/patch upgrades for GitHub (raw, .patch, Issues/PRs, Gist), GitLab, Codeberg, Stack Overflow (✅ Accepted Answer), npm, crates.io, Hugging Face (YAML model cards → GFM tables), Discourse, Lobsters, Reddit, Bluesky (ATProto), Mastodon, DEV.to, pkg.go.dev, PyPI, Wikipedia, IETF RFCs, ArXiv, PubMed/PMC, Hacker News, and YouTube transcripts.
  • Data-Gathering & SPA Reliability:
    • Native CDP SPA Engine (internal/browser/renderer.go): Tracks real CDP Network.requestWillBeSent / loadingFinished XHR/Fetch networkidle, triggers IntersectionObserver lazy-load scroll passes, waits on a MutationObserver DOM quiescence check, and flattens open Web Component shadowRoot trees.
    • SSR Hydration Recovery (Next.js App Router self.__next_f.push RSC, __NEXT_DATA__, Nuxt 3 __NUXT_DATA__, Remix, Apollo): Recovers full prose from thin SPA shells in 1 HTTP hop before launching Chromium.
    • Optional Ethical Heavy-SPA Fallback (Crawl4AI & Firecrawl — Cloud or Self-Hosted): Configure CRAWL4AI_API_URL and/or FIRECRAWL_API_URL + FIRECRAWL_API_KEY so heavy SPAs (or Lite-only fleets without local Chromium) fall back automatically after verifying robots.txt — while never delegating robots.txt blocks, 401/403/429, or anti-bot challenges.
    • Conditional 304 Not Modified Revalidation: Sends stored ETag / Last-Modified validators from META D1 and serves verified R2 Markdown immediately on 304.
    • 7-Vendor WAF Catalog + 4 Anti-False-Positive Layers: Accurately classifies Cloudflare, Akamai, DataDome, PerimeterX, AWS WAF, Imperva, and Vercel/Fastly/Kasada challenges while stripping <pre><code> / <blockquote> and vetoing 200 OK articles (>= 250 words, JSON-LD, <nav>) so security/Turnstile docs never false-positive.
    • Opt-in waybackFallback: Recovers 5xx / DNS-dead link rot from the Internet Archive (never invoked on robots.txt, 403/429, or challenges).
    • Token Efficiency: -91.6% LLM token reduction on our offline 8-fixture synthetic regression corpus (bench/RESULTS.md), preserving <T> generics (Promise<void>), GFM tables, KaTeX math, .ipynb outputs, and OpenAPI 3.x schemas.

Tested & Supported Platforms Matrix

Status legend:

  • ✅ Daily-Driver + CI Verified: Running daily on the author's 3-node personal fleet (arm64, Home Assistant OS x86_64, and Windows VM) since Nov 18, 2026 and gated by automated E2E / smoke suites on every PR.
  • ✅ Live-Tested + CI Verified: Executed end-to-end on real host OS/runtime during verification and covered by automated CI tests.
  • 🔧 Build & Manifest Verified: Static CGO_ENABLED=0 cross-compilation (go build / GoReleaser) and deployment manifests verified cleanly; community hardware reports welcome!
Platform / Target Role(s) Binary / Image Footprint Capabilities Verification Status
Cloudflare Workers + Hibernating DO + D1 (FTS5) + R2 + Workflows Cloud Hub (/mcp, /admin, OAuth 2.1, HUB_SHARDS) Serverless bundle (0 npm runtime deps) 12-tool MCP, Search, FTS5, R2 Artifacts, RFC 9421, OTLP ✅ Daily-Driver + CI Verified (workerd H7 & S1 suites)
Home Assistant OS (amd64 x86_64 & aarch64 RPi / NUC / VM) Outbound Home Worker (edgefetch_fat) Multi-arch Alpine + Chromium + SwiftShader + Poppler HTTP/2, artifacts-v1, browser-v1, interact-v1, 256 MiB PDF ✅ Daily-Driver + CI Verified (x86_64 & arm64 home nodes)
Windows 11 (windows/amd64 Desktop & VM) Local stdio MCP, desktop Launchpad, CLI & Fleet Worker ~8 MB (edgefetch.exe, 0 DLL deps) Auto-detects native Windows Chrome / Edge / Brave for CDP JS render ✅ Daily-Driver + CI Verified (Windows VM node + CLI E2E)
Linux Docker Worker — Fat (linux/amd64, linux/arm64) Standalone Worker (Dockerfile.agent) Non-root 65532:65532 + AppArmor + Chromium WebGL Full HTTP/2 + SSRF-proxied Headless Chromium + Poppler PDF ✅ Live-Tested + CI Verified (live AppArmor + WebGL CI smoke)
Linux Distroless Worker/Server — Lite (linux/amd64, linux/arm64) Ultra-light Worker / Server / Go Hub (Dockerfile) ~8 MB static binary (~15 MiB RAM) HTTP/2, Markdown, pure-Go PDF, --mode=hub, RFC 9421, OTLP ✅ Live-Tested + CI Verified (buildx multi-arch CI)
All-in-One Container (Dockerfile.all-in-one / Docker Compose) Full TypeScript Hub + node:sqlite FTS5 + Go Chromium Worker Node 26 Alpine + Go + Chromium + Poppler (0 Redis, 0 Postgres) /admin, OAuth 2.1, /mcp, SQLite FTS5, /data artifacts, JS render ✅ Live-Tested + CI Verified (V2-9 E2E + buildx CI)
Linux (ubuntu/wsl2 linux/amd64) --mode=hub, --mode=hub,agent HA Mesh, CLI (scrape, doctor) ~8 MB static ELF Active-active EDGEFETCH_HUB_URLS, Go Hub /mcp, Fuzz*, -race ✅ Live-Tested + CI Verified (Go 1.27 -race + govulncheck)
WebAssembly / WASI (wasip1/wasm) Sandboxed Offline HTML → Markdown / JSON Extractor cmd/edgefetch-wasm (make build-wasm) Runs inside wasmtime, wazero, or Wasm sandboxes ✅ Live-Tested + CI Verified (go build + unit test)
Chrome / Firefox Manifest V3 Extension Active-Tab FTS5 Clipper (POST /v1/ingest-tab) extensions/browser-clipper/ Clips authenticated tabs ethically into private FTS5 collections ✅ Live-Tested + CI Verified (V2-11 E2E)
Deno Deploy + Turso HTTP + S3 SigV4 Serverless Hub Adapter hub/portable/deno-server.ts, s3-bucket.ts, turso-d1.ts Zero-dep WebCrypto SigV4 (S3/R2/Tigris/MinIO/B2) + Turso /v2/pipeline ✅ Live-Tested + CI Verified (V2-11 unit + tsc)
macOS Apple Silicon & Intel (darwin/arm64, darwin/amd64) Local stdio MCP, desktop Launchpad, CLI & Worker ~7.5–8 MB static binary Auto-detects /Applications/Google Chrome.app, Edge & Brave 🔧 Build & Manifest Verified (.goreleaser.yaml)
Windows on ARM (windows/arm64) Desktop CLI, stdio MCP & Worker ~7.5 MB (edgefetch.exe) HTTP/2, Markdown, pure-Go PDF, OS Edge/Chrome detection 🔧 Build & Manifest Verified (.goreleaser.yaml)
Fly.io / Railway / Render / Coolify 1-Click Cloud Self-Hosted All-in-One Hub + Worker fly.toml, railway.json, render.yaml Persistent /data SQLite FTS5 + artifacts + Headless Chromium 🔧 Build & Manifest Verified (runs Dockerfile.all-in-one)
Kubernetes (amd64 / arm64 Clusters) Zero-Trust Fleet Worker Pod deploy/k8s/agent-deployment.yaml Read-only rootfs, seccomp: RuntimeDefault, NetworkPolicy (ingress: []) 🔧 Build & Manifest Verified
Unraid, CasaOS, Portainer, Synology, Proxmox LXC & Termux Homelab / NAS / LXC / Spare-Phone Worker deploy/, compose.agent.yaml Outbound WSS node (zero port forwarding) 🔧 Build & Manifest Verified
OpenWrt (linux/mipsle, armv7) & FreeBSD (freebsd/amd64, arm64) Router / Firewall Appliance Lite Worker ~7.5–8.5 MB static binary HTTP/2, Markdown, pure-Go PDF (CGO_ENABLED=0) 🔧 Build & Manifest Verified (.goreleaser.yaml)

Reproducible Go Hot-Path Microbenchmarks

Run go test -bench=. -benchmem ./... locally:

Go Benchmark (go test -bench=. -benchmem ./...) Latency Heap Allocations
BenchmarkParseRetryAfter 4.45 ns/op 0 B/op (0 allocs/op)
BenchmarkRobotsMatch (* wildcards + $ anchor) 52.17 ns/op 0 B/op (0 allocs/op)
BenchmarkIsPublicIP (IPv4 + NAT64 IPv6 check) 155.50 ns/op 0 B/op (0 allocs/op)
BenchmarkClassifyFeedJSON 1,229 ns/op 1,416 B/op (28 allocs/op)

Documentation & Contributing

License

MIT

About

Self-hosted, MIT web scraping + search MCP server for AI agents: one Go binary, zero open ports, $0 idle on Cloudflare.

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

1 star

Watchers

0 watching

Forks

Releases

Sponsor this project

Packages

Used by

Contributors

Languages