Skip to content

release(v2.6.2): Round 5 security closure, YAML site adapters, vetted Hub packages & FSL-1.1-MIT Hub license - #50

Merged
bartrumb merged 2 commits into
mainfrom
feat/adapter-devex-and-vetted-deps
Nov 21, 2026
Merged

bartrumb merged 2 commits into
mainfrom
feat/adapter-devex-and-vetted-deps

Conversation

@bartrumb

Copy link
Copy Markdown
Owner

Summary (v2.6.2 — SCRAPE_PIPELINE_VERSION = 29)

1. Round 5 Security & Functional Audit Closure (R5-H1..R5-H4, R5-M1..R5-M8 + Lows)

  • catalog memory, concurrency & pacing bounds (R5-H1, R5-M4): Forwards maxBytes: 16_777_216 through Hub fetch wrappers, enforces a 16 MiB per-run in-memory retention cap (32 MiB global LRU across 8 runs while streaming full NDJSON to R2), caps concurrent async runs at 2 per principal (429 catalog_busy), and enforces a 120s per-run sleep budget.
  • Linear Shopify image dedup & fence headers (R5-H2, R5-H3): Replaces quadratic allImageUrls.includes with a Set (<= 250 images/product) and bounds code-fence header cleaning at 512 chars.
  • CommonMark multi-line backticks, indented fences & residual javascript: (R5-H4, R5-M2, R5-M3): Tracks cross-line unmatched backtick runs (computeTickPoisonedLines), closes list-indented fences on dedent, strips residual nested-bracket & blockquote-reference javascript: links, strips entry NUL bytes before sentinel stashing, and sanitizes twin Markdown in /v1/extract.
  • Vetted Hub packages (R5-M1, R5-M5): Executes user regexes via Google RE2 (re2js), swaps hand-rolled RFC 6455 WebSocket parsing to ws, parses complex HTML adapters via htmlparser2 (parseHtmlDoc), and parses YAML manifests via yaml (maxAliasCount: 0, merge: false) + per-IP in-flight request slot limits & Host OAuth issuer hardening in hub/portable/*.
  • Go renderer, agent & CI hardening (R5-M6..R5-M8 + Lows): Launches Linux/macOS Chromium via --remote-debugging-pipe (0 listening TCP sockets on Chromium, 0700 userDataDir), updates initialOrigin on resumed navigate actions, strips caller headers after cross-origin 302 redirects, hoists dispatchCh across WebSocket reconnects, scrubs strconv.Quote'd URLs in error messages, preserves exact-1 MiB /robots.txt final rules, switches vouch.yml to safe pull_request_target, and gates release container tag jobs on environment: release.

2. Shared Go + TypeScript Declarative YAML Site Adapters & Authoring Workbench

  • Converts 10 site adapters (rubygems, packagist, hex-pm, dev-to, huggingface, lobsters, doi-crossref, usaspending, polymarket, openrouter) to #-commented YAML manifests in internal/extract/manifests/*.yaml.
  • Embeds manifests/*.yaml into edgefetch --mode=hub via //go:embed (0 Go dependencies added — go.mod remains strictly coder/websocket only) and exposes GET /v1/extractors + MCP find_extractors in pure Go.
  • Adds npm run adapter:test [<id>] [<https://url>] [--record] (390 adversarial bulkhead fuzz cases) and .yaml-by-default npm run adapter:new + docs/ADAPTERS.md.

3. Dual-Boundary SaaS-Protective Hub Licensing

  • Keeps cmd/, internal/, sdk/, and extensions/ under MIT (LICENSE), while licensing hub/ under FSL-1.1-MIT (hub/LICENSE — free to self-host personally or internally; prohibits third parties from offering hub/ as a competing commercial hosted SaaS; converts to MIT after 2 years).

… Hub packages & FSL-1.1-MIT Hub license

Signed-off-by: Brett Bartrum <17114576+bartrumb@users.noreply.github.com>
… crash & run vouch check on vouch.yml PRs

Signed-off-by: Brett Bartrum <17114576+bartrumb@users.noreply.github.com>
@bartrumb
bartrumb merged commit 2f8a7cf into main Nov 21, 2026
4 checks passed
@bartrumb
bartrumb deleted the feat/adapter-devex-and-vetted-deps branch November 21, 2026 17:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant