Spacrawl

_{"Corvus corone near Canford Cliffs" by — CC BY 2.0, via Wikimedia Commons.}

An SPA crawler for creating embeddable JSON for retrieval augmented generation. It was built with Pinecone and local FAISS vector stores in mind.

Description

This is a domain-scoped, path-bounded, JS-rendering doc crawler built with Playwright. Perfect for:

• LLM/RAG knowledge base: output is JSON with per-page Markdown + extracted code blocks, perfect for chunking and embedding.

• Documentation sites with client-side rendering (Mintlify, Docusaurus, GitBook, Nextra, ReadTheDocs SPA) where requests + BeautifulSoup returns a nothingburger.

• Bounded BFS crawls: when you want every page under /a/given/section/ and nothing else. Configurable allowlists make it safe to point at a docs site without scraping the blog, changelog, marketing pages, etc.

• Structured extraction: the DocPage dataclass is the shape; swap html_to_markdown for a domain-specific extractor and the crawl machinery stays the same.

• Polite scraping: robots.txt, per-request delay, retries with backoff, real UA. It won't get you rate-limited on a moderately-sized docs site.

It's not good for: paginated APIs, sites that require authentication, doomscroll feeds, or huge sites.

Spacrawl crawls a documentation site, extracts each page as Markdown plus structured code blocks. These blocks are immediately upsertable into a vector store, however the project comes with a sectioner.py module which splits those pages into heading-bounded sections sized for retrieval.

The sectioner.py module that improves retrieval by sectioning pages into smaller pieces for more accurate retrieval.

This project was built by using LangChain's Python docs (Mintlify) as the baseline, which render client-side and would cause requests + BeautifulSoup to return virtually nothing. It should work on any site with a stable content root like Docusaurus, GitBook, Nextra, ReadTheDocs SPA, etc. but has not yet been tested.

The Two Main Pieces

1. Crawl Playwright renders each page. The crawler extracts the main content as Markdown, isolates code blocks with language tags, and follows in-scope links until a page cap is reached. Outputs one JSON record per page.

2. Section Splits each page's Markdown at heading boundaries, merges undersized sections into their neighbors, and attaches page-level metadata. Outputs one JSON record per section.

Installation

Requires Python 3.10+ and Poetry.

poetry install
poetry run playwright install chromium
poetry run spacrawl

The second command downloads the Chromium binary Playwright drives. It is separate from the pip install and cached under ~/.cache/ms-playwright/

Configuration

Both stages read from a .env file in the project root. Copy .env.example and fill it in.

# Base URL for the docs site
BASE_URL=https://docs.langchain.com

# Where the crawl starts
START_URL=https://docs.langchain.com/oss/python/langchain/overview

# Only URLs on this domain are crawled
ALLOWED_DOMAIN=docs.langchain.com

# Comma-separated path prefixes. Everything outside these is ignored.
# Extend as needed, e.g. "/oss/python/,/oss/javascript/"
ALLOWED_PATH_PREFIXES=/oss/python/

# Default page cap. Override per-run with --max-pages.
MAX_PAGES=100

BASE_URL, START_URL, ALLOWED_DOMAIN, and ALLOWED_PATH_PREFIXES are required. MAX_PAGES defaults to 100.

Scoping the Crawl

START_URL and ALLOWED_PATH_PREFIXES work together. The crawler begins at START_URL and follows links, but only enqueues URLs whose path starts with one of the prefixes.

To crawl a single section, set both to that section:

START_URL=https://docs.langchain.com/oss/python/langchain/agents/overview
ALLOWED_PATH_PREFIXES=/oss/python/langchain/agents/

To crawl multiple sections, list them all and pick whichever one you want to start from:

START_URL=https://docs.langchain.com/oss/python/langchain/overview
ALLOWED_PATH_PREFIXES=/oss/python/langchain/,/oss/python/deepagents/

The crawl will cross between sections via any links it finds, as long as both are in the allowlist.

Usage

1. Crawl

poetry run spacrawl --max-pages 20 --output spacrawl_output.json

Flag	Purpose
--max-pages N	Override MAX_PAGES from .env
--output PATH	Output file (default: spacrawl_output.json)
--start-url URL	Override START_URL from .env

The crawl honors robots.txt, delays 1s between requests per worker, retries transient failures with exponential backoff, and exits non-zero if any page errored. Ctrl+C saves partial results before exiting.

2. Section (optional)

poetry run python -m sectioner.py spacrawl_output.json --output sections.json

Flag	Purpose
--scope-code-blocks	Attach only code blocks whose body appears in the section's text
--output PATH	Output file (default: stdout)

The crawler output is upsertable as-is - one vector per page. The sectioner is a quality improvement, not a prerequisite. It splits each page at heading boundaries so retrieval returns the relevant slice instead of the whole page, and it trims the metadata so Pinecone doesn't reject records for exceeding its size limit.

Without --scope-code-blocks, every section on a page carries every code block from that page. This is the "code as metadata" design - one retrieval hit returns prose and its code samples together - but it makes metadata scale with code-sample count rather than section count. For code-heavy pages (Deep Agents has 17 code blocks), this can push individual records over Pinecone's 40 KB metadata budget. Pass --scope-code-blocks if you hit that limit.

Budget validation

The sectioner measures each record's serialized metadata against Pinecone's 40 KB per-record filterable metadata limit and logs a warning for any record that exceeds it. It does not fail the run - the output file is still written, and the warnings identify which records to fix.

Reference: https://docs.pinecone.io/reference/api/database-limits/operation-limits#metadata-filter-limits

FAISS has no equivalent limit - metadata lives in a sidecar file, not in the index - so the check is a no-op if you're targeting a local index.

Output formats

Crawl output

A JSON array of page records:

[
  {
    "url": "https://docs.langchain.com/oss/python/langchain/overview",
    "title": "LangChain overview",
    "content_markdown": "# LangChain overview\n\nLangChain provides `create_agent`...",
    "code_blocks": [
      {"language": "python", "code": "from langchain.agents import create_agent\n..."}
    ],
    "metadata": {
      "url": "https://docs.langchain.com/oss/python/langchain/overview",
      "description": "LangChain provides create_agent: a minimal, highly configurable agent harness.",
      "content_selector": "#content-area"
    },
    "error": null
  }
]

Sectioner output

A JSON array of section records:

[
  {
    "id": "https://docs.langchain.com/oss/python/langchain/overview#1",
    "text": "# LangChain overview\n\n## Create an agent\n\nThis example demonstrates...",
    "url": "https://docs.langchain.com/oss/python/langchain/overview",
    "title": "LangChain overview",
    "heading_path": ["Create an agent"],
    "code_blocks": [
      {"language": "python", "code": "from langchain.agents import create_agent\n..."}
    ],
    "description": "LangChain provides create_agent: a minimal, highly configurable agent harness.",
    "content_selector": "#content-area"
  }
]

Embed text. Everything else is metadata for the vector store.

Tuning

Crawler pacing lives in src/spacrawl/crawler.py:

• REQUEST_DELAY - seconds between requests per worker (default 1.0)
• MAX_CONCURRENT - parallel browser pages (default 3)
• MAX_RETRIES - retries per page on transient failure (default 2)

Content extraction assumes Mintlify's #content-area root. For other static site generators, change CONTENT_SELECTOR in crawler.py.

Section size is controlled by two constants at the top of src/spacrawl/sectioner.py:

• MAX_SECTION_CHARS (default 3000) - sections above this are split at H3 boundaries.
• MIN_SECTION_CHARS (default 400) - sections below this merge into their neighbor.

Known limitations

• Content extraction is set for Mintlify. Other static site generators need a selector change.

• Code appearing inside prose is preserved in content_markdown but not in code_blocks. code_blocks only holds block-level <pre> samples.

• The --scope-code-blocks heuristic matches code blocks by string containment of their body in the section text. Duplicated code samples across tabs will attach to whichever section contains them first.

• No resume support. For crawls over a few hundred pages, run in the background and checkpoint manually.

---

---

Generalization roadmap

Spacrawl's crawler is made for LangChain's Mintlify documentation pages. The following are site-specific:

1. CONTENT_SELECTOR - the wrapper around the main content
2. _UI_NOISE_PATTERNS - regexes that strip Mintlify chrome
3. TITLE_SELECTOR - currently h1
4. SELECTOR_TIMEOUT_MS - 8s, tuned to Mintlify's hydration time

Everything else is already site-agnostic. The language detector walks the DOM for Shiki/Prism/highlight.js markers, the Markdown renderer handles standard HTML, and the crawl queue, URL filtering, and sectioner have no SSG coupling at all.

The possible approach: named profiles

Move the four site-specific values into a profiles.py module with one entry per supported SSG, selected by a single .env variable:

PROFILE=mintlify

Each profile would carry its selector, noise patterns, and timeout. The crawler would read the profile instead of module-level constants. Roughly 60 lines of net change; no logic rewrite.

Sketch of the shape (not yet implemented):

@dataclass(frozen=True)
class Profile:
    name: str
    content_selector: str
    title_selector: str = "h1"
    noise_patterns: tuple[re.Pattern, ...] = ()
    selector_timeout_ms: int = 8000

PROFILES = {
    "mintlify": Profile(
        name="mintlify",
        content_selector="#content-area",
        noise_patterns=(...),
    ),
    "docusaurus": Profile(
        name="docusaurus",
        content_selector="article.theme-doc-markdown",
    ),
    "mkdocs": Profile(
        name="mkdocs",
        content_selector=".md-content__inner",
    ),
    "generic": Profile(
        name="generic",
        content_selector="main article, article, main, [role=main]",
    ),
}

What is not planned

Auto-detection. An earlier iteration of this crawler tried a chain of candidate selectors and picked the first one with non-empty text. It matched a 150-character description stub instead of the 40,000- character content root, and produced near-empty output for every page. The fix was to use one stable selector.

Content-density scoring. Readability.js-style behavior works on the sites they were tuned against and fail silently elsewhere. Silent failure in RAG data is worse than the program screaming at me.

Caveat, TBH

Only the Mintlify profile has been verified against a real site. The Docusaurus, MkDocs, and generic entries sketched above are educated guesses based on those tools' common DOM structure. They have not been run, and I don't know of sites that I could run them against. Building the profile library the right way is empirical - one site per SSG, inspect the output, tune the profile, commit - not speculative.

What profiles wouldn't solve

• Auth-protected docs. Requires cookie or session injection. Different feature.
• Infinite scroll and paginated SPAs. Requires scroll-and-wait logic. Different feature.
• Anti-bot measures. Cloudflare, DataDome, and similar. No clean answer; out of scope.
• Sitemap-driven discovery. A crawl-strategy change, orthogonal to profiles. Worth adding separately if faster full-site coverage becomes a goal.

---

LICENSE