scrapeunblocker-haystack

Haystack integration for ScrapeUnblocker - fetch pages that block ordinary HTTP requests.

ScrapeUnblocker renders web pages in a real browser behind anti-bot protections such as Cloudflare, DataDome, PerimeterX and Akamai, so your pipeline gets the real content instead of a block page, a captcha, or an empty JavaScript shell.

Installation

pip install scrapeunblocker-haystack

Setup

Get an API key at scrapeunblocker.com and export it:

export SCRAPEUNBLOCKER_API_KEY=<your-api-key>

Both components read that variable by default, or accept a Secret explicitly.

Components

ScrapeUnblockerFetcher

Fetches URLs and returns one Document per page.

from scrapeunblocker_haystack import ScrapeUnblockerFetcher

fetcher = ScrapeUnblockerFetcher()
result = fetcher.run(urls=["https://example.com"])

print(result["documents"][0].content[:200])

Parameter	Default	Description
api_key	SCRAPEUNBLOCKER_API_KEY env var	ScrapeUnblocker API key
parsed_data	False	Return AI-parsed structured JSON instead of raw HTML
proxy_country	None	Two-letter country code of the country the request is made from
time_sleep	None	Seconds to wait after load before capturing
steps	None	Browser actions to run before the HTML is captured (see below)
list_elements	False	Return the parsed elements JSON instead of HTML (see below)
markdown	False	Return the page as clean Markdown instead of HTML (see below)
main_content_only	True	With markdown, keep only the main content (drop navigation, header, footer, sidebars)
base_url	https://api.scrapeunblocker.com	API base URL
timeout	180	HTTP timeout in seconds
raise_on_failure	False	Raise instead of skipping a URL that fails

By default a URL that cannot be fetched is logged and skipped, so one bad URL does not discard the rest of the batch.

Markdown for LLMs and RAG

Set markdown=True to get each page as clean Markdown instead of HTML: headings, lists, tables (GitHub-flavored), absolute links, images with their alt text and fenced code blocks, without scripts, styles, navigation, footers, forms or hidden elements. The Markdown goes straight into a prompt or a splitter, with no HTML converter, and is usually a small fraction of the HTML's size. It is billed like an HTML fetch.

from scrapeunblocker_haystack import ScrapeUnblockerFetcher

fetcher = ScrapeUnblockerFetcher(markdown=True)
result = fetcher.run(
    urls=["https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"]
)

doc = result["documents"][0]
print("# A Light in the Attic" in doc.content)  # True
print(doc.meta["content_type"])                 # text/markdown; charset=utf-8

By default only the page's main content is kept: navigation, the site header, footer and sidebars are dropped. Pass main_content_only=False to convert the whole page. markdown works with proxy_country, time_sleep and steps, but not with parsed_data or list_elements, which return their own JSON formats; combining them raises ValueError. A page that does not exist (404/410) is skipped or raises TargetNotFoundError, as below, with the site's own page as Markdown in body.

Browser steps

Pass steps to drive the page in a real browser before its HTML is captured - wait for content to appear, click, type, scroll, and so on. Each step is a dict with an action key plus its parameters:

from scrapeunblocker_haystack import ScrapeUnblockerFetcher

fetcher = ScrapeUnblockerFetcher(
    steps=[
        {"action": "wait_for", "selector": "#results"},
        {"action": "type", "selector": "input#q", "value": "laptops"},
        {"action": "press_key", "value": "Enter"},
        {"action": "wait_for_text", "value": "in stock"},
        {"action": "click", "selector": "button.load-more"},
        {"action": "scroll", "value": "bottom"},
    ]
)
result = fetcher.run(urls=["https://example.com/search"])

Available actions:

Action	Parameters
wait_for	selector, selector_type? (css|xPath|className|tagName), timeout_ms?
wait_for_text	value, timeout_ms?
wait	value (milliseconds)
click	selector, selector_type?, timeout_ms?
type	selector, selector_type?, value, clear?, timeout_ms? (human-like typing)
select	selector, selector_type?, value, timeout_ms?
press_key	value (Enter|Tab|Escape|Backspace|Delete|Space|Arrow*|Home|End|PageUp|PageDown)
scroll	value ("bottom" or an integer pixel offset)

Steps are non-idempotent. If a step fails, the fetch raises StepExecutionError (with step_index, action, selector and reason), which is logged and skipped by default, or re-raised when raise_on_failure=True.

Pages with nothing to parse

With parsed_data=True, a page that loads but holds no structured data still yields a Document. Its JSON carries "data_extracted": false and the rendered page in html, and meta["data_extracted"] is False, so you can route it to an HTML converter. The call is billed like a plain HTML fetch.

Pages that do not exist

If the site answers HTTP 404 or 410, there is no page to index: the URL is skipped with a warning, or TargetNotFoundError (with url, origin_status and the site's own body) is raised when raise_on_failure=True. That is the site's answer, not a block - the call is billed and a retry returns the same result.

List elements

Set list_elements=True to return the parsed elements JSON ({url, count, elements: [...]}) instead of the page HTML. The JSON is stored as the Document content, and the element count is available in meta["count"]:

fetcher = ScrapeUnblockerFetcher(list_elements=True)
result = fetcher.run(urls=["https://example.com"])

doc = result["documents"][0]
print(doc.meta["count"])          # number of elements
print(doc.content)                # {"url": ..., "count": ..., "elements": [...]}

ScrapeUnblockerWebSearch

Searches Google and returns the organic results as Document objects, with the snippet as content and title / link / position / displayed_url / domain / source (the site's name) / date / sitelinks / rating / rich_snippet (the facts line under the snippet) in the metadata.

from scrapeunblocker_haystack import ScrapeUnblockerWebSearch

search = ScrapeUnblockerWebSearch(top_k=5)
result = search.run(query="best web scraping api")

for doc in result["documents"]:
    print(doc.meta["title"], doc.meta["link"])

Parameter	Default	Description
api_key	SCRAPEUNBLOCKER_API_KEY env var	ScrapeUnblocker API key
pages_to_check	1	How many result pages to scrape
proxy_country	None	Two-letter country code for localised results
top_k	None	Keep at most this many results
full_urls	False	Wait until Google gives out every result's real URL (see below)
base_url	https://api.scrapeunblocker.com	API base URL
timeout	180	HTTP timeout in seconds

Every result's real URL

Google releases a results page's links a few at a time, the last ones up to about 40 seconds after the search. By default the search returns sooner, so most results carry their real URL in meta["link"], and a result whose link has not come yet has the site's home page there (or a Google redirect link when Google shows no site). Set full_urls=True when you need the exact page of every result, for example to fetch or cite them:

search = ScrapeUnblockerWebSearch(full_urls=True)

The call then takes up to about 40 seconds longer. It is billed the same.

In a pipeline

Fetch pages as Markdown and answer a question from them:

from haystack import Pipeline
from haystack.components.builders import ChatPromptBuilder
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.dataclasses import ChatMessage

from scrapeunblocker_haystack import ScrapeUnblockerFetcher

prompt = [
    ChatMessage.from_user(
        "Answer the question using the pages below.\n\n"
        "{% for doc in documents %}{{ doc.content }}\n{% endfor %}\n"
        "Question: {{ question }}"
    )
]

pipe = Pipeline()
pipe.add_component("fetcher", ScrapeUnblockerFetcher(markdown=True))
pipe.add_component("prompt_builder", ChatPromptBuilder(template=prompt, required_variables="*"))
pipe.add_component("llm", OpenAIChatGenerator())

pipe.connect("fetcher.documents", "prompt_builder.documents")
pipe.connect("prompt_builder.prompt", "llm.messages")

result = pipe.run(
    {
        "fetcher": {"urls": ["https://example.com"]},
        "prompt_builder": {"question": "What is this page about?"},
    }
)
print(result["llm"]["replies"][0].text)

Serialization

Both components implement to_dict() / from_dict(), so pipelines using them can be saved and reloaded. The API key is serialized as a Haystack Secret reference, not as its value.

Links

• ScrapeUnblocker: www.scrapeunblocker.com
• API documentation: docs.scrapeunblocker.com
• Haystack: https://haystack.deepset.ai

License

Apache-2.0