Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,11 @@ WELLFOUND_CDP_ENDPOINT=ws://{login}-zone-scraping_browser-country-us-pid-{profil
# user:pass URL is doing something.
WELLFOUND_PROXY=http://{user}:{password}@na.proxy.2captcha.com:2334

# Through a proxy, images, video/audio and fonts are not downloaded: the
# proxy is billed by traffic and nothing this scraper reads is in them.
# Uncomment to load them anyway (same as --load-media).
# WELLFOUND_LOAD_MEDIA=1

# --- Default target -------------------------------------------------------
# Saves retyping --url. Wellfound serves ONE host in ONE language: there is
# no per-country TLD, no locale segment, and the exit country changes nothing
Expand Down
25 changes: 25 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,31 @@ does it leads the release notes rather than being discovered from a bill.

## [Unreleased]

> **Behaviour changes.** (1) A run through a proxy now makes one pre-check
> request before starting a browser. A proxy that answers **HTTP 407** or
> **402** ends the run with **exit 5** and
> `<out>.latest_attempt.meta.json` (`status: failed`, `stop_reason:
> proxy_auth_failed` / `proxy_balance_exhausted`) instead of timing out on
> every retry. (2) Through a local proxy, images, video/audio and fonts are
> **no longer downloaded**; `--load-media` (or `WELLFOUND_LOAD_MEDIA=1`)
> restores them. Runs without a proxy or over `--cdp-endpoint` are unchanged.

### Added

- `proxy_check.py`: the proxy pre-check (one CONNECT plus `HEAD /robots.txt`
on the target host, no third-party service) and media blocking for all
three engines. The 407 message names `WELLFOUND_PROXY` and says the
gateway refused the credential (or the traffic balance is exhausted); a
402 says the proxy traffic/balance is exhausted. Neither prints the
credential. A proxy that does not answer keeps the existing retry
behaviour.
- `--load-media` on all three engines, and `WELLFOUND_LOAD_MEDIA` (commented
out) in `.env.example`.
- Offline checks: a stub proxy that answers 407/402/200 or nothing, each
engine driven to exit 5 without a browser on a 407, the attempt sidecar,
the Playwright route handler, the pyppeteer Fetch patterns, the Selenium
blocked-URL list, and when each engine installs blocking.

### Changed

- CI only: the whole canary, free and paid jobs alike, now runs weekly (Mondays 05:40 UTC) and on a manual dispatch; the daily schedule is removed.
Expand Down
2 changes: 1 addition & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ RUN pip install --no-cache-dir -r requirements.txt -r requirements-playwright.tx
# container that nothing in the repo would have noticed.
COPY captcha_solver.py env_config.py fingerprint_client.py output_writer.py \
page_flow.py playwright_scraper.py product_parser.py proxy_pool.py \
diff_runs.py ./
proxy_check.py diff_runs.py ./

ENTRYPOINT ["python3", "playwright_scraper.py"]
CMD ["--help"]
36 changes: 34 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -265,6 +265,34 @@ both directions.
unsatisfiable pins, and pyppeteer and selenium collide on `urllib3`. Use a
virtualenv per engine if you need more than one.

### Proxy traffic savers

A proxy is billed by traffic, so a run through `--proxy` / `--proxy-file` /
`WELLFOUND_PROXY` does two things to avoid paying for bytes it does not use:

- **One pre-check before any browser starts.** A CONNECT through the proxy
and a `HEAD /robots.txt` on the target host. Chromium reports a proxy's
HTTP 407 as a navigation timeout, so a wrong or exhausted credential used
to spend every retry before ending as a load failure. Now a **407** (the
gateway refused the credential, or the traffic balance is exhausted) or a
**402** (proxy traffic/balance exhausted) ends the run in seconds with
**exit 5** and `<out>.latest_attempt.meta.json` saying `status: failed`,
`stop_reason: proxy_auth_failed` (or `proxy_balance_exhausted`). Any
previous output and its `.meta.json` are left as they were. The message
names the variable and never prints the credential. If the proxy does not
answer at all, the run goes on as before and the browser makes its own
attempt. No pre-check runs without a proxy or with `--cdp-endpoint`.
- **Images, video/audio and web fonts are not downloaded** through a local
proxy: all three engines abort those requests (Playwright by resource type,
pyppeteer through CDP's Fetch domain, Selenium by file extension with CDP
`Network.setBlockedURLs`). Every row comes from JSON the server embeds in
the page, never from an image or a font, and Cloudflare's challenge is
scripts and an iframe, which are not blocked. This was decided from what
the parser reads: on 2026-10-20 a free run from a datacentre was refused by
Cloudflare before any listing was served, so a loaded-vs-blocked comparison
of live rows was not possible. `--load-media` (or `WELLFOUND_LOAD_MEDIA=1`)
loads them. Without a proxy, or over `--cdp-endpoint`, nothing is blocked.

---

## Pagination, and the trap in it
Expand Down Expand Up @@ -330,7 +358,9 @@ one page five times.
## Output contract

Exit codes: `0` ok · `1` crash · `2` bad usage · `3` blocked · `4` zero jobs
· `5` remote API error · `6` partial.
· `5` content not obtained (a remote API error, a load failure, or a proxy
that answered 407/402 to the pre-check: `stop_reason: proxy_auth_failed` /
`proxy_balance_exhausted`) · `6` partial.

* **A run that finds nothing writes nothing** — last night's good output is
never replaced with `[]`.
Expand All @@ -339,7 +369,9 @@ Exit codes: `0` ok · `1` crash · `2` bad usage · `3` blocked · `4` zero jobs
by number.
* **A failed run writes no sidecar**, so a `"failed"` file never sits beside
good data (the one exception is below, and only where no earlier output
exists).
exists). A run the proxy pre-check stopped writes
`<out>.latest_attempt.meta.json` instead, a separate file, and leaves
`<out>.meta.json` alone.
* **`--allow-empty` is for a genuine empty result only.** With it, a run
that finished and found nothing writes an empty JSON (`[]`), a CSV holding
only its header, and a sidecar with `status: empty` and its `stop_reason`,
Expand Down
2 changes: 2 additions & 0 deletions env_config.py
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,8 @@
"WELLFOUND_CDP_ENDPOINT": "cdp_endpoint",
"WELLFOUND_PROXY": "proxy",
"WELLFOUND_URL": "url",
# "1" keeps images/video/fonts loading through a proxy (--load-media).
"WELLFOUND_LOAD_MEDIA": "load_media",
}
# Deliberately NOT here: an output prefix. `--out` already carries a non-empty
# default, so `apply()` would never see it as unset and the variable would be
Expand Down
29 changes: 29 additions & 0 deletions output_writer.py
Original file line number Diff line number Diff line change
Expand Up @@ -471,6 +471,35 @@ def write_run_meta(out_prefix: str, meta: dict) -> str:
return path


ATTEMPT_META_SUFFIX = ".latest_attempt.meta.json"


def finish_without_fetch(out_prefix: str, *, stop_reason: str,
pages_requested: int, start_url: str,
mode: str = "listing", source: str = SOURCE_DEFAULT,
message: str = "") -> int:
"""End a run that was stopped before anything was fetched; exit 5.

Used by the proxy pre-check (a 407 or 402 from the proxy). The record
goes to `<out>.latest_attempt.meta.json`, NOT `<out>.meta.json`: the
latter describes the data file beside it, and a previous run's good
output is still there, untouched, so overwriting its metadata with
`status: failed` would make the pair contradict each other.
"""
meta = run_meta(status="failed", stop_reason=stop_reason,
pages_requested=pages_requested, pages_completed=0,
start_url=start_url, final_url=start_url, products=0,
mode=mode, source=source)
path = f"{out_prefix}{ATTEMPT_META_SUFFIX}"
with _atomic(path) as f:
json.dump(meta, f, ensure_ascii=False, indent=2)
print(f"[+] Wrote attempt metadata -> {path} (status=failed, "
f"stop_reason={stop_reason})")
print(f"[!] Failed run: nothing was fetched ({stop_reason}). "
f"{message} Any previous output and its .meta.json are untouched.")
return EXIT_FETCH_FAILED


def run_meta(status: str, stop_reason: str, pages_requested: int,
pages_completed: int, start_url: str, final_url: str,
products: int, pages_failed: Optional[List[int]] = None,
Expand Down
15 changes: 15 additions & 0 deletions playwright_scraper.py
Original file line number Diff line number Diff line change
Expand Up @@ -77,12 +77,14 @@
parse_listing_page, references_own_assets,
role_url, site_turnstile_sitekey)
from output_writer import (dedupe_by_key, finish_run, EXIT_API_ERROR,
finish_without_fetch,
SOURCE_DEFAULT)
import page_flow
from page_flow import MIN_CARD_MATCHES
from proxy_pool import (from_args as proxy_pool_from_args, to_playwright, mask,
ROTATE_MODES, ProxyError, ProxyPool)
import env_config
import proxy_check

logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
logger = logging.getLogger("playwright_scraper")
Expand Down Expand Up @@ -397,6 +399,8 @@ def _launch_local(pw, args, pool):
# static read, however careful — cannot produce a solvable task.
# It costs nothing on a page that never renders one.
context.add_init_script(TURNSTILE_INTERCEPT_JS)
if proxy_check.block_media(args, pool, remote=False):
proxy_check.install_playwright(context)
return browser, context, context.new_page()


Expand Down Expand Up @@ -1390,6 +1394,16 @@ def scrape(args) -> int:
"(~150-300MB each). Make sure the machine has the "
"memory for it.", concurrency, concurrency)

# One request through the proxy before a browser exists: a 407 or 402
# from the gateway ends the run here, in seconds, instead of surfacing
# as a navigation timeout after every retry.
refused = proxy_check.gate(pool, _target_url(args))
if refused:
return finish_without_fetch(
args.out, stop_reason=refused.stop_reason,
pages_requested=args.pages, start_url=_target_url(args),
mode=args.mode, source=SOURCE_DEFAULT)

with sync_playwright() as pw:
session = _BrowserSession(pw, args, pool,
remote=bool(args.cdp_endpoint)).open()
Expand Down Expand Up @@ -1736,6 +1750,7 @@ def parse_args():
help="reCAPTCHA v3 minimum score to request (0.3, 0.7 or 0.9 "
"— the API only accepts these three). Ignored for v2 "
"widgets.")
proxy_check.add_media_flags(p)
p.add_argument("--cdp-endpoint", default=None,
help="Connect to an already-running browser over CDP instead "
"of launching Playwright's bundled Chromium, e.g. "
Expand Down
Loading
Loading