Skip to content

About

Open-source SHEIN scraper in Python. Export products, prices, discounts, ratings, reviews and stock to JSON or CSV.

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

10 stars

Watchers

0 watching

Forks

Latest commit

 

History

36 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

SHEIN Scraper — Product Listings, Prices and Ratings in Python

release tests canary python licence engines

SHEIN scraper — an open-source (MIT) Python tool that reads us.shein.com search, category and product pages and writes one row per product — title, brand, current and original price, discount, rating, review count and stock flags — to JSON or CSV. SHEIN sends suspicious visitors to its own /risk/challenge gate: measured on 2026-09-30 with fresh US Scraping Browser API profiles, 3 of 5 profiles passed it, each on its third round, with the puzzle solved through 2Captcha. Use it for price monitoring and catalogue research, or to compare two runs with the bundled diff tool.

Scrape SHEIN product listings into clean JSON or CSV. Give it a search term, a category or a product URL; get back one row per product: title, brand, current and original price, discount, rating, review count, stock and shipping flags, image and product link.

It is built for price monitoring and catalogue research on us.shein.com:

  • Real data, not scraped markup. Listings come from the page's own embedded product data (window.gbRawData), product pages from their schema.org JSON-LD, so prices and ids are the site's own values.
  • Gets past SHEIN's bot check. SHEIN's own /risk/challenge gate is handled automatically: the "I am human" checkbox, the 3x3 "select the images matching the icon" grid, and the "click the icons in order" puzzle, the last two solved through 2Captcha.
  • Honest results. A blocked, rate-limited or partial run says so in its exit code and a .meta.json file next to the output. A run that finds nothing never overwrites your last good data.
  • Three browser engines (Playwright, Selenium, Puppeteer) running one shared scraping loop, a local browser or 2Captcha's Scraping Browser API over CDP, rotating proxies, and a run-to-run diff tool.

Example output

A real row from a run on 2026-09-30, every column in the output's order:

{
  "sku": "458057728",
  "source": "shein.com",
  "category": "Women Mini Dresses",
  "title": "Aloruh Women's Solid Color Sleeveless Mini Dress, Suitable For Beach Vacation,Dresses For Women Summer",
  "brand": "Aloruh",
  "price": 13.03,
  "currency": "USD",
  "price_source": "embedded_json",
  "product_url": "https://us.shein.com/Aloruh-Women-s-Solid-Color-Sleeveless-Mini-Dress-Suitable-For-Beach-Vacation-Dresses-For-Women-Summer-p-458057728.html",
  "image_url": "https://img.ltwebstatic.com/v4/j/pi/2026/05/02/f8/17776852344d34295349aad8f43f4fbd32e5a672d8_thumbnail_405x552.jpg",
  "scraped_at": "2026-09-30T12:56:49Z",
  "original_price": 20.89,
  "discount_pct": 38.0,
  "rating": 4.62,
  "review_count": 1001,
  "store_code": "4534970445",
  "in_stock": true,
  "is_clearance": false,
  "quickship": false
}

Full sample: sample_output.json · 3 rows, collected 2026-09-30 (a listing run; all three in «Women Mini Dresses»).

Field Meaning
sku SHEIN's goods_id, the same across a product's colours and sizes
category SHEIN's category name for the product
price / currency what the customer pays now
original_price / discount_pct the "was" price and the discount; null when there is no discount
price_source where the price came from: embedded_json (listing data), json_ld (product page) or json_ld_min_variant (lowest variant price of a product page)
rating / review_count average rating and number of reviews; rating is null when there are no reviews
store_code the seller's store id
in_stock, is_clearance, quickship SHEIN's own flags

Requirements and access

Runtime Python 3.9 or newer (the offline suite runs on 3.9 and 3.12 in CI); Playwright with its Chromium (recommended), pyppeteer, or Selenium (local Chrome only), one per virtualenv
Coverage us.shein.com (prices in USD): searches, categories and product pages; listings from the page's embedded product data, product pages from their JSON-LD. Other SHEIN country sites are not tested. See Coverage and limitations
Required for the documented scenario Nothing to start: the Quick start runs a local headless Chromium with no key or proxy. But SHEIN sends such clients to its /risk gateway often — a datacentre address was rate-limited on 2026-11-18 — so for regular use the README recommends a 2Captcha Scraping Browser API profile (SHEIN_CDP_ENDPOINT) and a 2Captcha key (TWOCAPTCHA_KEY)
Optional: 2Captcha Scraping Browser API A profile that has passed SHEIN's challenge keeps its cookies; 3 of 5 fresh US profiles passed on 2026-09-30, each on its third round
Optional: captcha solving The image grid and the icon-order puzzle are solved through 2Captcha; capped at --max-solves per run (default 8, 0 never pays); the "I am human" checkbox is clicked for free
Optional: proxies, fingerprint, Scraper API SHEIN_PROXY / --proxy-file for a local browser; --fingerprint applies a 2Captcha Fingerprint API user agent; --scraper-api fetches one page without a browser
Measured Challenge pass rate: 2026-09-30; a datacentre run without a key or proxy: 2026-11-18 at fcfd0e6 (rate-limited, exit 3). See Verification

Quick start

git clone https://github.com/2scraper/shein-scraper.git
cd shein-scraper
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements-playwright.txt
playwright install chromium
cp .env.example .env        # add your keys here, never on the command line

python3 playwright_scraper.py --query "summer dress" --max-results 20

Results land in shein_results.json, with shein_results.json.meta.json beside it.

Recommended setup for regular use: a 2Captcha Scraping Browser API profile in SHEIN_CDP_ENDPOINT and a key in TWOCAPTCHA_KEY. SHEIN challenges or rate-limits plain local browsers quickly; a Scraping Browser profile that has passed the challenge once keeps its cookies and usually scrapes without further challenges.

If the run does not end with exit 0:

  • exit 3, blocked or rate-limited — SHEIN sent the browser to its /risk gateway. A rate limit (/risk/action/limit) has nothing to solve: wait at least five minutes, or pass --rate-limit-cooldown 300. A challenge (/risk/challenge) needs a 2Captcha key and usually a Scraping Browser profile. See Getting past SHEIN's bot check.
  • exit 4, no products — the page loaded but had none: check the query or category.
  • exit 5 or 6, not obtained or partial — 5: a remote service failed, the page never loaded, or SHEIN said products exist and none could be parsed; 6: rows were written but stop_reason says why the run did not finish. See Run results and exit codes.

Setup, .env and exit codes for the whole family: How to run the 2scraper scrapers.

Coverage and limitations

The short version: us.shein.com only; one infinitely-scrolling listing per run, verified up to about 20 products; --sort is recorded but not sent; the bot check is not always solvable.

Known limitations

  • us.shein.com only. Other SHEIN country sites use different locales and have not been tested.
  • Listings beyond the first ~20 products are unverified. The page embeds its first batch of about 20 products, and every live run so far asked for 20 or fewer. The scraper scrolls for more and stops when nothing new appears, but whether scrolling actually loads further products has not been confirmed. The run warns, and reports partial, when it collects fewer than SHEIN says exist.
  • --sort is not sent to SHEIN yet. Its URL parameter has not been confirmed.
  • Profiles wear out. A Scraping Browser endpoint's credentials eventually stop working; how long they last has not been measured for this repo, so treat an HTTP 401 (reported with a clear message) as the signal to fetch a fresh endpoint. After roughly ten runs in a row a profile may be rate-limited for a few minutes.
  • The bot check is not always solvable — see above.

Examples

# a search
python3 playwright_scraper.py --query "linen pants" --max-results 60

# a category, as it appears in shein.com's own URLs
python3 playwright_scraper.py --category "Women Jeans-c-1934.html" --format csv --out jeans.csv

# a single product page (read from its JSON-LD)
python3 playwright_scraper.py --url "https://us.shein.com/dsbayvkj-p-33704388.html"

# through the Scraping Browser API (endpoint taken from SHEIN_CDP_ENDPOINT in .env)
python3 playwright_scraper.py --query "summer dress" --max-results 20

# the same with the other engines
python3 puppeteer_scraper.py --query "summer dress"
python3 selenium_scraper.py  --query "summer dress"      # local browser only, see "Engines"

# compare two runs of the same search
python3 diff_runs.py monday.json tuesday.json

Verification

The dependable path needs a paid Scraping Browser API profile and a 2Captcha key, so it was not re-run in this pass; the last dated measurement of it is from 2026-09-30, below.

Date, commit Check Conditions Result
2026-11-18, fcfd0e6 Quick start install a fresh virtualenv, Python 3.13, requirements-playwright.txt (Playwright 1.65.2) installs; every flag used in the Quick start and Examples is in --help
2026-11-18, fcfd0e6 Quick start command with --max-solves 0 local headless Chromium, no proxy, no key, no .env, one Hetzner datacentre address redirected to SHEIN's /risk/action/limit rate-limit gateway, exit 3, nothing written, no solve bought
2026-11-18, fcfd0e6 Offline suite python3 smoke_test.py (108/108) and python3 .github/ci_checks.py, core requirements only passed
2026-09-30 SHEIN's challenge fresh US Scraping Browser API profiles, puzzles solved through 2Captcha 3 of 5 profiles passed, each on its third round; the other two were refused on every answer. sample_output.json is from that day
2026-10-19 to 2026-11-16 Canary, weekly: canary-live must return rows over the Scraping Browser API failing since 2026-10-19 because the SHEIN_CDP_ENDPOINT repository secret is not set, so it tests nothing about the site; last green run 2026-10-16
2026-10-19 to 2026-11-16 the same, canary-datacentre a GitHub runner, no proxy, no key refused through the /risk gateway (exit 3) on 2026-10-19, 10-26, 11-02 and 11-09; served on 2026-11-16 (15 rows, complete, exit 0)

Getting past SHEIN's bot check

SHEIN sends suspicious visitors to its own risk gateway instead of the page. The scraper recognises both gates:

  • /risk/challenge, a verification step. The scraper clicks the "I am human" checkbox (free). If SHEIN escalates, it screenshots the puzzle and sends it to 2Captcha: the image grid as a GridTask, the icon sequence as a CoordinatesTask. It clicks the answer and repeats with a fresh puzzle if SHEIN refuses it (--risk-challenge-rounds, default 5).
  • /risk/action/limit, a rate limit. There is nothing to solve. The run stops with exit 3 (no output is written unless you pass --allow-empty, in which case the .meta.json says stop_reason: rate_limited), or, with --rate-limit-cooldown 300, waits and retries once.

What to expect, measured on 2026-09-30 with fresh US Scraping Browser profiles: 3 of 5 profiles passed the challenge, each on its third round. SHEIN usually refuses the first correct answer or two. The other two profiles were refused on every answer, including correct ones: SHEIN had flagged the profile itself, and only a different profile helps. The run then exits 3 rather than returning bad data.

Paid solves are capped per run with --max-solves (default 8, 0 never pays). The count actually spent is recorded as solves_spent in the .meta.json.

Run results and exit codes

Every run writes <out> and <out>.meta.json (status, stop_reason, product count, total_results reported by SHEIN, whether --max-results capped it, solves spent, and a hash of the output file). A run that collects nothing writes neither, so it never replaces your previous good file.

Exit Meaning
0 complete
6 partial: rows were written, but the run did not finish cleanly — stop_reason says why (blocked, rate_limited, remote_api_error, failed_pages, rejected_rows)
3 blocked or rate-limited, no rows
4 the page loaded but had no products
5 the content was never obtained, no rows: a remote service failed (Scraping Browser, Scraper API), the page never loaded, or SHEIN stated the listing holds products and none could be parsed (stop_reason: parser_found_nothing — the page layout has probably changed)
2 bad usage
1 crash (a bug; please report it)

Comparing runs

diff_runs.py old.json new.json compares two complete runs by sku: products added, removed, price changed. It refuses comparisons that would mislead: two runs of different searches, markets or --sort, or a .meta.json that does not match its file. A currency difference is reported as currency_changed, never as a price change. When a run was capped by --max-results, a product missing from the new run is listed as left_selection (it dropped out of the top N), not as removed. --fail-on-change exits 1 on a real price change, for use in a pipeline.

Options

Same flags for all three engines. Credentials go in .env (TWOCAPTCHA_KEY, SHEIN_PROXY, SHEIN_CDP_ENDPOINT), never on the command line.

Option Default
--query / --category / --url SHEIN_URL what to scrape (a typed --url wins; it can be a search, category or product URL). SHEIN_URL from the environment or .env is used only when none of the three is typed
--max-results 30 products to collect
--format / --out json / shein_results.<format> output format and path
--max-scrolls / --stall-rounds / --scroll-delay 20 / 4 / 1.5s how far to scroll a listing, and when to stop if nothing new appears
--cdp-endpoint SHEIN_CDP_ENDPOINT connect to a Scraping Browser API profile instead of launching a browser
--proxy / --proxy-file / --proxy-shuffle SHEIN_PROXY one proxy or a rotating pool (http://login:pass@host:port or host:port:login:pass)
--solve-captcha when-blocked off / when-blocked / always
--risk-challenge-rounds 5 puzzle rounds per SHEIN challenge (0 = don't solve)
--max-solves 8 paid 2Captcha solves per run (0 = never pay)
--block-retries 2 retries on the same session after a block with no products
--rate-limit-cooldown 0 seconds to wait and retry once after a rate limit (e.g. 300)
--retries / --retry-delay / --delay-jitter 2 / 3s / 0.3 navigation retries, and randomised waits
--fingerprint / --fp-tags / --fp-country off apply a 2Captcha Fingerprint API user agent (local browsers only)
--scraper-api off fetch through 2Captcha's Scraper API instead of a browser (no scrolling: one page per run)
--scraper-api-cdp off route --scraper-api through a Scraping Browser profile (--scraper-api-country, --scraper-api-profile-id, --scraper-api-account-id)
--dump-html off save the page HTML (and puzzle screenshots) next to the output, for debugging
--allow-empty off write an output file even when nothing was found
--headless / --headful headless show the browser window
--sort relevance recorded in the results only; not yet sent to SHEIN

There is no --pages, --concurrency or --mode: a SHEIN listing is one infinitely-scrolling page with no per-page address, so --max-results, --max-scrolls and --stall-rounds take the place of a page count, and nothing can be handed to a second worker. The kind of page (search, category or product) follows from the URL.

Run any engine with --help for the full list.

Engines

  • Playwright (playwright_scraper.py) is the recommended engine, with local Chromium or the Scraping Browser API.
  • Puppeteer (puppeteer_scraper.py, via pyppeteer) supports the same two modes. pyppeteer itself is no longer maintained.
  • Selenium (selenium_scraper.py) runs a local Chrome only. chromedriver cannot authenticate a Scraping Browser endpoint, and its --proxy-server cannot use a proxy password (the credentials are stripped, with a warning).

Install one engine per virtualenv (requirements-playwright.txt, requirements-selenium.txt, requirements-puppeteer.txt). Their dependencies conflict with each other.

All three engines share one scraping loop (page_flow.py), so they agree on results, exit codes and when money is spent. Docker: docker build -t shein-scraper . gives an image with Playwright and Chromium.

Development

python3 smoke_test.py            # 108 offline checks, no network, no engine needed
python3 .github/ci_checks.py     # credential scan

CI runs the offline suite on Python 3.9 and 3.12, installs the built wheel outside the checkout, builds the Docker image and launches Chromium in it, and runs each engine in its own virtualenv. TESTING.md describes live testing with real credentials; CHANGELOG.md has the history.

FAQ

Does SHEIN have a public API?

This repository does not use one. It reads what us.shein.com's own pages carry: listings from the page's embedded product data (window.gbRawData), product pages from their schema.org JSON-LD, so prices and ids are the site's own values.

Is it legal to scrape us.shein.com?

This is not legal advice. Scraping public pages is permitted in many jurisdictions, but SHEIN's terms of use, copyright in product texts and images, and data-protection law differ between countries. The output is product and price data from public pages, and the scraper does not log in. Keep the request rate low — SHEIN rate-limits fast clients — and do not collect personal data without a lawful basis.

Why does SHEIN block headless browsers or datacentre IPs?

SHEIN sends suspicious visitors to its own /risk gateway: a challenge to solve, or a rate limit to wait out. It does not publish what it scores. Measured: a Hetzner datacentre address was rate-limited on 2026-11-18; a GitHub runner was refused four weeks running and served on the fifth (2026-11-16). A Scraping Browser profile that has passed the challenge keeps its cookies, and 3 of 5 fresh US profiles passed it on 2026-09-30.

How do I run the SHEIN scraper on a server?

Put a Scraping Browser API endpoint in SHEIN_CDP_ENDPOINT and a key in TWOCAPTCHA_KEY in .env, then run the Quick start command; headless is the default and no display is needed. docker build -t shein-scraper . gives an image with Playwright and Chromium. Cap paid solves with --max-solves, and branch on the exit code and on status in the .meta.json.

What fields does the scraper return?

19 columns per product, in the same order in JSON and CSV: sku, source, category, title, brand, price, currency, price_source, product_url, image_url, scraped_at, original_price, discount_pct, rating, review_count, store_code, in_stock, is_clearance, quickship. The table under Example output explains them.

Can it scrape other SHEIN country sites?

Not as tested: this repository reads us.shein.com. Other SHEIN country sites use different locales and have not been tested.

How often does it break when SHEIN changes?

A weekly canary is meant to check that SHEIN still serves rows over the Scraping Browser API, but that job has had no endpoint secret since 2026-10-19 (see Verification), so it is not watching the site right now. When SHEIN states that a listing holds products and none can be parsed — the page layout has probably changed — the run ends with exit 5 and stop_reason: parser_found_nothing, not as an empty result. Fixes are recorded in CHANGELOG.md.

Related scrapers

Same CLI conventions, same output contract (one row per item, JSON or CSV, a .meta.json sidecar and shared exit codes):

All scrapers: github.com/2scraper · Need another site? Open a request

Licence

MIT, see LICENSE.

About

Open-source SHEIN scraper in Python. Export products, prices, discounts, ratings, reviews and stock to JSON or CSV.

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

10 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages