Claude/stress test concurrent users bc67c4 - #31
Merged
Conversation
Adds tooling to test the live system at ~20 simultaneous users, targeting the production Pages origin so requests pass through the Pages Function and its service binding — the path where Pyodide cold-start 500s appear. The Worker origin bypasses it and cannot reproduce them. - backend/scripts/seed_load_test_users.py seeds loadtest-NN@example.com accounts directly in D1. Going through /api/auth/register would take four hours at the 5/hour registration limit. Password comes from LOADTEST_PASSWORD, never the repo; generated SQL and session files are gitignored. - load-test/build-scenario.mjs derives the replayed request sequence from sessionStorage['studyplanner:api-request-log'], which the app already keeps, rather than hand-writing an assumed sequence. It excludes login, register, logout, feedback and client-errors, each of which is rate limited per IP or would break the session. - load-test/mint-sessions.mjs collects session cookies out of band, pacing to the fixed 15-minute login window. Logging in inside the test would hit the 10/15min limit at VU 11 and measure the limiter instead of the app. - load-test/scenario.js replays at 20 VUs with think time, failing the run on any 5xx or 429. - load-test/login-burst.js times 8 simultaneous logins (PBKDF2 at 310k iterations in Pyodide) with headroom under the limit. docs/load-test-2026-08.md records the one finding that needed no load run: enforce_rate_limit keys on the client IP, so 20 users behind one campus NAT share a 10-logins-per-15-minutes budget and users 11-20 get 429. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Phase 0 reconnaissance against production contradicted three assumptions the harness was built on. Findings and measurements are in docs/load-test-2026-08.md. The deployed frontend calls the Worker origin directly. VITE_API_BASE_URL is baked into the Pages build, so browsers never touch the same-origin /api/* Pages Function — the previous commit targeted the wrong host. Points scenario.js, mint-sessions.mjs and login-burst.js at studyplanner-api.*.workers.dev. A side effect worth recording: the caches.default catalog cache in functions/_shared/proxy.ts is therefore never exercised by web users. It still responds correctly when called directly, so this is dead weight rather than breakage. The catalog is fetched once per browser session, not per page view: responses carry s-maxage=900 and the frontend caches them in sessionStorage for 24h (shared/utils/sessionCache.ts, observed as a 1.43 MB entry). Replaying the full sequence every iteration would have invented backend load that does not exist, so build-scenario.mjs now splits a recording into firstLoad and steadyState and scenario.js runs firstLoad on __ITER === 0 only. That also matches the scenario under test more closely: twenty people opening the app at the start of a lecture is a burst of expensive session starts, not sustained traffic. Single-user latency, measured with no contention, dwarfs anything concurrency is likely to add: the 1.43 MB catalog took 12.7s cold and 3.3s warm, and /api/config takes 2.7s warm to return a single null field. Small sample, recorded as an order of magnitude for Phase B to replace with real percentiles. recorded-session.json now carries the real anonymous first-load steps and their observed timings; the authenticated steps remain assumed until a logged-in session can be recorded. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Session minting hit the failure the stress test was built to look for, before any concurrency was applied: HTTP 503 | Worker exceeded resource limits | ray=a275225218a7dc82-FRA It appeared on the 9th of a strictly sequential run of logins, and the immediate retry succeeded, so it depends on isolate state rather than on the request. The response is a Cloudflare interstitial, not an application error, so clients get no JSON body to act on. wrangler tail puts login at 421-538ms of CPU per request - PBKDF2 at 310,000 iterations under Pyodide, against single-digit milliseconds for a typical Worker request. Recorded in docs/load-test-2026-08.md along with the hypothesis that a cold isolate paying Pyodide init on top of that is what breaches the limit, and what confirming it would take. mint-sessions.mjs needed three fixes to get this far, all of which cost real evidence on the first attempt: - Cloudflare interstitials were truncated to 200 characters of HTML, hiding the error code and Ray ID. It now extracts both. - A single failure aborted the run before sessions.json was written, discarding four already-minted sessions. It now writes partial results, resumes from them, and reports failures at the end. - 5xx responses are retried with backoff, and a 429 waits for the next fixed window instead of being recorded as a failure. Retries consume rate-limit budget, which is itself worth knowing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
wrangler tail over the full minting run plus concurrent browser traffic captured 43 failing requests, and they are two separate bugs rather than one. The dominant mode, 20 of 43, is "Attempted to use PyProxy when Python GIL not held" - the known workerd Python-Workers defect. Its shape is the giveaway: median CPU of 2ms against a 2.5s wall time. These requests are not exhausting a budget, the Python event loop wedges (Exception in callback PyodideTask.task_wakeup), the request never completes, and Cloudflare kills it and reports exceededCpu. The exceededCpu outcome is a symptom, not the cause. It hits every endpoint and is not load-dependent. This corrects a prior assumption: the fault was believed to be specific to the Pages service-binding path. Every observation here is on direct workers.dev ingress, so switching ingress does not avoid it. The second mode is genuine CPU exhaustion from PBKDF2 at 310,000 iterations, 421-538ms of CPU per login. Real, but only 2 of 43 and confined to /api/auth/login. Two further findings recorded: - 24 of the 43 failures were POST /api/client-errors. When the Worker fails the browser reports each failure to the same wedged Worker, so a partial outage becomes a self-sustaining request storm. - Login 429s while registration keeps working because each policy has its own counter and no other endpoint is rate limited at all. That part is by design. The compounding part is not: enforce_rate_limit runs before authentication, so failed attempts count, and 5xx-driven retries lock users out for 15 minutes through no fault of their own. Phase D is superseded - the 500s reproduced with no concurrency at all. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The August 2026 concurrent-user test surfaced two backend defects and one non-fix. See docs/load-test-2026-08.md for the measurements. Login rate limiting punished the wrong people. Every policy keyed on sha256(CF-Connecting-IP), so twenty students behind one campus NAT shared a budget of 10 logins per 15 minutes and users 11-20 got 429. Worse, enforce_rate_limit ran before authentication, so a 5xx from a wedged isolate and every retry it provoked consumed the same budget as a wrong password: an outage locked users out for the rest of the window. auth_login is now keyed on the submitted account and split into a read (enforce_failed_attempt_limit) and a write (record_failed_attempt) that only fires on a genuine AuthenticationError. Successful logins and server-side failures cost nothing. Keying on the account would normally invite a targeted lockout; counting only failures is what removes that, because a user with the right password never touches the counter. The ceiling is raised to 500 per 15 minutes, and registration to 50 per hour - a deliberate call that frustrated users are the likelier harm here. Password hashing ran PBKDF2-HMAC-SHA256 at 310,000 iterations through hashlib, i.e. as interpreted Pyodide bytecode, costing 421-538 ms of CPU per login and failing outright with "Worker exceeded CPU time limit". password_hashing.py derives the same digest natively through crypto.subtle.deriveBits, with hashlib kept as a fallback so a broken fast path degrades into slow logins rather than failed ones. No hash migration is needed: same algorithm, salt and iteration count give byte-identical output, verified in the runtime rather than argued from the spec. Raising compatibility_date to 2026-04-01 was tried as a candidate fix for the Pyodide GIL fault (cloudflare/workerd#6624) and reverted - it failed 60/60 requests with a dedicated-snapshot error and flips entrypoint dispatch from on_fetch to fetch. That fault stays open upstream. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
24 of the 43 failures captured during the load test were POST /api/client-errors - the frontend's own error reporter, failing against the same wedged Worker it was reporting on. A partial outage became a self-sustaining request storm against the endpoint least able to absorb it. fetchJson now retries GET and HEAD up to three times on 5xx and transport failures. The Pyodide GIL fault (cloudflare/workerd#6624) wedges one request and serves the next normally, so most of it becomes latency the user never sees. Mutations are deliberately not retried: a POST that timed out may still have been applied. Every attempt is still written to the local request log, since the retries are themselves a signal, but only the final one is reported upstream - so a recovered blip generates no traffic at all. reportClientErrorToServer additionally caps reports at 10 per page load and drops 401 and 429 outright. Every anonymous visitor's session check is a 401 and a 429 is the limiter working as designed; both arrive in bursts and drown out the reports worth reading. It also no longer reads window.location unguarded. That runs inside fetchJson's failure path, where a ReferenceError would replace the ApiError callers expect. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds a Fixes section to the load-test report covering what changed, how each change was verified, and what remains open. Two corrections to the earlier findings. The first draft called the GIL fault "not load-dependent at all"; that was read off a run of sequential logins, but a browser was open against the same Worker throughout, so the run was never free of concurrency. Upstream describes it as needing 3-5 concurrent requests, which makes it a low-concurrency bug rather than a scale bug - below the threshold this test was built to probe. The second concerns the WebCrypto change: it adds an await to the login path, and the await boundary is exactly where the upstream race lives, so it was worth checking it does not make things worse. Twenty sequential logins on a cold isolate succeeded 3/20 with WebCrypto and 2/20 with hashlib forced - indistinguishable. That ~90% failure rate is a property of wrangler dev --remote preview isolates rather than production, and is recorded as a comparison between the two columns, not as a production figure. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The previous commit claimed login CPU was moved off hashlib onto native
WebCrypto. It was not. workerd rejects the call outright:
NotSupportedError: Pbkdf2 failed: iteration counts above 100000 are not
supported (requested 310000).
PASSWORD_PBKDF2_ITERATIONS is 310,000, so crypto.subtle.deriveBits threw on
every login, the fallback caught it, and hashing ran through hashlib exactly as
before. The change was inert.
The earlier verification proved the wrong thing. Its probe used 1,000
iterations, which is under the cap and therefore passed; it established that the
FFI plumbing works, not that it works at the count actually used. The
register/login round trip passed for the same reason - the fallback silently
made it work.
hash_password_hex now checks WEBCRYPTO_MAX_PBKDF2_ITERATIONS instead of
discovering the limit by exception once per isolate, so the path is explicitly
dormant. Two tests pin the constraint. Behaviour is unchanged.
Two further corrections. Pyodide's hashlib is compiled C in WASM at roughly 1.5x
native, not interpreted bytecode: the same work takes 313 ms as native OpenSSL,
421-538 ms in the Worker, and ~4,300 ms as a pure-Python loop. So WebCrypto was
worth about a third, not an order of magnitude. And reaching it would mean
dropping to <=100,000 iterations - a security decision that weakens hashing 3x
and needs a per-user iteration count with rehash-on-login, against a fault that
accounted for 2 of 43 observed failures. Not recommended.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ncurrent-users-bc67c4
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.