fix(api-client): bound HTTP timeouts and stop retry log spam - #813
Merged
RapidPoseidon merged 1 commit intoAug 13, 2026
Merged
Conversation
Uploads could stall indefinitely and then bury the real failures under thousands of warning lines. Two independent causes, both in the REST client. Timeouts were never applied. `_build_timeout` returned `None` whenever no `_request_timeout` was passed — which is the default for every generated call — and httpx reads an explicit `None` as "wait forever" rather than "use the client default". Observed effect on a benchmark participant upload: single requests hanging 87s, 290s and 314s, each holding one of the 25 upload worker threads open, so throughput collapsed and the run looked like a dead network. The client now carries a bounded default timeout and unset requests resolve to `USE_CLIENT_DEFAULT`. `ReadError` was never retried. `_is_retryable_error` covered `ConnectError`, `RemoteProtocolError`, `ConnectTimeout` and `ReadTimeout`, but `ReadError` and `WriteError` are the `NetworkError` siblings of `ConnectError` — a local firewall or TLS-inspecting proxy aborting a live request surfaces as one of those. `ReadError` accounted for 98 of ~100 failures on one Windows upload run and bypassed the transport retry entirely. `ReadError`, `WriteError` and `PoolTimeout` are now retryable; retrying them carries the same already-accepted caveat as `ReadTimeout` (the server may have received the request), so this widens no risk class. Retry logging is now aggregated. Every retried attempt logged at WARNING, and the default level is WARNING, so a flaky link produced one line per attempt across thousands of concurrent requests. Attempts now log at DEBUG and feed a throttle that warns once on the first transient error and then at most once per 30s with a per-reason breakdown. Failures that never recover are unaffected. Also demoted two per-item error logs that duplicate an authoritative post-batch summary, and capped the benchmark upload failure report so a batch-wide failure prints five error blocks plus a count instead of one per sample. `rest.py` is generated, so `openapi/templates/rest.mustache` carries the same change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: lino <68745352+LinoGiger@users.noreply.github.com>
LinoGiger
marked this pull request as ready for review
August 13, 2026 11:59
LinoGiger
approved these changes
Aug 13, 2026
RapidPoseidon
deleted the
fix(api-client)/http-timeouts-and-retry-log-noise
branch
August 13, 2026 12:06
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Recurring reports of "connection errors" during benchmark participant uploads, most recently adding Grok Image Imagine 2.0 (Thinking) to
bmk_1Pet6iHzbf6RQu— on a healthy connection.The server was not at fault. During that upload window
Rapidata.Asset.APIlogged zero errors and everyPOST /participant/{id}/samplethat arrived returned 200. All the failures were client-side transport errors (ReadError [WinError 10053],ConnectTimeout [WinError 10060]), and two REST-client defects turned a brief local hiccup into a stalled run buried in warnings.What was wrong
1. Timeouts were never applied.
_build_timeoutreturnedNonewhen no_request_timeoutwas passed — the default for every generated call — and httpx treats an explicitNoneas no timeout, not use the client default. Only omitting the argument inherits the default:Effect in traces: single upload requests hanging 87s, 290s and 314s, each pinning one of the 25 upload worker threads.
2.
ReadErrorwas never retried._is_retryable_errorcoveredConnectError,RemoteProtocolError,ConnectTimeoutandReadTimeout. ButReadError/WriteErrorare theNetworkErrorsiblings ofConnectError— a local firewall or TLS-inspecting proxy aborting a live request surfaces as one of those, not as a timeout.ReadErrorwas 98 of ~100 failures on one run and skipped the transport retry entirely.3. Retry logging was per-attempt at WARNING. The default log level is WARNING, so every retried attempt printed a line — across thousands of concurrent requests, the failures that never recovered were buried under the ones that did.
What changed
None→ wait foreverUSE_CLIENT_DEFAULT→connect=15s read=120s write=120s pool=60sReadError/WriteError/PoolTimeout... and N moreRetrying
ReadError/WriteErrormeans the server may already have received the request — the same caveatReadTimeoutretries already accept, so this widens no risk class. The sample endpoint is idempotent in effect (the backend rejects a duplicate with 409, which the SDK treats as success).rest.pyis generated, soopenapi/templates/rest.mustachecarries the identical change (verified in sync).Verification
tests/api_client/test_rest_retry.py— timeout resolution, the retryable-error matrix, and the throttle's suppression behaviour._get_session_defaults()+_build_timeout(None)composition now raisesReadTimeoutand classifies it retryable, where it previously hung indefinitely.pyright src/rapidata/rapidata_client→ 0 errors.blackapplied.tests/rapidata_client/audience/pre-exist on the base branch (verified by stashing) and are untouched by this change.Follow-up, not in this PR
upload.maxWorkersdefaults to 25, which is what makes a filtering middlebox abort connections in the first place. Lowering the default is a behaviour change for every user, so it is worth deciding separately.🔗 Session: session-5ba57819