Skip to content

fix(api-client): bound HTTP timeouts and stop retry log spam - #813

Merged
RapidPoseidon merged 1 commit into
mainfrom
fix(api-client)/http-timeouts-and-retry-log-noise
Aug 13, 2026
Merged

fix(api-client): bound HTTP timeouts and stop retry log spam#813
RapidPoseidon merged 1 commit into
mainfrom
fix(api-client)/http-timeouts-and-retry-log-noise

Conversation

@RapidPoseidon

Copy link
Copy Markdown
Contributor

Why

Recurring reports of "connection errors" during benchmark participant uploads, most recently adding Grok Image Imagine 2.0 (Thinking) to bmk_1Pet6iHzbf6RQu — on a healthy connection.

The server was not at fault. During that upload window Rapidata.Asset.API logged zero errors and every POST /participant/{id}/sample that arrived returned 200. All the failures were client-side transport errors (ReadError [WinError 10053], ConnectTimeout [WinError 10060]), and two REST-client defects turned a brief local hiccup into a stalled run buried in warnings.

What was wrong

1. Timeouts were never applied. _build_timeout returned None when no _request_timeout was passed — the default for every generated call — and httpx treats an explicit None as no timeout, not use the client default. Only omitting the argument inherits the default:

omit timeout kwarg       -> raised ReadTimeout after 2.0s
explicit timeout=None    -> still hanging when killed at 20s

Effect in traces: single upload requests hanging 87s, 290s and 314s, each pinning one of the 25 upload worker threads.

2. ReadError was never retried. _is_retryable_error covered ConnectError, RemoteProtocolError, ConnectTimeout and ReadTimeout. But ReadError/WriteError are the NetworkError siblings of ConnectError — a local firewall or TLS-inspecting proxy aborting a live request surfaces as one of those, not as a timeout. ReadError was 98 of ~100 failures on one run and skipped the transport retry entirely.

3. Retry logging was per-attempt at WARNING. The default log level is WARNING, so every retried attempt printed a line — across thousands of concurrent requests, the failures that never recovered were buried under the ones that did.

What changed

Before After
Unset request timeout None → wait forever USE_CLIENT_DEFAULTconnect=15s read=120s write=120s pool=60s
ReadError / WriteError / PoolTimeout not retried retried
Per-attempt retry log one WARNING each DEBUG + one throttled WARNING per 30s with a per-reason breakdown
Per-sample upload failure ERROR (even when the sweep recovered it) INFO — the post-sweep summary stays authoritative
Batch failure report one error block per failed sample first 5 + ... and N more

Retrying ReadError/WriteError means the server may already have received the request — the same caveat ReadTimeout retries already accept, so this widens no risk class. The sample endpoint is idempotent in effect (the backend rejects a duplicate with 409, which the SDK treats as success).

rest.py is generated, so openapi/templates/rest.mustache carries the identical change (verified in sync).

Verification

  • 16 new tests in tests/api_client/test_rest_retry.py — timeout resolution, the retryable-error matrix, and the throttle's suppression behaviour.
  • End-to-end against a socket that accepts and never replies: the real _get_session_defaults() + _build_timeout(None) composition now raises ReadTimeout and classifies it retryable, where it previously hung indefinitely.
  • pyright src/rapidata/rapidata_client → 0 errors. black applied.
  • Full suite: 102 passed. The 4 failures in tests/rapidata_client/audience/ pre-exist on the base branch (verified by stashing) and are untouched by this change.

Follow-up, not in this PR

upload.maxWorkers defaults to 25, which is what makes a filtering middlebox abort connections in the first place. Lowering the default is a behaviour change for every user, so it is worth deciding separately.


🔗 Session: session-5ba57819

Uploads could stall indefinitely and then bury the real failures under
thousands of warning lines. Two independent causes, both in the REST client.

Timeouts were never applied. `_build_timeout` returned `None` whenever no
`_request_timeout` was passed — which is the default for every generated call —
and httpx reads an explicit `None` as "wait forever" rather than "use the
client default". Observed effect on a benchmark participant upload: single
requests hanging 87s, 290s and 314s, each holding one of the 25 upload worker
threads open, so throughput collapsed and the run looked like a dead network.
The client now carries a bounded default timeout and unset requests resolve to
`USE_CLIENT_DEFAULT`.

`ReadError` was never retried. `_is_retryable_error` covered `ConnectError`,
`RemoteProtocolError`, `ConnectTimeout` and `ReadTimeout`, but `ReadError` and
`WriteError` are the `NetworkError` siblings of `ConnectError` — a local
firewall or TLS-inspecting proxy aborting a live request surfaces as one of
those. `ReadError` accounted for 98 of ~100 failures on one Windows upload run
and bypassed the transport retry entirely. `ReadError`, `WriteError` and
`PoolTimeout` are now retryable; retrying them carries the same
already-accepted caveat as `ReadTimeout` (the server may have received the
request), so this widens no risk class.

Retry logging is now aggregated. Every retried attempt logged at WARNING, and
the default level is WARNING, so a flaky link produced one line per attempt
across thousands of concurrent requests. Attempts now log at DEBUG and feed a
throttle that warns once on the first transient error and then at most once per
30s with a per-reason breakdown. Failures that never recover are unaffected.

Also demoted two per-item error logs that duplicate an authoritative
post-batch summary, and capped the benchmark upload failure report so a
batch-wide failure prints five error blocks plus a count instead of one per
sample.

`rest.py` is generated, so `openapi/templates/rest.mustache` carries the same
change.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: lino <68745352+LinoGiger@users.noreply.github.com>
@LinoGiger
LinoGiger marked this pull request as ready for review August 13, 2026 11:59
@LinoGiger
LinoGiger self-requested a review as a code owner August 13, 2026 11:59
@RapidPoseidon
RapidPoseidon merged commit 5968c9c into main Aug 13, 2026
2 checks passed
@RapidPoseidon
RapidPoseidon deleted the fix(api-client)/http-timeouts-and-retry-log-noise branch August 13, 2026 12:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants