Skip to content

fix(llm): honour the configured request timeout on generate - #240

Merged
wolfy-j merged 5 commits into
masterfrom
fix/llm-request-timeout
Nov 4, 2026
Merged

wolfy-j merged 5 commits into
masterfrom
fix/llm-request-timeout

Conversation

@wolfy-j

@wolfy-j wolfy-j commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Generate handlers set timeout = contract_args.timeout or 600, so a driver's configured timeout and its env variable (ANTHROPIC_TIMEOUT, BEDROCK_TIMEOUT, OPENAI_TIMEOUT, OPENAI_COMPAT_TIMEOUT) never applied to generate, only to structured output. Claude's manifest also declared 240 while the code used 600.

Changes

  • Claude, Bedrock, OpenAI and compat generate pass the caller's timeout through, so the client's configured value applies.
  • Order: options.timeout, route context.timeout, driver config or env, then 600 seconds.
  • Claude's declared default is 600, matching what runs.
  • transport.timeout_seconds parses every timeout: seconds, a numeric string, or a duration such as 20m through time.parse_duration. All drivers use it, including Google and typesafe.
  • deadline_ms compares numbers in the transport, so a duration string is capped correctly.
  • Every request sends its timeout as a millisecond duration, because the runtime reads a number as whole seconds. Sub-second timeouts and deadlines now hold.
  • The README states the order, the unit, and that the timeout covers a whole request including a stream, not idle time.

Behaviour change

  • An invalid timeout such as soon is now an invalid_request naming its source, where it used to fall back silently.
  • An app with one of those env variables set now gets that value on generate, which may be shorter or longer than the 600 it was actually getting. Nothing changes for an app that sets none.

Tests

src/llm/test/timeout_test.lua checks each precedence level at the HTTP boundary for generate and streaming on every driver. llm 2658 pass, 9 skipped, lint clean. Tests for the removed fallback cap moved to the transport. No live provider call.

Generate handlers forced a 600-second fallback, so a provider's timeout setting
and its env variable applied only to structured output. The client now resolves
one default, and the declared default matches it.
@wolfy-j
wolfy-j requested a review from skhaz November 4, 2026 21:28
KickSide added 4 commits November 4, 2026 16:58
Every driver accepts seconds or a duration string such as 20m through
transport.timeout_seconds, the deadline cap compares numbers, and an invalid
value is an invalid_request naming its source rather than a silent default.
An explicit empty setting stays empty, so an empty api_key cannot pick up a credential from the environment. Every HTTP request gets its timeout as a millisecond duration, because the runtime reads a number as whole seconds.
@wolfy-j
wolfy-j merged commit 14977b7 into master Nov 4, 2026
25 checks passed
@wolfy-j
wolfy-j deleted the fix/llm-request-timeout branch November 4, 2026 23:02
@wippy-releases wippy-releases Bot mentioned this pull request Nov 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants