Repository navigation
Conversation
The default profile set no attempt bounds, so every attempt got the engine's 30 s timeout_ms. A route that never answered used 30 s of the 50 s request deadline, and the next candidate had only 20 s before the caller received a timeout. Set first_token_timeout_ms to 15 s on the default profile, matching how the agent profile bounds attempts. A silent route now falls through after 15 s and the next candidate keeps its full 30 s budget. A slow generation that is already streaming is not affected.
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars.
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Callers on the default profile (the OpenAI-compatible endpoint with no policy, e.g.
studio-prd,studio-rally-prd, Deckard) intermittently get504s after exactly 50 s. On 2026-10-28, from about 07:30 UTC, devexp-devllm-router-apireturned 25 of these out of about 7,300 chat completions. Every one hit the 50 sROUTER_REQUEST_DEADLINE_MS.The default profile sets no attempt bounds, so each attempt gets the engine's default
timeout_ms = 30000(core/llm_policy.lua), which takes precedence over--timeout-s. A route that never answers uses 30 s of the 50 s deadline, and the next candidate then has only 20 s to finish before the caller gets a timeout. Thetimeout -> next_candidateretry action is in place, but the time budget leaves no room for it.Change
first_token_timeout_ms = 15000to the default profile, in bothconfig.live.luaand the publisheddefaulttemplate, which must stay identical.first_token_timeout_ms: codex, AntSeed/OpenAI-compatible, Bedrock, OpenRouter.Behaviour notes
first_token_timeout_msswitches OpenAI-compatible buffered calls to thebuffered_ssepath (upstreamstream: true, aggregated before the router responds).DEFAULT_FIRST_TOKEN_TIMEOUT_MSis the knob.Tests
test_live_default_profile_bounds_first_output_so_fallback_fits_the_deadline: the live default profile puts a 15 s first-token bound on every attempt, keeps the engine's 30 s attempt timeout, and falls through to the next candidate after a timeout.test_live_default_profile_is_exactly_the_published_default_templatestill passes.main. CI runs the Lua suite.