Skip to content

Bound first output on the default profile so fallback fits the deadline - #126

Open
rrabenda wants to merge 1 commit into
mainfrom
fix/default-first-token-timeout
Open

rrabenda wants to merge 1 commit into
mainfrom
fix/default-first-token-timeout

Conversation

@rrabenda

Copy link
Copy Markdown
Contributor

Problem

Callers on the default profile (the OpenAI-compatible endpoint with no policy, e.g. studio-prd, studio-rally-prd, Deckard) intermittently get 504s after exactly 50 s. On 2026-10-28, from about 07:30 UTC, devexp-dev llm-router-api returned 25 of these out of about 7,300 chat completions. Every one hit the 50 s ROUTER_REQUEST_DEADLINE_MS.

The default profile sets no attempt bounds, so each attempt gets the engine's default timeout_ms = 30000 (core/llm_policy.lua), which takes precedence over --timeout-s. A route that never answers uses 30 s of the 50 s deadline, and the next candidate then has only 20 s to finish before the caller gets a timeout. The timeout -> next_candidate retry action is in place, but the time budget leaves no room for it.

Change

  • Add first_token_timeout_ms = 15000 to the default profile, in both config.live.lua and the published default template, which must stay identical.
  • A silent route now falls through after 15 s, and the next candidate keeps its full 30 s within the 50 s deadline.
  • Only the wait for the first output is bounded. Once a provider starts streaming (reasoning deltas count), it keeps the full 30 s. Of the successful requests on the 28th, 215 took over 20 s, so a lower total attempt timeout would have broken legitimate slow generations; this change doesn't touch them.
  • Every provider kind in the default order supports first_token_timeout_ms: codex, AntSeed/OpenAI-compatible, Bedrock, OpenRouter.

Behaviour notes

  • Like the agent profile, setting first_token_timeout_ms switches OpenAI-compatible buffered calls to the buffered_sse path (upstream stream: true, aggregated before the router responds).
  • 15 s is more conservative than the agent profile's 10 s. If routes with long prefill on large prompts start falling through, DEFAULT_FIRST_TOKEN_TIMEOUT_MS is the knob.

Tests

  • New test_live_default_profile_bounds_first_output_so_fallback_fits_the_deadline: the live default profile puts a 15 s first-token bound on every attempt, keeps the engine's 30 s attempt timeout, and falls through to the next candidate after a timeout.
  • test_live_default_profile_is_exactly_the_published_default_template still passes.
  • I ran the full suite locally. The only failures are tests that need Postgres or network, and they fail identically on main. CI runs the Lua suite.

The default profile set no attempt bounds, so every attempt got the
engine's 30 s timeout_ms. A route that never answered used 30 s of the
50 s request deadline, and the next candidate had only 20 s before the
caller received a timeout.

Set first_token_timeout_ms to 15 s on the default profile, matching how
the agent profile bounds attempts. A silent route now falls through after
15 s and the next candidate keeps its full 30 s budget. A slow generation
that is already streaming is not affected.
@coderabbitai

coderabbitai Bot commented Oct 28, 2026

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 73fdb925-e7d3-473a-8afd-7320cf90e870
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant