Skip to content

Name the hung route on deadline errors and return timeouts as 503 - #127

Open
rrabenda wants to merge 1 commit into
mainfrom
fix/deadline-error-reporting
Open

rrabenda wants to merge 1 commit into
mainfrom
fix/deadline-error-reporting

Conversation

@rrabenda

Copy link
Copy Markdown
Contributor

Problem

Callers sometimes get a 504 from router.ygr.ai after exactly 50 s. On devexp-dev on 2026-10-28 there were 25 of them, all from the ROUTER_REQUEST_DEADLINE_MS deadline. Two things made them impossible to diagnose:

  1. No route in the error. When the outer deadline cancels the run, the engine's decision_path is lost with it. The response had decision_path: [] and provider: null. The auth-proxy log and the call ledger couldn't say which provider or model hung.
  2. No JSON body for callers. router.ygr.ai is behind Cloudflare, which replaces an origin 502/504 body with its own plain-text error code: 504. The pod's nginx logged a 274-byte JSON body. The caller (Deckard) received only the Cloudflare text.

Change

  • Attempt record. llm_router_host.attempt_log is a request-scoped ContextVar. When the shim sets a list, execute_async records each provider attempt in it: provider, family, start time, and the outcome once it returns. This adds no work when unset.
  • Rebuilt decision_path on deadline. The shim builds the path from that record. An attempt still waiting at the deadline appears as error_kind: "timeout", in_flight: true with its elapsed time. routing_summary already reads decision_path, so the call ledger now records these attempts too.
  • Clearer message. The deadline error now says router: request deadline of 50000 ms exceeded — <provider>/<family>=timeout "no response before the 50000 ms request deadline". The warning log line names the route it was waiting on.
  • Status 503 for timeout. This covers the deadline, exhausted: timeout, and flow-node timeouts. error.code still carries timeout, so clients can tell it apart from no_candidates and capacity 503s. OpenAI SDKs retry 503 the same way as 504.

Notes for reviewers

  • This changes the HTTP status for timeouts from 504 to 503. A client that branches on 504 specifically would need to key on error.code == "timeout" instead.
  • Cloudflare replaces 502 bodies too, so the generic 502 (e.g. exhausted: server_error, which carries the per-attempt summary) is still lost for public callers. That is left for a follow-up because it touches the general error mapping.
  • Companion PR: Bound first output on the default profile so fallback fits the deadline #126 adds a first-token bound to the default profile, so a silent route falls through before the deadline instead of hanging until it.

Tests

  • New test_request_deadline_names_the_route_it_was_waiting_on: the first attempt times out and the second hangs. The 503 response lists both attempts, marks the second as in flight, and the message names it.
  • Updated the existing deadline, flow-node timeout, and diagnostics tests from 504 to 503.
  • Full suite run locally against Postgres 16: 1268 passed. 12 errors in test_sources/test_wallet_keeper came from running with -p no:logging, which removes the caplog fixture, not from this change. CI runs the Lua suite.

When the 50 s request deadline cancels a router run, the engine's
decision_path is lost with it. The error carried an empty path and
provider null, so nobody could tell which route had hung. The host now
records each provider attempt in a request-scoped ContextVar. On a
deadline the shim rebuilds decision_path from that record: the attempt
still waiting is reported as an in-flight timeout. The error message
names the deadline and summarizes the attempts, and the call ledger's
routing_summary now lists them as well.

Timeouts now return 503 instead of 504. Cloudflare in front of
router.ygr.ai replaces an origin 502/504 body with a plain-text
"error code: 504", so callers never saw the JSON error. error.code is
still "timeout", so clients can tell it apart from other 503s.
@coderabbitai

coderabbitai Bot commented Oct 28, 2026

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: d08c2d0c-0f94-4b65-b351-6a1e8ceffdd9
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant