fix(llm, openrouter): Report mid-stream provider errors - #978
Open
JeanMertz wants to merge 2 commits into
Open
Conversation
A request that OpenRouter rejects after committing the response cannot
change its HTTP status, so the rejection arrives as an ordinary SSE chunk
carrying a top-level `error` next to an empty `choices` array. The
`ChatCompletion` response type had no `error` field, so serde discarded
the payload and the chunk mapped to zero events. The turn loop then saw a
stream that ended without a terminal event, classified the synthetic
sentinel as transient, and burned the full retry budget resending an
identical body before failing with "provider stream ended without a
terminal event".
An oversized prompt now fails once with what the provider actually said:
error API error (status 400): prompt is too long:
1284668 tokens > 1000000 maximum
Both error paths route through one `map_error` helper reusing the existing
`From<jp_openrouter::Error>` classification, so a status code means the
same thing wherever it arrives. The per-choice path previously sniffed the
message text for quota patterns and fell back to `Other` for everything
else, which made a mid-stream 503 non-retryable while an identical 503 at
connect time was retryable. Both are transient now, and a 400 is not
retryable on either path.
Signed-off-by: Jean Mertz <git@jeanmertz.com>
Signed-off-by: Jean Mertz <git@jeanmertz.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A request that OpenRouter rejects after committing the response cannot change its HTTP status, so the rejection arrives as an ordinary SSE chunk carrying a top-level
errornext to an emptychoicesarray. TheChatCompletionresponse type had noerrorfield, so serde discarded the payload and the chunk mapped to zero events. The turn loop then saw a stream that ended without a terminal event, classified the synthetic sentinel as transient, and burned the full retry budget resending an identical body before failing with "provider stream ended without a terminal event".An oversized prompt now fails once with what the provider actually said:
Both error paths route through one
map_errorhelper reusing the existingFrom<jp_openrouter::Error>classification, so a status code means the same thing wherever it arrives. The per-choice path previously sniffed the message text for quota patterns and fell back toOtherfor everything else, which made a mid-stream 503 non-retryable while an identical 503 at connect time was retryable. Both are transient now, and a 400 is not retryable on either path.