Repository navigation
fix: Restore Tool Rounds Exactly as the Live Request Sent Them - #615
Open
Shabonasar wants to merge 2 commits into
Open
Shabonasar wants to merge 2 commits into
Shabonasar wants to merge 2 commits into
Conversation
…Position Adds conversation_instructions: context that is stable within a conversation. It stays in the system message right after the static instructions and is never moved: its own content block with a cache breakpoint for Anthropic and OpenRouter, a block followed by a cache point for Bedrock, appended text for other providers. additional_instructions keeps its per-turn behaviour. A host that passes conversation_instructions gets the compaction summary, both when created inside a run and when restored as initialSummary, as one stable user message right after the system message, with a breakpoint of its own while the request has one left. Token accounting counts it as instruction overhead. Both paths render the same carrier text, so the requests after a compaction form one continuous prefix. Hosts that do not pass the field keep the earlier placement and accounting. Adds a prompt cache harness (src/specs/prompt-cache) that runs declarative scenarios through Run and each provider class, captures the serialized request before the network, and asserts the cache invariants per provider.
A restored turn has to serialize like the live requests of that turn, or the next turn stops being a prefix continuation at the first difference. Four things drifted: - Sequential tool rounds of one turn were merged into one parallel batch when restored. Rounds now carry a round id and restore as separate assistant messages; text emitted before a call stays with that call. - A Responses tool round restored from stored parts lost the provider items the live request carried (function_call ids and status, reasoning items, message ids) and gained an empty assistant message. After a response with tool calls the graph slices the output per call and hands the slices to the new ON_PROVIDER_OUTPUT_ITEMS event; the content aggregator stores them on the tool_call parts and formatAgentMessages puts them back. They are replayed only to the model and endpoint that produced them, never under zdrEnabled, only when every item is a reasoning, function_call, or message item under a size cap, and reasoning references older than the retention of stored responses are left out. The replay gate takes the current time as an argument. Hosts must route the event to aggregateContent and persist the parts unchanged. - An in-process tool result made of content blocks was stored as JSON and restored as one string. The completion now marks it with outputFormat 'blocks' and restore parses it back into the same blocks. - Anthropic sent a tool_result as a text block only while the tail cache marker sat on it, and OpenRouter with prompt caching did the same for tool messages. A single plain text tool_result is now always sent as its string (Anthropic), and every non-empty tool message as one text block (OpenRouter). Tests: a generic live-versus-restored suite runs real turns against scripted streams in each wire protocol and requires the first request of turn N+1 to start byte for byte with the last request of turn N; Responses specs cover the reasoning variants and the gate branches.
This was referenced Oct 15, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A turn restored from stored content parts has to serialize like the live requests of that turn, or the next turn stops being a prefix continuation at the first difference. Four things drifted: sequential tool rounds of one turn were merged into one parallel batch; an OpenAI Responses tool round lost the provider items the live request carried (ids, status, reasoning items) and gained an empty assistant message; a tool result made of content blocks was restored as one string; and Anthropic and OpenRouter sent a tool result in a different shape depending on where the tail cache marker sat. This PR makes the restored request equal the live one for these cases.
Related to LibreChat-AI/LibreChat#17102. Depends on #613 (prompt-cache harness). Used by the LibreChat PR "provider output items" behind
endpoints.agents.promptCache.stableContext; hosts must routeON_PROVIDER_OUTPUT_ITEMStoaggregateContentand persist the parts unchanged.How it works
roundId; calls of one round stay together, a new round starts a new assistant message; text emitted before a call stays with that call.zdrEnabled, only when every item isreasoning,function_callormessageand the slice is under a size cap; reasoning references older than the retention of stored responses are left out. The gate takes the current time as an argument.outputFormat: 'blocks'on completion and parsed back into the same blocks on restore. A single plain-text Anthropictool_resultis always sent as its string; every non-empty OpenRouter tool message as one text block.Measurements
No cache-share run isolates this PR: the backend used for G1-G3 speaks the OpenAI chat and Anthropic protocols, not Responses. The evidence is the live-versus-restored suite below, which requires the first request of turn N+1 to start byte for byte with the last request of turn N for each provider client.
Testing
liveRestore.test.ts: real turns against scripted streams in each wire protocol (Chat Completions, Responses, Anthropic Messages, Gemini, Bedrock Converse), parts stored by the aggregator, history restored byformatAgentMessages; parallel and sequential rounds, text and empty text before calls, text, multi-block and image tool results; signed and redacted thinking reach the follow-up request of their turn unchanged. This was not verified against the real Anthropic or Bedrock APIs.responsesToolRounds.test.ts(both): reasoning variants, restore without stored items, each branch of the replay gate (origin, ZDR, type allowlist, size cap, retention); each test fails when its check is removed.restoredToolRounds.test.ts: round grouping informatAgentMessages.Type of change
Checklist