fix: Keep Per-Conversation Context and the Summary at a Fixed System Position

[Shabonasar]

Summary

With Anthropic or OpenRouter prompt caching, AgentContext moves the dynamic system part into a user message before the newest human message, so context that is fixed for a whole conversation (start time, memory, file lists) shifts on every turn and the previous exchange is re-written to the cache each time. A summary created by a compaction inside a run was also sent as a user message while the next run rendered the same summary in system, so every compaction cost a second prefix rebuild. This PR adds AgentInputs.conversation_instructions, kept in the system message right after instructions and never moved, and one fixed placement for the summary. Hosts that do not pass the field get the same requests as before.

The fixed summary placement applies to hosts that pass conversation_instructions; LibreChat passes it for the providers where the SDK caches the conversation block (Anthropic, OpenRouter, Bedrock with promptCache). Other providers, and hosts without the field, keep the earlier placement, so a summary there can still render in two places and cost a second rebuild after a compaction.

Related to LibreChat-AI/LibreChat#17102. Used by the LibreChat PR "provider output items and conversation block" behind endpoints.agents.promptCache.stableContext. Base: main.

How it works

S static instructions, T per-conversation context, D per-turn context, bp cache breakpoint.

before (T in additional_instructions, moved under prompt caching):
  turn 2: [tools][S|bp] u1 a1 [T D] u2
  turn 3: [tools][S|bp] u1 a1 u2 a2 [T D] u3        T shifts every turn

after (T in conversation_instructions):
  turn 2: [tools][S|bp][T|bp] u1 a1 [D2] u2
  turn 3: [tools][S|bp][T|bp] u1 a1 u2 a2 [D3] u3   request N+1 starts with request N
  after compaction: [tools][S][T][summary msg] retained turns ...   same bytes in the run and in the next run

• Anthropic and OpenRouter with promptCache: a separate system content block with its own breakpoint. Bedrock with promptCache: a block followed by a cache point. Other providers: appended to the system text at the same position.
• With the field set, the summary (created in a run or restored as initialSummary) is one stable user message right after the system message, with a breakpoint while the request has one left (Anthropic allows four). Token accounting counts it as instruction overhead. setSummary loses its precedesMessages option.
• additional_instructions keeps its per-turn behaviour (moved under prompt caching).

Measurements

G1: plain chat, 10 turns, timestamp changing	dev	lever off	lever on	uncached prompt tokens, dev -> lever on
OpenAI chat format (LiteLLM endpoint)	0.0%	0.0%	56.5%	84240 -> 31966
Anthropic Messages format (LiteLLM endpoint, cache_control ignored by the backend)	0.0%	0.0%	54.9%	63275 -> 34217

G3: compaction, 12 turns, max context 12000	dev	lever off	lever on	uncached prompt tokens, dev -> lever on
OpenAI chat format (LiteLLM endpoint)	2.2%	2.1%	20.1%	142765 -> 139625
Anthropic Messages format (LiteLLM endpoint, cache_control ignored by the backend)	6.1%	4.3%	30.2%	147707 -> 122023

Only the requests that carry conversation_instructions change; G1 and G3 above come from the LibreChat stack that passes it for Anthropic. OpenAI chat does not use the field, so its G1 gain comes from the host keeping the system prompt stable, and its G3 share is lower because the summary still renders in two places there.

Run: group G1-G3 measured on Qwen3.8-27B (vLLM behind NVIDIA Dynamo, through LiteLLM), OpenAI chat and Anthropic Messages, dev / lever off / lever on; the same self-hosted model answers both formats, the real OpenAI and Anthropic APIs were not measured; full method in #17102.

Note: the backend caches in 1600-token blocks, so a request reuses cache only in whole blocks and the first turns of a short conversation (prompt under two blocks) cannot hit; shares are therefore lower on short chats than on long ones.

Testing

npx jest src/agents src/specs/prompt-cache
npx tsc --noEmit && npx eslint src/agents src/specs/prompt-cache

• AgentContext.test.ts: system blocks and breakpoints per provider, Bedrock cache points, plain-text order, token accounting, the summary position, a summary set in a run rendering exactly like the same summary restored by the next run, and the unchanged behaviour without the field.
• Prompt-cache harness (src/specs/prompt-cache, docs/prompt-cache-harness.md): each turn is built through a real Run and the real chat model class and the serialized request is captured before the network. Providers: OpenAI, OpenAI Responses, an OpenAI-compatible endpoint, Azure, DeepSeek, xAI, Moonshot, Anthropic, OpenRouter and Bedrock with and without caching, Google. Invariants: the conversation block stays in system, only additional_instructions moves, request N+1 starts with request N up to the per-turn part (including after a compaction inside a turn), a prefix shared across conversations, at most four breakpoints, strict alternation where required. Offline, in the default npx jest.
• Bedrock with promptCache is covered by the harness only; it was not run against a real Bedrock endpoint.

Type of change

• Bug fix
• Feature

Checklist

• I reviewed my own changes
• Relevant tests have been added or updated
• Existing relevant tests pass
• The change does not introduce new warnings or errors
• User-facing or complex behavior is documented where necessary
• Required dependency changes have been merged/published (none)