Repository navigation
feat: Add a Fixed Tool Dispatcher for Mid-Conversation Tools - #614
Open
Shabonasar wants to merge 2 commits into
Open
Shabonasar wants to merge 2 commits into
Shabonasar wants to merge 2 commits into
Conversation
…Position Adds conversation_instructions: context that is stable within a conversation. It stays in the system message right after the static instructions and is never moved: its own content block with a cache breakpoint for Anthropic and OpenRouter, a block followed by a cache point for Bedrock, appended text for other providers. additional_instructions keeps its per-turn behaviour. A host that passes conversation_instructions gets the compaction summary, both when created inside a run and when restored as initialSummary, as one stable user message right after the system message, with a breakpoint of its own while the request has one left. Token accounting counts it as instruction overhead. Both paths render the same carrier text, so the requests after a compaction form one continuous prefix. Hosts that do not pass the field keep the earlier placement and accounting. Adds a prompt cache harness (src/specs/prompt-cache) that runs declarative scenarios through Run and each provider class, captures the serialized request before the network, and asserts the cache invariants per provider.
Providers serialize `tools` ahead of the system prompt, so binding a tool mid-conversation resets the prompt cache from the first byte. With the new opt-in `AgentInputs.toolDispatch`, a fixed `call_tool` dispatcher is bound from the first turn and deferred registry tools are reached through it, so `tools` stays byte-identical while the host announces new tools in the history. - ToolNode canonicalizes every batch before hooks, approval, HITL, and execution: a dispatcher call becomes its target (same call id, arguments coerced and validated against the target schema), so PreToolUse, approval policies, `action_requests`, and ON_TOOL_EXECUTE see the real tool. The same `resolveDispatchedToolCall` is exported for permission layers outside ToolNode. - Results and completion events keep the model form: the ToolMessage answers the dispatcher call under its name, and the completed tool call carries the real tool in `dispatched`, so stored history names only declared tools. - A direct call to a dispatchable tool runs the same way and is rewritten into a dispatcher call in the assistant message, since the model's own message would otherwise put an undeclared tool name into the history. - Deferred tools are never bound while the dispatcher is enabled, even after discovery. tool_search results carry full names and parameter schemas and point the model to the dispatcher. - `LCTool.disabled` keeps a tool bound but refuses its calls with a tool error before any hook, without changing `tools`. - A schema the validator cannot apply, or any error while resolving a call, becomes a tool error message instead of an exception. `ToolDispatchConfig.canDispatch` limits dispatchable tools to those the host declared. - Dispatcher calls, dispatchable tools, and disabled tools are excluded from eager prestart. Tests: unit coverage in toolDispatch.test.ts; the prompt-cache harness gains a dispatched-call scenario for every provider case.
This was referenced Oct 15, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Providers serialize
toolsahead of the system prompt, so binding a tool in the middle of a conversation (an MCP server switched on, a tool found withtool_search) resets the prompt cache from the first byte. With the new opt-inAgentInputs.toolDispatch, a fixedcall_tooldispatcher is bound from the first turn; deferred registry tools are reached through it, sotoolsstays byte-identical while the host announces new tools in the history. Without the option nothing changes.Related to LibreChat-AI/LibreChat#17102. Depends on #613 (prompt-cache harness). Used by the LibreChat PR "tool groups and the SDK dispatcher" behind
endpoints.agents.promptCache.stableContext.How it works
resolveDispatchedToolCallis exported for permission layers outsideToolNode.tool_searchreturns full names and schemas and points to the dispatcher.LCTool.disabledkeeps a tool bound but refuses its calls with a tool error before any hook. A schema the validator cannot apply becomes a tool error, not an exception.ToolDispatchConfig.canDispatchlets a host narrow the targets.Measurements
Run: group G2 measured on Qwen3.8-27B (vLLM behind NVIDIA Dynamo, through LiteLLM), OpenAI chat and Anthropic Messages, dev / lever off / lever on; the same self-hosted model answers both formats, the real OpenAI and Anthropic APIs were not measured; full method in #17102. With the lever on, the 13 requests of each G2 run have one
toolssha and onesystemsha, and everycall_toolwas executed.Note: the backend caches in 1600-token blocks, so a request reuses cache only in whole blocks and the first turns of a short conversation (prompt under two blocks) cannot hit; shares are therefore lower on short chats than on long ones.
Testing
toolDispatch.test.ts: unwrapping, refusals, schema validation, hooks and HITL on the target, direct-call rewriting,tool_searchguidance, binding.tool-dispatch.test.ts(harness): for every provider case a deferred tool is announced and called through the dispatcher;toolsstay byte-identical, every request starts with the previous one, the result answers the dispatcher call and the history names only declared tools.Type of change
Checklist