Repository navigation
fix(llm,agent): make thinking replay and request shaping safe - #239
Merged
Merged
Conversation
added 30 commits
November 3, 2026 14:19
Claude and Bedrock requests that replay thinking blocks under adaptive or enabled thinking set thinking.block_binding.prefix_mismatch_behavior to drop_block and send the thinking-binding-controls-2026-08-01 beta, so the API drops blocks bound to a different conversation instead of rejecting the request. Responses report input_transformations in metadata. The Claude stream client keeps redacted_thinking blocks and returns thinking blocks and tool calls in response order.
…g and drivers Fold a configured anthropic-beta header into the merged beta list, keep a block_binding the payload already carries and send the beta whenever one is present, append Bedrock betas and response field paths instead of replacing them, replay stored blocks in the API shape of their type, and capture, order and replay Bedrock reasoning and redacted reasoning.
thinking = always sends the adaptive thinking object at every effort; an absent or zero effort sends no output_config so the model default effort applies. The object carries block_binding at zero effort, so default effort routes survive handoffs. adaptive, budget and none are unchanged; OpenAI, OpenAI-compatible and Google reject always.
…n redacted bytes Bind thinking as the last change to each driver's wire body so the binding follows the thinking object actually sent. Replay stored Claude thinking blocks unchanged apart from stale fields of an earlier shape. Join Bedrock streamed redactedContent deltas by their decoded bytes. Add thinking helper tests and a shared deep_equal test helper, and document steady-state drop_block behavior and enforcement opt-in.
Resolves the options.request passthrough (#233) against the thinking binding: on every Claude and Bedrock handler the overlay is applied first and bind_thinking is the last change before dispatch. Adds the cross-PR route_flow tests and declares thinking: always in the Sonnet 5.5 between_tools README example.
…as a leading run Positional system/developer notes (metadata.preserve_position) render as text blocks after the preceding user turn on Claude and Bedrock, as user turns on Google, and stay in place on OpenAI and OpenAI-compatible; they never enter the top-level system prompt. A positional note without a preceding user turn is invalid_request. Bedrock developer messages render as user text instead of system blocks. The Google mapper no longer clears caller message metadata. builder:bind_thinking strips thinking blocks from a leading run of non-current messages and rejects middle removal and open tool rounds. Golden tests pin message-derived bodies on all five drivers.
time_aware contributes agent_options.turn_context instead of prompt text; hosts render the note with wippy.agent:turn_context and persist it per user turn. Memory recall runs only for hosts that set persist_context_notes and is returned as positional context_notes the host persists before the step output. The tail cache marker skips trailing positional notes. Each step returns prefix_fingerprint over the system prompt, tools and model. Sorted serialization moves to wippy.agent:canonical and is shared with attention_read.
Developer messages render as their own text blocks after the user turn they follow (or open a user turn after an assistant turn) and never edit an existing block, so the same history renders identically wherever the rolling cache marker sits. The Claude golden body changes only for its developer messages. bind_thinking becomes drop_stale_thinking: empty block lists carry nothing, every block up to the last stale message is removed, and an assistant turn (consecutive assistant and function call messages, including function_call content parts) with an unanswered call refuses removal and leaves the history unchanged.
…errors prefix_fingerprint hashes the system text as drivers send it (compiled prompt plus hoisted conversation system messages) and each tool's name, description, schema and native definition; registry ids, registry metadata and the model are excluded. A failing prompt function fails the step with its kind and retryability. A failing trait build function fails compilation; time_aware rejects time_interval naming turn_context. turn_context.render validates timezone and format and keeps the guidance sentence. A test guards the llm >= 0.5.13 dependency on prompt.is_positional_note.
A bound Bedrock request whose additionalModelRequestFields.anthropic_beta or additionalModelResponseFieldPaths is not a list fails with invalid_request before HTTP. README notes the uncaptured fallback array, the between_tools effort limits and further input_transformations values.
Claude and Bedrock render a positional note at the start of the conversation or after an assistant or tool call turn as its own user turn instead of rejecting it, matching persisted checkpoint-resume and stop-boundary notes. drop_stale_thinking replaces stripped messages with copies so caller-held message tables keep their blocks. A Bedrock developer golden case pins the branch rendering.
The prefix fingerprint is computed before generation; with drop_stale_thinking the agent strips blocks whose stored metadata.prefix_fingerprint differs (or that thinking_is_current rejects) from a copy of the history, including checkpoint replacements, and fails the step on an open tool round. Prompt function errors keep their details. time_aware reads its layout from time_format so a generic agent context format key is not taken as a time layout.
Developer messages, positional or not, render on Claude and Bedrock as their own "<developer-instruction>...</developer-instruction>" text block after the user turn they follow, restoring the model-visible wrapper without editing earlier blocks. Positional system notes stay plain. The unreachable map_messages nil guards in the Claude and Bedrock handlers are removed. drop_stale_thinking names the unanswered call id and tool name when it refuses. BREAKING CHANGE: Claude and Bedrock render developer messages and positional notes differently: each developer message is its own wrapped text block after the user turn it follows (previously merged into the preceding text, or a separate user message after tool results or cache markers), Bedrock developer messages are no longer sent as system blocks, positional notes render in place instead of the top-level system prompt, and builder:drop_stale_thinking replaces the earlier bind_thinking draft. Stored conversations render a different prefix once after upgrading.
The agent calls prompt.is_positional_note, so it requires wippy/llm >= v0.5.13; the test app lock records 0.5.13, resolved from the local llm source through the workspace replacement. time_aware rejects a time_format that is not a non-empty string. Tests cover stale-thinking stripping of checkpoint replacement histories. The README states that the fingerprint belongs on the rows that carry thinking blocks. BREAKING CHANGE: step responses no longer include memory_prompt; memory recall runs only when the host sets persist_context_notes and is returned as positional context_notes the host persists before the step output. The time_aware trait adds no prompt text, declares agent_options.turn_context for hosts to render with wippy.agent:turn_context, and rejects time_interval. A trait build function that returns an error fails compilation, and a failing prompt function fails the step.
The breaking llm changes in this branch release as wippy/llm 0.6.0, so the agent requires >=v0.6.0.
The llm changes in this branch release as a patch, wippy/llm 0.5.13.
The last assistant turn with no user message after it is the round the request continues; the provider requires its thinking, so it is never stripped or consulted and stale blocks there are left to the provider's drop-on-mismatch handling. Earlier turns are stripped through the last stale one before it.
…urce lock Stale-thinking stripping leaves the current tool round intact. The package source lock records wippy/llm 0.5.13 resolved from the local llm source, matching the manifest requirement.
… current drop_stale_thinking scans back over the trailing non-assistant messages; the last assistant turn stays the current round unless a user message follows it with no function result in between. Steering messages and the image message get_messages inserts after a tool result join the tool-result turn, so the round keeps its thinking.
…ults Covers stale-thinking stripping when a user message follows the round's tool results and documents the rule.
drop_stale_thinking counts any message that renders as user content (user, developer, positional system note) when deciding whether the last assistant turn is still the current round; hoisted system messages and cache markers do not.
A checkpoint replacement history recomputes prefix_fingerprint before its thinking blocks are judged, so the returned fingerprint matches the prefix actually sent. The lifecycle contract states that per-step lifecycle messages change the prefix unless the host persists them in place as positional notes.
Memory recall runs whenever the agent has a memory contract, as before this branch. The recall note is sent at the conversation tail as a positional developer note and returned in context_notes on normal, truncated and cancelled results for the host to persist in place. The persist_context_notes runtime option is removed. BREAKING CHANGE: step responses return the memory recall note in context_notes instead of memory_prompt; hosts persist each context note after the messages they passed and before the step output.
tool_caller:set_result_listener(fn) receives (call_id, result) once per call with the final normalised result as soon as that call finishes. Parallel execution collects responses in completion order via channel.select over pending responses and the cancel channel; sequential execution delivers before the next call starts. A listener failure stops the batch, cancels in-flight calls, skips after_execute wrappers and makes execute return nil and the error. Argument, context and executor preparation is shared by both strategies.
cancel_pending takes responses that have already arrived and cancels only commands that are still running, so a finished command is never cancelled.
…local/integration
Stale current-round blocks: the agent marks them and Claude and Bedrock remove marked blocks when the final thinking object cannot carry a binding. Interleaved responses: extraction records block positions on all four paths and consolidation rebuilds the response order; stored rows without positions keep their order. OpenAI-compatible tool rounds stop at a non-tool message once every call has its result, so a positional note between rounds stays in place. One blank rule across drivers: blank user, system and developer text never reaches a driver, and an assistant turn with no visible text and no call renders as nothing. Claude invents no text for it. Effective output is planned once before the card limit and admission, from the route and the driver's output policy; mappers read it and adjusted.max_tokens.sent is the wire value. Requests with no mapped turn fail before HTTP on Claude, Bedrock, Google and OpenAI. drop_stale_thinking handles string content. The prefix fingerprint covers only what reaches the wire.
A recorded layout renders only visible text, and a row whose text differs from the layout only in outer whitespace keeps its layout. The API compares neither whitespace-only text blocks nor a block's outer whitespace, so neither is an edit to the prefix. The README now limits the order guarantee to calls of one response that consolidate into one assistant turn.
Whitespace-only assistant text is not rendered inside a turn with a call, whether or not the row has a recorded layout. Claude and Bedrock leave the block out, and OpenAI-compatible sends the empty content its calls need. The API does not compare whitespace-only text blocks, so thinking blocks stay valid; a stored row with one restarts its prompt cache once, as the README notes.
added 7 commits
November 4, 2026 09:32
A truncated step discards its calls but kept their thinking and layout, so every later request sent an assistant turn ending in thinking, and the fingerprint still matched. The agent now takes the calls and the thinking that led to them out of the response metadata. Rows already stored get the same rule at replay: a call the conversation no longer holds leaves the layout, and thinking behind the last remaining text or call is not sent. The Bedrock JSON path no longer reports a length stop with tool calls as a tool_call stop, which hid truncation and let incomplete calls run.
Bedrock sent call ids verbatim, so an id from another provider could break its 64-character limit or its character set, and OpenAI Responses has the same length limit. The id mapper moves from the Claude mapper to a shared util and takes each provider's limits. Calls and results share a translation derived from a digest of the id, stable across requests and collision-checked. Ids within the limits, and all Claude ids, stay byte-identical.
… listeners run A turn the drivers emit never ends in thinking, whatever the row shape. Stale thinking scans treat cache markers and hoisted system messages as part of the assistant turn. The prefix fingerprint separates arrays from objects. Stop cancels all unfinished calls before any result listener runs.
… and keep notes in tool rounds
skhaz
approved these changes
Nov 4, 2026
added 4 commits
November 4, 2026 13:39
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes thinking replay and request shaping safe across handoffs, cut-off responses and foreign call ids. Follows #235, #237 and #238, which are merged. These ship in one release.
Changes
noneorbetween_toolsno longer sends an unbound block.metadata.content_positionsrecords interleaved text, thinking and calls, and replay rebuilds that order.ensure_content_existsis gone.invalid_request.route.plan_outputcomputes effective output once, before the card limit and admission.adjusted.max_tokens.sentis the wire value.util/tool_id.luais shared, and Bedrock and OpenAI translate foreign ids. Compat sends them unchanged.falseresult staysfalsein parallel execution, and Stop cancels every unfinished call before any listener runs.time_interval: ignored bytime_awarerather than failing compilation.Behaviour changes
strictrejects a budget raise ofmax_tokens.max_tokensreserves the 2000-token default in admission.Tests
src/llm/test/replay_safety_test.luaandsrc/agent/test/replay_safety_test.lua. llm 2417 and agent 619 pass, excluding skips, lint clean. Across 840 request bodies, the only differences from master are the blank-turn removals and Bedrock id translation.