Repository navigation
fix: support structured output on current Claude models - #262
Conversation
|
|
||
| ### Fixes | ||
|
|
||
| - **Structured output works with Claude revisions that reject forced tool choice**: schema-mode requests targeting Claude Sonnet 5.5, Opus 5.5, Fable 5.1, or Mythos 5.1 now use automatic tool choice for the initial request and corrective turns. Matching uses the API model ID, including aliases and routed provider IDs. The model must still call `StructuredOutput` with a result-file path; schema validation, retry limits, missing-result failures, and permission checks remain in force. Other model revisions retain their existing forced tool choice. |
There was a problem hiding this comment.
🟡 New changelog bullet missing trailing PR reference.
🤖 Fix with your agent
Fix this code review finding (aictrl-dev/cli PR #262, CHANGELOG.md:19):
Problem: New changelog bullet missing trailing PR reference
Detail: Every other bullet in CHANGELOG.md — across all sections, including the other bullets added in this same release cut — ends with a trailing issue/PR reference such as (#256) or (#243). The new "Structured output works with Claude revisions that reject forced tool choice" bullet added by this PR ends with "...retain their existing forced tool choice." and no reference, breaking the file's uniform convention and leaving the fix untraceable to its PR.
Suggested fix: Append the PR reference to the bullet's last sentence, e.g. "...Other model revisions retain their existing forced tool choice. (#262)"
Implement the fix on the PR head branch and add a regression test that fails before the fix and passes after.
Why this matters
Every other bullet in CHANGELOG.md — across all sections, including the other bullets added in this same release cut — ends with a trailing issue/PR reference such as (#256) or (#243). The new "Structured output works with Claude revisions that reject forced tool choice" bullet added by this PR ends with "...retain their existing forced tool choice." and no reference, breaking the file's uniform convention and leaving the fix untraceable to its PR.
### Fixes
- **Structured output works with Claude revisions that reject forced tool choice**: schema-mode requests targeting Claude Sonnet 5.5, Opus 5.5, Fable 5.1, or Mythos 5.1 now use automatic tool choice for the initial request and corrective turns. Matching uses the API model ID, including aliases and routed provider IDs. The model must still call `StructuredOutput` with a result-file path; schema validation, retry limits, missing-result failures, and permission checks remain in force. Other model revisions retain their existing forced tool choice.
- **Built-in guidance follows the available tools**: system instructions and recovery messages no longer tell the model to delegate, search, or edit with tools absent from its request catalog. Custom instructions and plugin text remain unchanged. (#252)
Code reviewVerdict: Looks good — only minor / nit comments below. · 🔴 0 · 🟠 0 · 🟡 1 · ⚪ 0 · 0/1 resolved
🤖 Fix all 1 open findings with your agent📋 Out-of-diff findings (1)
Reviewed 6 files · 0 inline · view all 1 findings ↗ · AI-generated — verify findings before acting · AI guidance and feedback aictrl · AI code review for fast-moving teams · aictrl.dev |
Review response — PR #262Verified the single changelog consistency finding against the reviewed commit and addressed it on this PR branch. Issues addressed (pushed to this PR)
Review claims verified false (no change needed)None. Not addressed hereNone. Validation: |
Refs #138 and #140. This is a compatibility follow-up; it requests no new issue closure.
Problem
PR Explanation on the managed
anthropic/claude-sonnet-5-5model reaches the executor, then fails before any tool calls withtool_choice: type "tool" and "any" are not supported for this model.Schema mode unconditionally sends required tool choice, and its corrective turn forcesStructuredOutputby name. These Claude revisions reject both forms regardless of their thinking setting.Behavior
Use automatic tool choice for schema-mode requests targeting Claude Sonnet 5.5, Opus 5.5, Fable 5.1 and Mythos 5.1. Match the actual API model ID, including routed and configured aliases. Compatible models keep their required initial choice and named corrective choice.
The prompt still requires the
StructuredOutputtool and a result-file path. File/schema validation, final revalidation, bounded corrective attempts, permission handling and truthful failure on missing output remain in force. A prose answer does not become a successful structured result.Scope
0.5.5metadata. The release also includes changes already merged since0.5.4: headless tool selection, explicit stdin/command handling, config/auth hardening, domain/upgrade corrections and GPT verbosity defaults.CHANGELOG.mdrecords that complete release scope.Rollout and risks
This candidate is based on CLI
main, commitb9172499a2825030d17b6dd1f02c51e1773177f0; PR head is88348989d8bf49f388222b4b7df4bb7307053eb5. Anthropic recommends automatic tool choice for these revisions: tool-choice guidance.Publication and the application executor update are separate release steps. Require the repository's native current-versus-previous comparison, CI/review, npm publication/install verification and the application Docker pin/rebuild. This PR alone does not update the production executor. No model downgrade or schema-validation bypass is introduced.
Verification
0.5.5binary built from this exact commit with Bun1.4.3; version/hash/provenance verified. Candidate-versus-published0.5.4comparison passed six sessions and cleanup. Default reference-token deltas: first input -4, session input -16, output 0, total -16. The previous binary has no--tools, so its selected arm is explicitly unsupported.{ "answer": 4 }on attempt 1, two tool calls and no session error.verifyhit a fixture child-process deadline after successful terminal output; the single retry completed successfully and the timeout did not recur. Automated AI review failed without posting findings; it is not counted as approval.Test plan
0.5.4evidence and token deltas.