Skip to content

fix: support structured output on current Claude models - #262

Merged
byapparov merged 2 commits into
mainfrom
fix/sonnet-55-structured-tool-choice
Nov 4, 2026
Merged

byapparov merged 2 commits into
mainfrom
fix/sonnet-55-structured-tool-choice

Conversation

@byapparov

@byapparov byapparov commented Nov 4, 2026 •

Copy link
Copy Markdown
Member

Refs #138 and #140. This is a compatibility follow-up; it requests no new issue closure.

Problem

PR Explanation on the managed anthropic/claude-sonnet-5-5 model reaches the executor, then fails before any tool calls with tool_choice: type "tool" and "any" are not supported for this model. Schema mode unconditionally sends required tool choice, and its corrective turn forces StructuredOutput by name. These Claude revisions reject both forms regardless of their thinking setting.

Behavior

Use automatic tool choice for schema-mode requests targeting Claude Sonnet 5.5, Opus 5.5, Fable 5.1 and Mythos 5.1. Match the actual API model ID, including routed and configured aliases. Compatible models keep their required initial choice and named corrective choice.

The prompt still requires the StructuredOutput tool and a result-file path. File/schema validation, final revalidation, bounded corrective attempts, permission handling and truthful failure on missing output remain in force. A prose answer does not become a successful structured result.

Scope

  • Provider capability predicate and schema-mode tool-choice selection only; no schema or public event format changes.
  • Native Anthropic HTTP regression fixture reproduces the production 400, then verifies validation repair, missing-result handling and budget exhaustion under automatic choice. A compatible Sonnet 4.5 case preserves forced choice.
  • Prepare CLI 0.5.5 metadata. The release also includes changes already merged since 0.5.4: headless tool selection, explicit stdin/command handling, config/auth hardening, domain/upgrade corrections and GPT verbosity defaults. CHANGELOG.md records that complete release scope.

Rollout and risks

This candidate is based on CLI main, commit b9172499a2825030d17b6dd1f02c51e1773177f0; PR head is 88348989d8bf49f388222b4b7df4bb7307053eb5. Anthropic recommends automatic tool choice for these revisions: tool-choice guidance.

Publication and the application executor update are separate release steps. Require the repository's native current-versus-previous comparison, CI/review, npm publication/install verification and the application Docker pin/rebuild. This PR alone does not update the production executor. No model downgrade or schema-validation bypass is introduced.

Verification

  • Reproduced RED through the actual Anthropic SDK and local HTTP server: three new cases failed with the exact provider 400, while the compatible-model case passed.
  • Native Anthropic regression selector after fix: 4 passed.
  • Eight relevant schema/provider/prompt regression files: 291 passed, 4 existing live-provider tests skipped, 0 failed.
  • Complete CLI suite: 2,301 passed, 10 skipped, 0 failed across 160 files.
  • Required release regression list: 95 passed. Workspace forced typecheck: 6/6 tasks passed. Frozen installation, formatting and whitespace checks passed.
  • Independent source review found no material issue; validation and terminal guards remain unchanged.
  • Native 0.5.5 binary built from this exact commit with Bun 1.4.3; version/hash/provenance verified. Candidate-versus-published 0.5.4 comparison passed six sessions and cleanup. Default reference-token deltas: first input -4, session input -16, output 0, total -16. The previous binary has no --tools, so its selected arm is explicitly unsupported.
  • One minimal live Sonnet 5.5 check using the exact candidate binary passed: exit 0 in 5.01 seconds, accepted { "answer": 4 } on attempt 1, two tool calls and no session error.
  • Hosted verification CI and all CodeQL checks passed on this exact head: build 3/3, workspace typecheck 6/6, SDK-prefix token budgets, loopback/fault controls and full tests (2,301 passed, 10 skipped). Initial verify hit a fixture child-process deadline after successful terminal output; the single retry completed successfully and the timeout did not recur. Automated AI review failed without posting findings; it is not counted as approval.

Test plan

  • Reproduce and fix initial/corrective Sonnet 5.5 requests through the native Anthropic adapter.
  • Preserve rejection of invalid/missing output and forced tools for compatible models.
  • Run complete CLI suite, release regression list and workspace typecheck.
  • Review native candidate-versus-published 0.5.4 evidence and token deltas.
  • Complete hosted verification CI and CodeQL on this head.
  • Obtain automated review or owner acceptance; the failed AI review posted no findings.
  • Verify a minimal live Sonnet 5.5 structured result with the candidate binary.
  • Publish/install the exact CLI version, update the app executor, verify sandbox, then repeat the authorized production PR explanation.

@byapparov byapparov self-assigned this Nov 4, 2026
Comment thread CHANGELOG.md Outdated

### Fixes

- **Structured output works with Claude revisions that reject forced tool choice**: schema-mode requests targeting Claude Sonnet 5.5, Opus 5.5, Fable 5.1, or Mythos 5.1 now use automatic tool choice for the initial request and corrective turns. Matching uses the API model ID, including aliases and routed provider IDs. The model must still call `StructuredOutput` with a result-file path; schema validation, retry limits, missing-result failures, and permission checks remain in force. Other model revisions retain their existing forced tool choice.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 New changelog bullet missing trailing PR reference.

🤖 Fix with your agent
Fix this code review finding (aictrl-dev/cli PR #262, CHANGELOG.md:19):

Problem: New changelog bullet missing trailing PR reference
Detail: Every other bullet in CHANGELOG.md — across all sections, including the other bullets added in this same release cut — ends with a trailing issue/PR reference such as (#256) or (#243). The new "Structured output works with Claude revisions that reject forced tool choice" bullet added by this PR ends with "...retain their existing forced tool choice." and no reference, breaking the file's uniform convention and leaving the fix untraceable to its PR.
Suggested fix: Append the PR reference to the bullet's last sentence, e.g. "...Other model revisions retain their existing forced tool choice. (#262)"

Implement the fix on the PR head branch and add a regression test that fails before the fix and passes after.
Why this matters

Every other bullet in CHANGELOG.md — across all sections, including the other bullets added in this same release cut — ends with a trailing issue/PR reference such as (#256) or (#243). The new "Structured output works with Claude revisions that reject forced tool choice" bullet added by this PR ends with "...retain their existing forced tool choice." and no reference, breaking the file's uniform convention and leaving the fix untraceable to its PR.

### Fixes

- **Structured output works with Claude revisions that reject forced tool choice**: schema-mode requests targeting Claude Sonnet 5.5, Opus 5.5, Fable 5.1, or Mythos 5.1 now use automatic tool choice for the initial request and corrective turns. Matching uses the API model ID, including aliases and routed provider IDs. The model must still call `StructuredOutput` with a result-file path; schema validation, retry limits, missing-result failures, and permission checks remain in force. Other model revisions retain their existing forced tool choice.
- **Built-in guidance follows the available tools**: system instructions and recovery messages no longer tell the model to delegate, search, or edit with tools absent from its request catalog. Custom instructions and plugin text remain unchanged. (#252)

@aictrl-dev

aictrl-dev Bot commented Nov 4, 2026

Copy link
Copy Markdown

Code review

Verdict: Looks good — only minor / nit comments below. · 🔴 0 · 🟠 0 · 🟡 1 · ⚪ 0 · 0/1 resolved

  • 🟡 CHANGELOG.md:19 — New changelog bullet missing trailing PR reference
🤖 Fix all 1 open findings with your agent
Fix the following code review findings on aictrl-dev/cli PR #262 (head branch).
Run the relevant tests/linters after each change.

1. CHANGELOG.md:19 — New changelog bullet missing trailing PR reference
   Detail: Every other bullet in CHANGELOG.md — across all sections, including the other bullets added in this same release cut — ends with a trailing issue/PR reference such as (#256) or (#243). The new "Structured output works with Claude revisions that reject forced tool choice" bullet added by this PR ends with "...retain their existing forced tool choice." and no reference, breaking the file's uniform convention and leaving the fix untraceable to its PR.
   Suggested fix: Append the PR reference to the bullet's last sentence, e.g. "...Other model revisions retain their existing forced tool choice. (#262)"
📋 Out-of-diff findings (1)
Sev Location Finding
🟡 CHANGELOG.md:19 New changelog bullet missing trailing PR reference

Reviewed 6 files · 0 inline · view all 1 findings ↗ · AI-generated — verify findings before acting · AI guidance and feedback


aictrl · AI code review for fast-moving teams · aictrl.dev

@byapparov

Copy link
Copy Markdown
Member Author

Review response — PR #262

Verified the single changelog consistency finding against the reviewed commit and addressed it on this PR branch.

Issues addressed (pushed to this PR)

  • Missing trailing PR reference — CHANGELOG.md:19: appended (#262) to the structured-output fix entry (commit a54e28009225eff49cbed4a8d980b11b0a10b95c). Runtime behavior is unchanged.

Review claims verified false (no change needed)

None.

Not addressed here

None.

Validation: git diff --check passed. Commit and push used the ordinary repository hooks; fresh CI and automatic review are tracked on the new commit.

@byapparov
byapparov merged commit 719a750 into main Nov 4, 2026
@byapparov
byapparov deleted the fix/sonnet-55-structured-tool-choice branch November 4, 2026 11:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant