[Sayan Shaw (sayanshaw24)]

Why

Native model templates define tool-call protocols and correlated result history. Bypassing those contracts on continuation turns can make a model repeat completed calls, consume its budget in hidden reasoning, or miss legitimate pending work.

This PR targets main directly and consolidates the native speculative host integration formerly reviewed in #1307.

What

• Project qualified Qwen tools, assistant calls, correlated results, and supported reasoning history through the native template.
• Support native function/parameter XML and tagged-JSON tool-call dialects, including required and named choices for the qualified speculative model.
• Preserve legitimate repeated work, including the same tool with different arguments or an intentional identical call.
• Keep original-schema validation authoritative; malformed unexecuted calls may receive one bounded recovery attempt without discarding already streamed semantic output.
• Preserve template-qualified activation, literal prompt-data ownership, and fail-closed malformed-batch handling.
• Constrain the existing Qwen2 mandatory-tool JSON-fence format with grammar guidance, including inherited session choices and request overrides.
• Honor GenAI's per-model external-prefill capability when available. A compile-and-link feature probe preserves the existing defaults with older runtimes, including pinned GenAI 0.19.0.
• Fix a Windows test-variable shadowing warning treated as an error.

Runtime requirements

New model-specific prefill capability reporting requires microsoft/onnxruntime-genai#2909 and a runtime package containing it. Building against an older package remains supported; that does not make the new runtime/model available on the older package.

This is not a PR into another feature branch. #1307's changes are incorporated here; no separate child PR is needed.

Validation

Consolidated branch, built against the actual pinned GenAI 0.19.0 headers and library:

• SDK build completed through the repository build wrapper.
• 439 focused protocol, prompt, session, grammar, search-option, and capability tests passed; one optional package-fixture test skipped.
• All 9 real-model CPU tool-call SDK tests passed, including required-choice inheritance/restoration, request overrides, streaming, and result continuation.

Previously qualified combined-stack H200 runs passed 16/16 HTTP gates each for Qwen 3.8 27B, Qwen 3.8 Flash Next, and the native speculative model. These cover streaming/non-streaming, automatic/required/named choice, serial and parallel calls/results, same-tool/different-arguments work, and failed-result no-retry. GPU qualification was not repeated solely for branch consolidation.

Continuation fidelity is not a completed-call blacklist or a guarantee against model-selected repetition. An intentional identical call remains legal.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
foundry-local	Ready	Preview	Nov 10, 2026 1:49am UTC

[Copilot]

🟡 Changes recommended

Unresolved critical issues can cause freed-memory reads and route valid tool calls as ordinary text, bypassing parsing, validation, and repeat checks.

14 open issues

• Avoid reading a string view after modifying its backing string · New
• Preserve token provenance when replaying buffered continuations · New
• Fall back when native XML cannot represent the tool schema · New
• Limit completed-call interception to qualified native models · New
• Do not infer request completion from tool-name coverage · New
• Preserve parameterless schemas when adding retry exclusions · New
• Switch native XML retries to canonical JSON consistently · New
• Keep reasoning recovery within the original output budget · New
• Honor explicit response guidance during native activation · New
• Honor required and forced tool choices in native prompts · New
• Scope completed calls to the latest user request · New
• Preserve JSON escaping when restoring templated values · New
• Require decoder-compatible framing when qualifying XML templates · New
• Use the string-aware scanner to find the closing marker · New

What changed in this PR

This PR extends Foundry Local’s C++ chat pipeline to use template-qualified native Qwen tool protocols for multi-turn continuations, building on the protocol detection work in #1231.

Changes:

• Adds native XML and tagged-JSON prompt projection, parsing, and configurable tool boundaries.
• Tracks completed calls and introduces repeat-call and hidden-reasoning recovery.
• Expands capability and continuation tests, and includes the Qwen2 required/named fenced-output compatibility path.

File	Description
sdk_v2/​cpp/​test/​internal_api/​toolcalling/​tool_call_stream_accumulator_test.cc	Tests tagged-JSON parsing and custom boundaries.
sdk_v2/​cpp/​test/​internal_api/​genai_model_instance_test.cc	Tests template-qualified native capabilities.
sdk_v2/​cpp/​test/​internal_api/​chat/​chat_template_test.cc	Tests fenced required/named prompt instructions.
sdk_v2/​cpp/​test/​internal_api/​chat/​chat_session_test.cc	Tests repeat interception, retries, and capability precedence.
sdk_v2/​cpp/​src/​inferencing/​generative/​toolcalling/​tool_call_context.h	Adds native protocol and continuation state.
sdk_v2/​cpp/​src/​inferencing/​generative/​toolcalling/​qwen_xml_tool_call_decoder.h	Declares tagged-JSON parsing and configurable boundaries.
sdk_v2/​cpp/​src/​inferencing/​generative/​toolcalling/​qwen_xml_tool_call_decoder.cc	Implements tagged-JSON parsing and generalized XML boundaries.
sdk_v2/​cpp/​src/​inferencing/​generative/​genai_model_instance.h	Exposes inferred tool-protocol capabilities.
sdk_v2/​cpp/​src/​inferencing/​generative/​genai_model_instance.cc	Qualifies protocols through rendered-template probes.
sdk_v2/​cpp/​src/​inferencing/​generative/​chat/​prepared_chat_prompt.cc	Projects native tool definitions and conversation history.
sdk_v2/​cpp/​src/​inferencing/​generative/​chat/​chat_template.cc	Adds fenced tool-only instructions.
sdk_v2/​cpp/​src/​inferencing/​generative/​chat/​chat_session.h	Declares tool-support resolution helper.
sdk_v2/​cpp/​src/​inferencing/​generative/​chat/​chat_session.cc	Integrates native parsing, completion tracking, and recovery.

_{Review effort: Balanced}

[Sayan Shaw (sayanshaw24)]

Addressed the three additional findings from the latest review summary in 78c783b, with complete schema literal-data coverage in 87023c5:

• XML boundary ownership no longer interprets raw parameter quotes as JSON strings. A production regression on both backends and API shapes accepts an unmatched raw quote with a token-owned close, and rejects the same call with an ordinary-character closing marker without publication.
• Native schema dictionary keys are literal-protected, including properties and unused $defs names. Stable marker identities preserve template required-name membership; rendered definitions and references round-trip unchanged. Object-valued const/enum/default/examples are treated as data, not structural fields. Tests check actual control-token IDs against a benign baseline.
• XML activation uses the same shared name predicate as parsing/grammar generation, so unsupported function/property names retain canonical projection rather than selecting an undecodable native protocol.

Final expanded focused selection: 291 passed, 1 optional package test skipped. Fresh unchanged H200 HTTP gates on the final combined source: Qwen 27B 16/16, Flash Next INT4 16/16, tagged-JSON Generator 16/16, Harmony 14/14 supported cases. Main is current, the PR stacks and integration branch are updated, and all test-owned GPU servers are stopped. CI is still running.

[Copilot]

🔵 Needs a closer look

Human assessment is needed because native tool-call recovery can discard already-streamed text, while deeply nested arguments can exhaust the stack before validation.

No open issues 🎉

_{Review effort: Balanced}

[Sayan Shaw (sayanshaw24)]

Addressed both latest summary-review concerns in 3c00a74. Recovery now requires an exact token-prefix checkpoint whenever semantic output has already been emitted, including required/named choices; it cannot clear emitted text by falling back to whole-turn regeneration. Expanded the production regression across both backends, typed and Chat Completions APIs, and automatic/required/named choices. Supplied function argument history now uses the existing 64-level JSON nesting policy before DOM parsing or prompt projection; the existing schema scanner is shared rather than duplicated. Tests cover the exact limit, limit+1, 10,000 nested levels, and bracket characters inside strings. Final combined-stack validation passed 518 focused tests on both the updated runtime and the exact pinned GenAI 0.19.0 runtime (one optional package test skipped on each). No valid protocol selection, generation algorithm, completed-call filtering, or model gate prompt/assertion was changed.

[Sayan Shaw (sayanshaw24)]

Final post-fix real-model regression is complete on the latest combined stack: Flash Next 16/16, Qwen 27B 16/16, and the speculative native model 16/16 HTTP gates passed. Prompts/assertions were unchanged; coverage includes both streaming modes, required/named choice, result continuation, serial and parallel calls/results, same-tool/different-arguments, and failed-result no-retry. The published integration head 63d28ef has an SDK tree byte-identical to the tested source, including the latest compatibility/provenance fixes. All test-owned GPU servers have been stopped.

[Copilot]

🔵 Needs a closer look

Human assessment is needed to evaluate stack-exhaustion risks from unchecked JSON nesting and prompt-integrity risks from schema-token handling and NUL truncation.

No open issues 🎉

_{Review effort: Balanced}

[Sayan Shaw (sayanshaw24)]

Follow-up 15e905a closes a concrete C-string truncation edge case from the latest summary: every restored prompt fragment is validated before tokenizer entry. Raw embedded NUL now produces an explicit INVALID_ARGUMENT rather than truncating the prompt, while template-serialized JSON U+0000 remains lossless. The production fixture checks the exact raw-fragment error and the preserved JSON-escaped spelling. Historical function arguments and admitted schemas retain their existing 64-level tool nesting policy; the shared scan runs before argument DOM construction. These changes do not claim a new generic HTTP JSON transport policy. The final focused combined-stack suite now passes 519 tests (one optional package skip).

[Copilot]

🔵 Needs a closer look

Human assessment is needed because tagged-JSON serialization can exhaust the stack before depth validation, and reasoning-only continuation recovery is not connected in either request path.

No open issues 🎉

_{Review effort: Balanced}

[Baiju Meswani (baijumeswani)]

Summary

This is a useful improvement, but I cannot yet recommend approval as production Flash coding support.

The PR uses qualified model templates for tool definitions and conversation history, supports native XML and tagged JSON through Generator and Engine, and keeps shared schema validation. Generated-token provenance, literal-safe history, legitimate repeated calls, and prefix-preserving recovery are good improvements. The per-model prefill capability added in the latest delta also moves in the right direction.

Revision and validation scope

I checked the current source at 5e58bab8 against the previously tested 15e905a3. The Windows variable-shadowing issue is fixed in the new head, so I am not raising that resolved concern again. The decoder/admission and guided-recovery paths discussed inline remain unchanged.

The following real-model results are from 15e905a3, not a rerun of the latest head. Both SDK variants used the same controlled source-built runtime: microsoft/onnxruntime-genai#2896 with stream/warmup patches, guidance enabled, and pinned ORT. This is not qualification of the default released stack or latest ORT main.

Gate	Result
27B protocol comparison	Main 33/41; PR 35/41, with no newly failing cases
27B, unmodified Copilot CLI tool schemas	PR passed: 11 local requests, real error and failing-test recovery, exact source preservation, and seven independent passing tests. Main timed out at 600 seconds
27B, added minLength and oneOf profiles	Both timed out at 600 seconds
Supplied Flash package	Missing published tool-marker IDs; 0/6 schema controls and all three coding profiles failed
Separate Flash marker-ID diagnostic	6/41 protocol cases; 0/6 schema controls; all three coding profiles failed

The diagnostic Flash coding run read and searched the fixture and observed the deliberate error and failing tests, but then hit unsupported-guidance errors before completing the repair. Fixing package metadata alone is not enough. The constrained 27B coding profiles were not run on main, so I am not calling those timeouts a demonstrated regression from main.

Before approval

Please address the inline codec, admission, recovery, depth-limit, and dead-code concerns. Then verify a clean build and CI on the current head, including the new Qwen2 mandatory-choice and Generator prefill cases. Re-run the coding gate with normal and schema-rich tools on both Qwen models using the actual model packages and a documented runtime combination. A few simple tool calls are not enough to establish production coding support.

Keep the design small

I suggest four internal pieces:

1. An immutable model/backend capability snapshot: published controls, qualified formats, and support for guidance and prefix-safe recovery.
2. Small protocol descriptors for qualification, history projection, lossless representability, framing, and decoding.
3. One immutable per-turn decision that selects projection, output protocol, and supported recovery before generation.
4. The original shared ToolCallPlan as the semantic authority, with common atomic publication, provenance, prefix, cancellation, and budget handling.

An enum/descriptor and a small policy object are enough; there is no need for a plugin framework or a class per model. Remove unreachable retry logic and unused context fields/legacy guidance entry points rather than growing a second schema implementation. A shared conformance corpus should cover each qualified protocol on every supported backend, including a backend without guidance.

[Baiju Meswani (baijumeswani)]

Please make recovery depend on the loaded backend's capabilities instead of always clearing both native modes and enabling guidance.

The tested qwen38_flash_graph backend rejects guidance. The Flash BYOK diagnostic run got through reads, searches, and failing tests, but then received unsupported-guidance errors before completing the repair. Initial fallback after native-schema refusal has the same capability gap.

Select the protocol and its supported fallback/recovery together before generation. Use guided recovery only when supported; otherwise use a qualified, bounded native correction where it can preserve values and the published prefix, or reject an unsupported request clearly before generation. Add tests with a guidance-unavailable backend for both native-schema refusal and recovery from rejected native output. This avoids routing an accepted coding request into a recovery path that cannot run.

[Baiju Meswani (baijumeswani)]

Please separate XML representability from the legacy semantic-schema support checks used by schema.second.valid.

I reproduced this with the unchanged decoder: an ordinary string parameter is admitted, but adding minLength: 1 makes CanDecodeQwenXmlToolSchemas return false, even though a nonempty string has exactly the same wire representation and the original shared ToolCallPlan accepts it. This needlessly changes the protocol and can select unsupported guidance on Flash.

Keep the framing/type checks actually needed for lossless decoding, but let the immutable original ToolCallPlan enforce semantic constraints. Ensure the decoder uses that same separation, rather than only relaxing this gate. Add a test where adding a valid constraint leaves a representable payload on the native path and shared admission still rejects values that violate the constraint.

[Baiju Meswani (baijumeswani)]

The framing guard is useful, but the new native admission contract needs to account for the values this format cannot preserve.

With the unchanged decoder, an unrestricted string schema is admitted, but string data containing literal <tool_call> or <parameter=x> is rejected. Those spellings can legitimately occur in source files. Separately, an admitted string-or-integer anyOf decodes raw XML 123 as the integer, although the string "123" is also schema-valid. Post-validation cannot identify the intended type when both values are valid.

Please keep the guard and original-schema validation. Reject ambiguous scalar unions unless the wire format distinguishes their types, and qualify an explicitly escaped/self-delimiting format or a supported lossless alternative for arbitrary source strings. Add round-trip tests for numeric-looking strings, reserved markup, quotes, backslashes, newlines, and Unicode. Do not repair this by guessing types or changing source data.

[Baiju Meswani (baijumeswani)]

Please apply the shared bounded-depth scan before this native DOM parse and the later arguments.dump(), not only during shared call admission.

A depth-70 probe reached the native parsed-call stage and was rejected only afterward by the shared 64-level check. No invalid call was published and I did not reproduce a crash; the issue is that the limit currently comes after the parsing/serialization work it should bound.

Reuse IsJsonWithinDepthLimit before native JSON DOM construction, ideally in the common parse helper so structured XML parameter values are covered too. Handle envelope depth consistently and add near-limit and over-limit cases. This matches the earlier guard already added for transcript arguments.

[Baiju Meswani (baijumeswani)]

This native tagged-JSON retry branch is unreachable: the same function unconditionally sets native_tagged_json_tool_calls to false at its start, and nothing sets it back before this check.

Please remove the dead branch, or replace the implicit boolean transitions with the explicit supported recovery decision described above if native retry is intended. Keeping a substantial unreachable parser path makes it harder to know which recovery behavior is actually exercised and to add another model safely. Add coverage for the selected recovery strategy rather than retaining a branch that production cannot enter.

[bmehta001]

Found a reproducible native-prompt regression while qualifying this branch (5e58bab84c213cb082e4dc66645e3f44acf54944) with the real Copilot CLI.

Nullable schema type arrays break protected-fragment restoration. A minimal tool parameter such as "path": {"type": ["string", "null"]} triggers:

prepared_chat_prompt.cc: chat template changed a protected fragment's serialization

The restricted PowerShell-only Copilot offer worked, but its full default tool offering failed before generation. The same failure reproduced with a minimal nullable-type tool schema.

In PrepareTextChatPrompt, scalar schema type names are already treated as structural. Arrays of schema type names are instead protected as data; a native template rendering the array with | string changes the serialization witness. Treating schema type arrays consistently with scalar types fixes this without relaxing witness checks, original-schema validation, or protection of argument/data fields named type:

-                if (argument_data || dictionary_keys || key != "type" || !item.is_string()) {
+                // Schema type names are structural, including nullable arrays; argument "type" values are still data.
+                if (argument_data || dictionary_keys || key != "type" || (!item.is_string() && !item.is_array())) {

Self-contained regression for the existing QwenNativeProductionIntegrationTest fixture

TEST_F(QwenNativeProductionIntegrationTest, NativeProjectionPreservesSchemaTypeArraysRenderedAsText) {
  const auto directory = model_directory_ / "nullable-type-template";
  ASSERT_TRUE(std::filesystem::create_directory(directory));
  for (const auto& entry : std::filesystem::directory_iterator(model_directory_)) {
    if (entry.is_regular_file()) {
      std::filesystem::copy_file(entry.path(), directory / entry.path().filename());
    }
  }
  const auto tokenizer_path = directory / "tokenizer_config.json";
  auto tokenizer = nlohmann::json::parse(ReadText(tokenizer_path));
  tokenizer["chat_template"] = tokenizer["chat_template"].get<std::string>() +
      R"({% for tool in tools %}{% set function = tool.function if tool.function is defined else tool %})"
      R"({% for parameter_name, parameter in function.parameters.properties | items %})"
      R"(<schema-type>{{ parameter.type | string }}</schema-type>{% endfor %}{% endfor %})";
  WriteText(tokenizer_path, tokenizer.dump(2));
  auto loaded = load_manager_->LoadModel(directory.string(), "nullable-type-template", ExecutionProvider::kCPU);
  ASSERT_EQ(loaded.status, ModelLoadManager::LoadStatus::kSuccess);

  ToolCallContext context;
  context.native_qwen_xml_tool_calls = true;
  context.tool_output = true;
  context.tool_call_start = "<tool_call>";
  context.tool_call_end = "</tool_call>";
  context.tools_json = R"([{"type":"function","function":{"name":"read","parameters":{
    "type":"object","properties":{"path":{"type":["string","null"]}},"required":["path"]}}}])";
  context.tool_plan = ToolCallPlan::Create(context.tools_json);
  const auto prepared = PrepareTextChatPrompt(
      chat_internal::PrepareChatMessages({{FOUNDRY_LOCAL_ROLE_USER, "read the file"}}, false),
      *loaded.model, context);
  const auto baseline = loaded.model->GetPreprocessor().ApplyChatTemplateWithOptions(
      R"([{"role":"user","content":"read the file"}])", context.tools_json.c_str(), nullptr, true);
  const auto begin = baseline.find("<schema-type>");
  const auto end = baseline.find("</schema-type>", begin);
  ASSERT_NE(begin, std::string::npos);
  ASSERT_NE(end, std::string::npos);
  EXPECT_NE(prepared.prompt.find(baseline.substr(begin, end - begin)), std::string::npos);
  EXPECT_EQ(prepared.prompt.find("__foundry_literal_"), std::string::npos);
}

This uses only the fixture's checked-in tiny CPU model, not an external model download.

Validation after this change: 314 focused tests passed on Linux Release and Windows RelWithDebInfo, including the new regression and the existing adversarial literal/data protection tests; zero skipped or disabled tests. With the full default tool offering, real Windows Copilot successfully executed a nonce-file read and a separate exact-path file creation/read-back workflow against the Linux GPU runtime.

This is smoke qualification, not a claim that the full coding-agent benchmark has passed. I checked the other open PRs touching this area (#973 and #1313) and did not find an equivalent fix.

The independent fix for preserving Bolt's configured greedy default is in #1315; it does not depend on this native-tool PR.