Repository navigation
Conversation
Bolt manages prefill internally and rejects external chunk_size settings. Preserve its model default while retaining explicit settings and the existing default chunking policy for other decoder types. Files changed: - sdk_v2/cpp/src/inferencing/generative/chat/search_options.cc - sdk_v2/cpp/test/internal_api/chat/search_options_test.cc Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3b7a6cfe-bac8-4d81-a588-6eb681fdf36d
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues are recorded for the changes to Nemotron Bolt’s prefill defaults.
No open issues 🎉
What changed in this PR
This PR adjusts Foundry Local’s prefill defaults for Nemotron Bolt, which manages prefill internally and rejects external chunking.
Changes:
- Avoid injecting the default
chunk_size=2048fornemotron_bolt. - Preserve positive, explicitly configured chunk sizes and existing defaults for other model types.
- Add focused regression tests using the checked-in tiny CPU model, without GPU dependencies or model downloads.
| File | Description |
|---|---|
sdk_v2/cpp/test/internal_api/chat/search_options_test.cc |
Adds CPU-backed tests for Bolt, other-model defaults, and explicit chunk sizes. |
sdk_v2/cpp/src/inferencing/generative/chat/search_options.cc |
Excludes Bolt from default external prefill chunking. |
Review effort: Balanced
bmehta001
changed the title
Rui Ren (rui-ren)
self-requested a review
November 10, 2026 06:05
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Do not inject the default external
chunk_size=2048formodel.type="nemotron_bolt". Bolt manages prefill internally, and ORT GenAI rejects external chunking when creating its generator.Preserve explicitly configured chunking and the existing defaults for other model types. Add focused regression tests using the checked-in tiny CPU model, without GPU dependencies, model downloads, or skipped cases.
Validation
./foundry_local_tests '--gtest_filter=ChunkedPrefillOptionsTest.*:SearchOptionsParsingTest.*:DynamicEngineChatTest.*'— 41 passed, zero skipped/disabled..\foundry_local_tests.exe '--gtest_filter=ChunkedPrefillOptionsTest.*:SearchOptionsParsingTest.*:DynamicEngineChatTest.*'— 41 passed, zero skipped/disabled.clang-format --dry-run --Werrorchecks andgit diff --check— passed.finishReason="stop".Scope
The H100 smoke used greedy decoding and the standalone CUDA EP. The custom launcher also disabled cuDNN flash attention to avoid a separate allocation-during-CUDA-graph-capture failure. This PR changes only Foundry's default prefill policy and its tests; it does not change CUDA packaging, attention dispatch, or sampling defaults.