Repository navigation
feat: Decision 3.0 (d3-lite/nano/mini) Core ML runtime - #37
Merged
Merged
Conversation
D3Manager runs vLLM Semantic Router's Decision 3.0 models on the GPU through Core ML: text graph per length bucket (E256/E640/E1536, fp16 compute, int8 weights), fp32 vision tower per patch bucket (P320/P1536/P3072), host-side embeddings, M-RoPE positions and the 255-way FP32 readout, one row per question. D3ModelStore downloads pinned, checksummed snapshots from the lite/, nano/ and mini/ folders of FluidInference/d3-coreml (revision 31d0a53b; all 69 pinned checksums verified against the repo). D3Check: 'example lite|nano|mini' and 'parity' against the Python Core ML host (0 token mismatches, 0 flips). Answers vs upstream PyTorch (bf16, MPS): text 2/0/1 flips of 100, images 1/24, 0/24, 0/48; ~46/65/160 ms per text question on an M5 Pro.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
D3Manager, which runs vLLM Semantic Router's Decision 3.0 models (d3-lite 0.85B, d3-nano 2.2B, d3-mini 4.5B) on the GPU through Core ML. A request takes text or JSON state, optional images and any number of choice, yes/no or score questions, and gets a probability for every option.D3ModelStoredownloads pinned, checksummed snapshots from thelite/,nano/andmini/folders of FluidInference/d3-coreml, pinned to revision31d0a53b.D3Check example lite|nano|minidownloads a model and answers a sample request.D3Check paritycompares the Swift host with the Python Core ML host.D3PromptTestscovers prompt building.Flips are answers that differ from upstream PyTorch (bf16 on MPS).
Verified locally:
swift build -c releasepasses.D3Check example litere-downloaded files from the new repo and answered (route=returns 0.979, 142 ms).Tests were not run locally because XCTest isn't installed here, so CI will be the first run.
🤖 Generated with Claude Code