Repository navigation
Faster prefill, tokenizer fix, draft-model speculation (1.3.0) - #9
Merged
Merged
Conversation
The first sweep ran cold (first touch of the loaded buffers, GPU clock ramp) and read 2-3x slower than the same matvec inside a decode step. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…dentical) qgemm is now one template over the per-simdgroup accumulator grid: the 32x32 tile (2x2 accumulators) replaces the old qgemm, and a 64x64 tile (4x4 accumulators, 2 MMAs per fragment load instead of 1) runs long batches. Both keep each output's chain (fp32 dequant, k ascending in 8-steps), so results are bit-identical to the old kernel and to each other. Full unrolling is what makes the 4x4 tile work (4x slower without it); padded rows and register prefetch measured slower on M2 Pro and are not used. Linear.useGemm64 picks the 64x64 tile when it pads at most 1.2x more rows and yields >= 64 threadgroups. flash_prefill stages K/V as half (exact: the cache is half) in two buffers with 16-byte loads of the next block during the current one's MMAs (one barrier per block), rescales O element-wise instead of by a diagonal MMA, and schedules the latest query blocks first. Same key blocking and softmax order, bit-identical output; ~1.9x faster at D=128, ~1.4-1.6x at D=256. Tests: both tiles against the CPU reference (odd O, K % 64 != 0, T up to 517), tiles agree bitwise across batch sizes, and prefill vs incremental decode for a 600-token prompt (two batches, the second at offset 512). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Replaces qmvb4; maxVerifyTokens 4 -> 8. Correct (kernel tests pass), not yet faster. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ng-context verify bench Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… split attention) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A small model of the same family drafts greedy tokens (its own KV cache, catch-up and k-1 GPU-sampled decode steps queued back to back, only the ids read back); the target verifies them in one batched pass with the exact acceptance rule. Prompt lookup stays as a free hybrid source. Load checks tokenizer compatibility and the memory of both models; usage reports draft_tokens / accepted_draft_tokens. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…f-draft greedy identity Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…EADME Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
🚀 Deploying Preview to Cloudflare 🚀Preview Deployments by commit
|
String equality is canonical equivalence, so Gemma's ; and U+037E collided: which id a semicolon got varied per process, and the losing id decoded to nothing. Token ids now match transformers on the tested Gemma and Qwen3 tokenizers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The draft catches up on the prompt after the first token, a non-4-bit target is refused before loading, a saved pick that does not fit is dropped instead of failing the load, tokenizer hashing leaves the main actor, and draft statistics move from env-gated stderr into GenerationStats. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jkyberneees
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
flash_prefill. Qwen3 0.6B 2,040 → 3,020 tok/s at 512 tokens, Mistral 7B 166 → 247. Decode is unchanged.;and U+037E) collided, so a;could get the wrong id, varying per process. Lookups now compare exact text.--draft <model|auto>, app picker, server usage fields). Output is identical, it is off by default, and it is currently slower: Qwen3 8B goes 38.1 → 27.9 tok/s on a short prompt, because verifying 3–8 tokens costs about 3 decode steps and about 45% of drafts are accepted.qmvx4handles 3–8 tokens and is ALU-bound at about 3× a step.Test plan
swift test: 153 tests pass locally with installed models;tokens with the wrong id;: identical to 1.2.2--draft autoon Qwen3 8B matches the plain run on short and 1,890-token prompts, with the draft active🤖 Generated with Claude Code