cuda : backport GGML_CUDA_FA_QUANTS from upstream (ggml-org/llama.cpp#28079)

[cheese-cakee]

Overview

Fixes #267.

Backport of upstream ggml-org#28079 (5a4d0fe, by @pwilkin). With the default CUDA build, a K/V type pair without a compiled vector kernel (for example q8_0/q4_0) is now converted to f16 on the GPU with a one-time warning, instead of being rejected by the CUDA backend and running flash attention on the CPU without any message:

ggml_cuda_flash_attn_ext_vec: no FlashAttention vector kernel compiled for K/V types q8_0-q4_0, converting K and V to f16 instead (slow). Add "q8_0-q4_0" to GGML_CUDA_FA_QUANTS to compile it.

GGML_CUDA_FA_QUANTS selects which pairs get native kernels (default q4_0-q4_0;q8_0-q8_0;f16-f16;bf16-bf16, or all). GGML_CUDA_FA_ALL_QUANTS stays as a deprecated alias for all.

Additional information

The cherry-pick conflicted only in the docs/build.md options table (kept GGML_CUDA_PEER_MAX_BATCH_SIZE). fattn.cu merged without conflicts. The in-place q4_0/q8_0 MMA path from #221 still requires K->type == V->type, so mixed pairs take the f16 conversion path.

Testing on RTX 4050 Laptop (cc 8.9), CUDA 12.6, Linux (WSL2), -DCMAKE_CUDA_ARCHITECTURES=89:

• test-backend-ops -o FLASH_ATTN_EXT: prism default build 2994/2994 with all mixed cases "not supported"; this PR default build 4007/4007, mixed cases pass against CPU. prism with GGML_CUDA_FA_ALL_QUANTS=ON is also 4007/4007.
• Qwen3-0.6B-Q8_0, -ngl 99 -fa 1, GGML_SCHED_DEBUG=2: on prism the FLASH_ATTN_EXT nodes of all 28 layers run on CPU for q8_0/q4_0, f16/q8_0 and q4_0/q8_0; with this PR they run on CUDA0, and the warning is printed once per process.
• The shape from cuda: avoid short-query MMA FA crash with quantized V #189 (hs 256, GQA 6, kv 1280, 2-5 queries, f16 K with q8_0/q5_1/q4_0 V) passes.
• cuda: use the MMA flash attention kernel for GQA above 4 with quantized K/V on Ada #307, cuda: smaller KV tile for the 8-column MMA flash attention config (head size 256) #308 and cuda: avoid short-query MMA FA crash with quantized V #189 apply cleanly on top of this PR.
• Not tested: HIP and MUSA (their CMake files change the same way), other GPU architectures.

llama-bench, Qwen3-0.6B-Q8_0, -ngl 99 -fa 1 -t 8 -r 3, q8_0 K / q4_0 V, t/s, median of 5 alternating rounds:

test	prism default (FA on CPU)	this PR, default (f16 conversion)	prism + GGML_CUDA_FA_ALL_QUANTS=ON
pp512	888	13937	14378
pp512 @ d4096	47	8631	8881
tg128	78.7	190.7	186.5
tg128 @ d4096	17.3	99.6	143.3

The f16 conversion is much faster than the CPU fallback, but decode at long context is still slower than a compiled pair, which is what the warning says. For q8_0/q8_0 (already supported) the three builds are within run-to-run spread, except pp512 at depth 0, which varies by up to 15% between runs on this laptop.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES. AI was used for the backport and for running the tests; I checked and verified the change and the results afterwards.

[bri-prism]

@cheese-cakee @sb32445 @professorpalmer @Wyzix33, several open PRs touch flash attention with quantized K/V: #317 (upstream backport), #307 (MMA for GQA above 4), the FA commit inside #285, and #189. Our plan is to land #317 first, since it matches upstream.

• @Wyzix33, cuda : backport GGML_CUDA_FA_QUANTS from upstream (ggml-org/llama.cpp#28079) #317 handles the short-query shape from cuda: avoid short-query MMA FA crash with quantized V #189. In testing cuda: avoid short-query MMA FA crash with quantized V #189 also hit an illegal memory access in FLASH_ATTN_EXT and lowered MTP decode, so we'd like to close it in favor of cuda : backport GGML_CUDA_FA_QUANTS from upstream (ggml-org/llama.cpp#28079) #317 unless you see a case cuda : backport GGML_CUDA_FA_QUANTS from upstream (ggml-org/llama.cpp#28079) #317 misses.
• @sb32445 and @professorpalmer, cuda: use the MMA flash attention kernel for GQA above 4 with quantized K/V on Ada #307 and the FA commit in cuda+server: q8_0 K/V at the full 262k window on 12 GB (tiered VMM KV cache), MTP drafting at every depth #285 change the same routing. cuda+server: q8_0 K/V at the full 262k window on 12 GB (tiered VMM KV cache), MTP drafting at every depth #285's version already checks ggml_cuda_fattn_mma_kv_native_supported(dst), which cuda: use the MMA flash attention kernel for GQA above 4 with quantized K/V on Ada #307 needs too: without it, types the MMA kernel can't handle in place (head size 64, q5_x, mixed K/V after cuda : backport GGML_CUDA_FA_QUANTS from upstream (ggml-org/llama.cpp#28079) #317) would convert the whole cache to f16 on every decode step. Could you agree on one change, probably cuda: use the MMA flash attention kernel for GQA above 4 with quantized K/V on Ada #307 with that guard?
• @professorpalmer, cuda+server: q8_0 K/V at the full 262k window on 12 GB (tiered VMM KV cache), MTP drafting at every depth #285 bundles several independent features. Splitting it (tiered KV, MTP depth, server flags) would let the pieces land separately.

[professorpalmer]

Agreed on the order: #317 first.

On #307 vs the FA commit in #285 (quantized-KV GQA decode on the in-place MMA kernel): one change, #307 with the ggml_cuda_fattn_mma_kv_native_supported(dst) guard, is fine by me, and I am not opening a separate PR for that commit. @sb32445, if you would rather take the guard as a patch against your branch than fold it in yourself, say so and I will send it.

On splitting #285: the branch was six commits, each of which applies cleanly on its own and on the current prism head (eaecb50c7), so #285 is now replaced by one PR per feature, each with the receipts that belong to it:

• llama/cuda: tiered KV cache (--kv-vram-cells): VRAM head, pinned-host tail in one VMM range #319 llama/cuda: tiered KV cache (--kv-vram-cells): VRAM head, pinned-host tail in one VMM range
• speculative: --spec-draft-window (MTP drafting at every depth) #320 speculative: --spec-draft-window and --spec-draft-n-max-tail: MTP drafting at every depth
• server: --reasoning-effort-allow/-fallback and --reasoning-max-tokens-floor #321 server: --reasoning-effort-allow/-fallback and --reasoning-max-tokens-floor
• cuda: BATCH_INVARIANT: occupancy-independent FA KV split, PTQ1_0 mat-vec up to 8 columns #322 cuda: BATCH_INVARIANT: occupancy-independent FA KV split, PTQ1_0 mat-vec up to 8 columns
• cuda: GGML_CUDA_OP_TIMING=1 per-node GPU time breakdown #323 cuda: GGML_CUDA_OP_TIMING=1 per-node GPU time breakdown (diagnostics; drop it if you would rather not carry it)

The FA GQA-decode commit stays out, per the above. On the two speculative-deferral defects from the #221 review: both are fixed in prism as merged (begin() clears catchup_failed with the other per-task state; draft() marks the stash failed on a failed decode instead of dropping it), so none of these PRs touch that path.

[sb32445]

@professorpalmer @bri-prism thanks. I built the rule from your FA commit in #285 into my branch behind a temporary switch and compared the two on the 4070 (same binary, Bonsai 2 27B, q4_0 K/V).

• Normal operation: same result. For batch dimension 1 the two rules route identically (486 FLASH_ATTN_EXT shapes, also at kv=256) and run at the same speed: depth 0 with MTP n-max 2, 8 pairs, -0.02 % (CI [-0.10, +0.06] %); decode at 16k with one query 57.6 against 57.6 tok/s (prism before cuda: use the MMA flash attention kernel for GQA above 4 with quantized K/V on Ada #307: 51.7); 16k with MTP n-max 2 109.3 against 109.4. FLASH_ATTN_EXT passes with both rules (3022/3022 with 28 temporary cases for GQA 6 and 1 to 8 queries).
• One difference. Your block sits before the ggml_cuda_batch_invariant() check, so with GGML_CUDA_BATCH_INVARIANT=1 it sends 1 to 8 queries to the MMA kernel. With that mode on, MTP on and off gave the same text for 4 of 4 prompts with the vector kernel, and for 3 of 4 with your rule (3 runs, same prompt every time); your rule is 2.5 % faster there (98.6 against 96.2 tok/s). If that mode is meant to give identical output with and without MTP, the block belongs after the check.
• Architectures. Yours sits in the generic Turing branch, mine only in the Ada branch. I cannot test Turing or Ampere. Do you have 1-query decode numbers with and without the MMA route on your 2060? If it helps there, I am happy to extend the rule beyond Ada; your GGML_CUDA_FA_MMA_DECODE_MIN_KV makes the A/B easy.

My suggestion: keep the guard that is in #307 now (Ada, Q->ne[1] <= 2 && (gqa_ratio <= 4 || !native)) and extend it to Turing and Ampere in a follow-up once there are numbers. If you prefer your block, I would take it with the order fixed.

For the record, on the 4070 the guard does not change speed on top of the kTrain stack, which already has the MMA route from #307: depth 0 with MTP n-max 2, 12 pairs, -0.04 % (CI [-0.10, +0.02] %, n.s.); 16k without MTP 59.3 against 59.2 tok/s, same output hash. With q4_0 K/V the MMA kernel reads in place, so the guard is not reached. It only matters when K/V cannot be read in place.

The existing FLASH_ATTN_EXT cases only use quantized K/V at head size 64 and 72 with GQA up to 4, so they do not cover this route; I can add a few cases for it if you want them. Fine with #317 going first, I will rebase #307 and #308 on it once it is merged.

The checks were run with Claude Code (AI assistance).

[professorpalmer]

@sb32445 Turing numbers for 1-query decode, with and without the MMA route. On this card the MMA route is a clear gain, and the gain increases with depth.

RTX 2060 SUPER (sm_75), our 8 GB engine (bonsai-8gb, which contains the FA commit from #285), Bonsai 2 27B PTQ1_0, GQA 6, no MTP head, -c 32768, all in VRAM, stock clocks, greedy, 3 prompts x 400 tokens per depth. The server restarts for each run. Two rounds, alternated. "no MMA" is GGML_CUDA_FA_MMA_DECODE_MIN_KV=0. The default (256) takes the MMA route.

K/V	route	4k decode tok/s	16k decode tok/s
q4_0	MMA	43.5 / 43.8	38.9 / 39.0
q4_0	no MMA	41.2 / 41.3	32.9 / 32.9
q8_0	MMA	43.7 / 43.7	38.9 / 38.9
q8_0	no MMA	42.2 / 42.2	34.3 / 34.2

MMA vs no MMA, mean: q4_0 +6 % at 4k and +18 % at 16k. q8_0 +4 % at 4k and +14 % at 16k. I did not compare the output texts in this test.

On the batch-invariant point you are correct. With GGML_CUDA_BATCH_INVARIANT=1 the block must not change the route, so it belongs after the check. I agree with your plan: keep the guard that is in #307 now, and extend it to Turing (and Ampere, when somebody measures it) in a follow-up. These numbers are for that follow-up. I can run more shapes or depths on this card if you need them.