llama : decline the KV precision tail when no device serves it natively

[yogthos]

Overview

--kv-tail-tokens N is only a win when some device can run the tail with native tail attention. When no device can, planning still succeeds and every layer takes the generic tail route. llama-kv-cache.cpp already logs that route as "catastrophic generic attention", and the run then continues at a large throughput cost for a flag that was set to improve quality.

The Metal backend exports no tail-attention entry point at all, so every Metal build takes the generic route whenever a tail is requested. grep -rl ggml_backend_kv_tail_segmented_attention_supported ggml/src/ returns CUDA, Vulkan and CPU, and nothing under ggml-metal.

This adds a check in llama_context::llama_context before the cache is built. If no device backing a KV layer exports any tail-attention entry point, the tail is declined and the reason is logged. LLAMA_KV_TAIL_ALLOW_GENERIC=1 keeps the previous behaviour. An environment variable was used so there is no change to llama_context_params or to the CLI.

This follows the KVarN block a few lines below, which already declines itself when its runtime requirements do not hold, and the AGENTS.md invariant that unsupported placements fail closed rather than silently degrade.

Backends that implement tail attention are unaffected. Only the presence of the entry point is checked, so the per-layer planner still makes every finer decision. Loading with -ngl 0 puts the layers on CPU, which exports the entry point, and the new path stays quiet.

Additional information

Measurements from the same binary and the same command line. The only difference between the two rows is the environment variable.

• Hardware: Apple M1 Max, 64 GB, macOS 26.3, Metal backend
• Commit: 49e2b29
• Model: Qwen3.8-27B-Q8_0.gguf
• Prompt: 6066 tokens of filler followed by "Now write a 100-word paragraph about rivers.", max_tokens 160
• Sampling: --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 0.0 --repeat-penalty 1.0

build/bin/llama-server -m Qwen3.8-27B-Q8_0.gguf --port 11434 \
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 \
  --presence-penalty 0.0 --repeat-penalty 1.0 --parallel 1 \
  --n-gpu-layers 99 --batch-size 1024 --ubatch-size 256 \
  --flash-attn on --spec-type draft-mtp --spec-draft-n-max 2 \
  --cache-type-k q5_0 --cache-type-v q5_0 --kv-tail-tokens 1024 \
  --ctx-size 100000 --fit-ctx 100000 --jinja \
  --chat-template-kwargs '{"preserve_thinking": true, "reasoning_effort": "medium"}' \
  --chat-template-file chat_template.jinja \
  --no-mmproj-offload --threads 7 --threads-batch 8

	prompt	eval
generic tail route, LLAMA_KV_TAIL_ALLOW_GENERIC=1	26.6 tok/s	5.08 tok/s
tail declined, this PR	142.1 tok/s	11.92 tok/s

With the tail declined the "catastrophic generic attention" warning no longer appears, and the new line reads:

no device provides native KV tail attention; disabling the 1024-token KV precision
tail. The generic tail route costs several times more than plain quantized attention,
so it is not enabled by default. Set LLAMA_KV_TAIL_ALLOW_GENERIC=1 to keep it anyway.

One separate observation found while measuring this, not addressed here. On Metal, --cache-type-k and --cache-type-v must be the same type or flash attention is refused, because ggml-metal-device.m rejects GGML_OP_FLASH_ATTN_EXT when src[1]->type != src[2]->type. The q5_0 and q4_1 pair in the AGENTS.md example is fine on CUDA and silently costs flash attention on Metal. Both problems have to be fixed to reach the numbers above. I can open that separately if it is wanted.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES. The investigation, the patch and the benchmarks were produced with Claude Code. I reviewed the change and reran the measurements before submitting.