KVarN + draft-mtp recipe benchmark on RTX 5080 Laptop: quality and VRAM reproduce, speed does not (data + repro)

[brennenawana]

Hi — I benchmarked the KVarN/IQ4_XS-Smaller/draft-mtp recipe (as seen in the recent "Qwen 3.8 27B at 50 tok/s on 16GB VRAM" video) on the same GPU class (RTX 5080 Laptop 16GB) and got results well below the claimed speeds. Posting full data in case it's useful; happy to be told I misconfigured something.

Setup

• beellama.cpp built from main (2026-09-03), CUDA sm_120, driver 610.57.04 / CUDA 13.3.1
• jrell/Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller.gguf (13.5 GB)
• Flags: -fa on -ctk kvarn5 -ctv kvarn4 --kv-tail-tokens 1024 --kv-tail-type f16 -b 1024 -ub 256, MTP cells add --spec-type draft-mtp --spec-draft-n-max 2
• Probe: streaming OpenAI API, 3 runs/cell, medians; short = ~500-token prompt, long = 28k-token prompt, recall = 3 planted facts + code-navigation question

Results (short gen / long gen, tok/s)

Config	Claimed	Measured	VRAM
64k + MTP	50	16.7 / 5.0	13.4 GB
128k, no MTP	~30	10.2 / 1.6	15.6 GB
96k, no MTP	—	15.3 / 3.4	14.7 GB
Stock llama.cpp, unsloth UD-IQ4_XS, 64k, q4_0 KV	—	18.4 / 15.8	15.3 GB

Reproduced: quality (3/3 recall + nav in every cell — the IQ3_S FFN trade is probe-safe) and VRAM efficiency (128k boots at 15.6 GB).

Did not reproduce: speed. Even short-context decode is ~3x below claim, and long-context decode collapses to 1.6-5 tok/s — KVarN cache reads look ~4.6x slower than stock q4_0 at the same prompt depth. draft-mtp adds only a small gain (16.7 vs ~15.3 short at 64k).

Ruled out background load: A/B rerun of stock vs beellama under a 15.0 load average reproduced both sides' numbers exactly.

Questions:

1. Is there a known commit/flag combination that fixes the long-context KVarN read path?
2. Any known interaction between KVarN caches and -fa on / batch flags that would tank decode at depth?
3. Would profiling guidance help (I can capture timings if useful)?

Thanks for the project — the memory results are genuinely impressive; I'd love to make the speed part work on this hardware.

[Anbeeld]

First, please try https://github.com/Anbeeld/beellama.cpp/releases/tag/preview-v0.4.5, it has KVarN performance improvements.

For the rest, I've no idea what the video was about, but it was probably based on this post so please refer to it for more details.

[brennenawana]

Follow-up with more data points:

Build matrix, same recipe (64k ctx + MTP, short/long gen tok/s):

Variant	Measured
main @ cd3c41e, source build	16.7 / 5.0
main + -DGGML_NATIVE=ON -DGGML_CUDA_FA=ON -DCMAKE_BUILD_TYPE=Release	16.0 / 5.6
preview-v0.4.5 prebuilt (Ubuntu CUDA 13.3)	11.1 / 3.8
Stock llama.cpp, unsloth UD-IQ4_XS, 64k, q4_0 KV	18.4 / 15.8

Two findings that may help:

1. Timeline: the Reddit post's claimed config can't be older than KVarN's cache-type integration — the Aug-26 commit (925e117) rejects kvarn5 ("Unsupported cache type"), so kvarn args landed in the last week of Aug. preview-v0.4.5 (Sep 1) is the only public kvarn-era artifact from that window, and it's the slowest variant we measured.

2. CPU-bound signature on Blackwell: during decode the GPU sits near-idle (12-60% util, 25-35 W, quiet fans) while tok/s stays at 4-17 — KVarN cache management appears to fall back to CPU/inefficient paths on sm_120 rather than the fattn-mma-kvarn-decode kernels. Stock q4_0 keeps the GPU saturated at 15.8 tok/s at the same prompt depth. This matches the pattern of Bug Report: kvarn6 KV-quantization causes massive performance regression on ROCm (RX 7900 XTX) #122 (ROCm regression).

Also confirmed the OP's exact command doesn't fit a 5080 laptop at -ngl 99 + 100k ctx + MTP (~300 MB over our usable VRAM vs a 4070 Ti Super); --fit-ctx handles it if -ngl is left unpinned.

Closing this out from my side unless there's a config that should change the picture — happy to re-test a specific commit or flag set. Quality-wise KVarN is solid: 3/3 recall + nav in every cell, so the precision-tail claims hold; it's purely the throughput path.