Eval bug: Segmentation Fault using kvarn with mtp enabled on gemma-4-31B

[Dewamade]

Name and Version

./llama-cli --version
version: 11207 (ba27eda)
built with GNU 13.3.0 for Linux x86_64

Operating systems

Linux

GGML backends

CUDA

Hardware

RTX 3090

Models

gemma-4-31B-it-qat-UD-Q4_K_XL.gguf

Problem description & steps to reproduce

Got seg fault when using kvarn with MTP enabled on gemma-4-31B, with no MTP enable it works or using standard kv cache it works with MTP enabled

First Bad Commit

No response

Relevant log output

Logs

./beellama-unsloth-gemma4-31B.sh: line 28: 41424 Segmentation fault      (core dumped) /home/user/llama/beellama.cpp/build/bin/./llama-server -m /home/user/windows-drive/Ai-Model/unsloth/gemma-4-31B-it-qat-GGUF/gemma-4-31B-it-qat-UD-Q4_K_XL.gguf -md /home/user/windows-drive/Ai-Model/unsloth/gemma-4-31B-it-qat-GGUF/mtp-gemma-4-31B-it.gguf -ngl 999 -ngld 999 --ctx-size 163840 -b 2048 -ub 512 -np 1 -ctk kvarn5 -ctv kvarn4 -ctkd q8_0 -ctvd q8_0 --spec-type draft-mtp --spec-draft-n-max 1 --spec-draft-n-min 0.0 -fa on --jinja --chat-template-kwargs '{"enable_thinking":true}' --chat-template-file /home/user/windows-drive/Ai-Model/unsloth/gemma-4-31B-it-qat-GGUF/chat_template.jinja --temp 1.0 --top-p 0.95 --top-k 64 --cont-batching -cram 8192 --cache-reuse 256 --slot-save-path ~/.cache/slot --ctx-checkpoints 32 --metrics

[Anbeeld]

Current v0.4.4 contains several fixes in the KVarN + MTP paths that may also fix this crash, particularly around state synchronization, rollback, and KVarN stage ownership.

I don't have enough evidence yet to call this specific issue fixed, though. Could you retest the original command on v0.4.4 release? If it still crashes, please post the new log/backtrace and I'll investigate from there.

[Our0b0r05]

I'm also having this issue using v0.4.5 on windows.

[bethe410]

Current v0.4.4 contains several fixes in the KVarN + MTP paths that may also fix this crash, particularly around state synchronization, rollback, and KVarN stage ownership.

I don't have enough evidence yet to call this specific issue fixed, though. Could you retest the original command on v0.4.4 release? If it still crashes, please post the new log/backtrace and I'll investigate from there.

v0.4.4 KVarN + MTP crash log (same on v0.4.5-preview) in my box:

log-run-test.txt

hope it can be fixed! (●'◡'●)

[Anbeeld]

Pushed what should be a fix into v0.4.5 branch, please try it.

[Our0b0r05]

seems to be working on my machine. though setting --kv-tail-tokens 1024 still causes a crash. To be fair I don't actually know what this does and I just set it because it's in the example.

[bethe410]

Same as @Our0b0r05, without --kv-tail-tokens 1024 it works. with it still crash

0.04.736.258 I load_tensors: offloading output layer to GPU
0.04.736.271 I load_tensors: offloading 3 repeating layers to GPU
0.04.736.271 I load_tensors: offloaded 5/5 layers to GPU
0.04.736.279 I load_tensors:        CUDA0 model buffer size =   226.90 MiB
0.04.736.282 I load_tensors:    CUDA_Host model buffer size =   144.00 MiB
0.04.738.513 D load_all_data: using async uploads for device CUDA0, buffer type CUDA0, backend CUDA0
0.04.785.145 D load_all_data: buffer type CUDA_Host is not the default buffer type for device CUDA0 for async uploads
0.04.815.068 W llama_init_from_model: KVarN is target-context-only; disabling it for this auxiliary context
0.04.815.113 I llama_context: constructing llama_context
0.04.815.124 I llama_context: n_seq_max             = 1
0.04.815.124 I llama_context: n_ctx                 = 168192
0.04.815.125 I llama_context: n_ctx_seq             = 168192
0.04.815.125 I llama_context: n_batch               = 2048
0.04.815.126 I llama_context: n_ubatch              = 512
0.04.815.126 I llama_context: causal_attn           = 1
0.04.815.127 I llama_context: flash_attn            = enabled
0.04.815.128 I llama_context: kv_unified            = false
0.04.815.133 I llama_context: freq_base             = 1000000.0
0.04.815.134 I llama_context: freq_scale            = 1
0.04.815.135 I llama_context: n_rs_seq              = 0
0.04.815.136 I llama_context: n_outputs_max         = 1
0.04.815.136 I llama_context: n_outputs_max_per_seq = 1
0.04.815.136 I llama_context: n_ctx_seq (168192) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
0.04.815.190 D set_abort_callback: call
0.04.815.335 I llama_context:  CUDA_Host  output buffer size =     1.00 MiB
0.04.815.345 I llama_kv_cache_iswa: creating non-SWA KV cache, size = 168192 cells
0.04.817.406 D llama_kv_cache: layer   0: filtered
0.04.817.420 D llama_kv_cache: layer   1: filtered
0.04.817.422 D llama_kv_cache: layer   2: filtered
0.04.817.422 D llama_kv_cache: layer   3: filtered
0.04.817.427 I llama_kv_cache: size =    0.00 MiB (168192 cells,   0 layers,  1/1 seqs), K (none):    0.00 MiB, V (none):    0.00 MiB
0.04.817.427 I llama_kv_cache: attn_rot_k = 0, n_embd_head_k_all = 0
0.04.817.428 I llama_kv_cache: attn_rot_v = 0, n_embd_head_k_all = 0
0.04.817.431 I llama_kv_cache_iswa: creating     SWA KV cache, size = 1536 cells
Segmentation fault

[Anbeeld]

--kv-tail-tokens 1024 should be working now, please try it.