[AMD] Add kimik3-fp4-mi355x-vllm-agentic-dspark (AgentX, conc 1/2/4/8) - #2551
[AMD] Add kimik3-fp4-mi355x-vllm-agentic-dspark (AgentX, conc 1/2/4/8)#2551Fangzhou-Ai wants to merge 4 commits into
Conversation
AMD MI355X (gfx950) sister of kimik3-fp4-b300-vllm-agentic-dspark: DSpark level-2 speculative decoding on the Kimi-K3 MXFP4 checkpoint, agentic-coding scenario, GPU-resident KV, conc sweep [1, 2, 4, 8]. Validated on rocm/pytorch-private:hy-kk-08092026 (TP8) at native 1M context with AITER asm-padded MLA, fp8 KV cache, gpu-memory-utilization 0.95, max-num-batched-tokens 4096, and max-num-seqs 8 to bound long-context prefill activations. Co-authored-by: Cursor <cursoragent@cursor.com>
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
ROCm sister of kimik3_fp4_b300_vllm_mtp.sh: serves moonshotai/Kimi-K3 (MXFP4, TP8) with the Inferact/Kimi-K3-DSpark level-2 draft head for the kimik3-fp4-mi355x-vllm-agentic-dspark recipe. Models are resolved by HF id. Encodes the validated MI355X (gfx950) config: AITER asm-padded MLA + AITER MoE, fp8 KV cache, gpu-memory-utilization 0.95, max-model-len 1048576, max-num-batched-tokens 4096, and max-num-seqs capped at 8 to bound long-context prefill activations. GPU-resident KV only. DSpark spec-config uses probabilistic drafting with synthetic acceptance pinned to golden AL 2.51 (real block verification under EVAL_ONLY). Co-authored-by: Cursor <cursoragent@cursor.com>
Follow the AMD agentic harness convention from dsv4_fp4_mi355x_vllm_mtp.sh: set -euo pipefail, require EP_SIZE/DP_ATTENTION in check_env_vars (pure-TP8 profile guards them), the PARALLEL_ARGS pattern, and the ROCm AITER env block before launch. DSpark stays at num_speculative_tokens=2 (NSPEC=2) as adopted for this recipe. Co-authored-by: Cursor <cursoragent@cursor.com>
Document the new MI355X Kimi-K3 agentic recipe: DSpark level-2 speculative decoding, AITER asm-padded MLA kernel, and fp8 KV cache at native 1M context. Co-authored-by: Cursor <cursoragent@cursor.com>
2 participants
Summary
Adds
kimik3-fp4-mi355x-vllm-agentic-dsparktoconfigs/amd-master.yaml— the AMD MI355X (gfx950) sister of the NVIDIAkimik3-fp4-b300-vllm-agentic-dsparkrecipe. Kimi-K3 MXFP4 with DSpark level-2 speculative decoding on theInferact/Kimi-K3-DSparkdraft head,agentic-codingscenario, GPU-resident KV, concurrency sweep[1, 2, 4, 8].DSpark is expressed as
spec-decoding: mtp(matching the NVIDIA-dsparkentry), level 2 (num_speculative_tokens 2), probabilistic drafting with synthetic acceptance pinned to the committed golden AL 2.51.Validated serve config (rocm/pytorch-private:hy-kk-08092026, vLLM 0.26.1rc1.dev, TP8, native 1M context)
VLLM_ROCM_USE_AITER=1/_MOE=1/_MLA=1, asm-padded MLA (VLLM_ROCM_AITER_MLA_ASM_PADDING=asm)--kv-cache-dtype fp8— halves KV footprint; required for a stable AITER MLA warmup at 1M--max-model-len 1048576— model-nativemax_position_embeddings--gpu-memory-utilization 0.95— the asm-MLA 1M warmup only converges at this util--max-num-batched-tokens 4096— chunked prefill--max-num-seqs 8— bounds concurrent long-context prefills so MoE/linear activations stay within the ~15 GB/GPU headroom left after fp8 KVReference data point (conc=4, 900s, semianalysis_cc_traces_weka_062126)
p90_e2e_normalized_interactivity_tok_s_user: 11.58total_throughput_tok_s_chip: 946.88 (8 GPUs)Notes
AgenticCodingConfig(each arm carries explicitkv-offloading;dram-utilizationomitted since there is no DRAM arm).hy-kkimage before the full sweep is scheduled.Test plan
hy-kkimage with the settings aboveMade with Cursor