-
Notifications
You must be signed in to change notification settings - Fork 253
[AMD] [AgentX] Add MI355X Qwen3.5 FP4 SGLang MTP #2562
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
cquil11
wants to merge
1
commit into
main
Choose a base branch
from
codex/mi355x-qwen35-fp4-agentx-mtp
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+203
−1
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
178 changes: 178 additions & 0 deletions
178
benchmarks/single_node/agentic/qwen3.5_fp4_mi355x_sglang_mtp.sh
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,178 @@ | ||
| #!/usr/bin/env bash | ||
| set -euo pipefail | ||
| set -x | ||
|
|
||
| # AgentX trace replay for Qwen3.5-397B-A17B MXFP4 on MI355X with SGLang | ||
| # native EAGLE MTP. Throughput uses the committed golden synthetic | ||
| # acceptance length; evaluation retains real target-model verification. | ||
|
|
||
| source "$(dirname "$0")/../../benchmark_lib.sh" | ||
|
|
||
| export EVAL_FRAMEWORK="lm-eval" | ||
|
|
||
| check_env_vars \ | ||
| MODEL TP CONC EP_SIZE KV_OFFLOADING \ | ||
| TOTAL_CPU_DRAM_GB RESULT_DIR DURATION | ||
|
|
||
| SCHEDULER_RECV_INTERVAL=${SCHEDULER_RECV_INTERVAL:-30} | ||
|
|
||
| if [[ -n "${SLURM_JOB_ID:-}" ]]; then | ||
| echo "JOB $SLURM_JOB_ID running on ${SLURMD_NODENAME:-unknown}" | ||
| fi | ||
|
|
||
| if [[ -n "${MODEL_PATH:-}" ]]; then | ||
| if [[ ! -d "$MODEL_PATH" || -z "$(ls -A "$MODEL_PATH" 2>/dev/null)" ]]; then | ||
| hf download "$MODEL" --local-dir "$MODEL_PATH" | ||
| fi | ||
| else | ||
| hf download "$MODEL" | ||
| export MODEL_PATH="$MODEL" | ||
| fi | ||
|
|
||
| rocm-smi || true | ||
| amd-smi || true | ||
|
|
||
| export WEKA_LOADER_OVERRIDE=semianalysis_cc_traces_weka_062126_256k | ||
| resolve_trace_source | ||
| install_agentic_deps | ||
|
|
||
| export AIPERF_SERVER_METRICS_URLS="http://localhost:${PORT}/metrics" | ||
| export AIPERF_REQUIRED_SERVER_METRIC_PREFIX="sglang:" | ||
|
|
||
| SERVER_LOG="$RESULT_DIR/server.log" | ||
| mkdir -p "$RESULT_DIR" | ||
|
|
||
| SERVER_PID="" | ||
| cleanup_agentic_services() { | ||
| local exit_code=$? | ||
| trap - EXIT INT TERM | ||
| set +e | ||
| stop_background_process_tree "$SERVER_PID" "SGLang server" 60 | ||
| exit "$exit_code" | ||
| } | ||
| trap cleanup_agentic_services EXIT | ||
| trap 'exit 130' INT | ||
| trap 'exit 143' TERM | ||
|
|
||
| CACHE_ARGS=() | ||
| WARMUP_ARGS=() | ||
| PAGE_SIZE=16 | ||
| if require_agentic_kv_offload_backend hicache; then | ||
| # Qwen3.5 creates target KV and Mamba host pools per rank. NEXTN adds a | ||
| # one-layer draft KV pool, or 1/15 of target KV. Convert the node-level | ||
| # workflow budget to SGLang's per-rank --hicache-size and retain the | ||
| # established MI355X 180 GB per-pool ceiling. | ||
| HICACHE_ALIGNMENT_RESERVE_GB=$TP | ||
| HICACHE_USABLE_TOTAL_GB=$((TOTAL_CPU_DRAM_GB - HICACHE_ALIGNMENT_RESERVE_GB)) | ||
| if [ "$HICACHE_USABLE_TOTAL_GB" -lt 1 ]; then | ||
| echo "Error: insufficient DRAM after HiCache alignment reserve." >&2 | ||
| exit 1 | ||
| fi | ||
| MAX_HICACHE_SIZE_GB=$((HICACHE_USABLE_TOTAL_GB * 15 / TP / 31)) | ||
| HICACHE_MAX_SIZE_GB_PER_RANK_POOL=${HICACHE_MAX_SIZE_GB_PER_RANK_POOL:-180} | ||
| if [ "$MAX_HICACHE_SIZE_GB" -gt "$HICACHE_MAX_SIZE_GB_PER_RANK_POOL" ]; then | ||
| MAX_HICACHE_SIZE_GB="$HICACHE_MAX_SIZE_GB_PER_RANK_POOL" | ||
| fi | ||
| HICACHE_SIZE_GB="${HICACHE_SIZE_GB:-$MAX_HICACHE_SIZE_GB}" | ||
| if [ "$HICACHE_SIZE_GB" -lt 1 ] || [ "$HICACHE_SIZE_GB" -gt "$MAX_HICACHE_SIZE_GB" ]; then | ||
| echo "Error: HICACHE_SIZE_GB=$HICACHE_SIZE_GB outside 1..$MAX_HICACHE_SIZE_GB." >&2 | ||
| exit 1 | ||
| fi | ||
| PROJECTED_HICACHE_TOTAL_GB=$(((HICACHE_SIZE_GB * TP * 31 + 14) / 15 + HICACHE_ALIGNMENT_RESERVE_GB)) | ||
| if [ "$PROJECTED_HICACHE_TOTAL_GB" -gt "$TOTAL_CPU_DRAM_GB" ]; then | ||
| echo "Error: projected HiCache use ${PROJECTED_HICACHE_TOTAL_GB} GB exceeds configured ${TOTAL_CPU_DRAM_GB} GB." >&2 | ||
| exit 1 | ||
| fi | ||
| echo "HiCache pools: ${HICACHE_SIZE_GB} GB per rank; projected node total ${PROJECTED_HICACHE_TOTAL_GB} GB." | ||
|
|
||
| # Offloaded runs use page size 1 with the proven ROCm direct/layer_first | ||
| # host-transfer path. | ||
| PAGE_SIZE=1 | ||
| CACHE_ARGS=( | ||
| --enable-hierarchical-cache | ||
| --hicache-size "$HICACHE_SIZE_GB" | ||
| --hicache-io-backend direct | ||
| --hicache-mem-layout layer_first | ||
| --hicache-write-policy write_through_selective | ||
| ) | ||
| WARMUP_ARGS=(--skip-server-warmup) | ||
| fi | ||
|
|
||
| PARALLEL_ARGS=( | ||
| --tp "$TP" | ||
| --dp 1 | ||
| --ep-size "$EP_SIZE" | ||
| ) | ||
|
|
||
| TOKENIZER_ARGS=() | ||
| if [ "$TP" -ge 4 ]; then | ||
| TOKENIZER_ARGS=(--tokenizer-worker-num 6) | ||
| fi | ||
|
|
||
| MAX_RUNNING_REQUESTS=$((2 * CONC)) | ||
| CUDA_GRAPH_MAX_BS="$CONC" | ||
| [ "$CUDA_GRAPH_MAX_BS" -gt 64 ] && CUDA_GRAPH_MAX_BS=64 | ||
| if agentic_kv_offload_enabled && [ "$CUDA_GRAPH_MAX_BS" -gt 16 ]; then | ||
| CUDA_GRAPH_MAX_BS=16 | ||
| fi | ||
|
|
||
| export PYTHONNOUSERSITE=1 | ||
| export SGLANG_USE_AITER=1 | ||
| export SGLANG_USE_AITER_UNIFIED_ATTN=1 | ||
| export AITER_FLYDSL_FORCE=1 | ||
| export SGLANG_MAMBA_SSM_DTYPE=bfloat16 | ||
| export SGLANG_TIMEOUT_KEEP_ALIVE=1800 | ||
|
|
||
| if [ "${EVAL_ONLY:-false}" != "true" ]; then | ||
| export SGLANG_SIMULATE_ACC_LEN=3.39 | ||
| export SGLANG_SIMULATE_ACC_METHOD=match-expected | ||
| export SGLANG_SIMULATE_ACC_TOKEN_MODE=real-draft-token | ||
| fi | ||
|
|
||
| SGLANG_CMD=( | ||
| python3 -m sglang.launch_server | ||
| --model-path "$MODEL_PATH" | ||
| --served-model-name "$MODEL" | ||
| --host 0.0.0.0 | ||
| --port "$PORT" | ||
| --trust-remote-code | ||
| "${PARALLEL_ARGS[@]}" | ||
| --attention-backend aiter | ||
| --mem-fraction-static 0.80 | ||
| --model-loader-extra-config '{"enable_multithread_load": true}' | ||
| --watchdog-timeout 1200 | ||
| --page-size "$PAGE_SIZE" | ||
| --cuda-graph-max-bs "$CUDA_GRAPH_MAX_BS" | ||
| --max-running-requests "$MAX_RUNNING_REQUESTS" | ||
| --max-prefill-tokens 32768 | ||
| --chunked-prefill-size 32768 | ||
| --scheduler-recv-interval "$SCHEDULER_RECV_INTERVAL" | ||
| --stream-interval 50 | ||
| "${TOKENIZER_ARGS[@]}" | ||
| --tokenizer-path "$MODEL" | ||
| --reasoning-parser qwen3 | ||
| --tool-call-parser qwen3_coder | ||
| --speculative-algorithm EAGLE | ||
| --speculative-num-steps 3 | ||
| --speculative-eagle-topk 1 | ||
| --speculative-num-draft-tokens 4 | ||
| --enable-metrics | ||
| --enable-cache-report | ||
| "${CACHE_ARGS[@]}" | ||
| "${WARMUP_ARGS[@]}" | ||
| ) | ||
|
|
||
| printf '%q ' "${SGLANG_CMD[@]}" | tee "$RESULT_DIR/sglang_command.txt" | ||
| printf '\n' | tee -a "$RESULT_DIR/sglang_command.txt" | ||
| "${SGLANG_CMD[@]}" > "$SERVER_LOG" 2>&1 & | ||
| SERVER_PID=$! | ||
|
|
||
| wait_for_server_ready --port "$PORT" --server-log "$SERVER_LOG" --server-pid "$SERVER_PID" | ||
|
|
||
| if [ "${EVAL_ONLY:-false}" = "true" ]; then | ||
| run_eval --port "$PORT" | ||
| else | ||
| build_replay_cmd "$RESULT_DIR" | ||
| REPLAY_CMD+=" --apply-chat-template" | ||
| run_agentic_replay_and_write_outputs "$RESULT_DIR" | ||
| fi | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🔴 Line 177 appends
REPLAY_CMD+=" --use-chat-template"afterbuild_replay_cmd, butbuild_replay_cmd(benchmark_lib.sh:1927) builds anaiperf profileinvocation for the agentic /v1/chat/completions replay path, not therun_benchmark_serving(benchmark_serving.py) path where this flag is actually recognized. This will makeaiperf profilereject the unrecognized argument and exit non-zero, failing every non-eval (throughput) concurrency point in this new recipe. Fix: delete line 177 entirely.Extended reasoning...
build_replay_cmdinbenchmark_lib.sh:1887-2053constructsREPLAY_CMDas an$AIPERF_CLI profile --scenario inferencex-agentx-mvp ...invocation, targeting--endpoint /v1/chat/completionswith--endpoint-type chat. Nowhere in this function (or anywhere in theaiperf profileargument set) is--use-chat-templatehandled — that flag is defined and consumed exclusively insiderun_benchmark_serving(benchmark_lib.sh:520for parsing,:636-638for application), which drivesutils/bench_serving/benchmark_serving.pyfor thefixed-seq-lenscripts, not the agentic replay path.In
qwen3.5_fp4_mi355x_sglang_mtp.sh:176-179, the non-EVAL_ONLYbranch callsbuild_replay_cmd "$RESULT_DIR"and then appendsREPLAY_CMD+=" --use-chat-template"before callingrun_agentic_replay_and_write_outputs "$RESULT_DIR". That function executes$REPLAY_CMDliterally atbenchmark_lib.sh:2112($REPLAY_CMD 2>&1 | tee "$result_dir/benchmark.log") and gates success on the exit code (replay_rc). Sinceaiperf profilehas no such flag, it will reject the unrecognized CLI argument and exit non-zero, causingrun_agentic_replay_and_write_outputsto treat the run as failed.This is not a stylistic mismatch — it's a straightforward "passing an unsupported flag to a CLI tool" bug. The only reason it isn't obviously wrong from a shallow read is that
--use-chat-templateis a real, valid flag elsewhere in the same file, which makes it plausible at a glance that it belongs here too. But tracing which commandREPLAY_CMDactually becomes shows it doesn't apply to this call site.Corroborating evidence from the rest of the codebase: (1) the direct sibling B200 recipe
qwen3.5_fp4_b200_sglang_mtp.shappends--server-metricsafterbuild_replay_cmd, never--use-chat-template; (2) no other agentic*_mtp.shscript (dsv4 variants, glm5.2, etc.) adds this flag toREPLAY_CMD; (3)kimik3_fp4_b300_vllm_mtp.sh:35-38has an explicit comment stating exactly why agentic recipes must not add it: "AGENTS.md requires MTP scripts to pass --use-chat-template to run_benchmark_serving. Agentic recipes never call it -- the replay drives AIPerf against /v1/chat/completions, so prompts are already chat-formatted ... Nothing to add here."Step-by-step proof of the failure: (1) Script reaches the
elsebranch (EVAL_ONLYdefaults tofalse, so this is the default/majority path — the sweep'sagentic-codingscenario inconfigs/amd-master.yamlruns manyconc-listthroughput points, e.g.conc-list: [1, 4, 8, 12, 16, ...]). (2)build_replay_cmd "$RESULT_DIR"setsREPLAY_CMD="$AIPERF_CLI profile --scenario inferencex-agentx-mvp --url http://localhost:$PORT --endpoint /v1/chat/completions --endpoint-type chat ... --output-artifact-dir $RESULT_DIR/aiperf_artifacts ...". (3) Line 177 appends--use-chat-templateto this string. (4)run_agentic_replay_and_write_outputsruns$REPLAY_CMD, i.e. invokesaiperf profile ... --use-chat-template. (5)aiperf's CLI parser (argparse or similar) has no such option registered for theprofilesubcommand and will error out ("unrecognized arguments: --use-chat-template") with a non-zero exit code. (6)replay_rcbecomes non-zero, and the run is recorded as failed — for every throughput concurrency point across all foursearch-spacerows in the newqwen3.5-fp4-mi355x-sglang-agentic-mtpconfig entry.The fix is simply to delete line 177 (
REPLAY_CMD+=" --use-chat-template"), leavingbuild_replay_cmd's output untouched, matching every other agentic*_mtp.shscript's pattern.