[Tianlei Wu (tianleiwu)]

Summary

This draft PR consolidates and optimizes CPU/GPU hybrid expert offload for com.microsoft::MoEPacked (GGUF sub-byte block-packed experts) and com.microsoft::QMoE (RC5 mixed-precision INT4/INT3/INT2 experts) across 16 GB (RTX 5060 Ti), 24 GB (RTX 4090), 32 GB (RTX 5090), and H200 GPUs.

Stacked On

• [CUDA] Improve QMoE offload decode throughput via AVX2 spill elimination and working-set retention #33935 ([CUDA] Improve QMoE offload decode throughput via AVX2 spill elimination and working-set retention)
• [CUDA] Tail expert substitution for QMoE offload decode (stacked on #33935) #33944 ([CUDA] Tail expert substitution for QMoE offload decode (stacked on #33935))
• Support direct external initializer buffers in AddExternalInitializers #33877 (Support direct external initializer buffers in AddExternalInitializers)
• [CUDA] Add GatherPacked, MatMulPacked, and MoEPacked contrib ops for block-packed weights #33953 ([CUDA] Add GatherPacked, MatMulPacked, and MoEPacked contrib ops for block-packed weights)

Key Changes

1. End-to-End CPU/GPU Hybrid Offload for com.microsoft::MoEPacked (session_state.cc, moe_packed.h, moe_packed.cu, moe_packed.cc):
   • Places MoEPacked expert weight initializers (gate_weights, up_weights, down_weights) on CPU (zero-copy file mapping) and implements IMoeOffloadNode admission with QMoESessionController.
   • Implements CUDA-graph-capturable hybrid doorbell decode (MoEPackedOffloadClassifyAndBuildTilesKernel, MoEPackedDoorbellWaitKernel, MoEPackedOffloadFinalizeVec8Kernel, and LaunchQMoETailSubstitution integration) and 8,192-row double-buffered 32-expert wave-streaming prefill with D2D adaptive hot-expert promotion.
2. Lock-Free qmoe::SpinPool & Register-Accumulator AVX2+FMA+F16C GGUF Cold-Expert Kernels (moe_packed.cc, cmake/onnxruntime_providers_cuda*.cmake):
   • Backs MoEPackedParallelPool with qmoe::SpinPoolRegistry (qmoe_spin_pool.h) and pins MoEPackedDoorbellService to CpuForWorker(0).
   • Implements pure register-accumulator 2-channel (GgufDualDot2ChAvx2Tpl<RT>) and 4-column (IQ4_NL/Q2_0) AVX2+FMA+F16C kernels covering IQ2_S, IQ2_XXS, IQ2_XS, IQ3_XXS, IQ4_XS, IQ4_NL, Q2_0, Q5_0, and mixed gate_type != up_type layers, cutting iq3_xxs CPU cold-expert latency 4.8x (245.5 us/exp -> 49.2 us/exp) and boosting iq3_xxs decode throughput 4.1x (16.2–20.8 tok/s -> 67.2–81.7 tok/s lossless, 179.96 tok/s with tail substitution) and iq1_s decode throughput up to 142.91 tok/s lossless (261.65 tok/s with tail substitution) on RTX 5090.
3. QMoE INT2 Prefill & AVX-512 Cold-Expert Optimizations + 16 GB / 24 GB / 32 GB WDDM Reserve Calibration (qmoe_offload.cc, moe_gemv_tc.cuh, qmoe_shared_layout_int2_avx512.cc, qmoe_session_controller.h):
   • Dispatches MoeGemvTcInt2Kernel on sorted expert_map prefill waves (+24% to +30% faster 8K/32K prefill on 2b375, reaching 2,560 tok/s).
   • Extends Int2Group8Avx512 to RT = 1..4 with factored scale FMA.
   • Calibrates WDDM post-session reserves across 16 GB, 24 GB, and 32 GB GPUs for both QMoE and MoEPacked and auto-detects 8-byte vs. 4-byte STRP v1 popularity profiles.

---

Accuracy & Performance Impact of Tail Expert Substitution ( hreshold = 0.02, min_rank = 2) on 2b375 (RTX 5090 32 GB)

Evaluated on qwen_flash_next_int4_int8_rc5_contrib_dflash2_qmoe_2b375_x02_y14_z32_dp23_b64_unified (xhigh reasoning, greedy decoding):

• MMLU-Pro (400 samples, max_new_tokens = 32,768):
  • Without TailSub ( hr = 0.0): 82.75% (331/400, strict 81.75%), median 162.09 tok/s, token-weighted 230.99 tok/s
  • With TailSub ( hr = 0.02, r = 2): 83.25% (333/400, strict 82.25%) (+0.50 pp, 12W / 10L, McNemar p = 0.8318), median 255.64 tok/s (1.58x faster), token-weighted 328.52 tok/s (1.42x faster)
• LiveCodeBench v6 (175 problems, 256K = 262,144 limit):
  • Paired 161 / 175 problems on RTX 5090: 83.85% (135/161) With TailSub vs. 83.23% (134/161) Without TailSub (+0.62 pp, 13W / 12L, McNemar p = 1.0000), median 217.27 tok/s (1.54x faster), token-weighted 248.82 tok/s (1.65x faster)
  • Full 175 / 175 problems (at 256K): 77.14% (135/175, 43/43 = 100% Easy, 42/52 = 80.8% Medium, 50/80 = 62.5% Hard, conditional 88.24% when code is emitted) vs. 82.86% (145/175) lossless H200 reference.