[Rui Ren (rui-ren)]

Description

Reuses the existing CPU/CUDA com.microsoft::MoEGroupRouter from #33817 instead
of adding MoERouterPacked. Router weights remain ordinary dense, unquantized
FP32/FP16/BF16 weights. No new standalone router schema or kernel registration
is introduced.

The backward-compatible extensions are an optional dense projection input,
paired optional INT32 expert IDs / FP32 mixture weights, and router_scale.
The original grouped dense-log-score interface and dispatch remain unchanged.
Compact routes currently support one group, up to 512 experts and top-k <= 32.
The projection path accepts rank-2 activations and widths divisible by 256.

The compact path preserves selection-only correction bias, lower-ID ties,
exceptional-score ordering, unbiased sigmoid weights, sequential selected-rank
FP32 normalization and post-normalization scaling. Routing is expressed through MoEGroupRouter followed by preselected MoEPacked.
MoEPacked no longer exposes an integrated sigmoid-router interface; its expert
arithmetic and packed layouts remain unchanged.

Shared CUDA routing kernels live in onnxruntime/contrib_ops/cuda/moe/.
The existing MoEGroupRouter wrapper remains in its legacy cuda/bert/ location;
MoEPacked reuses only the shared vector-load primitive for IQ2 experts;
it does not perform projected sigmoid routing.

Why optional projection? A separate FP32 MatMul/Sigmoid probe on 12 real
BF16 geometries, with TF32 both disabled and enabled, selected the same IDs but
did not preserve bitwise FP32 weights in any of the 24 comparisons
(maximum absolute difference 1.55e-6). This is an arithmetic-preservation
choice, not router quantization. No token-divergence claim is inferred from
that probe, and no log/exp round trip is used for compact weights.

Graph wiring: MoEGroupRouter([router_input, correction_bias, dense_weight], outputs=["", expert_ids, expert_weights], top_k=k, router_scale=scale).
Connect the compact outputs to MoEPacked inputs 5/6; omit its router_probs, set normalize_routing_weights=0, and preserve
the expert activation/quantization attributes. Consumers needing the original
dense output can still request output 0.

Dependency: #33969 in
native stack #33967.
No MoEPacked schema, packed payload or GenAI source change in this layer.

Current layer head: 1e5355fcfc4737c16899b4fec10d9c8ec7f09eba. The composition regression below is
validated in the combined stack; this does not claim a separate rerun of the
historical #33994-only benchmark.

Router separation verification

Owning-layer head: f3b1c3100df803646f9302175d924c1928219dbf. Combined-stack head (#34065): de9e2b2dc1766ee58e427850d34bf208729e5642.
The official production Release CUDA source build was run at both heads, with
Flash enabled, QUICK_BUILD disabled and no binary-object overlays.

• 19 focused owning-layer provider tests and 51 combined-stack
  provider/schema tests passed, with zero failures or skips.
• The compiled MoEPacked schema has 5..7 inputs and exactly six attributes.
  Removed router attributes and the former ten-input signature are rejected.
  Scoped schema/kernel documentation matches the repository generators.
• The chained router regression checks expert IDs, FP32 mixture coefficients
  and FP16/BF16 expert outputs against the independent host reference.
• Current frozen stack before/after controls matched all tokens, first-call
  emission counts and integer speculative counters in 18 complete request
  pairs across ordinary/k7 modes, generation, placement and short timing smoke checks.
  All three native capture/reset, candidate-prefix replay and opt-in R1/R8
  committed-state tests passed on both runtimes. Required operators ran on CUDA.
• Six admitted H100s were used in parallel; physical GPU1 and the foreign GPU0
  client were left untouched. Runtime library hashes and immutable payload hashes
  were checked. Existing fixture value-1 MatMulPacked rounding attributes were
  removed for both controls because the current parent already makes that the
  default; no production MatMulPacked precision code changed.
• Changed-source clang-format, whitespace and six-layer preservation audits passed.
  Expert CUDA kernels are byte-identical to their respective frozen layers.

Evidence: /home/mobius/bolt-moe-api-20261110-c8741c5a/.
These are focused source-bound checks, not independent HuggingFace parity,
statistically qualified performance results or full-CI approval.

Previous router-sharing qualification and benchmark (not current-head acceptance)

Previous verification

Commit: c7f2ea87c25a2b814db726fd4a0c36706c8286b7. The committed C++ files are verified byte-for-byte
against the executing source snapshot used for the source-built Release CUDA
runtime. The initial clean build was incrementally rebuilt for the updated
PR head 6c1fc43a30c7cee4e893b22b909362d144727f89; no binary object overlay
was used, and earlier-head passes are not substituted for current-head gates.
GenAI remains pinned at db6ffd29fc96e319f645ff461efa05c643235f29.
H100, CUDA 12.8.93, built-in CUDA EP, Flash enabled, QUICK_BUILD disabled;
no benchmark object overlay.

• 40 native MoEGroupRouter / MoEPacked-routing / MatMulPacked-precision tests passed, zero skips.
• 36 real component cases: compact INT32 IDs and FP32 weights matched the frozen
  original router exactly in a separate original-runtime process; public
  one-output MoEPacked integrated/reused BF16 outputs matched exactly.
• Same-mode A/B live prefixes matched 891 complete finite logits/cache/state
  tensors per mode over nine snapshots.
• All four CUDA placement checks passed, including all 40 reused routers.
• 240 complete same-mode natural-prompt generations, integer counters,
  self-consistency and reset/replay checks passed.
• Native capture/reset and identical-candidate prefix replay passed in both
  arms, including 368 bitwise live replay rows.
• Changed-source clang-format and whitespace checks passed. Schema/kernel
  documentation was generated from this runtime and scoped to MoEGroupRouter.

All six selected GenAI native tests passed (three per arm), including the
opt-in R1/R8 committed-state test with capture disabled/enabled and reset/replay.
Independent record checks confirm 776 R1/R8 boundary comparisons with no
differences and 8 matching complete native sequences. Ordinary/k7 tokens also match
in all 30 sampled natural-prompt cases on this updated stack. The older-head
cross-mode failure is retained historically, not waived or attributed to this head.
These are bounded quantized-runtime checks, not independent HuggingFace parity,
a new accuracy score, universal lossless speculation, or full-CI acceptance.

Benchmark scope: this candidate ends at PR #33994 above #33969. It does not
include downstream performance layers #34002/#34003. Absolute rates are not a
rerun of the earlier full-stack TPS tables; no performance attribution is made
for unrelated lower-stack changes.

Historical failed harness/qualification attempts remain in the earlier evidence
directory, including the invalid private three-output MoEPacked observer
(the published operator has one output). Current component checks use the public
expert output and a separate frozen-router process. No numerical threshold was
relaxed.

Short TTFT / decode benchmark

Before = integrated routing; after = reused MoEGroupRouter, on the same
candidate runtime. Warm-session synchronized TTFT includes append/prefill
through the first public emission, excluding loading and generator construction.
Decode TPS excludes the actual number emitted by the first call, not an assumed
one token.

Input tokens	Mode	TTFT before -> after (s)	Decode before -> after (tok/s)
1,024	ordinary	2.759 -> 2.767	65.79 -> 65.53
1,024	k7	2.756 -> 2.764	164.96 -> 164.38
2,048	ordinary	5.506 -> 5.522	65.84 -> 65.50
2,048	k7	5.494 -> 5.513	270.60 -> 269.35
8,192	ordinary	22.198 -> 22.244	65.17 -> 64.85
8,192	k7	22.173 -> 22.250	204.64 -> 204.07

Exact 1,024 / 2,048 / 8,192-token inputs use natural MMLU-Pro prompt text
concatenated and clipped to length, not repeated-token synthetic input and not
a scored single-question MMLU accuracy run. Natural EOS remains enabled with a
512-output-token cap; actual generated lengths were 75 at 1,024 input tokens; 512 at 2,048 input tokens; 512 at 8,192 input tokens.
Twelve same-GPU AB/BA pairs used all seven allowed H100s, reserving physical
GPU1: 120 measured generations plus 48 excluded warmups. All complete same-mode
A/B tokens, first-call emission counts and integer/histogram counters matched.
Two paired sessions and ten measured requests per arm/mode/length are a small
sample, not a significance or population-level claim. Compare within each
mode only; these same-mode performance results do not establish independent
HuggingFace parity or lossless speculation.

Evidence: /home/mobius/bolt-group-router-v2-20261109-c8741c5a/.
results.json, before-after.csv, evidence-seal.json, and
independent-audit.json preserve raw checks, recomputed metrics and hashes.

The existing stack and all seven members, including merged #33953, are
preserved. Active downstream #34002/#34003 branches are not force-rebased;
their owners need to synchronize them with this updated parent before
combined-stack qualification.

Historical measurements and qualification evidence (not current-head acceptance)

These previously reported results are retained for provenance. They have not been rerun on the rebased head above; earlier CI/build status and benchmark results do not establish current-head CI, accuracy, or merge readiness.

Earlier build and integration evidence

Fresh complete ORT Release CUDA source build on H100, with CUDA 12.8.93, CUDA90/Flash compilation, no QUICK_BUILD, no kernel-benchmark overlay and no reused benchmark runtime objects. Actual core/CUDA/shared-provider/Python-binding hashes and loaded mappings are recorded. Unchanged complete source-built GenAI is pinned at f1a3c6c79532bc2db8b809ce99d1357a453bd722.

• 12 native MoERouterPacked/MoEPacked routing tests passed, zero skips, including contiguous-K IQ2 cancellation.
• Four production GenAI ordinary/k7 integrated/standalone prefixes matched 891 complete logits/cache/state tensors each over nine snapshots.
• Standalone raw replay matched all 1,827 observer tensors exactly, including 720 router tensors, against preserved standalone evidence.
• 240 complete same-mode natural-prompt generation sequences, counters and reset/replay checks passed against the preserved qualified control.
• Both placements passed; all 40 standalone routers and coupled packed/Mamba/attention computations ran on CUDA.
• All twelve owned qualification GPU jobs drained. Scoped C++ lint and whitespace checks passed. Schema/kernel documentation is scoped to the new operator; descriptions stripped from the runtime registry are restored from the source schema.

This is equivalence to preserved quantized production execution, not an independent HuggingFace reference. The inherited 14/30 ordinary-versus-k7 differences remain; this PR does not establish lossless speculation, production qualification, new accuracy scores or full-CI acceptance.

Routing-boundary measurements

Before = integrated router; after = standalone router. These completed measurements use the earlier frozen extraction build, not a new performance measurement of this production refactor. Full-model timings include routing and MoEPacked's preselected validation/copy costs; production capture/provider policy is unchanged.

Input tokens	Ordinary before -> after tok/s	k7 before -> after tok/s
1,024	54.63 -> 54.32	342.60 -> 341.66
8,192	54.00 -> 53.76	346.12 -> 345.38
12,288	53.62 -> 53.42	346.22 -> 345.25
24,576	52.44 -> 52.27	343.90 -> 343.18
32,768	51.81 -> 51.65	342.12 -> 341.68
126,976	45.38 -> 45.24	310.58 -> 309.86

The long-context extension used two reversed-order epochs, four assigned H100s, twenty measured requests per arm/mode/input, two warmups per session and 512 generated tokens. All 448 complete sequences, including warmups, and integer/histogram counters matched same-mode A/B; all 64 GPU jobs drained. Paired session/request bootstrap confidence intervals are recorded. Observed decode overhead is small, not zero: approximately 0.30-0.36% ordinary and 0.13-0.28% k7 at these longer inputs; some k7 intervals include zero. k7 timing remains observational because the separate cross-mode failure is retained.

Evidence roots:

• /home/mobius/bolt-router-pr-20261108-c8741c5a/validation/
• /home/mobius/bolt-router-ab-20261108-c8741c5a/
• /home/mobius/bolt-router-long-context-20261108-c8741c5a/

The long-context final audit preserves the original parent seal and explicitly records one incidental regenerated Python bytecode cache. All 1,070 durable parent artifacts were unchanged; all ten cached code objects match the unchanged source. No numerical, token/counter, package, source, provider or request-count criterion was relaxed.

Checklist

• Tests added/updated
• Existing grouped dense routing remains compatible; optional interface documented
• Documentation updated
• Full CI and independent HuggingFace/lossless-speculation acceptance (not claimed)

[Copilot]

Note

Copilot was unable to run its full agentic suite in this review.

Copilot review overview

3 open issues

• The scratch buffer is allocated as uint8_t and then reinterpret_cast to float*. This assumes… · New
• This CU file uses expf, isfinite, and INFINITY but does not include a header that clearly… · New
• The 1e-20f additive constant is a magic number. Please replace it with a named constant (and… · New

What changed in this PR

Adds a new CUDA contrib operator MoERouterPacked to provide projected-sigmoid routing for MoEPacked, including schema/registration, shared CUDA implementation, tests, and documentation updates.

Changes:

• Introduce MoERouterPacked operator schema and CUDA kernel registration.
• Implement routing CUDA kernels (sigmoid scoring + deterministic top-k) and refactor MoEPacked sigmoid-routing path to reuse them.
• Add CUDA-focused unit tests and update operator documentation.

File	Description
onnxruntime/​test/​contrib_ops/​moe_router_packed_test.cc	Adds CUDA tests for routing behavior, tie-breaking, edge cases, and validation.
onnxruntime/​core/​graph/​contrib_ops/​ms_opset.h	Registers the new operator schema in the Microsoft opset.
onnxruntime/​core/​graph/​contrib_ops/​contrib_defs.cc	Defines the MoERouterPacked operator schema, constraints, and documentation.
onnxruntime/​contrib_ops/​cuda/​moe/​moe_router_packed_impl.h	Declares shared CUDA launcher and packed-load helper.
onnxruntime/​contrib_ops/​cuda/​moe/​moe_router_packed_impl.cu	Implements sigmoid projection and deterministic top-k selection CUDA kernels.
onnxruntime/​contrib_ops/​cuda/​moe/​moe_router_packed.h	Declares the CUDA kernel wrapper class for the new op.
onnxruntime/​contrib_ops/​cuda/​moe/​moe_router_packed.cc	Implements kernel validation, scratch allocation, and launch wiring.
onnxruntime/​contrib_ops/​cuda/​moe/​moe_packed.cu	Refactors existing sigmoid-routing to call the shared router launcher; reuses packed-load helper.
onnxruntime/​contrib_ops/​cuda/​cuda_contrib_kernels.cc	Registers MoERouterPacked CUDA kernels for FP16/BF16.
docs/​OperatorKernels.md	Documents kernel availability/signature for MoERouterPacked.
docs/​ContribOperators.md	Adds generated operator documentation section for MoERouterPacked.

_{Review effort: Lite}

[Rui Ren (rui-ren)]

Stack #33967 review — parent 5809f2531dcb -> head ebdbb8548801.

Reviewed the standalone schema/kernel registration, FP16/BF16 projection, selection-only bias, finite/exceptional ranking and lower-ID ties, selected-rank FP32 normalization, and the integrated MoEPacked call into the same shared implementation. No new actionable correctness finding in this extraction.

The current byte-scratch alignment comment is not a demonstrated defect: GetScratchBuffer delegates to the same CUDA allocator, and changing T changes the allocation's count units, not its alignment. The normal CUDAAllocator uses cudaMalloc; the arena preserves allocator alignment. A typed float scratch buffer can improve clarity, but it does not itself repair an alignment failure. Likewise, naming the inherited 1e-20f stabilizer may clarify intent, but preserve its value and normalization order when doing so.

This is static review feedback, not a new clean build, standalone-op runtime test, or approval of outstanding CI.

[Tianlei Wu (tianleiwu)]

Reviewed this layer (stack #33967, layer 4) at ebdbb85488.

Extraction fidelity. I diffed the removed MoEPackedSigmoid* kernels in moe_packed.cu against moe_router_packed_impl.cu after renames: kernel bodies, launch geometry and error checks are identical, and MoEPacked now goes through the shared LaunchMoERouterPackedCuda. I also ran the unmodified kernels on an H200 against an independent host reference (1,200 randomized FP16/BF16 cases with ties, +/-inf and NaN bias and activations, 16-byte-misaligned pointers, widths 256-2048, experts 1-512, k 1-32) and found no selection or weight mismatches. Schema, kernel registration and op-set registration look correct.

No blocking issues. Main requests (all inline):

1. The new tests use all-zero (or single-element) data, so common kernel regressions still pass them.
2. The ContribOperators.md entry cannot be regenerated from the schema; doc-gen validation will report a diff once this PR targets main.
3. Smaller API-robustness items: a lumped validation message and missing shape inference.

Non-blocking design notes

• The standalone op rejects rows > 2048, while MoEPacked chunks internally at 2048 rows, so a graph that swaps in MoERouterPacked must pre-chunk long prefills itself. If that is intentional, say "callers must chunk" in the schema text and pin it with a rows = 2049 rejection test; otherwise the same chunk loop can be used here, since the kernels have no row limit other than grid.x (the cap only bounds the FP32 score scratch).
• LaunchMoERouterPackedCuda is now a shared entry point whose memory-safety preconditions (width % 256 == 0, experts <= 512, 1 <= k <= min(32, experts)) are enforced only by its callers (moe_router_packed.cc for the op, DispatchMoEPackedCuda and moe_packed.cc for the integrated path). A host-side guard in the launcher is cheap and keeps the two call sites from drifting.
• The BF16 SM80 check runs in ComputeInternal on every Run(), whereas MoEPacked enforces it in the constructor, so an unsupported device fails at the first inference rather than at session creation.

On the existing Copilot comments: I agree the scratch-alignment concern is not a defect (same allocator, and arena chunks are 256-byte aligned); the <cmath> and epsilon-naming items are optional.

[Rui Ren (rui-ren)]

Updated in c7f2ea8 to reuse the existing CPU/CUDA MoEGroupRouter from #33817, with optional ordinary dense projection and compact IDs/FP32 weights. Router weights are not quantized; MoERouterPacked is removed and the legacy grouped interface remains supported.

Fresh source-built validation passed 40 provider/schema tests, all six native GenAI capture/state tests (including R1/R8), 36 real component cases and 240 same-mode generation checks. The audited seven-H100 1k/2k/8k TTFT/TPS table is in the updated description; observed decode overhead versus integrated routing is 0.28–0.51% in this small sample. These rates are for this layer, not the downstream performance-stack tip.

Coordination: #34002/#34003 branches and the complete native stack membership are untouched. Please synchronize those active downstream layers with the updated parent before combined-stack qualification; I have not force-rebased or merged them.

[Tianlei Wu (tianleiwu)]

Re-review of 232a1dfa59.

Verdict: COMMENT — but two compile errors introduced in this layer must be fixed before merge: they break the Windows CUDA and wasm32 builds, and #34002 / #34003 inherit them.

The standalone MoERouterPacked op from the previous revision is gone, so my earlier threads on it are moot. Their equivalents on the new code are in place: observed-value diagnostics in ValidateMoEGroupRouterProjection, shape inference that validates every known operand, a randomized comparison against an independent host reference, and the schema text as the source of truth for docs/ContribOperators.md (I compared the generated MoEGroupRouter section with bert_defs.cc; it matches).

Blocking (inline)

1. moe_group_router_impl.cu:157 — int64_t grid size → MSVC C4244 / C2220 on every Windows CUDA job (example).
2. linear_attention_gates.cc:300,302 — int64_t index into an InlinedVector → -Werror,-Wshorten-64-to-32 on wasm_Debug, wasm_Release and wasm_Release_static_library (example).

In the #34003 Windows CUDA job (which contains this layer's code) the other new/changed CUDA translation units (moe_packed.cu, matmul_packed.cu, gather_packed.cu, fused_state_space_impl.cu) compiled without warnings, so these two lines look like the whole delta for those platforms.

Independent verification (stack tip 623a725f5c, which contains this layer; Linux, CUDA 13.0, sm_80;sm_90a, Release; A100-SXM4-80GB)

• Build OK; the stack's 60 provider tests pass, including MoEGroupRouterTest (9), MoEGroupRouterProjectionShapeInferenceTest (2) and MoEGroupRouterProjectionTest (11: randomized reference, ties, non-finite inputs, chained-vs-integrated, runtime-geometry negatives); compute-sanitizer synccheck, racecheck and initcheck are clean on them.
• The CPU compact path runs in the same tests and also passed in the Windows CPU debug job of #34003 (onnxruntime_provider_test passed there; that job's only failure is onnxruntime_shared_lib_cpuinfo_dlopen_test, which this stack does not touch).

Non-blocking

• Test gap for the compact-path guards — inline on CompactOutputsMustBePaired.
• Public API surface: with compact outputs feeding MoEPacked inputs 5/6 (the "Graph wiring" in the description), MoEPacked's integrated sigmoid router (routing_function, router_scale, inputs 7–9, added in #33963) is a second public form of the same computation. See my inline comment on #33963; worth deciding before either ships.
• Nit: the CPU compact path allocates two InlinedVectors of num_experts elements per row inside the parallel-for (heap allocations once num_experts exceeds the inline capacity); per-batch scratch hoisted out of the row body would avoid them. Low priority for a reference path.

[Tianlei Wu (tianleiwu)]

Suggestion: the runtime guards of the compact path have no negative tests — this pairing test is the only one. Please add cases (CPU and CUDA) for:

• (a) compact outputs with n_group > 1 or topk_group > 1 — the compact kernels are single-group only, so relaxing that check would silently route wrongly;
• (b) projection_weight without compact outputs;
• (c) router_scale != 1 without compact outputs;
• (d) top_k > 32 / experts > 512 on the non-projected compact path (shape inference only checks these when projection_weight is present).

Each is a few lines on top of RunCompactCase / RunRouterAttributeFailure.