[Rui Ren (rui-ren)]

Description

Extends com.microsoft::MatMulPacked (CUDA) with the scaling and operand-precision
contract needed by Nemotron Bolt, without repacking native weights. The optional scalar
FP32 NVFP4 global_scale is applied after FP32 accumulation and before the output cast.

Complete decoded weights now round to the activation dtype by default. The precision
policy is session-only: ep.cuda.matmul_packed_allow_unrounded_gemv="0" (or unset)
retains rounding, while "1" permits eligible GEMV kernels to skip it. Tensor-core
kernels may still round. GEMM/prefill always rounds, including with FP32 scratch buffers.
Only "0"/"1" are accepted; invalid values fail kernel/session initialization.

The dequantize_to_activation_dtype node attribute is removed. Existing exports carrying
that attribute must remove it or be regenerated; it is no longer an operator attribute.

The BF16 decode path preserves contiguous K partitions and MMA operand ordering.
Tensor Core dispatch checks compiled-image support as well as GPU capability so
compute_75-only PTX correctly uses SIMT on newer GPUs. Prefill retains bounded tiled
dequantization/GEMM, including the FP16 FP32-accumulation path.

Dependency: first follow-up to merged #33953; this PR now targets main.
Part of native stack #33967.
No CPU-format expansion, weight repacking, or GenAI source change.

Summary of Changes

Operator Contract and Documentation

File	Change
onnxruntime/core/graph/contrib_ops/contrib_defs.cc	Add the optional NVFP4 multiplier; define rounded operands by default without a precision attribute.
docs/ContribOperators.md, docs/PackedWeightFormats.md	Document the scale input, default rounding, session opt-out, and FP16 overflow semantics.

CUDA Implementation and Regression Tests

The session key is declared in include/onnxruntime/core/session/onnxruntime_session_options_config_keys.h and resolved once per kernel, without an environment alias or model attribute.

File	Change
onnxruntime/contrib_ops/cuda/quantization/matmul_packed.{cc,h,cu}	Validate the scale, apply it before the output cast, retain bounded prefill, and select rounded BF16 TC/SIMT paths using compiled kernel capability.
onnxruntime/contrib_ops/cuda/quantization/gguf_quant_cuda.cuh	Add the NVFP4 eight-element sub-block decode used by the projection path.
onnxruntime/test/contrib_ops/matmul_packed_precision_test.cc	Cover default/explicit 0/1 policies, invalid values, FP16/BF16 complete operands, prefill boundaries, scale-before-cast, nonzero fallbacks, and cancellation.

Testing

Session-policy update: owning head 2936957a4d9bfccaa070429938446c0c1482228b; optimized stack tip a3f2076b65e0d1a5c7e9b82165ffab014e233876.

• Owning layer: 10 native CUDA + 10 compute75-only PTX tests, plus 3 Python tests
  covering all packed formats, session-policy dispatch boundaries, and FP16 operand overflow.
• Optimized stack tip: 15 native CUDA + 15 compute75-only PTX tests, 5 Python tests
  including byte-offset IOBinding fallbacks, and 1 actual internal CUDA cancellation test.
• Zero failures/skips. Production host/CUDA/schema/test sources compile; generated MatMulPacked
  documentation matches the production schema. CLANGFORMAT, Ruff/format and whitespace checks pass.

Runtime checks use fresh policy objects and the exact MatMulPacked schema registration over
ABI-matched preserved support. Not a clean full-source build, full-CI approval, independent
HF parity, performance remeasurement, or new end-to-end model qualification.
NVFP4 global-scale placement and unrelated MoE/router ownership remain unchanged.

Historical measurements and qualification evidence (not current-head acceptance)

Earlier review qualification (historical)

Review-fix snapshot: 1929a7099606c61f00a40f2e199007ee090eda3e.

Documented NVFP4 ID/layout, bit7-clear scale domain, scale edge cases, K geometry, CUDA-only availability and FP32 global-scale placement. Added default/opt-in scale tests, large-N default dispatch coverage, and an explicit failing switch default. Revised the cancellation fixture so both the native TC and compute75 PTX SIMT paths assert 2.

Owning-layer validation: 8 native + 8 compute75-only PTX projection tests; zero failures/skips.

Integrated tip regression: 51 C++ provider/schema tests, 5 actual internal CUDA tests and 2 Python IOBinding tests passed, zero failures/skips. All owning-layer provider cases are retained. The final overlay recompiles the changed CUDA/host/schema/test sources over ABI-matched preserved ORT support; not a clean full-source build, full-CI approval, independent HF parity or new end-to-end qualification. Targeted configured clang-format/Ruff and whitespace checks passed.

These previously reported results are retained for provenance. They have not been rerun on the rebased head above; earlier CI/build status and benchmark results do not establish current-head CI, accuracy, or merge readiness.

Historical policy labels: the table predates this session-policy change. 'Default' below denotes the former unrounded-GEMV policy (now permitted by the session opt-out); 'Precision opt-in' denotes rounded operands (now the default). The old attribute is no longer available. These timings have not been remeasured.

Measured CUDA kernel latency - 2026-11-07

BF16 inputs/output and FP32 accumulation, Q8_0/NVFP4 packed weights; NVFP4 global scale 0.75. Default dispatch is compared with dequantize_to_activation_dtype=true. For rows >8 both paths use full dequantization plus cuBLAS (and the NVFP4 output-scale pass).

This is a precision option, not a universal speedup: single-row latency regresses in all four shapes/formats; Q8_0 eight-row dispatch improves, NVFP4 eight-row results depend on shape, and 32-row results are approximately unchanged. Current operand rounding is now the default. All 12 synthetic cases have bitwise-equal complete outputs in both runs; these representable synthetic scales do not replace the separate adversarial precision tests.

Format	N x K	Rows	Default us	Precision opt-in us	Default / opt-in
Q8_0	18560 x 4096	1	53.82	95.33	0.56x
Q8_0	18560 x 4096	8	108.98	95.49	1.14x
Q8_0	18560 x 4096	32	162.65	162.57	1.00x
Q8_0	4096 x 8192	1	34.77	39.17	0.89x
Q8_0	4096 x 8192	8	71.73	40.88	1.75x
Q8_0	4096 x 8192	32	80.23	80.23	1.00x
NVFP4	18560 x 4096	1	51.80	132.81	0.39x
NVFP4	18560 x 4096	8	119.29	134.85	0.88x
NVFP4	18560 x 4096	32	154.32	154.29	1.00x
NVFP4	4096 x 8192	1	29.83	54.09	0.55x
NVFP4	4096 x 8192	8	66.63	56.90	1.17x
NVFP4	4096 x 8192	32	71.00	70.98	1.00x

H100 80GB HBM3, CUDA 12.8.93, driver 580.159.04, GCC 11.4.0. CUDA-event median per dispatch; 64 dispatches per captured CUDA graph, 50 warmups, three graph warmups, and 15 paired timed batches alternating AB/BA. Four rotating allocation banks contain deterministic synthetic weights (seed 617); this is not a cold-cache guarantee or a checkpoint-weight benchmark. Raw p95 values describe batch-average latency, not individual-invocation tails. No profiler runs in the timing window. The assigned GPU had empty admission, one stable compute client during execution, and clean release afterward; clocks were not locked. Two independent 39-case runs passed, including the published replay runner.

Scope: same-build production CUDA dispatch alternatives, not historical parent-commit binaries. The measured runtime is an isolated CUDA provider relink with preserved core/support, not a clean full-source ORT build or end-to-end model qualification. Each measured operator's CUDA source hash matches its owning PR head. A clean source build is a separate replication variant and must be measured again. Kernel-only timings do not establish model TTFT/tok/s, MMLU/LCB, speculative integration, or full-model readiness.

Complete primary/repeat results, numerical metrics, source hashes and benchmark code. All 39 cases passed in both supervised runs; full-model work remains paused. No source push, stack change, merge, auto-merge or ready-for-review promotion is included.

Checklist

• Review feedback addressed with focused regressions
• Documentation and supported contracts updated
• Native stack order and immediate-parent dependencies preserved
• Clean full-source/full-CI qualification (not claimed by focused overlays)

[Rui Ren (rui-ren)]

The native kernel optimizations are now pushed in #34002 and registered as the
new top of merge stack #33967.
This changes the intended merge stack, not alternative #33940. Existing layer
heads and their history are preserved.

Actual #33994-tip baseline versus the new native candidate, real 1028-1663-token
inputs: 54.25 -> 92.04 ordinary tok/s; 201.48 -> 318.91 k=7 tok/s.
Both arms used the same native package and corrected GenAI. All 240 complete
measured generations/counters, 1,782 full live tensor/state hashes per arm and
22 native CUDA regressions passed; no skips.

The earlier ~92 tok/s private baseline already contained these optimizations:
it was not the unmodified remote stack. The separate #33940 ~109/~370 result is
still not a result of this native candidate. That remaining gap is outstanding.
Full protocol, source pins and limits are in #34002; it remains draft.

[Copilot]

🟡 Changes recommended

The BF16 Tensor Core path can silently return zeros for 75-virtual builds on SM80+ GPUs because dispatch checks hardware capability but not whether the compiled kernel supports BF16 MMA.

3 open issues

• Gate BF16 Tensor Core dispatch on compiled kernel capability · New
• Correct GQA documentation for unsupported batch and sequence behavior · New
• Regenerate WebGPU operator table from supported registrations · New

What changed in this PR

Extends the packed-weight operators introduced by #33953 with Bolt-specific projection precision controls. The main addition is optional NVFP4 global scaling and complete-weight rounding in CUDA MatMulPacked, while preserving the default dispatch behavior.

Changes:

• Add FP32 global scaling before the output cast and opt-in activation-dtype weight rounding.
• Add contiguous-K BF16 dispatch, alignment-safe packed reads, and precision regression tests.
• Update operator documentation and pin the transformer-test SentencePiece dependency.

File	Description
tools/​ci_build/​requirements/​transformers-test/​requirements.txt	Pin SentencePiece for CI protobuf compatibility.
onnxruntime/​test/​python/​transformers/​test_gguf_packed_ops.py	Test packed operators, formats, and validation.
onnxruntime/​test/​contrib_ops/​matmul_packed_precision_test.cc	Cover scaling, rounding, and cancellation behavior.
onnxruntime/​core/​graph/​contrib_ops/​ms_opset.h	Register packed operator schemas.
onnxruntime/​core/​graph/​contrib_ops/​contrib_defs.cc	Define packed schemas and projection precision options.
onnxruntime/​contrib_ops/​cuda/​quantization/​matmul_packed.h	Declare projection parameters and launch interface.
onnxruntime/​contrib_ops/​cuda/​quantization/​matmul_packed.cu	Implement projection dispatch, rounding, and scaling.
onnxruntime/​contrib_ops/​cuda/​quantization/​matmul_packed.cc	Validate inputs and allocate projection workspace.
onnxruntime/​contrib_ops/​cuda/​quantization/​gguf_quant_cuda.cuh	Provide shared packed decoders and alignment-safe reads.
onnxruntime/​contrib_ops/​cuda/​quantization/​gather_packed.h	Declare CUDA packed-gather interface.
onnxruntime/​contrib_ops/​cuda/​quantization/​gather_packed.cu	Gather and decode packed rows on CUDA.
onnxruntime/​contrib_ops/​cuda/​quantization/​gather_packed.cc	Register and validate CUDA packed gather.
onnxruntime/​contrib_ops/​cuda/​moe/​moe_packed.h	Declare packed-expert parameters and interfaces.
onnxruntime/​contrib_ops/​cuda/​moe/​moe_packed.cu	Implement packed-expert routing and computation.
onnxruntime/​contrib_ops/​cuda/​moe/​moe_packed.cc	Validate expert inputs and manage chunked execution.
onnxruntime/​contrib_ops/​cuda/​cuda_contrib_kernels.cc	Register CUDA packed kernels.
onnxruntime/​contrib_ops/​cpu/​quantization/​gather_packed.cc	Implement CPU packed gather and dequantization.
onnxruntime/​contrib_ops/​cpu/​cpu_contrib_kernels.cc	Register CPU packed-gather kernels.
docs/​OperatorKernels.md	Update packed-kernel listings and WebGPU type tables.
docs/​ContribOperators.md	Document packed contracts and update GQA descriptions.

_{Review effort: Balanced}

[Rui Ren (rui-ren)]

Stack #33967 review — parent 548473fa9111 -> head 9e194f240c12.

Reviewed this layer against its immediate parent, including scaling before the final cast, bounded prefill workspace, opt-in operand rounding, compiled-image BF16 capability gating, and the added tests.

One reproducible test/configuration issue, detailed inline: RoundedBf16ContiguousKCancellation assumes Tensor Core reduction order even when a compute_75-only image correctly selects SIMT. On H100, rebuilding the production MatMulPacked TU with compute_75-only PTX and running that test produces 1 instead of the asserted 2. The previously checked nonzero/complete-operand rounding fallback regressions pass; this is a fixture capability-gating problem, not the old zero-output dispatch bug.

Validation: targeted H100 reproduction using the ABI-matched focused runtime overlay, not a full clean-source ORT build. No source changes made for this review.

[Tianlei Wu (tianleiwu)]

Reviewed this layer (base main, bottom of stack #33967) at 9e194f240c.

I traced the new NVFP4 decode, the contiguous-K BF16 Tensor Core kernel (operand pairing, decode indices for 64/128-K chunks, tile tails, reduction order), global_scale application on all four routes (GEMV, TC, FP32 prefill, cuBLAS prefill), workspace sizing, and cuBLAS math-mode restore on the error paths. I found no correctness defect in the kernels, and the default dispatch is unchanged.

Should be resolved before merge

1. The PTX-only fixture issue in RoundedBf16ContiguousKCancellation is still open (see the existing thread on matmul_packed_precision_test.cc; the PR checklist item is unchecked). With a compute_75-only image on SM80+, the SIMT fallback's lane-strided reduction combines the 1 at k=128 with 2^25 at k=0 first, giving 1 instead of 2.
2. docs/PackedWeightFormats.md does not define NVFP4 or the new rounding/scale semantics, although MatMulPacked now accepts packed_type == 40 (inline on matmul_packed.cc).

Suggestions / nits (inline): default-path global_scale test coverage, UE4M3 scale-byte parity with the pinned reference, and the missing default in LaunchMatMulPackedBf16ContiguousK. Also, docs/ContribOperators.md gains two trailing blank lines at EOF.

[Tianlei Wu (tianleiwu)]

Re-review of a213df5bb5.

Verdict: COMMENT — no blocking findings in this layer; all earlier feedback is addressed.

Earlier threads, re-checked on this head

• NVFP4 (ID 40) row, layout, K % 64 and scale arithmetic in docs/PackedWeightFormats.md — fixed (I re-derived 0x01 → 2^-9 and 0x7e → 448).
• global_scale on the default route — fixed (TestNvfp4GlobalScale now runs default and opt-in; Nvfp4GlobalScaleLargeNDefaultDecode covers N = 65536).
• Scale bytes with bit 7 set — resolved by contract (bit 7 must be clear, with known-answer tests for 0x00, 0x01, 0x7e, 0x7f).
• switch without default in the contiguous-K launcher — fixed (ORT_THROW).

I also re-read the epilogues: the FP32 global_scale is applied before the cast on every path (GEMV, warp-K-split tensor-core reduction, cuBLAS prefill through the FP32 output tile + MatMulPackedCastTileKernel, FP32 prefill), and the workspace sizing accounts for the extra FP32 tile.

Independent verification (stack tip 623a725f5c, which contains this layer; Linux, CUDA 13.0, sm_80;sm_90a, Release; A100-SXM4-80GB)

• onnxruntime_provider_test builds; the stack's 60 provider tests (including this layer's 8 MatMulPackedPrecisionTest cases) and 5 internal CUDA tests pass, also 3x in shuffled order. The pre-existing *Packed* suites (655 tests) show 0 failures.
• compute-sanitizer synccheck, racecheck and initcheck: no findings on the MatMulPacked tests.
• CI on this head has no failures so far; the Windows CUDA and wasm build failures in this stack start at #33994.

Suggestions (non-blocking)

1. The schema text for K still says "CUDA requires a positive multiple of 32"; NVFP4 needs a multiple of 64 (enforced via GgmlRowBytes, documented only in PackedWeightFormats.md). A parenthetical "(64 for NVFP4)" keeps the op reference self-consistent.
2. "global_scale requires NVFP4" depends only on the node's inputs and packed_type, so it can be rejected at session creation (check info.node().InputDefs() in the constructor) rather than at the first Run.

[Rui Ren (rui-ren)]

Implemented the session-only MatMulPacked precision policy.

• Removed dequantize_to_activation_dtype from the schema, kernel parsing and test exporters.
• Complete decoded weights round to the activation dtype by default. ep.cuda.matmul_packed_allow_unrounded_gemv="1" permits skipping rounding only in eligible GEMV; unset/"0" retains rounding. GEMM/prefill always rounds, including FP32 scratch. Invalid values fail initialization. NVFP4 global-scale placement is unchanged.
• Owning layer: 10 native CUDA + 10 compute75 PTX tests and 3 Python tests. Latest Stream exact NVFP4 operands in full BF16 projection tiles #34065 tip: 15 native + 15 PTX tests, 5 Python tests and 1 actual internal CUDA cancellation test; zero failures/skips. CLANGFORMAT/Ruff/whitespace and schema/documentation consistency checks pass.

Published atomically across all seven currently unmerged layers, preserving the concurrent #33994 CI fix and newly appended #34065. Native stack #33967 retains all eight members, including merged #33953; nothing merged or auto-merged.

Existing exports containing the removed attribute need that attribute removed or re-exporting. Validation uses fresh changed objects over ABI-matched preserved support, not a clean full-source build, full CI, fresh performance measurements or full-model generation qualification.

[Tianlei Wu (tianleiwu)]

Reviewed f51abbca077f against base 3bde254157f3, including all 11 changed files and the latest session-only precision policy.

No new actionable findings. I traced default/unset/"0" rounding and the "1" GEMV relaxation, complete-weight rounding in FP16's FP32 prefill scratch, BF16 contiguous-K operand mapping and compiled-image capability gating, and NVFP4 scaling before the output cast across decode and tiled prefill. Workspace sizing and tile tails are consistent with those routes.

Independent validation: compiled this head's unchanged production matmul_packed.cu and decoder header with CUDA 13.0, then ran a standalone direct-launch harness on H200:

• 248 cases passed with native sm_90a, with BF16 Tensor Core support reported as available.
• 248 cases passed with compute_75-only PTX, with BF16 Tensor Core support correctly reported as unavailable. cuobjdump confirmed sm_75 PTX and no native device ELF.
• Covered F32/Q8_0/IQ2_XXS/NVFP4 packed weights, FP16/BF16 activations, both rounding settings, M=1/7/8/9 plus M=129 tile crossing, K=64/128/256, and N=17/4099/8192/65536. NVFP4 cases included the FP32 global scale; comparisons used dtype-appropriate tolerances.

This harness passes dispatch parameters directly and reuses existing ORT support libraries; it does not exercise model/session initialization or constitute a clean full-source build, the repository's complete test suite, or full-CI qualification.

The two suggestions in the previous review remain non-blocking: clarify NVFP4's K multiple in the schema text, and reject a supplied global_scale for non-NVFP4 nodes at kernel creation. No duplicate inline threads added.