[Rui Ren (rui-ren)]

Description

Stream small restored NVFP4 operand pairs directly into the existing BF16 Tensor Core operations for aligned, full eight-row projection tiles on SM90.

• Preserve both BF16 scaling operations, operand restoration before MMA, contiguous warp/K partitions, MMA order and FP32 accumulation/reduction order.
• Compact the zero/sign reconstruction while retaining GGUF code 8's positive-zero behavior.
• Keep partial-row, tail and unaligned paths unchanged. Do not modify the current IQ2 CTA/prefill fixes.
• Extend the existing packed-operand-extremes regression to N=35 and N=8192, exercising the full-tile specialization as well as boundary paths.

Historical benchmark pins: parent #34003 at 623a725f5cf51ceeaba5725720c3c6effa2fbf3b, measured candidate 61d27b9cad1a90fbf0d319048e47ca784875e5f2. The throughput, profiling and full-model qualification below apply to these pins, not the current rebased head.

Motivation and Context

Reduce the NVFP4 projection cost in fixed-k7 Nemotron-Bolt speculative decoding without changing precision, routing, cache semantics, package graphs, generated output or workload counters.

Measured end-to-end result: a small incremental improvement, not a full resolution of the speculative TPS regression. On H100, 30 frozen prompts, ordinary and fixed-k7 modes, 240 complete measured generations and two reversed same-physical-GPU passes:

Input tokens	Baseline k7 tok/s	Candidate k7 tok/s	Gain
178-254	352.8195	355.6267	+0.80%
427-932	291.3841	294.3088	+1.00%
1028-1663	316.6777	320.6373	+1.25%

Both individual passes were positive in every bucket. Ordinary decoding was essentially unchanged. Throughput is pooled as (generated_tokens - 1) / sum(decode_seconds), excluding the first token consistently with the synchronized timer. Frozen prompts, four warmups, mode order, EOS/512-token cap and all speculative workload counters are unchanged. No profiler or owned compilation ran during timing.

Separate production CUPTI profiling verified the exact full-tile NVFP4 specialization: 1,200 calls, 47.1110 -> 41.7313 ms total affected-kernel time (-11.42%), with no dropped/invalid records. This is kernel time, not an end-to-end throughput percentage. Registers increased from 64 to 72; this is an operand-scheduling optimization, not a measured register-count reduction.

Historical benchmark validation (pins above)

• Exact current CMake compile/link/archive closure, CUDA 12.8.1, sm90a and unchanged production numerical flags; incremental builds, not two independent clean builds.
• 49 native tests per arm, no skips; four candidate compute-sanitizer memchecks, zero errors.
• 1,782 complete output/state tensors per arm matched bitwise, including preserved frozen ordinary outputs.
• Live cache/state/rollback tests and full generation pilots passed. All 240 benchmark sequences matched the frozen ordinary token sequences; same-mode speculative counters matched across arms and reversed passes.
• Changed-region clang-format and git diff --check passed.

The common GenAI binaries were actually built at b9c6ee9ac579c2c2bff59804ce82e6c1f1c2a3f5. Live stack #2924 head 7b7cc5a2821d15136904c1ebb66400c976593a87 has byte-identical inference sources; only test include wiring and test whitespace differ. These binaries are not relabeled as a fresh build of that newer head.

No new independent HF-reference validation or cross-GPU-architecture qualification is claimed. Original failed build/recovery attempts are preserved; recovered objects lack individual exit-code receipts from the original failed coordinator and were separately checked for source/recipe/log/ELF/hash consistency before all subsequent native/model/runtime gates.

Read-back-verified local evidence: 690 files, 27,562,698 uncompressed bytes. Archive SHA256: 194e008fc7c32751554fc7c34ee2691fd26dc0cbef9c70009228f1031bfc046a. Benchmark statistics SHA256: aa034152a1e7e561f025c26264935a386020a1e4e0ec4e872dacc4051010f566.

The previously measured ~2.3% improvement was a different, combined older-source candidate; it is not the measured gain for this isolated follow-up.

Session-policy rebase validation

Current head: a3f2076b65e0d1a5c7e9b82165ffab014e233876, based on #34003 at 7b39b3f69b72812f563f8b727e81c4296c6ce362.
Preserves the concurrently published #33994 CI narrowing-conversion fix and this
layer's streamed NVFP4 operands/full-tile regression.

Inherits #33962's default-rounded operands and session-only
ep.cuda.matmul_packed_allow_unrounded_gemv policy ("0" default, "1" permits
eligible unrounded GEMV). Prefill always rounds. The removed model attribute is
not emitted by tests; existing exports carrying it need removal/re-export.

15 native CUDA + 15 compute75-only PTX projection tests, 5 Python format/policy/alignment
tests and 1 actual internal CUDA cancellation test passed, zero failures/skips.
CLANGFORMAT, Ruff/format and whitespace checks passed. These checks use fresh
changed objects over ABI-matched preserved support, not a clean full-source build,
full CI, performance remeasurement, or fresh full-model generation qualification.

[Copilot]

🟢 Approval recommended

No active review issues remain that require resolution before approval.

No open issues 🎉

What changed in this PR

This PR builds on #34003 to reduce NVFP4 projection overhead by streaming restored operand pairs into existing BF16 Tensor Core operations for aligned, full eight-row tiles.

Changes:

• Adds compact zero/sign reconstruction and pairwise operand decoding while preserving BF16 scaling, MMA order, and FP32 accumulation order.
• Keeps partial-row, tail, and unaligned fallback paths unchanged.
• Extends packed-operand-extremes coverage to both N=35 and N=8192, exercising boundary and full-tile paths.

File	Description
onnxruntime/​test/​contrib_ops/​matmul_packed_precision_test.cc	Extends operand-extremes tests to full projection widths.
onnxruntime/​contrib_ops/​cuda/​quantization/​matmul_packed.cu	Streams restored NVFP4 operand pairs into full-tile BF16 operations.

_{Review effort: Balanced}

[Rui Ren (rui-ren)]

Review-fix handoff for stack #33967: six remaining review threads have isolated, tested patches on this branch. Aggregate diff.

• Extend MoEPacked for Bolt ReLU-square routing and BF16 IQ2 precision #33963: independent FC2 cancellation fixture, intermediate=1024, eight warps, decode and populated prefill.
• Add snapshot-aware Mamba2Verify CUDA contrib operator #33969: actionable head/group diagnostics and negative attribute coverage.
• Reuse MoEGroupRouter for projected sigmoid routing #33994: compact-router rejection cases on both CPU and CUDA.
• Speed up native Bolt packed projections and IQ2 experts #34002: Hopper-only fast-path guards, exhaustive unsigned-scale decoder oracle, contract-valid scale fixtures, and fetch-count assertion.

Validation at 1835243421cc20bef869349e81aaf62075d987e1: 57 native tests passed with no skips; four CUDA memchecks passed with zero errors; configured formatter checks passed. The decoder oracle checks all 128 valid scale bytes × 16 codes × 16 positions (32,768 BF16 bit comparisons).

The integrating workstream published the router-API removal while these tests were running and resolved that separate thread. All four per-PR patches apply cleanly to the new owning heads, through stack top de9e2b2dc1766ee58e427850d34bf208729e5642, but that new combination has not been runtime-retested.

Integration is still pending. I did not overwrite any active PR branch, change stack membership, or resolve the six remaining threads prematurely. Local per-PR patches, exact commands, hashes, XML results and preserved failures are in /home/mobius/stack33967-review-fixes-20261110-0322/HANDOFF.json.