Repository navigation
cuda: ptq1 sm86 rows1 - #340
Conversation
bri-prism
left a comment
bri-prism
left a comment
There was a problem hiding this comment.
No blocking correctness findings. The sm_86-only guard addresses the scope problem in #335: H200 and other compute capabilities retain ROWS=4, and multi-column schedules remain unchanged. The shared-memory guard and launcher use the same selected row count. The dot helper and kernel body are unchanged.
Independent validation:
- 320 ROWS=1 versus ROWS=4 comparisons were byte-identical, covering tail rows, bias, three fused GLU variants, and default/invariant reductions.
- 128 invariant comparisons at 2/3/4/8 columns passed.
- Both sets passed with normal and production CUDA compiler flags; Compute Sanitizer reported zero errors in both builds.
- Native sm_86 compilation reproduced 76 versus 108 registers for the unfused one-column kernels, with no spills.
- 163,840 host dispatch/shared-memory combinations and 224 tail/indexing cases passed.
These were direct kernel probes using sm_86 PTX forward-JITed on H200, not a full production-library or end-to-end model test. They validate the scheduling arithmetic but do not independently confirm the reported RTX 3080 speedup. The remaining performance check is a native sm_86 A/B; another sm_86 card would also help establish how broadly the gain applies. The slow-tail caveat in the description is appropriate.
prism PrismML-Eng#340 added rows_per_item (by compute capability) to ptq1_0_pt_smem_bytes(). The guard of the gate fused 2..4 column mat-vec (ggml_cuda_mmvq_ptq1_0_can_fuse_mc) still passed four arguments; it now uses the same rule as the launcher, so guard and launch agree on the shared memory. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2 participants
Title: cuda: use one-row scheduling for PTQ1_0 planar GEMV on sm_86
Overview
Use one output row per work item for the one-column PTQ1_0 planar-transposed GEMV path only on NVIDIA compute capability 8.6 (sm_86), tested on a GeForce RTX 3080 (GA102). Other compute capabilities keep the four-row schedule, including Hopper. This reduces register use and exposes more independent row work on the tested Ampere batch-1 decode path. The PTQ1_0 dot arithmetic and activation layout are unchanged; multi-column paths keep their existing 4/2-row schedule.
Results
Measured on an NVIDIA GeForce RTX 3080 (GA102, compute capability 8.6) with Ternary Bonsai 2 27B PTQ1_0:
This A/B used seven repetitions per process. The reported gains use medians. Context-4096 had slow-tail samples in both variants; one pair had a lower ROWS=1 mean despite its higher median, so the median improvement does not establish a tail-latency improvement.
The compiled kernel uses 76 registers per thread for ROWS=1 versus 108 for ROWS=4, with no spills.
Measurement method
The final comparison used a rebuilt source-default candidate and an isolated ROWS=4 baseline on the RTX 3080. Each
llama-benchprocess generated 128 tokens at contexts 512 and 4096, with seven repetitions and the same 60 C / 5% idle start gate. The model was fully offloaded (99 GPU layers), with Flash Attention enabled, F16 KV cache, batch/microbatch 2048/512, and 8 CPU threads.LD_LIBRARY_PATH,ldd, and loader traces verified that each process loaded its intended CUDA library. The earlier two-pair comparisons alternated process order.Correctness
The recorded final sm_86 ROWS=1 build passed four selected CTests, and 96/96 CUDA-versus-CPU PTQ1_0/PQ2_0 matmul cases. No new operator or quantization type is introduced. Full CI and perplexity were not rerun for this draft: this session's
nvidia-smicannot communicate with the driver. The change keeps the per-row dot and reduction order unchanged.Scope
The one-row schedule applies only to the one-column planar path on NVIDIA sm_86, the RTX 3080 (GA102) used for these measurements. Every other compute capability retains the four-row schedule. The existing host layout policy is unchanged, so other architectures may still select the planar path (for example, invariant-mode Hopper), but they use four rows. Multi-column 4/2-row schedules are unchanged.
Requirements