Hand-written CUDA and Triton kernels for LLM inference, checked against a PyTorch reference and benchmarked against PyTorch and cuBLAS.
A personal project, organised as a lab notebook: each chapter in docs/ covers one
kernel, from motivation over implementation to correctness check and benchmark.
Work in progress.
- LayerNorm: raw CUDA kernel with hand-written warp shuffles, a CUB block-reduce
variant and a Triton port, benchmarked against
torch.nn.LayerNormover GPT-2 prefill and decode shapes. - Tiled matmul (FFN): shared-memory block tiling, register tiling, a Triton port
with a block-size sweep, and a raw
cublasSgemmcall as the reference, all in true fp32. - int8 dequantizing GEMV (decode): weight-only int8 with group-wise scales, split-K, 1.8 to 1.9x faster than fp16 cuBLAS against a ceiling of 1.97x from the bytes alone.
- In progress, not in this repo yet: int4 dequantizing GEMV, fused residual-add+RMSNorm, online softmax.
Benchmarks against torch and cuBLAS are a calibration, not a race: the vendor library is the realistic ceiling for a plain dense GEMM. The point is to see where each level of the memory hierarchy shows up in the numbers. Wins over the library come from elsewhere: the int8 GEMV beats cuBLAS by moving fewer bytes, not by out-tuning it.
So far the kernels are tested and benchmarked one at a time. The plan is to put them into real models, step by step:
- GPT-2 with the real HF weights as the correctness reference. The model and the
weight loading are in
model/; next is swapping kernels in and checking the model output end to end. - A modern dense model (Llama or Qwen), which needs RMSNorm and the dequantizing GEMV for decode.
- A Mixture-of-Experts model (OLMoE-1B-7B) as the long-term goal.
Inference only, on a single GPU. No training, no distributed setup.
- 00: Overview goal, model plan, chapter plan
- 02: LayerNorm first raw CUDA kernel
- 03: Tiled Matmul shared-memory and register tiling, Triton, cuBLAS reference
- 04: Dequantizing GEMV int8 weights for decode, measured against the bandwidth ceiling
More chapters as kernels get built.
kernel-forge/
├── model/ reference PyTorch architecture + HF weight loading (the correctness oracle)
├── kernels/ custom CUDA/Triton kernels, one file per op
├── tests/ per-op correctness checks vs. the PyTorch reference
├── bench/ benchmark and tuning scripts, results in bench/results/
└── docs/ chapter-by-chapter writeup
Needs an NVIDIA GPU with the CUDA toolkit; the CUDA kernels are compiled as torch extensions on first use.
uv syncOn a machine with CUDA and torch already installed system-wide, reuse that instead of pulling the multi-GB CUDA wheel a second time:
uv venv --system-site-packages
uv sync --no-install-package torchRun the correctness tests and a benchmark:
uv run pytest
uv run bench/matmul_benchmark.pyApache License 2.0, see LICENSE.
model/gpt2.py is adapted from Sebastian Raschka's
LLMs-from-scratch (Apache License 2.0),
renamed to GPT-2's original parameter names so the HF checkpoint loads directly.