Skip to content

About

Hand-written CUDA and Triton kernels for LLM inference, checked against PyTorch and benchmarked against cuBLAS and torch.compile. Written up chapter by chapter as a lab notebook.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

25 Commits

Folders and files

Repository files navigation

Kernel Forge 🛠️

Hand-written CUDA and Triton kernels for LLM inference, checked against a PyTorch reference and benchmarked against PyTorch and cuBLAS.

A personal project, organised as a lab notebook: each chapter in docs/ covers one kernel, from motivation over implementation to correctness check and benchmark. Work in progress.

Status

  • LayerNorm: raw CUDA kernel with hand-written warp shuffles, a CUB block-reduce variant and a Triton port, benchmarked against torch.nn.LayerNorm over GPT-2 prefill and decode shapes.
  • Tiled matmul (FFN): shared-memory block tiling, register tiling, a Triton port with a block-size sweep, and a raw cublasSgemm call as the reference, all in true fp32.
  • int8 dequantizing GEMV (decode): weight-only int8 with group-wise scales, split-K, 1.8 to 1.9x faster than fp16 cuBLAS against a ceiling of 1.97x from the bytes alone.
  • In progress, not in this repo yet: int4 dequantizing GEMV, fused residual-add+RMSNorm, online softmax.

Benchmarks against torch and cuBLAS are a calibration, not a race: the vendor library is the realistic ceiling for a plain dense GEMM. The point is to see where each level of the memory hierarchy shows up in the numbers. Wins over the library come from elsewhere: the int8 GEMV beats cuBLAS by moving fewer bytes, not by out-tuning it.

Direction

So far the kernels are tested and benchmarked one at a time. The plan is to put them into real models, step by step:

  • GPT-2 with the real HF weights as the correctness reference. The model and the weight loading are in model/; next is swapping kernels in and checking the model output end to end.
  • A modern dense model (Llama or Qwen), which needs RMSNorm and the dequantizing GEMV for decode.
  • A Mixture-of-Experts model (OLMoE-1B-7B) as the long-term goal.

Inference only, on a single GPU. No training, no distributed setup.

Docs

More chapters as kernels get built.

Layout

kernel-forge/
├── model/       reference PyTorch architecture + HF weight loading (the correctness oracle)
├── kernels/     custom CUDA/Triton kernels, one file per op
├── tests/       per-op correctness checks vs. the PyTorch reference
├── bench/       benchmark and tuning scripts, results in bench/results/
└── docs/        chapter-by-chapter writeup

Setup

Needs an NVIDIA GPU with the CUDA toolkit; the CUDA kernels are compiled as torch extensions on first use.

uv sync

On a machine with CUDA and torch already installed system-wide, reuse that instead of pulling the multi-GB CUDA wheel a second time:

uv venv --system-site-packages
uv sync --no-install-package torch

Run the correctness tests and a benchmark:

uv run pytest
uv run bench/matmul_benchmark.py

License

Apache License 2.0, see LICENSE.

model/gpt2.py is adapted from Sebastian Raschka's LLMs-from-scratch (Apache License 2.0), renamed to GPT-2's original parameter names so the HF checkpoint loads directly.

About

Hand-written CUDA and Triton kernels for LLM inference, checked against PyTorch and benchmarked against cuBLAS and torch.compile. Written up chapter by chapter as a lab notebook.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages