Skip to content

[CUDA] Sequential kernel for gated_delta_update - #4783

Open
zcbenz wants to merge 1 commit into
mainfrom
cuda-gdn
Open

zcbenz wants to merge 1 commit into
mainfrom
cuda-gdn

Conversation

@zcbenz

@zcbenz zcbenz commented Oct 28, 2026 •

Copy link
Copy Markdown
Member

Add a simple CUDA kernel for gated_delta_update that updates states sequentially, I will add a chunked kernel in follow up PR.

This kernel is modified from the submission of @kamahori from MLSys 2026 Competition, and the modification was done by myself instructing AI.

Benchmarks by running benchmarks/python/gated_delta_bench.py:

B T Hk Hv Dk Dv fallback (ms) seq (ms) seq (speedup)
1 8 16 32 128 128 2.239 0.049 45.52x
1 32 16 32 128 128 9.161 0.104 88.14x
1 64 16 32 128 128 18.124 0.125 144.53x
1 128 16 32 128 128 38.162 0.206 185.11x
1 256 16 32 128 128 79.476 0.495 160.55x
1 512 16 32 128 128 164.421 0.827 198.74x
8 8 16 32 128 128 7.096 0.210 33.73x
8 32 16 32 128 128 25.544 0.482 52.98x
8 64 16 32 128 128 51.001 0.721 70.72x
8 128 16 32 128 128 101.674 1.419 71.65x
8 256 16 32 128 128 203.280 2.360 86.13x
8 512 16 32 128 128 406.669 4.646 87.53x

@zcbenz
zcbenz force-pushed the cuda-gdn branch 5 times, most recently from c50f270 to 175b135 Compare October 29, 2026 11:08
@ArupKantiDas

Copy link
Copy Markdown
Contributor

I ran this on an M5 Pro (applegpu_g17s, which has NAX), since none of the CI runners have NAX.

I used Claude Code to build and run the tests above.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants