Repository navigation
cuda: fix sparse F16 compilation and temporary buffer leaks - #287
Open
dandan20031031 wants to merge 1 commit into
Open
dandan20031031 wants to merge 1 commit into
dandan20031031 wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Enabling
LLAMA_CUDA_F16exposes compilation errors in the sparse CUDA paths. Once those errors are corrected, repeated single-vector sparse operations retain an activation buffer on every call. This change makes the F16 configuration buildable and releases the affected temporary storage.Validated on Windows with RTX 5080, CUDA 13.3.73, MSVC 19.44, and SM120. After warmup, 2048 calls with input width 16384 and 64 stored weight rows produced:
The before-memory-fix baseline includes only the compilation prerequisite changes; the original upstream F16 configuration could not be built in this environment. Both direct and mapped row cases reproduced the vector-path growth.
Validation:
LLAMA_CUDA_F16=ONandOFF.test-quantize-fnspassed.3.04e-5for Q4_0 vectors, zero for the other cases.git diff --checkpassed.Run instructions are in
tests/README-cuda-sparse.md. The memory regression usescudaMemGetInfoand should run on an otherwise idle GPU. Validation covers single-GPU synthetic operator inputs; full-model generation, hybrid offloading, multi-GPU operation, and Linux have not been tested.