Skip to content

cuda: fix sparse F16 compilation and temporary buffer leaks - #287

Open
dandan20031031 wants to merge 1 commit into
Tiiny-AI:mainfrom
dandan20031031:fix/cuda-sparse-temporaries
Open

dandan20031031 wants to merge 1 commit into
Tiiny-AI:mainfrom
dandan20031031:fix/cuda-sparse-temporaries

Conversation

@dandan20031031

Copy link
Copy Markdown

Enabling LLAMA_CUDA_F16 exposes compilation errors in the sparse CUDA paths. Once those errors are corrected, repeated single-vector sparse operations retain an activation buffer on every call. This change makes the F16 configuration buildable and releases the affected temporary storage.

  • Remove the unused F16 activation conversion from the Q4_0/Q8_1 vector path.
  • Return the required F16 activation buffer to the existing CUDA buffer pool after the F16 vector kernel is enqueued.
  • Resolve ambiguous half comparisons and keep batched sparse activations as FP32, matching the caller's buffer type.
  • Add model-free regression cases for Q4_0/F16 weights, vector/batch inputs, direct/mapped rows, and changing activation masks.

Validated on Windows with RTX 5080, CUDA 13.3.73, MSVC 19.44, and SM120. After warmup, 2048 calls with input width 16384 and 64 stored weight rows produced:

F16-enabled path Before memory fix After memory fix
Q4_0, one vector 70 MiB growth 0 bytes observed growth
F16, one vector 68 MiB growth 0 bytes observed growth
Q4_0/F16, three vectors 0 bytes observed growth 0 bytes observed growth

The before-memory-fix baseline includes only the compilation prerequisite changes; the original upstream F16 configuration could not be built in this environment. Both direct and mapped row cases reproduced the vector-path growth.

Validation:

  • All 16 sparse regression cases passed across LLAMA_CUDA_F16=ON and OFF.
  • Existing test-quantize-fns passed.
  • Four mapped-row Compute Sanitizer memcheck runs (Q4_0/F16, vector/batch) reported zero errors.
  • Numerical errors on the synthetic inputs were unchanged: maximum absolute error about 3.04e-5 for Q4_0 vectors, zero for the other cases.
  • git diff --check passed.

Run instructions are in tests/README-cuda-sparse.md. The memory regression uses cudaMemGetInfo and should run on an otherwise idle GPU. Validation covers single-GPU synthetic operator inputs; full-model generation, hybrid offloading, multi-GPU operation, and Linux have not been tested.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant