[tetsuo]

A regression test added for the Metal addmm fix (#4769) found an existing CUDA error.

Some cuBLAS split-K algorithms store intermediate results in FP16. This can cause rounding errors or overflow before the final result is calculated.

The error was found on x86_64 with a Tesla T4. The cause is CUDA algorithm selection, not the CPU architecture. Apple Silicon uses Metal and does not use this code. Other NVIDIA GPUs can select different algorithms.

This change:

• Keeps reductions in FP32 for FP16/BF16 outputs, using separate cached preferences without changing FP32 or complex64 policy.
• Adds a CUDA test for rounding, overflow, transposed inputs, strided bias, batches, and changes between data types.

Tested and validated on T4/CUDA 13.

Full CUDA 12 builds and tests are not complete. Other NVIDIA GPU architectures still need tests.

Focused benchmarks show roughly a 5–6% increase in execution time for one FP16 matrix shape.

[zcbenz]

Thanks for looking into this, but I think we should not turn this on by default for CUDA because of the performance hit, which would match the default behavior of PyTorch:
https://docs.pytorch.org/docs/2.15/notes/numerical_accuracy.html#reduced-precision-reduction-for-fp16-and-bf16-gemms

If there are actual requests for float32 precision for reduction, we can consider adding a new option for it.