vulkan: allow multiple devices to access gpu-visible host memory

[0cc4m]

Overview

This has been on my todo list for a while: Overhaul the Vulkan backend host-visible memory management to allow multiple devices to access host memory. On master only the first device can directly read from host memory, which increases load times in multi-gpu systems and may make transfers slower.

This branch changes it to use VK_EXT_external_memory_host to use a regular aligned CPU allocation and access it from all devices. In my tests it reduces non-mmap multi-GPU load times by around 20%. No effect on inference performance.

Single-GPU systems should be unaffected.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES

[0cc4m]

@jeffbolznv Can you review this when you have time?

[jeffbolznv]

Overall it looks fine to me. Codex had a few good finds that I didn't notice:

• P1: Allocation failure becomes an invalid buffer. ggml/src/ggml-vulkan/ggml-vulkan.cpp:13078: aligned allocation can return nullptr, but the caller only handles exceptions. It
creates a non-null buffer whose get_base() subsequently asserts. Handle the null return explicitly.

• P2: Explicit device selection disables the feature. ggml/src/ggml-vulkan/ggml-vulkan.cpp:5392: multi_device is assigned only in the default-selection branch. With
GGML_VK_VISIBLE_DEVICES=0,1, it stays false and device 1 continues staging. Move the assignment outside the selection branches.

• P2: Insufficient alignment without host-import support. ggml/src/ggml-vulkan/ggml-vulkan-buffers.cpp:283: when no selected device advertises the extension, alignment falls to
8 bytes. The CPU buffer wrapper requires 32-byte alignment and can assert. The minimum must also satisfy ggml’s alignment requirements.

On the first one, IMO we should throw rather than codex's suggestion of returning null and handling null in the caller. On the third one, maybe we should check for any device advertising the extension before enabling this path.

[0cc4m]

Good points, should be fixed now. I went with the nullptr handling that can catch both the Vulkan error and the alloc nullptr result.

[CISC]

Not seen this one before:
https://github.com/ggml-org/llama.cpp/actions/runs/38209201416/job/114679365164?pr=29741#step:3:36502

[0cc4m]

Seems to be some kind of Nvidia cleanup crash again, not caused by this PR.

[jeffbolznv]

If somebody can capture a crash dump or call stack for one of these intermittent crashes, I can share it internally.