Repository navigation
feat(gpu-operator): migrate GPU sharing from device plugin to DRA - #2620
Merged
Merged
Conversation
Contributor
|
Warning
Note +8 added · 18 changed · −3 removed: 29 resources across 21 apps Blast radius
konflate 0.6.4 · rendered |
jfroy
force-pushed
the
feat/gpu-operator-dra
branch
6 times, most recently
from
September 9, 2026 02:55
63a85d9 to
94ca726
Compare
Switch the gpu-operator from the ClusterPolicy (device plugin) stack to the GPUCluster (DRA) stack introduced in chart v26.7.0, and convert every GPU consumer from the nvidia.com/gpu extended resource to DRA resource claims. GPUCluster and ClusterPolicy may not exist at the same time, so this is a single cutover with no canary. It drops the device plugin, gpu-feature- discovery, the operator validator and the MIG manager, and replaces them with the NVIDIA DRA driver kubelet plugin (dra-driver-nvidia-gpu v0.5.0). Rollback is a second cutover. Replicating the time-slicing configuration ------------------------------------------ The device plugin's `replicas: 8` has no equivalent in the DRA driver's `sharing.strategy: TimeSlicing`, which only calls SetTimeSlice to set the CUDA compute time-slice duration and does nothing to make a GPU allocatable to more than one claim. Oversubscription comes from the driver's ConsumableShares feature gate instead: with CONSUMABLE_SHARES set, the kubelet plugin publishes each GPU with allowMultipleAllocations and a request policy on its capacities. An integer N would be the literal equivalent of `replicas: N` (a `shares` capacity of N, one consumed per claim); `unlimited` instead gives the memory capacity a request policy defaulting to 0, so claims consume nothing and sharing is uncapped. A full non-MIG GPU publishes only the `memory` capacity, which matters because a capacity with no request policy defaults to full-device consumption and would silently cap sharing at one claim. computeDomains is disabled: it manages multi-node NVLink and there is no NVSwitch fabric here. Prerequisites are already met on k8s 1.36 and no talconfig change is needed: DynamicResourceAllocation is GA and locked, DRAConsumableCapacity and KubeletPodResourcesDynamicResources are beta and on by default. Claim template -------------- All nine GPU pods want the same device, so they share one namespace-scoped ResourceClaimTemplate, `any-nvidia-gpu-claim`, delivered by its own small Kustomization following the media-pv pattern. Each pod still gets its own generated ResourceClaim. The template takes any device in the gpu.nvidia.com class; a workload needing a particular card should get its own template with a CEL selector on the device attributes rather than narrowing this one. Note that ResourceClaimTemplate.spec is immutable, so changing the request later needs `force: true` on the Kustomization or a one-time manual delete. stash's `stash` container and `nams` sidecar share a single claim; they ran on the same physical GPU under time-slicing anyway. Other notes ----------- driver.enabled and toolkit.enabled stay false because Talos installs both the driver and the container toolkit as system extensions. driver.enabled is not merely a ClusterPolicy input and does not become dead config here: templates/nvidiadriver.yaml gates the NVIDIADriver CR on it alone, independent of clusterPolicy.deployCR, and upstream defaults it to true. dcgm-exporter switches from KUBERNETES_VIRTUAL_GPUS, which only maps device-plugin time-sliced replicas, to KUBERNETES_ENABLE_DRA. That needs read access to resource.k8s.io objects, which the chart does not grant, hence app/rbac.yaml. The nvidia.com/gpu.* node labels gpu-feature-discovery left on kantai1 go stale once GFD is removed. Nothing in this repo selects on them; they can be deleted by hand afterwards. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012XxqQRjawjipsCtiVhTLcC Signed-off-by: Jean-Francois Roy <jf@devklog.net>
jfroy
force-pushed
the
feat/gpu-operator-dra
branch
from
September 9, 2026 05:02
94ca726 to
f4ecc09
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Switch the gpu-operator from the ClusterPolicy (device plugin) stack to
the GPUCluster (DRA) stack introduced in chart v26.7.0, and convert every
GPU consumer from the nvidia.com/gpu extended resource to DRA resource
claims.
The device plugin's time-slicing
replicas: 8has no equivalent in theDRA driver's
sharing.strategy: TimeSlicing, which only sets the CUDAcompute time-slice duration. Oversubscription comes from the driver's
ConsumableShares feature gate instead: with CONSUMABLE_SHARES set, the
kubelet plugin publishes each GPU with allowMultipleAllocations and a
request policy on its memory capacity.
unlimiteddefaults each claim to0 consumed memory, so any number of claims can share the RTX 4000 Ada.
Cluster prerequisites are already met on k8s 1.36: DynamicResourceAllocation
is GA and locked, DRAConsumableCapacity and KubeletPodResourcesDynamicResources
are beta/on by default. No talconfig change is needed.
Notes:
cutover. It also removes gpu-feature-discovery, the device plugin, the
operator validator and the MIG manager; the stale nvidia.com/gpu.*
node labels GFD left behind can be dropped by hand afterwards.
KUBERNETES_ENABLE_DRA and needs RBAC on resource.k8s.io, which the
chart does not grant.
stashcontainer andnamssidecar share one claim; they ranon the same physical GPU under time-slicing anyway.
Co-Authored-By: Claude Opus 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_012XxqQRjawjipsCtiVhTLcC