[Feature]: Driver upgrades cannot free GPU consumers that lack an nvidia.com/gpu request or run as DaemonSets

[ctrlsam]

What

Request a supported way for the driver upgrade flow to free every consumer holding the GPU kernel module open. Two cases the current flow cannot handle:

1. Pods that use the GPU via runtimeClassName: nvidia with no nvidia.com/gpu resource request (exclusive mode, no time-slicing / MPS).
2. DaemonSet GPU consumers (for us, a standalone dcgm-exporter).

Both keep the module refcount above zero, so an in-place driver unload/upgrade stalls. This is the underlying cause behind the stuck-driver symptom in #2549 (the init container "fails to unload the driver while workloads are active").

Why the current flow doesn't cover these

There are two eviction paths, and neither can free these consumers:

• Node drain (k8s-operator-libs pkg/upgrade/drain_manager.go): PodSelector is configurable and defaults in gpu-operator (controllers/upgrade_controller.go) to nvidia.com/gpu-driver-upgrade-drain.skip!=true. But IgnoreAllDaemonSets: true is hardcoded, so DaemonSet consumers are never evicted by drain.
• Targeted pod deletion (pkg/upgrade/pod_manager.go): uses an injected PodDeletionFilter. gpu-operator's filter keys on the nvidia.com/gpu resource, so pods without that request are never selected. IgnoreAllDaemonSets: true is hardcoded here too.

Net: a DaemonSet GPU consumer cannot be freed by any path, and a non-resource GPU pod is only reachable via a broad node drain - heavier than needed, and in our testing it did not free our runtime-direct Deployments either.

Proposal

1. Allow declaring additional GPU consumers to evict during an upgrade by label/selector, surfaced through driver.upgradePolicy, so they do not have to request nvidia.com/gpu. This complements the existing nvidia.com/gpu-driver-upgrade-drain.skip opt-out with an opt-in.
2. Provide handling for declared DaemonSet GPU consumers during the upgrade window - e.g. an opt-in to cordon/park them (or make the DaemonSet-ignore configurable for a named set), restored on upgrade-done. The operator already quiesces its own operands; this extends that to declared third-party consumers.

Where the change lives

• Core: NVIDIA/k8s-operator-libs pkg/upgrade (drain/pod managers; the hardcoded IgnoreAllDaemonSets: true).
• Surface: NVIDIA/gpu-operator driver.upgradePolicy API and the injected PodDeletionFilter in controllers/upgrade_controller.go.

Workaround today

We run a small sidecar controller that watches nvidia.com/gpu-driver-upgrade-state and, on any transition away from upgrade-done, deletes the runtime-direct pods and parks the DaemonSet consumer (non-matching nodeSelector), then restores them on upgrade-done. With those consumers freed, the operator unloads, rebuilds and reloads the driver in place with no node reboot (boot_id unchanged across the upgrade). Happy to share it.

Related: #2549.

[ctrlsam]

Have created a PR in k8s-operator-libs that will enable this feature to be added here:
NVIDIA/k8s-operator-libs#150

[tariq1890]

You have mentioned dcgm-exporter as an example of the GPU consumer. The k8s-driver-manager init-container takes care of evicting this during an in-place driver upgrade. Do you have another GPU consumer that the k8s-driver-manager doesn't touch?

[tariq1890]

Ah, I see that you have mentioned "standalone dcgm-exporter". Can we understand why you deploy dcgm-exporter separately from gpu-operator?

[ctrlsam]

Good questions, thanks.

You're right about the operator's own dcgm-exporter - the driver-manager evicts it during an in-place upgrade. Ours is a standalone dcgm-exporter deployed outside the operator, so the driver-manager doesn't know about it and doesn't evict it. But I'll be upfront that dcgm-exporter is the weaker example, and we do intend to move to the operator's bundled exporter once a site is fully on the operator (more on the "why standalone" below).

The primary consumers this PR is for are our inference workloads, not dcgm-exporter:

• Triton and the NVIDIA VST (vms) pods run with runtimeClassName: nvidia and no nvidia.com/gpu resource request. We run them in exclusive mode (no time-slicing / MPS) so each gets native performance and full VRAM on small edge GPUs (e.g. an 8 GB Quadro RTX 4000), with several consumers sharing the device through the runtime rather than the device plugin.
• Because they don't request nvidia.com/gpu, the targeted GPU-pod eviction (which keys on that resource via the PodDeletionFilter) never selects them. In our in-place upgrade testing on an edge node, these pods kept the driver module pinned and the upgrade stalled at pod-restart-required until we deleted them by hand. Freeing them let the operator unload, rebuild and reload the driver in place with no reboot.

So the gap is GPU consumers that use the device via the nvidia runtime without a resource request: they're invisible to the upgrade eviction. The PodSelector lets us declare them so the operator frees them itself, without forcing them onto a resource-request / time-slicing model we deliberately avoid on these GPUs.

On why the standalone dcgm-exporter: we run GPU metrics across a fleet of 400+ edge servers, most of which are still on bare-metal NVIDIA drivers rather than the operator. The standalone exporter gives us one metrics pipeline that behaves the same against bare-metal and operator-managed drivers. Moving to the operator's bundled dcgmExporter has to be a paired, per-site change (enable the operator's exporter and remove the standalone, since the two can't co-exist on a node), which we can't roll out fleet-wide while most sites are still bare-metal. It's transitional - we expect to retire it per site as sites finish migrating. The durable justification for this PR is the runtime-direct inference workloads above.