Is this a new feature, an improvement, or a change to existing functionality?
New Feature
How would you describe the priority of this feature request
High, we will not adopt slurm-bridge without it :( After it is added, we would be very eager to integrate it.
Please provide a clear description of problem this feature solves
We run a multi-tenant Kubernetes service cluster alongside a Slurm HPC cluster, and want tenants to run short GPU workloads from Kubernetes without writing batch scripts. The blocker is granularity of sharing.
Our GPU nodes have 4x H100, all MIG-partitioned. A typical workload from Kubernetes needs about a quarter of one H100, so roughly 1/16th of a node. Under the planned time-only sharing model, placing that pod requires every native Slurm job on the node to terminate first. We would be blocking 100% of a node to run a workload that needs almost none of it.
Our Slurm cluster serves our largest user group and our Kubernetes cluster is small by comparison. We cannot remove GPU nodes from Slurm, even temporarily, and a static whole-node carve-out would idle capacity we cannot spare. Slurm alone currently packs these nodes at MIG granularity with high efficiency, so any model requiring whole-node exclusivity is a large utilisation regression for us.
The practical result is that we cannot adopt slurm-bridge for GPU work. We stay on our current workaround: tenant pods mount the Slurm binaries and submit jobs to slurmctld directly. That keeps node efficiency because it is Slurm-native, but no Kubernetes object represents the running workload, users must build apptainer or podman containers, and job materials have to be staged outside the cluster. This is exactly the situation slurm-bridge would otherwise solve for us.
Follow-up to the discussion in #41, requested by @killianmuldoon.
Feature Description
As an administrator running Slurm and Kubernetes on shared GPU nodes, I want bridge-managed pods and native Slurm jobs to run on the same node at the same time on disjoint resources, so that a small Kubernetes workload does not require draining an entire multi-GPU node.
EARS form: while a native Slurm job is running on a subset of a hybrid node's resources, slurm-bridge shall allow an external job to be allocated the node's remaining unallocated resources and bind its pods to that node.
Describe your ideal solution
Dynamic co-residency. slurmctld tracks both native and external allocations on the node and packs them together, with external jobs submitted non-exclusively. This gives better utilisation because the Kubernetes share is not stranded when idle, but it needs coordinated GPU device identity and aligned CPU sets.
Describe any alternatives you have considered
Dedicating whole GPU nodes to Kubernetes. Rejected. Our Kubernetes cluster is small relative to Slurm demand, so the idle GPU time would be unacceptable to our user community.
Time-only sharing as currently planned. Rejected for GPU work because of the roughly 16x over-allocation described above. It would work for CPU-only workloads.
Our current workaround, pods that mount the Slurm binaries and submit to slurmctld directly. Preserves node efficiency but gives up essentially everything Kubernetes-native: no Kubernetes object for the running workload, apptainer or podman required inside the job, and materials staged outside the cluster.
Additional context
Hardware context: GPU nodes with 4x H100, statically MIG-partitioned in different configurations. Typical Kubernetes workloads target a single or multiple MIG instances. Native Slurm jobs on the same nodes range from single-MIG up to full-node multi-GPU.
Related: #41
Code of Conduct
Is this a new feature, an improvement, or a change to existing functionality?
New Feature
How would you describe the priority of this feature request
High, we will not adopt slurm-bridge without it :( After it is added, we would be very eager to integrate it.
Please provide a clear description of problem this feature solves
We run a multi-tenant Kubernetes service cluster alongside a Slurm HPC cluster, and want tenants to run short GPU workloads from Kubernetes without writing batch scripts. The blocker is granularity of sharing.
Our GPU nodes have 4x H100, all MIG-partitioned. A typical workload from Kubernetes needs about a quarter of one H100, so roughly 1/16th of a node. Under the planned time-only sharing model, placing that pod requires every native Slurm job on the node to terminate first. We would be blocking 100% of a node to run a workload that needs almost none of it.
Our Slurm cluster serves our largest user group and our Kubernetes cluster is small by comparison. We cannot remove GPU nodes from Slurm, even temporarily, and a static whole-node carve-out would idle capacity we cannot spare. Slurm alone currently packs these nodes at MIG granularity with high efficiency, so any model requiring whole-node exclusivity is a large utilisation regression for us.
The practical result is that we cannot adopt slurm-bridge for GPU work. We stay on our current workaround: tenant pods mount the Slurm binaries and submit jobs to slurmctld directly. That keeps node efficiency because it is Slurm-native, but no Kubernetes object represents the running workload, users must build apptainer or podman containers, and job materials have to be staged outside the cluster. This is exactly the situation slurm-bridge would otherwise solve for us.
Follow-up to the discussion in #41, requested by @killianmuldoon.
Feature Description
As an administrator running Slurm and Kubernetes on shared GPU nodes, I want bridge-managed pods and native Slurm jobs to run on the same node at the same time on disjoint resources, so that a small Kubernetes workload does not require draining an entire multi-GPU node.
EARS form: while a native Slurm job is running on a subset of a hybrid node's resources, slurm-bridge shall allow an external job to be allocated the node's remaining unallocated resources and bind its pods to that node.
Describe your ideal solution
Dynamic co-residency. slurmctld tracks both native and external allocations on the node and packs them together, with external jobs submitted non-exclusively. This gives better utilisation because the Kubernetes share is not stranded when idle, but it needs coordinated GPU device identity and aligned CPU sets.
Describe any alternatives you have considered
Dedicating whole GPU nodes to Kubernetes. Rejected. Our Kubernetes cluster is small relative to Slurm demand, so the idle GPU time would be unacceptable to our user community.
Time-only sharing as currently planned. Rejected for GPU work because of the roughly 16x over-allocation described above. It would work for CPU-only workloads.
Our current workaround, pods that mount the Slurm binaries and submit to slurmctld directly. Preserves node efficiency but gives up essentially everything Kubernetes-native: no Kubernetes object for the running workload, apptainer or podman required inside the job, and materials staged outside the cluster.
Additional context
Hardware context: GPU nodes with 4x H100, statically MIG-partitioned in different configurations. Typical Kubernetes workloads target a single or multiple MIG instances. Native Slurm jobs on the same nodes range from single-MIG up to full-node multi-GPU.
Related: #41
Code of Conduct