Add slurm node name labeling to k8s nodes for StatefulSet-mode NodeSets for slurm-bridge interoperability

[arsdragonfly]

Summary

Adds slurm node name labeling to k8s nodes for StatefulSet-mode NodeSets for slurm-bridge interoperability. Applicable when pinToNode is true and oversubscription is turned off.

Checklist

• I have read the
  CONTRIBUTING.md
  and the
  Code of Conduct.
• New or existing tests cover these changes (where applicable).
• Documentation is updated if user-visible behavior changes.

Additional Context

An important nuance regarding why StatefulSet-mode NodeSets could be more appealing (even in a slurm-bridge setting), compared to DaemonSet-mode NodeSets, emerges when one considers the scenario of workload-aware scale-in of the k8s cluster itself (think of saving costs on the cloud).
With Karpenter or Cluster Autoscaler, which work via eliminating idle nodes without running pods, StatefulSet-mode NodeSet scale-in is already workload-aware in slurm-operator, after which scale-in would then trivially work in a workload-aware fashion as well. A working example can be found here.
With DaemonSet-mode NodeSet (which, unlike real DaemonSet, is actually also workload-bearing), the decision of "which node to evict during scale-in" no longer has a nice workload-aware k8s-native answer, and one would find oneself needing awkward custom solutions.

[killianmuldoon]

Thanks for this @arsdragonfly - I'll pull it internally and incorporate the changes.

[killianmuldoon]

I'm not totally comfortable with including this change. What we want here is to encode a contract slurm-bridge uses for hybrid nodes. Today this has to be done manually by cluster admins, I'm not sure encoding it inside slurm-operator is the right choice here.

The current implementation creates a very contract for the StatefulSet NodeSet - leaking slurm-bridge implementation details into the way the operator works.

I do understand the pain point here and would love to find a solution - I wonder if this be done more suitably with an admission controller or some other custom piece of orchestration?

[arsdragonfly]

I'm not totally comfortable with including this change. What we want here is to encode a contract slurm-bridge uses for hybrid nodes. Today this has to be done manually by cluster admins, I'm not sure encoding it inside slurm-operator is the right choice here.

The current implementation creates a very contract for the StatefulSet NodeSet - leaking slurm-bridge implementation details into the way the operator works.

I do understand the pain point here and would love to find a solution - I wonder if this be done more suitably with an admission controller or some other custom piece of orchestration?

Not sure why the cluster-admin-tunable label isn't a proper contract already.
slurm-operator owns the lifecycle of its pods and hence should manage the lifecycle of mappings.

[killianmuldoon]

slurm-operator owns the lifecycle of its pods and hence should manage the lifecycle of mappings

Agreed - but this is a slurm-bridge mapping rather than something related to operator. I'm trying to understand what would be the right approach here - we don't want tight coupling between the two projects without a strong motivation.

[killianmuldoon]

@arsdragonfly - can I ask what kind of use case you're trying to unlock here?

• Should the slurm-operator slurmd nodes scheduled by slurm-bridge?
• Are you looking to have a single slurmd pod per Kubernetes node?
• Is the main focus having a slurm-bridge hybrid cluster but with the right metadata to allow autoscaling?

[arsdragonfly]

Should the slurm-operator slurmd nodes scheduled by slurm-bridge?

Yes, that should be a valid scenario.

Are you looking to have a single slurmd pod per Kubernetes node?

That's what oversubscribe set to false already supports, yes.

Is the main focus having a slurm-bridge hybrid cluster but with the right metadata to allow autoscaling?

Yes, with particular focus on sane workload-aware scale-down behavior, as stated in the additional context in the beginning. (Scale-up-wise DaemonSet is also doable if you have something like KEDA -> CAPI but its scale-down isn't trivially graceful and workload-aware)

we don't want tight coupling between the two projects without a strong motivation

Not sure if that had been a goal at all from other maintainers' perspective. DaemonSet relied on an even more implicit pod hostname - k8s node name matching.

[killianmuldoon]

I wonder if a simpler solution would be to always set the slurmd node name to the Kubernetes nodeName in this case instead of relying on an override label?

This would make StatefulSet slurmd nodes with pinToNode=true, and oversubscribeNode=false look similar to DaemonSet nodes, but with more dynamic scheduling.

We could pass the KUBERNETES_NODE_NAME through the downward API and use it to register the node name.

The end result would be StatefulSet nodes with slurmNodeName == kubernetesNodeName. KEDA scheduling should work without any changes, and slurm-bridge should be able to match nodes using the existing flow through nodeName. Does this solution make sense?

[arsdragonfly]

It would be doable, though at the cost of mass pod replacement when one flips those flags to that specific combination.

[arsdragonfly]

see #264 for the alternative solution

[killianmuldoon]

see #264 for the alternative solution

Thanks for putting that together - we'll probably discuss this internally and find a solution that suits overall. Are you equally happy with a solution that uses the nodename as in #264?

[arsdragonfly]

I'm okay with either implementation (#264 is just effectively extending DaemonSet's convention to eligible StatefulSet)