[BUG]: Race condition prevents low priority pods from being scheduled

[cgetzen]

Version

1.2.1

Which installation method(s) does this occur on?

No response

Describe the bug.

While some pods are waiting for Slurm capacity (their external jobs are queued on Priority/Resources), lower-priority pods stop being scheduled at all, even when the resources they need are free.

The starved pods show no FailedScheduling events, empty status.conditions, and their names never appear in the scheduler log. They are never popped from the scheduling queue.

Contributing factors:

1. A pod whose external job has no allocation yet is retried forever. On every failed cycle, PostFilter calls activatePod() -> fwk.Handle.Activate(). In kube-scheduler this guarantees the pod re-enters the active queue within podMaxBackoffSeconds (default 10s).
2. The chart's default profile uses strict PrioritySort as the only queue ordering, so each retry re-enters at the front of the queue, ahead of every lower-priority pod.
3. Each retry is expensive: validatePodToJob fetches and JSON-decodes the entire response and rebuilds the pod->job map on every scheduling cycle. On our production cluster that is ~30 MB (~900 job records), 1.4s per attempt at 4 CPUs.
4. The scheduling loop is serial. Once blocked pods × per-attempt cost exceeds the 10s retry cap, the loop is permanently occupied by the waiting pods, and no lower-priority pod is ever attempted.

v1.1.1 had the same behavior through a different path (PreFilter returned fwk.Pending, which requeues with no backoff at all); ac755a9 changed the mechanism to activatePod() and the behavior persisted.

Minimum reproducible example

1. Give the scheduler a small CPU limit so per-attempt cost is measurable (ours ran at 0.25 vCPU):

kubectl -n slurm-bridge set resources deploy/slurm-bridge-scheduler --limits=cpu=250m
# optional: to observe attempt cadence add --v=4 to the scheduler args

2. Inflate the /jobs payload

kubectl -n slurm exec statefulset/slurm-controller -c slurmctld -- bash -c \
  'for i in $(seq 1000); do sbatch --hold -p shared --wrap "sleep 10" >/dev/null; done'

3. Create a high PriorityClass and 20 "waiting" pods

kubectl create priorityclass high-priority --value=100000
for i in $(seq 1 20); do cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: blocked-$i
  annotations:
    slurmjob.slinky.slurm.net/priority: "0"
spec:
  schedulerName: slurm-bridge-scheduler
  priorityClassName: high-priority
  restartPolicy: Never
  containers:
  - name: c
    image: registry.k8s.io/pause:3.9
    resources: {requests: {cpu: 100m, memory: 64Mi}}
EOF
done

4. Wait until all 20 pods carry the scheduler.slinky.slurm.net/slurm-jobid

5. Create the pod that never gets scheduled

kubectl run victim --image=registry.k8s.io/pause:3.9 --restart=Never \
  --overrides='{"spec":{"schedulerName":"slurm-bridge-scheduler"}}'

6. The victim stayed Pending for 12+ minutes with zero scheduling attempts

Relevant log output

$ kubectl describe pod <starved-pod>
...
Status:       Pending
Conditions:   <none>
Events:       <none>

On pod creation, we will see:

I0827 20:34:39.397042  eventhandlers.go:214] "Add event for unscheduled pod" pod="<ns>/<starved-pod>"

Meanwhile, we will see these logs for the other pods:

I0827 20:33:50.262595  schedule_one.go:123] "Attempting to schedule pod" pod="<ns>/blocked-6"
I0827 20:33:51.672311  schedule_one.go:123] "Attempting to schedule pod" pod="<ns>/blocked-17"
I0827 20:33:53.170623  schedule_one.go:123] "Attempting to schedule pod" pod="<ns>/blocked-18"

Additional environment details

Other/Misc.

No response

Code of Conduct

• I agree to follow Slurm Bridge's Code of Conduct
• I have searched the open bugs and have found no duplicates for this bug report