conformance: add the all-or-nothing negative case to gang-scheduling (CNCF bar) #3603

Activity

1. yuanchen8911

   added

   area/validator

   theme/validationConstraint evaluation, health checks, and conformance evidenceConstraint evaluation, health checks, and conformance evidence

   on Nov 3, 2026
2. yuanchen8911

   added the

   Enhancement

   issue type on Nov 3, 2026
3. yuanchen8911

   added a parent issue on Nov 3, 2026

   • [Epic]: Single-path CNCF AI conformance validation and submission evidence #3601
4. github-actions

   added

   needs-triage

   and removed

   needs-triage

   on Nov 3, 2026
5. yuanchen8911

   changed the title [-]conformance: add gang-scheduling-gpu-barrier (port the script's two-phase all-or-nothing test)[/-] [+]conformance: add the all-or-nothing negative case to gang-scheduling (CNCF bar)[/+] on Nov 4, 2026
6. yuanchen8911 commented on Nov 4, 2026

   

   yuanchen8911

   on Nov 4, 2026

   Last edited by yuanchen8911

   ContributorAuthor

   More actions

   Scope update from the CNCF bar audit (#3619, 2026-11-04). Verified against the kubernetes-sigs/ai-conformance tests at fd9ba2d and the upstream validator at c64e5bc; the full table is on #3619.

   Add:

   • A negative phase after the positive case: a PodGroup with minMember = N and N CPU-only members whose combined CPU and memory requests cannot fit the cluster (size against the sum of allocatable minus allocated across schedulable nodes, or use upstream's shape of many small members; either is at the bar). Hold for a verification window (upstream default 30s) and fail if any member has status.phase=Running or spec.nodeName set.
   • Resource requests on the positive-case pods. Today they request nothing; upstream uses 100m CPU and 128Mi so the scheduler has something to co-schedule against.
   • A positive-case completion bound (upstream: Complete within 2 min) in place of only the co-schedule window.

   Keep, above the bar but the vacuity control: credit the negative phase only when the PodGroup status.schedulingConditions carries KAI's affirmative gang refusal, so an unbound pod from a stalled or absent scheduler does not pass.

   Drop: the GPU barrier, idle-node selection, capacity blocker and nodeName pinning from collect_gang. Delete the section without a port. No nvidia.com/gpu request anywhere in the check.

   Acceptance:

   • Positive: both members Succeeded, each with spec.nodeName, within the completion bound.
   • Negative: zero members bound or Running for the full window, and the PodGroup refusal present. The refusal condition is the primary oracle on every cluster. Where a member can fit an eligible node (heterogeneous clusters), a bound member additionally fails fast with its name in the message; on homogeneous clusters no member fits any node, so that signal cannot fire and the refusal condition alone decides.
   • With gang scheduling disabled (default scheduler on the same pods), the negative phase fails: no refusal condition is written, so the phase is not credited; where a member fits a node it also fails by the bound member's name.
   • No GPU requested; runtime under three minutes; the evidence page and e2e.log show both phases.
7. yuanchen8911

   self-assigned this

   on Nov 4, 2026
8. yuanchen8911

   added a commit that references this issue on Nov 4, 2026

   feat(conformance): add the all-or-nothing negative case to gang-sched…

   ...

   This issue will close once commit d351d89 is merged into the 'main' branch.

   d351d89
9. yuanchen8911 commented on Nov 4, 2026

   

   yuanchen8911

   on Nov 4, 2026

   ContributorAuthor

   More actions

   Scope note: deleting the collect_gang section and manifests/gang-scheduling-test.yaml from the bash collector moves to #3602, which retires the collector as a whole. While --cncf-submission still runs the script as the only submission path, removing the section alone would drop the gang_scheduling MUST page from a submission. PR #3637 covers the Go check.
10. yuanchen8911

    added 6 commits that reference this issue on Nov 4, 2026

    feat(conformance): add the all-or-nothing negative case to gang-sched…

    ...

    This issue will close once commit 755f968 is merged into the 'main' branch.

    755f968

    feat(conformance): add the all-or-nothing negative case to gang-sched…

    ...

    This issue will close once commit 9cc544e is merged into the 'main' branch.

    9cc544e

    feat(conformance): add the all-or-nothing negative case to gang-sched…

    ...

    This issue will close once commit 80e3355 is merged into the 'main' branch.

    80e3355

    feat(conformance): add the all-or-nothing negative case to gang-sched…

    ...

    This issue will close once commit f694eab is merged into the 'main' branch.

    f694eab

    feat(conformance): add the all-or-nothing negative case to gang-sched…

    ...

    This issue will close once commit d6d5d74 is merged into the 'main' branch.

    d6d5d74

    feat(conformance): add the all-or-nothing negative case to gang-sched…

    ...

    This issue will close once commit a4575d7 is merged into the 'main' branch.

    a4575d7
11. mchmarny

    closed this as completedin #3637on Nov 5, 2026