[Epic]: Single-path CNCF AI conformance validation and submission evidence #3601

Outcome

aicr validate --phase conformance is the only runner for CNCF AI conformance. A flag decides whether evidence is written and in what layout; for the same selected checks it never changes what a check does. One run on a cluster produces a submission directory that passes the upstream validator with no hand-run checks, no hand-written PRODUCT.yaml, and no hand redaction. The requirement set tracks the CNCF version being submitted; a requirement is Implemented only when every expected check is present and passed, an incomplete applicable MUST fails the run, and an unclaimed SHOULD renders Not Implemented without failing it.

Where things stand

AICR has two unrelated implementations of every CNCF requirement.

• Path A: Go validators in validators/conformance/, run by aicr validate --phase conformance. Their stdout artifacts become evidence/*.md through pkg/evidence/cncf/renderer.go when --evidence-dir is set. This is what nightly UAT runs.
• Path B: the 3,548-line bash script pkg/evidence/cncf/scripts/collect-evidence.sh, embedded in the binary and run by --cncf-submission. That flag short-circuits validation (pkg/cli/validate.go, the "Short-circuit" block): under it the Go checks never run.

Both deploy workloads and keep separate verdict logic; docs/conformance/cncf/index.md documents them as "Two Modes". The script is deeper for gang scheduling (two-phase GPU barrier on a pinned node vs CPU-only PodGroup in Go), HPA (CUDA stress vs metrics-API probe), and metrics (DCGM and Prometheus query capture vs an 80-line Go check). Go is deeper for secure access (1,687 lines, sibling-container isolation), cluster autoscaling (AKS branch, fail-closed list errors), and the gateway (data-plane readiness, exposure policy).

The NVKE v1.36 submission still needed four hand-run checks on top of the script (gateway traffic, RuntimeClass bypass probe, TrainJob reconciliation, autoscaling N/A text), plus hand-written PRODUCT.yaml, README, platform-configuration.md and hand redaction. review of #3455 caught a PRODUCT.yaml note (SR-IOV resources) whose cited evidence did not show the claim. #3155 sets the bar: incomplete results cannot masquerade as passing, and evidence must describe what was actually checked. The v1.2 release update names the next NVKE items: explicit secondary-network attachment checks and durable Multus token rotation.

Upstream requirement drift (checked 2026-11-03)

Validation gaps

Design: one path

Principle: aicr validate --phase conformance is the only runner. The recipe defines which conformance checks run; every run executes all of them at the depth the CNCF bar requires (#3619). Flags never change what a check does or how it decides. Nightly UAT and a CNCF submission run the identical check set; the submission run additionally writes the submission directory. The bash collector and its --feature section interface are deleted.

Commands:

• aicr validate -r recipe.yaml -s snapshot.yaml --phase conformance: every check the recipe lists. CTRF report and recipe-evidence attestation as today. No evidence files. This is the nightly run.
• aicr validate -r recipe.yaml -s snapshot.yaml --phase conformance --cncf-submission --evidence-dir <dir>: the same run, plus PRODUCT.yaml, README.md (with the platform facts PRODUCT.yaml notes cite), junit.xml, e2e.log and the existing evidence/<requirement>.md pages, redacted in one pass, then the pinned upstream validator as the last step. --cncf-submission and --evidence-dir require each other; --recipe is required; --skip-check and --no-cluster are refused. Vendor fields and the conditional cluster_autoscaling N/A justification come from the validate config. Statuses come from the run (evidence: fail-closed requirement status aggregation scoped by level #3612): an incomplete applicable MUST fails the run; a SHOULD with incomplete or absent checks renders Not Implemented without failing.

Not in the design (removed 2026-11-04 after the CNCF-bar audit, #3619): depth tiers or sibling checks, a nightly --skip-check list, a weekly full-depth lane, typed artifacts in the signed bundle, platform-configuration.md, the collector's GPU gang barrier, CUDA stress and nvidia-smi/DCGM captures. --skip-check remains only for lanes that cannot host a check at all (Kind, KWOK).

Nightly UAT budget

Nightly UAT already runs --phase all on cells with two GPU nodes. The parity ports to the CNCF bar add one CPU-only gang Job, one HPA cycle on 2 GPUs, one TrainJob and gateway traffic on CPU backends, estimated at 5 to 8 minutes with the HPA scale-down as the long pole; recipes that gain checks through #3605's wiring (EKS, AKS and NVKE training bases) see the largest delta. #3609 measures the cost and raises the step budget. Checks run serially on a cell.

Options considered on 2026-11-03 (one depth always-on; two tiers as sibling checks; a depth flag; delegating deep MUSTs to the upstream suite) are recorded in the ADR-007 amendment (#3614). The two-tier option was chosen that day and superseded on 2026-11-04 once the audit showed the expensive tests exceeded the CNCF bar; AICR's junit carries AICR check names referenced by file://...#<check> fragments, and the upstream suite remains optional independent verification.

Plan (sub-issues, in merge order)

1. docs(adr): amend ADR-007 for single-path CNCF conformance evidence #3611 docs(adr): amend ADR-007. Lands first.
2. evidence: key the CNCF requirement map by Kubernetes version and drop vendor names from titles #3606 evidence: complete upstream checklist per Kubernetes version with expected checks per requirement; vendor names out of titles; pinned schemas; unsupported versions refused.
3. conformance: align checks and submission output to the CNCF requirement bar #3619 conformance: the CNCF-bar audit (decision record for the scope cuts below). evidence: fail-closed requirement status aggregation scoped by level #3612 evidence: fail-closed status aggregation scoped by level. Gate: lands before any parity port.
4. Parity ports to the CNCF bar, in place (no sibling checks, no tiers), each deleting its script section: conformance: add the all-or-nothing negative case to gang-scheduling (CNCF bar) #3603 all-or-nothing negative case in gang-scheduling (CPU-only); conformance: bring pod-autoscaling and the two metrics checks to the CNCF bar #3604 HPA on GPU-requesting pods with a custom metric, plus metrics capture; conformance: own the checks hand-run for NVKE AI conformance (gateway traffic, CR reconciliation, RuntimeClass extra, autoscaling claim) #3428 gateway traffic (on feat(validator): gateway-neutral inference-gateway check #3589), CR reconciliation to completion, the RuntimeClass probe as an AICR extra, cluster-autoscaling constrained claim; inference-gateway check counts HTTPRoutes attached by parent name only, ignoring namespace #3597 and [Feature]: Gateway selection for inference recipes and a gateway-neutral inference-gateway check #3588 gateway fixes.
5. validate: run CNCF submission evidence from the conformance phase and retire collect-evidence.sh #3602 validate: --cncf-submission --evidence-dir renders from the conformance run (each requires the other; evidence only for submissions); delete the collector and remaining script; --feature removed; facade migration semantics specified. Gate: a requirement-by-requirement comparison of a Go run against a script run attached before deletion, scoped to the CNCF bar.
6. evidence: generate the CNCF submission directory (PRODUCT.yaml, README, junit.xml, e2e.log, redaction) #3605 evidence: submission packaging (PRODUCT.yaml, README with the platform facts, junit.xml, e2e.log with #<check> references, one redaction pass, pinned upstream validator gate; vendor fields from config) and the NVKE proof run.
7. NVKE proof (in evidence: generate the CNCF submission directory (PRODUCT.yaml, README, junit.xml, e2e.log, redaction) #3605): wire NVKE recipes to the expected checks, regenerate v1.36 through the single path, explain every status difference by evidence, dry-run the v1.37 shape.

In parallel at any point: #3615 v1.38 readiness (driver-runtime-management; workload sandboxing, where a CPU-pool runtime suffices and a missing runtime is Not Implemented, never N/A); #3509 node image and bootstrap in evidence; #3474 secondary-network attachment probe (v1.2 "Next" item).

Related, owned elsewhere (not sub-issues): #3588 gateway selection for inference recipes (iracic82), #3474 secondary-network attachment exercise (atif1996), #3509 node image and bootstrap version in evidence; follow-ups and gaps after this epic are tracked under #3710 (sub-issues #3704, #3705, #3709); #3155 (parent outcome), #1044, #2137 and #2877 (children of #3155), #2882, #3482.

NVKE v1.36 validation run, for provenance (closed): #3439 RDMA fabric gate stability window, #3452 KWOK NVKE cleanup on the SR-IOV pre-delete hook, #3361 check-nvidia-smi skipped every GPU node when one was busy, #3384 sensitive information removed from docs and issue text. #3389 (the NVKE conformance tracker) is attached as a sub-issue.

Upstream validator assumptions (checked 2026-11-04)

The submission generator targets the upstream validator at cncf/k8s-ai-conformance commit c64e5bc (scripts/validate.go, instructions.md) and the kubernetes-sigs/ai-conformance tests at fd9ba2d. The pinned revision is recorded in the submission README. These are the behaviors the design relies on; a bump of the pin re-checks each one.

• Fragments on every artifact reference. A reference to junit.xml or e2e.log without #<check> is resolved through the validator's upstream-test-name map and fails "not found" for an AICR-named testcase. The generator writes file://...#<check> on every reference, e2e.log included.
• e2e.log shape. The validator parses go test -v output: one --- PASS|FAIL|SKIP: <check> line per check. The log is emitted from the same CTRF report as junit.xml with those lines, so both artifacts agree by construction.
• Whole-artifact failure. Any failing testcase anywhere in a referenced artifact fails that artifact before fragments are resolved. This is why the export rule is mandatory: non-passing testcases behind a Not Implemented SHOULD or an N/A-overridden MUST export as skipped with a reason; an applicable MUST failure fails the run and nothing is written.
• Required vendor metadata. k8sConformanceUrl and productLogoUrl must be present and well-formed; they come from spec.validate.evidence.cncf.vendor and are refused up front when missing.
• Scheduler neutrality. The requirement is scheduler-neutral and KAI satisfies it. The upstream suite's gang adapter supports only kueue and volcano, so on KAI the AICR-named junit testcase is the sole machine-checked evidence for gang_scheduling, and the optional upstream-suite add-on is partial by design. Our check must therefore meet the upstream assertion on its own (conformance: add the all-or-nothing negative case to gang-scheduling (CNCF bar) #3603).
• ai_inference map gap. ai_inference is not in the validator's auto-tested map at the pinned revision. Its testcase is still referenced with a fragment so a later map change does not break an existing submission.

Risks and open decisions

• GPU hardware is needed to verify every parity port; the NVKE GB200 cluster must stay available with free GPUs.
• The conformance validator ships as its own image; each parity-port PR must rebuild it and run live, not only make test.
• The 83% coverage floor applies to new pkg/ code; validators are excluded.
• Whether to run the upstream sigs suite as optional independent verification (decided in evidence: generate the CNCF submission directory (PRODUCT.yaml, README, junit.xml, e2e.log, redaction) #3605); AICR's junit is the default artifact.
• workload_sandboxing at v1.38 needs a product answer from NVKE; a CPU-pool sandbox runtime satisfies the upstream test.
• Regenerate nim-eks conformance evidence in place with the strengthened behavioral tests #2137 (regenerate v1.35 nim-eks): close as superseded once the single path exists, or regenerate at the next NIM submission.