evidence: generate the CNCF submission directory (PRODUCT.yaml, README, junit.xml, e2e.log, redaction)

Part of #3601. Scope set by #3619 (CNCF bar). Depends on #3612 and #3606.

Summary

Generate the complete CNCF submission directory from the conformance run under --cncf-submission: everything that was written by hand for the NVKE v1.36 submission (#3455).

Today

pkg/evidence/cncf/renderer.go writes evidence/<requirement>.md and index.md from CTRF results. PRODUCT.yaml, README.md and platform-configuration.md were authored by hand, redaction was done by hand, and the extra artifacts (routing behavior, RuntimeClass isolation, TrainJob reconciliation) were hand-run files.

Change

No platform-configuration.md: the few facts PRODUCT.yaml notes cite (component versions, node allocatable) are rendered into README.md from the snapshot and the installed-component inventory (pkg/inventory), so every note can cite a captured value (the docs: add NVKE v1.36 AI conformance submission #3455 review finding).
PRODUCT.yaml and README.md from templates. Vendor fields (name, contact, URLs, platformVersion) come from the validate config (spec.validate.evidence.cncf.vendor), so the command stays at two flags. status fields come from the aggregation in evidence: fail-closed requirement status aggregation scoped by level #3612 and are never typed, with one explicit exception: spec.validate.evidence.cncf.notApplicable maps a conditional MUST id (today only cluster_autoscaling) to its justification and renders that requirement N/A, taking precedence over a check that ran; any other id is refused.
junit.xml and e2e.log emitted from the CTRF report with AICR check names; PRODUCT.yaml references every supporting check explicitly with file://...#<check> fragments, which the upstream validator honors before its upstream-name fallback.
Exit status. A submission run derives its exit status from the requirement aggregation, superseding --fail-on-error's per-check rule: flag and input refusals and the readiness pre-flight exit 2 as today; an incomplete applicable MUST exits non-zero; a write or redaction error, or the pinned upstream validator failing or unable to run (for example it cannot fetch the CNCF landscape), exits non-zero as an internal error; a non-passing check behind a SHOULD does not change the exit status.
Export rule for unsuccessful optional checks. The pinned upstream validator rejects any referenced artifact that contains a failing testcase, before it resolves a fragment (Test artifact ... contains failing tests). So a failed check behind a SHOULD that evidence: fail-closed requirement status aggregation scoped by level #3612 renders Not Implemented (for example the sandbox probe with no runtime, conformance: v1.38 readiness, driver-runtime-management and workload-sandboxing checks #3615) must not appear as a failure in the submission junit.xml or e2e.log. Rule: the raw CTRF report keeps the real verdict; the submission artifacts export a non-passing testcase of a Not Implemented SHOULD as skipped with the original verdict and reason in the message, and PRODUCT.yaml does not reference it. A non-passing check behind a requirement overridden to N/A through notApplicable exports the same way as a Not Implemented SHOULD, as skipped with "not applicable: " and the original verdict in the message, unreferenced by PRODUCT.yaml and without effect on the exit status. A failing testcase of an applicable MUST (one not overridden to N/A) is never exported as anything but a failure, and the run exits non-zero before a submission directory is declared complete.
Manual attestation rule. A requirement whose only evidence is observational (the cluster-autoscaling constrained claim, conformance: own the checks hand-run for NVKE AI conformance (gateway traffic, CR reconciliation, RuntimeClass extra, autoscaling claim) #3428) cites provider documentation in PRODUCT.yaml and does not reference its testcase as automated proof; the aggregation (evidence: fail-closed requirement status aggregation scoped by level #3612) marks such results so the generator applies this.
README.md is an AICR packaging choice, not an upstream requirement: upstream requires PRODUCT.yaml and recommends junit.xml and e2e.log. README carries the platform facts the notes cite.
Optional, pinned run of the kubernetes-sigs/ai-conformance suite as independent verification with its own provenance; upstream testcase names appear only for tests actually executed from that suite. Upstream recommends the suite for v1.37+ and does not require it.
One redaction pass (pkg/evidence/redact) over PRODUCT.yaml, README.md, junit.xml, e2e.log and the evidence pages before anything is written. The existing evidence/<requirement>.md pages stay as the human-readable companion and are not extended.
Run upstream cncf/k8s-ai-conformance/scripts/validate.go (pinned revision) against the output as the final step; fail the run if it fails.

Upstream validator assumptions (checked 2026-11-04)

The submission generator targets the upstream validator at cncf/k8s-ai-conformance commit c64e5bc (scripts/validate.go, instructions.md) and the kubernetes-sigs/ai-conformance tests at fd9ba2d. The pinned revision is recorded in the submission README. These are the behaviors the design relies on; a bump of the pin re-checks each one.

Fragments on every artifact reference. A reference to junit.xml or e2e.log without #<check> is resolved through the validator's upstream-test-name map and fails "not found" for an AICR-named testcase. The generator writes file://...#<check> on every reference, e2e.log included.
e2e.log shape. The validator parses go test -v output: one --- PASS|FAIL|SKIP: <check> line per check. The log is emitted from the same CTRF report as junit.xml with those lines, so both artifacts agree by construction.
Whole-artifact failure. Any failing testcase anywhere in a referenced artifact fails that artifact before fragments are resolved. This is why the export rule is mandatory: non-passing testcases behind a Not Implemented SHOULD or an N/A-overridden MUST export as skipped with a reason; an applicable MUST failure fails the run and nothing is written.
Required vendor metadata. k8sConformanceUrl and productLogoUrl must be present and well-formed; they come from spec.validate.evidence.cncf.vendor and are refused up front when missing.
Scheduler neutrality. The requirement is scheduler-neutral and KAI satisfies it. The upstream suite's gang adapter supports only kueue and volcano, so on KAI the AICR-named junit testcase is the sole machine-checked evidence for gang_scheduling, and the optional upstream-suite add-on is partial by design. Our check must therefore meet the upstream assertion on its own (conformance: add the all-or-nothing negative case to gang-scheduling (CNCF bar) #3603).
ai_inference map gap. ai_inference is not in the validator's auto-tested map at the pinned revision. Its testcase is still referenced with a fragment so a later map change does not break an existing submission.

Acceptance

A fixture with every MUST passing and one SHOULD whose check failed renders that SHOULD Not Implemented, exports its testcase as skipped, and passes the pinned upstream validator; the same fixture with a failed MUST exits non-zero.
A single aicr validate ... --cncf-submission on the NVKE GB200 cluster produces a directory that passes the upstream validator without edits (see the NVKE section below).

Proving it on NVKE (absorbed from #3610)

Regenerate the NVKE v1.36 evidence with one aicr validate --phase conformance --evidence-dir ... --cncf-submission run on the GB200 NVKE cluster and compare it with the merged submission in docs/conformance/cncf/v1.36/nvke/ (upstream cncf/k8s-ai-conformance#159). Then produce the v1.37 shape ahead of NVKE's next Kubernetes version.

Wire the NVKE recipes to every check the expected-check table requires (the kubeflow leaf today declares no gang, HPA, secure-access or gateway check).
Run the single path on the NVKE GB200 cluster (gb200-nvke-ubuntu-training-kubeflow). Prerequisites: free GPUs for the HPA and TrainJob checks; sign evidence with a non-personal identity.
Compare with the merged v1.36 directory: every hand-run artifact (gateway routing, RuntimeClass isolation, TrainJob reconciliation) must have a generated counterpart. Every status that differs must be explained by captured evidence; the old statuses are not the oracle.
Run the pinned upstream validator on the output.
Produce the v1.37 layout and confirm the validator credits the file://...#<check> references.
Decide whether to refresh the committed v1.36 directory in place.

Acceptance: zero hand steps between aicr validate and a validator-clean submission directory on NVKE, and every status difference accounted for by evidence.

Scope update from the CNCF bar audit (#3619, 2026-11-04). In addition to the "Upstream validator assumptions" section in the body:

Record the pinned upstream validator revision (today c64e5bc) in the submission README.
index.md is not written into the submission directory; README.md is the index.
Requirements that cited platform-configuration.md in the NVKE submission (driver_runtime_management, cluster_autoscaling) cite README.md or a documentation URL instead.
The validator checks k8sConformanceUrl and productLogoUrl for URL format and reachability, not only presence.

feat(validate): render CNCF submission from the conformance run

Review notes from the epic-wide audit of the single-path implementation (2026-11-06), for the NVKE step that stays open after #3701 merges:

Recipe wiring (step 1). nvke.yaml plus gb200-nvke-ubuntu-training(-kubeflow) resolve to platform-health, gpu-operator-health, dra-support, accelerator-metrics, ai-service-metrics (plus robust-controller on the kubeflow leaf). Without gang-scheduling, pod-autoscaling, secure-accelerator-access, driver-runtime-management and an inference gateway, five requirements render NOT TESTED and the run exits 8 with nothing promoted. pod-autoscaling needs kube-prometheus-stack with the prometheus-adapter rule in the recipe. ai_inference needs a gateway component on the submission recipe, or the submission must come from an inference recipe; decide which recipe is the NVKE submission recipe before wiring.
cluster_autoscaling on bare metal. NVKE nodes carry a nico:// providerID, which the check skips as "no autoscaler probe for this provider". The truthful declaration is spec.validate.evidence.cncf.notApplicable (fixed-size cluster, no autoscaler), which the upstream validator accepts with a reviewer warning. Do not use the documented attestation here; upstream frames manual attestation for requirements without an automated test and may challenge it.
Checklist version. The run selects the nearest carried checklist with a warning; a 1.38 NVKE cluster would render the v1.37 shape. Refusal of an above-latest minor is tracked in validator: fail-closed result handling and probe hardening for conformance checks #3703 (pre-flight items); run the regeneration on 1.37 or wait for that fix.
Troubleshooting rows a first NVKE run will hit (docs, before the run): gang negative phase (bound member, missing KAI refusal), gateway traffic (route not Accepted/ResolvedRefs, listener allowedRoutes, split out of tolerance, foreign backend), TrainJob reconciliation (missing torch-distributed, Failed=True), HPA stuck at one replica (second accelerator not allocatable), the nico:// autoscaling skip.
Still manual after the run: copy --evidence-dir into a fork of cncf/k8s-ai-conformance as vX.Y/<product>/ and open the PR; author the vendor block (landscape-exact name, reachable SVG logo, matching k8s-conformance URL); egress from the runner to raw.githubusercontent.com and every cited URL; review the directory for identifiers the scrubber does not cover (see evidence: CNCF submission follow-ups (redaction, pre-flight version, README facts, upstream suite) #3704); remove any .incomplete leftover.
The committed docs/conformance/cncf/v1.36/nvke/ README and PRODUCT.yaml still cite evidence/platform-configuration.md, which the generator no longer produces (step 6 decides the refresh).

NVKE regeneration done (2026-11-07, main `b1c0806`)

The single-path submission ran on the NVKE GB200 cluster nvke022-ai-20261026-134532 (Kubernetes v1.36.3, 1 control plane + 2 GB200 nodes, Ubuntu 24.04, arm64) and the upstream validator accepted the directory on the first run.

Preparation. Snapshot and recipe generated fresh (leaf gb200-nvke-ubuntu-training-kubeflow, 16 components). Drift report of the deployed stack against the recipe before any change: the only drift the run needed was prometheus-adapter (chart 5.3.0 already deployed, but its values predated the ai_conformance_hpa_queue_depth rule from #3647). Upgraded that one release with the bundle's own install script (same chart version, new values). Not changed: nodewright-operator v0.20.0 (pin moved to v0.21.0 in #3696, not needed for the run), gpu-operator tolerations, kubeflow-trainer restated defaults, nvsentinel init-image digest. Full table: /tmp/aicr/nvke022/3605/drift-report.md.

Recipe used. A copy of the generated recipe with the 12 conformance checks declared (the NVKE overlays on main declare neither gang-scheduling, pod-autoscaling, secure-accelerator-access, driver-runtime-management nor inference-gateway) plus the already-deployed agentgateway component so the inference-gateway check could run. The proposed overlay change is at /tmp/aicr/nvke022/3605/recipe-wiring.diff (touches gb200-nvke-ubuntu-training.yaml and gb200-nvke-ubuntu-training-kubeflow.yaml); whether it becomes a PR is a separate decision.

Config. spec.validate.evidence.cncf.vendor for NVIDIA / NVKE 0.22.0 and notApplicable: {cluster_autoscaling: ...} (fixed-size bare-metal cluster with nico:// nodes, no cluster autoscaler or Karpenter). No documented attestation.

Run. aicr validate --phase conformance --cncf-submission --evidence-dir ... --config config.yaml with validator images sha-6e79158ff... (the published main build; #3701 changed no validator code). Exit 0, 16m40s, 11 passed, 1 skipped.

Check	Result	Duration
platform-health	passed	<1s
gpu-operator-health	passed	<1s
dra-support	passed	18s
accelerator-metrics	passed	<1s
ai-service-metrics	passed	216s
gang-scheduling	passed	38s
pod-autoscaling	passed	566s
cluster-autoscaling	skipped (exported N/A)	0s
secure-accelerator-access	passed	13s
driver-runtime-management	passed	14s
inference-gateway	passed	38s
robust-controller	passed	11s

Upstream validator: "Validation successful with 1 warning(s)" (the expected cluster_autoscaling MUST marked N/A review warning); staging promoted to /tmp/aicr/nvke022/3605/submission/ with PRODUCT.yaml, README.md, junit.xml (13 testcases), e2e.log, evidence/ (9 pages).

PRODUCT.yaml statuses: Implemented: ai_inference, gang_scheduling, pod_autoscaling, accelerator_metrics, ai_service_metrics, secure_accelerator_access, robust_controller (all MUST) and driver_runtime_management (SHOULD). N/A: cluster_autoscaling. Not Implemented (SHOULD, no check): gpu_sharing, virtualized_accelerator, advanced_inference_ingress, high_performance_networking, disaggregated_inference.

Against the committed docs/conformance/cncf/v1.36/nvke/ (#3455): every requirement keeps its status; driver_runtime_management moves from a hand-cited note to a generated check page; every hand-run artifact (gateway routing, RuntimeClass isolation, TrainJob reconciliation, platform-configuration) now has a generated counterpart; README replaces index.md and platform-configuration.md. Full comparison and the v1.37 shape note: /tmp/aicr/nvke022/3605/comparison.md.

Before this replaces the committed directory or goes upstream:

Regenerate from a stamped release or CI build: the local build renders AICR: dev (commit unknown) in README, so provenance is not real yet.
README Components prints recipe pins, not installed versions (nodewright-operator v0.21.0 while the cluster runs v0.20.0); evidence: CNCF submission follow-ups (redaction, pre-flight version, README facts, upstream suite) #3704 item 3.
Redaction gap: ResourceSlice names carry the cluster name (nvke022-ai-20261026-134532...) on the dra-support and secure-accelerator-access pages; added to evidence: CNCF submission follow-ups (redaction, pre-flight version, README facts, upstream suite) #3704.
The N/A justification is text only; if reviewers want the NVKE node-pool documentation cited, the string must carry the URL.
The RuntimeClass probe ran on one node and the TrainJob demonstration is CPU-only; the old hand run covered both workers and GPU execution. Upstream does not require either.
Manual steps unchanged: copy into a fork of cncf/k8s-ai-conformance as v1.36/nvke/ and open the PR; vendor logo and URL ownership.

A second run with --emit-attestation is in progress for the attestation artifact; nothing is signed or published. Cluster left as found apart from the adapter upgrade; two cluster-admin ClusterRoleBindings from --no-cleanup runs on 2026-10-28 (aicr-validator-20261028-230835-..., aicr-validator-20261028-232623-...) predate this work and were left in place.

Addendum: the --emit-attestation run on the same recipe and snapshot also exited 0 (11 passed, 1 skipped, 13m49s). The unsigned bundle at /tmp/aicr/nvke022/3605/attestation/summary-bundle/ carries statement, manifest, recipe, snapshot, BOM and the conformance CTRF; all manifest digests match, the subject is recipe:gb200-nvke-ubuntu-training-kubeflow, and the validator images are recorded by immutable sha- tag. Not pushed, not signed, not verified against a registry. Cluster audit after both runs: no validator namespaces or Jobs left, 0 GPU-requesting pods, 3 nodes Ready.

Two facts for whoever refreshes the committed directory: the vendor block I used (www.nvidia.com, docs.nvidia.com/, yuanc@nvidia.com, repo NVIDIA/aicr) differs from the committed PRODUCT.yaml (docs.nvidia.com/dsx and /nvke, eboyd@nvidia.com, repo dsx-ai-factory/nvke), so the config must be aligned with the owners before anything goes upstream; and cluster-autoscaling skips on NVKE with unrecognized-platform (no autoscaler probe for nico://), so the N/A rests on the notApplicable justification alone, with no evidence page.

Navigation Menu

Repository navigation

evidence: generate the CNCF submission directory (PRODUCT.yaml, README, junit.xml, e2e.log, redaction) #3605

Summary

Today

Change

Upstream validator assumptions (checked 2026-11-04)

Acceptance

Proving it on NVKE (absorbed from #3610)

NVKE regeneration done (2026-11-07, main `b1c0806`)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

evidence: generate the CNCF submission directory (PRODUCT.yaml, README, junit.xml, e2e.log, redaction) #3605

Description

Summary

Today

Change

Upstream validator assumptions (checked 2026-11-04)

Acceptance

Proving it on NVKE (absorbed from #3610)

Activity

yuanchen8911 commented on Nov 4, 2026

yuanchen8911 commented on Nov 7, 2026

yuanchen8911 commented on Nov 7, 2026

NVKE regeneration done (2026-11-07, main b1c0806)

yuanchen8911 commented on Nov 7, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

NVKE regeneration done (2026-11-07, main `b1c0806`)