Repository navigation
Description
Part of #3601. Lands after the parity ports have demonstrated coverage, before #3605.
Summary
Make aicr validate --phase conformance the only runner for CNCF AI conformance. --cncf-submission stops bypassing validation and becomes an output option on that run. Delete what remains of pkg/evidence/cncf/scripts/collect-evidence.sh and its collector.
Today
--cncf-submission short-circuits validation in pkg/cli/validate.go and runs the embedded bash collector through pkg/evidence/cncf/collector.go. The Go checks in validators/conformance/ never run on a submission. docs/conformance/cncf/index.md documents the two as "Two Modes".
Order
The parity-port issues (#3603, #3604, #3428) each delete the script section they replace; sections whose depth exceeds the CNCF bar (GPU gang barrier, CUDA stress, nvidia-smi and DCGM captures) are deleted without a port (#3619). This issue removes the collector, the remaining sections and the flag plumbing once no section is still the only implementation of a requirement.
Change
- The decision record is docs(adr): amend ADR-007 for single-path CNCF conformance evidence #3611 (ADR-007 amendment) and lands first.
--cncf-submissionand--evidence-dirrequire each other (today only the first implies the second); evidence is written only for a submission.--skip-checkand--no-clusterare refused with them (already true). The run is the conformance phase, then render; the renderer itself is evidence: generate the CNCF submission directory (PRODUCT.yaml, README, junit.xml, e2e.log, redaction) #3605. A configevidence.cncf.dirwithoutcncfSubmissionis refused and documented as the one deprecation.--cncf-submissionrequires--recipe(or a config recipe input) and is refused withErrCodeInvalidRequestwithout one: the single runner selects checks from the recipe, and the collector's capability-detection path goes away with it. Update the examples indocs/conformance/cncf/index.mdanddocs/user/validation.mdthat show a submission run without--recipe, and the--cncf-submissionflag usage text.- Remove
--featureandevidence.cncf.features. Client.CollectCNCFEvidencekeeps its signature as a wrapper over run-plus-render somake api-diffstays green. Migration semantics to specify and test: a nilRecipeResultis rejected withErrCodeInvalidRequest(the run needs a recipe to select checks; the old capability-detection path is gone); a non-emptyFeaturesis rejected withErrCodeInvalidRequestnaming the removed option;NoClustertrue is rejected as today. Document indocs/integrator/go-library.md.- Delete
pkg/evidence/cncf/scripts/,collector.go, and the script-only tests (claim_shape_test.go). - Rewrite
docs/conformance/cncf/index.mdto one mode, including its "Structural Validation (CI)" example that shows--evidence-dirwithout--cncf-submission; updatedocs/user/cli-reference.md,docs/user/validation.md,docs/user/cli-config.md,docs/integrator/go-library.md. - Exit status of a submission run derives from the requirement aggregation (evidence: fail-closed requirement status aggregation scoped by level #3612) and supersedes
--fail-on-error's per-check rule; see evidence: generate the CNCF submission directory (PRODUCT.yaml, README, junit.xml, e2e.log, redaction) #3605 for the full list of non-zero causes.
Gate: demonstrated parity
Completing the new checks is not sufficient to delete the collector. Before this issue merges, one cluster run with the Go checks and one with the script (last revision) are compared section by section, and each MUST the script evidenced has a Go-produced counterpart at the CNCF bar (#3619): the verdict, the upstream-test behavior it demonstrates, and the measured values the PRODUCT.yaml note cites. Collector captures that exceed the bar (GPU barrier, CUDA stress, nvidia-smi and DCGM dumps) are not part of the comparison. The comparison is attached to this issue.
Acceptance
aicr validate -r recipe.yaml --phase conformance --evidence-dir d --cncf-submissionruns the Go conformance checks and writes evidence; no bash is executed.- For the same selected checks, verdicts are byte-identical with and without
--cncf-submission; the submission run may still exit non-zero on incomplete coverage or a failed artifact validation. - No file under
pkg/evidence/cncf/scripts/remains. - The parity comparison is attached and shows no MUST whose only evidence at the CNCF bar came from the script.
Activity
docs(adr): address review on the ADR-007 amendment
docs(adr): mark the ADR-007 amendment as decided, implementation pending
docs(adr): scope the exit causes and require --recipe for a submissio…
docs(adr): address the approval-round review on the ADR-007 amendment
yuanchen8911 commentedon Nov 4, 2026
Scope update from the CNCF bar audit (#3619, 2026-11-04). Verified against the kubernetes-sigs/ai-conformance tests at
fd9ba2dand the upstream validator atc64e5bc; the full table is on #3619.Add:
collect_gangbarrier and blocker,collect_hpaCUDA stress,collect_accelerator_metricsPrometheus queries and port-forward dumps,collect_service_metrics_dynamo,_nim,_trainer,collect_cluster_autoscalingAWS CLI and GKE ConfigMap captures,collect_dra(supporting evidence only).--cncf-submissionusage text, which still reads "Deploys GPU workloads, captures nvidia-smi output, Prometheus queries, and HPA scaling tests".Keep everything else as written.
Follow-ups from the #3627 review that land here with the flag contract:
CNCF MUST requirement not establishedwarnings when no conformance tests ran (today--evidence-diron a non-conformance phase logs one per MUST).pkg/evidence/cncf/aggregate.gocover exactly the checks on each requirement's evidence page, so a check added to a page cannot be silently unrequired.feat(validate): render CNCF submission from the conformance run
feat(validate): render CNCF submission from the conformance run
feat(validate): render CNCF submission from the conformance run
feat(validate): render CNCF submission from the conformance run
feat(validate): render CNCF submission from the conformance run
feat(validate): render CNCF submission from the conformance run
feat(validate): render CNCF submission from the conformance run
feat(validate): render CNCF submission from the conformance run
feat(validate): render CNCF submission from the conformance run
fix(validate): record every NVIDIA#3602 contract change and align exi…
feat(validate): render CNCF submission from the conformance run
fix(validate): record every NVIDIA#3602 contract change and align exi…
feat(validate): render CNCF submission from the conformance run
fix(validate): record every NVIDIA#3602 contract change and align exi…