Skip to content

validate: run CNCF submission evidence from the conformance phase and retire collect-evidence.sh #3602

Closed

Description

@yuanchen8911

Part of #3601. Lands after the parity ports have demonstrated coverage, before #3605.

Summary

Make aicr validate --phase conformance the only runner for CNCF AI conformance. --cncf-submission stops bypassing validation and becomes an output option on that run. Delete what remains of pkg/evidence/cncf/scripts/collect-evidence.sh and its collector.

Today

--cncf-submission short-circuits validation in pkg/cli/validate.go and runs the embedded bash collector through pkg/evidence/cncf/collector.go. The Go checks in validators/conformance/ never run on a submission. docs/conformance/cncf/index.md documents the two as "Two Modes".

Order

The parity-port issues (#3603, #3604, #3428) each delete the script section they replace; sections whose depth exceeds the CNCF bar (GPU gang barrier, CUDA stress, nvidia-smi and DCGM captures) are deleted without a port (#3619). This issue removes the collector, the remaining sections and the flag plumbing once no section is still the only implementation of a requirement.

Change

  • The decision record is docs(adr): amend ADR-007 for single-path CNCF conformance evidence #3611 (ADR-007 amendment) and lands first.
  • --cncf-submission and --evidence-dir require each other (today only the first implies the second); evidence is written only for a submission. --skip-check and --no-cluster are refused with them (already true). The run is the conformance phase, then render; the renderer itself is evidence: generate the CNCF submission directory (PRODUCT.yaml, README, junit.xml, e2e.log, redaction) #3605. A config evidence.cncf.dir without cncfSubmission is refused and documented as the one deprecation.
  • --cncf-submission requires --recipe (or a config recipe input) and is refused with ErrCodeInvalidRequest without one: the single runner selects checks from the recipe, and the collector's capability-detection path goes away with it. Update the examples in docs/conformance/cncf/index.md and docs/user/validation.md that show a submission run without --recipe, and the --cncf-submission flag usage text.
  • Remove --feature and evidence.cncf.features.
  • Client.CollectCNCFEvidence keeps its signature as a wrapper over run-plus-render so make api-diff stays green. Migration semantics to specify and test: a nil RecipeResult is rejected with ErrCodeInvalidRequest (the run needs a recipe to select checks; the old capability-detection path is gone); a non-empty Features is rejected with ErrCodeInvalidRequest naming the removed option; NoCluster true is rejected as today. Document in docs/integrator/go-library.md.
  • Delete pkg/evidence/cncf/scripts/, collector.go, and the script-only tests (claim_shape_test.go).
  • Rewrite docs/conformance/cncf/index.md to one mode, including its "Structural Validation (CI)" example that shows --evidence-dir without --cncf-submission; update docs/user/cli-reference.md, docs/user/validation.md, docs/user/cli-config.md, docs/integrator/go-library.md.
  • Exit status of a submission run derives from the requirement aggregation (evidence: fail-closed requirement status aggregation scoped by level #3612) and supersedes --fail-on-error's per-check rule; see evidence: generate the CNCF submission directory (PRODUCT.yaml, README, junit.xml, e2e.log, redaction) #3605 for the full list of non-zero causes.

Gate: demonstrated parity

Completing the new checks is not sufficient to delete the collector. Before this issue merges, one cluster run with the Go checks and one with the script (last revision) are compared section by section, and each MUST the script evidenced has a Go-produced counterpart at the CNCF bar (#3619): the verdict, the upstream-test behavior it demonstrates, and the measured values the PRODUCT.yaml note cites. Collector captures that exceed the bar (GPU barrier, CUDA stress, nvidia-smi and DCGM dumps) are not part of the comparison. The comparison is attached to this issue.

Acceptance

  • aicr validate -r recipe.yaml --phase conformance --evidence-dir d --cncf-submission runs the Go conformance checks and writes evidence; no bash is executed.
  • For the same selected checks, verdicts are byte-identical with and without --cncf-submission; the submission run may still exit non-zero on incomplete coverage or a failed artifact validation.
  • No file under pkg/evidence/cncf/scripts/ remains.
  • The parity comparison is attached and shows no MUST whose only evidence at the CNCF bar came from the script.

Activity

  1. added theissue type on Nov 3, 2026
  2. yuanchen8911 commented on Nov 4, 2026

    @yuanchen8911
    ContributorAuthor

    Scope update from the CNCF bar audit (#3619, 2026-11-04). Verified against the kubernetes-sigs/ai-conformance tests at fd9ba2d and the upstream validator at c64e5bc; the full table is on #3619.

    Add:

    • The list of sections deleted without a port, so the parity gate does not look for them: collect_gang barrier and blocker, collect_hpa CUDA stress, collect_accelerator_metrics Prometheus queries and port-forward dumps, collect_service_metrics_dynamo, _nim, _trainer, collect_cluster_autoscaling AWS CLI and GKE ConfigMap captures, collect_dra (supporting evidence only).
    • Parity gate criterion: for each MUST, the Go run demonstrates the upstream assertion named in the conformance: align checks and submission output to the CNCF requirement bar #3619 audit table; the script's verdict is not the oracle.
    • Update the --cncf-submission usage text, which still reads "Deploys GPU workloads, captures nvidia-smi output, Prometheus queries, and HPA scaling tests".

    Keep everything else as written.

    Follow-ups from the #3627 review that land here with the flag contract:

    • Gate the per-MUST CNCF MUST requirement not established warnings when no conformance tests ran (today --evidence-dir on a non-conformance phase logs one per MUST).
    • Add a consistency test that the alternative proof paths in pkg/evidence/cncf/aggregate.go cover exactly the checks on each requirement's evidence page, so a check added to a page cannot be silently unrequired.
  3. self-assigned this
    on Nov 5, 2026
  4. added 14 commits that reference this issue on Nov 5, 2026
    b024d14
    cba78f4
    46e2014
    89e74e5
    64e05f3
    863b958
    3445a7c
    2811cfd
    be76492
    d926de4
    6dee352
    11f76c4
    19a8536
    b602612
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions