[BUG]: Published BENCHMARK.md does not identify the evaluated source revision

[holgerroth]

What happened?

Published BENCHMARK.md files do not unambiguously identify the source repository revision that was evaluated.

In several cards produced by the NVSkills publication pipeline with SkillEvaluator 1.3.2, the Execution and Provenance section contains:

• Git commit: 0117bc2e3e54da4244a656466526c5b1b5a559ea
• a SkillEvaluator container tag ending in sha-0117bc2e3e54da4244a656466526c5b1b5a559ea

The identical value appears for unrelated skills evaluated from different repositories and source commits. It therefore appears to identify the evaluator/container build, not the evaluated skill content. Labeling it simply Git commit makes it look like the evaluated source revision.

The actual evaluated source SHA is known by the requesting and orchestration workflows, but it is not included in the committed benchmark card.

Public examples

Skill	Evaluated source SHA	Published benchmark
nvflare-autofl	NVIDIA/NVFlare@2263a2e (request run, central run)	BENCHMARK.md reports Git commit: 0117bc2...
nemo-fabric-build-adapter	NVIDIA/NeMo-Fabric@134c829 (request run, central run)	BENCHMARK.md reports the same 0117bc2...
nemotron-speech	nvidia-riva/Nemotron-speech-skills@eb2e23e (request run, central run)	BENCHMARK.md reports the same 0117bc2...

In each case, the commit adding the generated benchmark has the evaluated source revision as its parent, but the benchmark does not record that revision.

The newer public benchmark renderer on main no longer emits the ambiguous Git commit field. However, it still does not include an evaluated source repository, commit SHA, or skill-content digest in its evaluation metadata:

• Benchmark metadata renderer
• PASS-card publication requirements
• Current golden benchmark card

The publication gate verifies evaluator version, dataset digest, agents, task count, attempts, and environment, but not the evaluated source revision.

Actual behavior

• Published cards can omit the evaluated source identity.
• Some published cards label the evaluator/container revision as Git commit.
• Reviewers cannot determine from the card which source tree was evaluated or whether the result is stale.

Expected behavior

A published benchmark should distinguish at least:

• Evaluated source repository
• Evaluated source commit SHA, or a canonical digest of the evaluated skill content
• Evaluator version
• Evaluator implementation/container revision or immutable image digest

For example:

Evaluated source: NVIDIA/NVFlare
Evaluated source commit: 2263a2ebdab903e87f7e7c0a001d22c3a926a9cf
Evaluator version: 1.3.2
Evaluator container revision: 0117bc2e3e54da4244a656466526c5b1b5a559ea

The source identity should be carried from the orchestration input through the evaluation-result contract and included in the benchmark before signing or publication. It should not be inferred from repository state during rendering.

Suggested acceptance criteria:

• A generated BENCHMARK.md records an immutable evaluated source identity.
• Evaluated-source provenance and evaluator/container provenance use separate, unambiguous labels.
• Publication fails closed when required source provenance is missing or conflicts with the orchestration input.
• Signed output covers the recorded source identity.
• Regression coverage verifies that two source repositories evaluated with the same evaluator image retain different source SHAs.
• Existing cards using Git commit for the evaluator/container revision are renamed or regenerated.

Without source binding, reviewers cannot determine from a committed benchmark which version of the skill was evaluated or whether the benchmark became stale after source changes. The card can authenticate the evaluation artifact while still leaving the evaluated content ambiguous.

Related but distinct work:

• [BUG]: Pull request reports can contain broken source links #59 / fix(reporting): use commit SHA for pull request links #60 fixed commit-pinned source links in reports. That addresses broken links, not source provenance in BENCHMARK.md.
• feat(reporting): roll out unified benchmark publication card #47 introduced the newer decision-first benchmark card and publication gate, but the current metadata contract still lacks an evaluated source SHA.

Reproduction steps

1. Open any two of the public BENCHMARK.md examples above.
2. Observe that both report Git commit: 0117bc2e3e54da4244a656466526c5b1b5a559ea.
3. Observe that this value exactly matches the SHA embedded in the adjacent SkillEvaluator container tag.
4. Follow the linked request and central runs and confirm that the evaluated source SHAs differ between the repositories.
5. Confirm that none of those evaluated source SHAs appears in the corresponding committed BENCHMARK.md.
6. Inspect the current public renderer and publication gate and observe that neither requires an evaluated source SHA or skill-content digest.

SkillEvaluator version or commit

• Affected published cards report SkillEvaluator 1.3.2.
• Public upstream inspected at e3910c29e17424c7aab445e06ac8c04ff91d5350.

Environment

• Published GitHub skill repositories evaluated through the NVSkills CI workflow.
• The exact deployed publication integration is not present in the public repository, so part of the fix may belong in the downstream publication pipeline.
• The reproduction above uses only public GitHub Actions runs, commits, and committed benchmark artifacts.

Before submitting

• I agree to follow this project's Code of Conduct
• I have searched the open issues and found no duplicate of this report