Model-comparison report publishes vacuous rubric, coverage and confident-wrong columns when cases lack ground_truth

[Smana]

Summary

lore eval --compare renders a report whose quality columns are structurally empty or hardcoded whenever the case corpus lacks ground_truth — which is the case for every shipped replay case. The report gives no indication this happened, and benchmarking.md instructs users to commit and publish it.

Observed

A real 4-model comparison run today (glm-4.5-air, glm-4.5, gemini-2.5-flash, gemini-2.5-flash-lite; -cases examples/eval -n 3; judge glm-4.5-air) produced:

model	pass rate	root_cause	evidence	solution	description	calibration	coverage	confident-wrong
glm-4.5-air	100%	—	—	—	—	—	100%	0
glm-4.5	100%	—	—	—	—	—	100%	0
gemini-2.5-flash	0%	—	—	—	—	—	100%	0
gemini-2.5-flash-lite	100%	—	—	—	—	—	100%	0

graded_runs: 0 for every entry in the JSON. The header still reads "blind grading (the judge never sees which model produced a result)".

7 of the 14 columns carry no information, and two of them read as strong results:

• the five rubric dimensions are — because the judge never ran
• coverage is hardcoded to 1.0
• confident-wrong is judge-derived, so it is 0 by construction

Cause

1. Grading is gated on ground truth — internal/eval/compare_run.go:117: if cr.Judge != nil && c.GroundTruth != nil. Neither shipped replay case (examples/eval/harbor-chart-bump.yaml, examples/eval/poisoned-recall-verify.yaml) carries a ground_truth: block; they only have expected:. So the judge is constructed, never invoked, and no warning is emitted.

2. Coverage defaults to perfect — internal/eval/coverage.go:

   if len(expected) == 0 {
       cov.Ratio = 1.0
   }

   With no ground truth, expected is empty, so every model scores 100% coverage regardless of which data sources it actually touched.

Why this matters

benchmarking.md tells users to publish this artifact:

Versioned report. Commit the generated eval/reports/<stamp>-compare.md and .json so a published claim points at a reproducible artifact.

Following that instruction publishes a table asserting 100% data-source coverage and zero confidently-wrong answers for every model compared — both artifacts of missing ground truth rather than measurements. For a project whose positioning is honest measurement, that is the wrong failure mode: it silently overstates.

The pass-rate and token/cost columns remain valid; only the judge- and ground-truth-derived ones are affected.

Suggested fix

1. Say so. When graded_runs == 0, state it plainly in the report ("rubric grading skipped: no case carries ground_truth") instead of printing dashes under a "blind grading" header. Same for coverage — render n/a rather than 100% when expected is empty.
2. Warn at run time when a judge is configured but no case can be graded — currently silent.
3. Add ground_truth to the shipped replay cases so the rubric and coverage columns become real. This is the substantive fix; 1 and 2 are the honesty guard that should exist regardless.

Related: #391, #392 (provider compatibility). A separate --compare defect — the documented command could not run at all without a runlore.yaml — is already fixed on feat/proof-assets.

Found 2026-08-02 while benchmarking models for the nightly eval.