fix(evals): keep sampled-out items out of the experiment pass rate

[CTWalk]

PR Checklist

Please check if your PR fulfills the following requirements:

• The commit message follows our guidelines: https://voltagent.dev/docs/community/contributing/#commit-convention

Bugs / Features

• Related issue(s) linked
• Tests for the changes have been added
• Docs have been added / updated
• Changesets have been added https://voltagent.dev/docs/community/contributing/#creating-a-changeset

What is the current behavior?

An experiment item whose scorers were all sampled out counts as a pass in
summary.passRate.

evaluateItemStatus() sees a score list with no thresholdPassed === false
and returns passed, so the item lands in successCount
(packages/evals/src/experiment/run-experiment.ts:456). passRate is then
successCount / completedCount (aggregator.ts:144), and the item is in both
the numerator and the denominator — even though nothing about it was evaluated.

The same criterion resolves two different ways today:

function resolvePassRate(summary, scorerId) {
  if (!scorerId) return summary.passRate; // sampled-out items count as passes
  return summary.scorers[scorerId]?.passRate ?? null; // skipped results excluded
}

A scorer's own passRate is passCount / (passCount + failureCount)
(aggregator.ts:179) — skipped results are already left out of the
denominator. The item-level pass rate does not do the same, so
{ type: "passRate", min: 1 } and
{ type: "passRate", min: 1, scorerId: "x" } can disagree on one run.

Repro

const result = await runExperiment(
  createExperiment({
    id: "sampled-out",
    dataset: { items: [{ id: "item-1", input: "hello", expected: "hello" }] },
    runner: async ({ item }) => ({ output: item.input }),
    scorers: [
      {
        id: "sampled",
        threshold: 0.5,
        scorer: {
          id: "sampled",
          name: "sampled",
          sampling: { type: "never" },
          scorer: () => ({ status: "success", score: 1 }),
        },
      },
    ],
    passCriteria: { type: "passRate", min: 1 },
  }),
);

result.summary.passRate; // 1 — nothing was scored
result.summary.criteria[0].passed; // true

Because sampling is a cost control, the realistic shape is partial: with 100
items at 10 % sampling, ~90 unevaluated items are counted as passes, so the
pass rate sits near 1 regardless of how the evaluated 10 did. The practical
effect is that a passRate criterion gets easier to satisfy the less you
evaluate — including in the setup the evaluation docs suggest for a regression
test (passCriteria: { type: "passRate", min: 1.0 }).

What is the new behavior?

Items whose every scorer was skipped are excluded from the pass rate, on both
sides of the fraction. A run with nothing evaluated reports passRate: null,
the same "no data" value already used for meanScore and for a scorer-scoped
pass rate with no attempts. With that, the two spellings of the criterion agree.

Run	passRate before	after
1 item, its only scorer sampled out	1	null
1 below threshold + 1 sampled out	0.5	0
1 passed + 1 failed + 1 sampled out	0.67	0.5
100 items, 10 % sampling	≈ 1	passes among the ~10 scored
no sampling anywhere	unchanged	unchanged

successCount, failureCount, errorCount, and skippedCount keep their
current meanings, and item status is untouched.

Added two regression tests beside the existing experiment specs: a mixed run
(one item below threshold, one errored, one skipped) that reported 0.33
before and reports 0 now, and a fully sampled-out run that pins
passRate: null plus agreement between the scoped and unscoped criteria.

Notes for reviewers

The change is confined to aggregator.ts, which the package does not export,
so no public type changes and no changeset beyond a @voltagent/evals patch.

One consequence worth flagging: with sampled-out items present,
summary.passRate is no longer successCount / completedCount. Anything
downstream that recomputes a pass rate from those counts — including consumers
of the summary voltops/run.ts uploads — would differ from the reported
passRate by exactly the sampled-out items. mapSummary currently sends
successCount, failureCount, meanScore and passRate; happy to add
skippedCount there too if that is useful on your side.

One neighbouring question this PR deliberately does not decide: a
sampled-out item still has item status passed, while the offline-evaluation
docs list skipped as an item status and define skippedCount as "items with
status skipped". Reconciling those would mean adding "skipped" to
EvalResultStatus / VoltOpsEvalResultStatus, which is forwarded to VoltOps
(packages/evals/src/voltops/run.ts:407) — whether the API accepts that value
is something only you can tell from here. Glad to follow up either way; this
fix holds under both outcomes.

Developed with AI assistance. I ran the repro above and the @voltagent/evals
suite myself against main, and I'm answerable for everything in the diff.

---

Summary by cubic

Fixes inflated experiment pass rates by excluding items whose scorers were all sampled out. Pass criteria now reflect only evaluated items and match scorer-scoped pass rates.

• Bug Fixes

  • Exclude fully sampled-out items from summary.passRate (both numerator and denominator).
  • Report passRate: null when nothing was evaluated.
  • Keep successCount, failureCount, errorCount, and skippedCount unchanged.
  • Added regression tests and updated docs in website/evaluation-docs.

• Migration

  • summary.passRate may no longer equal successCount / completedCount when sampling is used. Update any downstream recomputations accordingly.

^{Written for commit 35a5fdd. Summary will update on new commits.}

Summary by CodeRabbit

• Bug Fixes

  • Pass rates now exclude items whose scorers were entirely sampled out.
  • Runs with no evaluated items now report a null pass rate instead of an inflated result.
  • Other evaluation counts remain unchanged.

• Documentation

  • Clarified pass-rate behavior for offline evaluations.

[changeset-bot]

🦋 Changeset detected

Latest commit: 35a5fdd

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@voltagent/evals	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 567b3e03-0353-4325-a72d-254618dc3dec

📥 Commits

Reviewing files that changed from the base of the PR and between 9aedd49 and 35a5fdd.

📒 Files selected for processing (4)

• .changeset/pass-rate-excludes-sampled-out-items.md
• packages/evals/src/experiment/aggregator.ts
• packages/evals/src/experiment/run-experiment.spec.ts
• website/evaluation-docs/offline-evaluations.md

---

📝 Walkthrough

Walkthrough

The experiment aggregator now calculates pass rates from evaluated items. Items with all scorers sampled out do not affect pass rates. Runs with no evaluated items return passRate: null. Tests, documentation, and a changeset cover the updated behavior.

Changes

Pass-rate correction

Layer / File(s)	Summary
Track evaluated items in aggregation packages/evals/src/experiment/aggregator.ts	ExperimentAggregatorState tracks evaluated items and evaluated successes. All-skipped items are excluded from evaluation counts. Pass rates return null when no items were evaluated.
Validate and document pass-rate behavior packages/evals/src/experiment/run-experiment.spec.ts, website/evaluation-docs/offline-evaluations.md, .changeset/pass-rate-excludes-sampled-out-items.md	Integration tests cover sampled-out items and runs with no evaluated items. Documentation and the changeset describe the updated pass-rate rules.

Estimated code review effort: 3 (Moderate) | ~20 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely describes the main change: excluding sampled-out items from experiment pass rates.
Description check	✅ Passed	The description follows the template and documents the bug, new behavior, tests, documentation, changeset, and reviewer notes; no related issue is linked.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[cubic-dev-ai]

All reported issues were addressed across 4 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>