verifying-claims is a portable Agent Skill that gives an engineering agent a way to check its work: exercise the real artifact, compare the result with an explicit expectation, capture evidence, and report only what that evidence establishes.
Engineering agents often make a correct observation and then overstate what it proves:
| Observation | Unsupported conclusion | Permitted conclusion |
|---|---|---|
| 18 focused tests passed | “The iOS and Android feature fully works.” | “The tested module passes; native integration and runtime remain unverified.” |
| A banner returned after login | “The CDN cache was the root cause.” | “The recovery correlates with login; the root cause remains unconfirmed.” |
| One reload showed no SDK request | “The website does not use the SDK.” | “No request appeared in the inspected load, profile, and time.” |
The skill connects each claim to an observable check, runs it when authorized, preserves the evidence after cleanup, and returns the strongest conclusion that the result permits. It also audits supplied evidence when the request is limited to that material.
- reuses project verification instructions, feature maps, and existing harnesses;
- opens a UI, performs the changed interaction, and captures and inspects screenshots;
- runs a script or CLI, records exit status and stdout/stderr, and inspects output files;
- calls a service or library's public interface and compares responses and state changes with expectations;
- separates direct evidence from interpretation;
- distinguishes focused tests, type-checks, builds, runtime checks, deployments, and production observations;
- distinguishes implemented, tested, committed, merged, released, and deployed states;
- preserves conflicts and missing evidence instead of smoothing them into confidence;
- identifies the one missing check that could materially change a verdict.
Surface-specific instructions live in verification recipes. A screenshot of a settings page establishes rendering; confirming Save persists settings requires an actual save and reload. Likewise, a script saying “Dry run: no files changed” needs a before/after state check.
When execution is blocked by tools, credentials, or permissions, the result stays unverified and names the blocker. A failed check remains a failure. Repairing it belongs to tasks that authorize implementation; auditing a claim does not imply permission to fix unrelated code.
The canonical skill is skills/verifying-claims/SKILL.md.
Install with the Skills CLI and select your agent interactively:
npx skills@latest add ruofanwei/verifying-claims-skill --skill verifying-claimsOr target Codex and Claude Code:
npx skills@latest add ruofanwei/verifying-claims-skill --skill verifying-claims -a codex -a claude-codeFrom a checkout of this repository, copy to the relevant path in your target project:
mkdir -p /path/to/project/.agents/skills /path/to/project/.claude/skills
cp -R skills/verifying-claims /path/to/project/.agents/skills/
cp -R skills/verifying-claims /path/to/project/.claude/skills/Codex reads the copy under .agents/skills; Claude Code reads the copy under .claude/skills. Install only the path for the agent you use. For personal installation across repositories, use $HOME/.agents/skills for Codex or $HOME/.claude/skills for Claude Code.
Invoke it explicitly when the conclusion matters. Preliminary smoke tests found that installation alone did not reliably preserve every evidence boundary:
Use $verifying-claims to verify the settings page change. Open the page,
save a new value, reload, and capture screenshots of the observed result.
Use $verifying-claims to run the report script against sample input,
inspect the generated report, and include the command and actual output.
Use $verifying-claims to decide whether these tests prove the feature is complete.
Claude Code also exposes an installed skill as /verifying-claims. Both agents can select it implicitly when a request matches the skill description, but implicit selection is not guaranteed. Use explicit invocation for incident closure, root-cause, release, deployment, and completion claims.
Two different claims require different evidence:
- Format compatibility: validate
SKILL.mdmetadata and installation layout. - Behavioral effectiveness: run the same cases with the skill absent, implicitly available, and explicitly invoked.
The repository includes an executable cross-agent harness. Requires Node.js 20 or later, with no npm dependencies for local runner tests:
node --test evals/run.test.mjs
node evals/run.mjs --dry-run --agent all --mode allThe first command uses stand-in CLIs to test the runner, including real fixture outputs, cleanup, failures, and timeouts. The dry-run prints the plan without calling agents or writing a report. Neither establishes real-agent effectiveness.
With an authenticated Codex or Claude Code CLI, run the disposable execution cases:
node evals/run.mjs --agent codex --mode explicit --suite execution
node evals/run.mjs --agent claude --mode explicit --suite executionThese calls use the configured provider and can consume account usage. The fixtures themselves require no network or external services. Each run has a fresh workspace. Execution cases permit file inspection and standalone Node commands. Claude restricts shell execution to node commands; Codex uses its workspace-write sandbox. These are different permission models: the local-only prompt is not an operating-system sandbox for Claude.
The runner prints the JSON report path. Review its tool transcripts, responses, embedded artifacts, and expected invariants. Artifact contents are captured before workspace cleanup, so proof survives the run. Word matching and an agent's self-report are insufficient to judge reasoning or execution.
Report results with the agent version, skill hash, mode, case count, repeat count, and date. A passing run supports only those versions, prompts, and conditions; it does not prove universal behavior. Follow the evaluation protocol for comparisons.
executionStatus: "completed" means the agent CLI exited successfully, not that the behavioral check passed. A missing tool invocation, denied permission, or absent artifact can still leave a check unverified. Authentication errors, sandbox failures, and timeouts describe the runner environment; keep them separate from behavioral failures. Failed or timed-out CLI runs make the runner exit nonzero while retaining the report.
Dated behavioral reviews are recorded under evals/results/. The 2026-10-19 smoke review records real Codex and Claude script execution, captured output, and observed blockers. Older reports describe earlier revisions. Keep preliminary runs labelled as such; do not replace the evaluation protocol with a favorable example.
The two suites serve different checks:
audit: reason from supplied evidence, with skill loading allowed.execution: run a report script and detect a deliberately misleading dry-run that writes a file.
--suite all is the default. Use --case <id> to select individual cases.
The three modes answer different questions:
absent: how the base agent performs without this skill;implicit: whether installation plus description-based triggering is enough;explicit: whether the skill works when deliberately invoked.
The harness refuses an absent run when it finds verifying-claims in the agents' known personal skill paths, because that would contaminate the control. --allow-contaminated-control overrides that guard and is recorded in the plan; it does not produce a clean control. A plugin, bundled skill, or undocumented discovery path can still affect a control; record the active agent configuration with published results.
The repository's own verification skill and feature map provide maintained commands and observable checks.
skills/verifying-claims/SKILL.md Portable skill instructions
skills/verifying-claims/agents/openai.yaml Optional Codex UI metadata
skills/verifying-claims/references/ Verification recipes by surface
.agents/skills/verify-verifying-claims/ Repository verification instructions
evals/cases.json Prompts and review invariants
evals/fixtures/ Disposable executable fixtures
evals/run.mjs Codex and Claude Code eval runner
evals/run.test.mjs Deterministic runner tests
evals/results/ Dated, scoped evaluation reports
Inspired by pstack's create-verification-skill: derive checks from the real project, drive user-facing entry points, preserve observable proof, and execute the verification instructions before treating them as verified. This skill applies that loop across projects and retains its evidence-bound claim audit.
MIT. See LICENSE.