A reality-checker for AI agents.
Grade an agent's answer against what its tools actually returned.
Documentation · Getting started · JSON output · Code review mode
attest is an open-source Python library and CLI for AI agent evaluation. It checks a tool-using agent's run against the evidence it recorded: every claim in the answer against the tool outputs, every tool call against the allowed tools and its errors, the content the agent read for prompt injection, and the agent's behaviour against its system prompt. Each finding maps to a pass, review or block verdict, and every command prints JSON.
Full documentation: peace-orefejo.github.io/attest, including how to evaluate AI agents and a comparison with other agent evaluation tools such as DeepEval, Ragas, LangSmith and Arize Phoenix.
attest grades a run: a JSON record of one task, with the question, each tool call and what it returned, and the agent's final answer (example).
pip install agent-attest
export ANTHROPIC_API_KEY=sk-ant-...
attest run your-run.jsonEvery command prints JSON. A false claim comes back with the tool output that contradicts it:
{
"severity": "fail",
"verdict": "unsupported",
"subject": "Paris is larger than Berlin",
"reason": "The evidence shows Berlin has a population of 3,677,000 while Paris has 2,103,000, meaning Berlin is larger than Paris, not the other way around.",
"evidence": "Paris population (city proper): 2,103,000",
"step": null,
"metadata": {}
}Most agent evals ask another model whether the answer looks good. A confident answer can pass even when a detail inside it is wrong; rewriting an agent's reasoning alone can raise an LLM judge's false-positive rate by up to 90% (Khalifa et al., 2026).
attest checks each claim against the real tool outputs instead. Its grading model sees only the claim and the evidence, never the agent's reasoning.
attest is an evaluation toolkit, not a guarantee: it trusts that the recorded run is authentic
and complete, and its accuracy hasn't been measured on a labelled benchmark yet. See
Limitations. The benchmark that will
measure it, per check, is being built in benchmarks/.
| Check | Question | API key |
|---|---|---|
| Faithfulness | Is every claim backed by a tool output? | ✓ |
| Tool use | Were the tools allowed, and were their errors handled? | — |
| Prompt injection | Did tool outputs carry hidden instructions, and were they obeyed? | --deep only |
| Role adherence | Did the agent stay inside its system prompt's job? | ✓ |
| Policies (Python only) | Did the run follow your project's own rules? | evidence and rubric rules only |
attest stats turns a pass count into a rate with a 95% confidence interval, so you can tell a
real improvement from noise.
Command line
attest run run.json # faithfulness + tool use, with an overall score
attest tools run.json # tool use only (no key)
attest injection run.json # injection scan only (no key)
attest review review.json # check an AI code reviewer's outputsPython
from attest import Attest
grader = Attest() # or Attest(provider="openai" | "gemini")
report = grader.evaluate(run)
print(report.verdict) # pass, review or blockIn Python a run is a Trajectory. examples/quickstart.py builds
one by hand, and from_langgraph_messages and
from_haystack_messages convert LangGraph and Haystack runs.
Code review
result = Attest().review(pr, outputs, guidelines="REVIEW.md")
post = result.filter(outputs) # the outputs worth showing| Install | Adds |
|---|---|
pip install agent-attest |
Anthropic |
pip install "agent-attest[openai]" |
OpenAI |
pip install "agent-attest[gemini]" |
Gemini |
pip install "agent-attest[files]" |
PDF and image scanning for injection |
pip install "agent-attest[all]" |
Everything |
uv sync
uv run pytestTests run offline with no API key. See CONTRIBUTING.md.
Copyright © 2026 Adepeju Orefejo. Licensed under the GNU AGPL v3.0: you can use, modify and share attest, but any version you distribute or run as a network service must be released under the same license, with its source. Versions 0.6.0 and earlier were MIT.