omind audit measures every omind surface that spends the agent's context or
attention, judges each against a declared threshold, and exits non-zero when
any of them fails. It exists because of #321:
about 3.4M tokens of unrequested recall at ~25% precision sat invisible for months
inside a usage ledger that had been written faithfully since 4.0.0 and never read.
From the outside, a memory tool quietly poisoning its own context window looks
exactly like a model that got worse.
A dashboard that can only look green is decoration. This one is allowed to say "the preflight costs more than it returns — turn it off."
omind audit # verdict table; exit 1 if any surface fails
omind audit --days 7 # narrower look-back (default 30)
omind audit --json # for cron / CIEach row is one number, the limit it is judged against, and a verdict:
| verdict | meaning |
|---|---|
ok |
measured, enough samples, inside the limit |
FAIL |
measured, enough samples, outside the limit — the row names a remedy |
?? (unmeasured) |
not judged: too few samples, an instrument that errored, or a question the recorded telemetry cannot answer. Never rounded up to ok. |
Only FAIL affects the exit code. unmeasured rows are printed with what is
missing, because a gap nobody can see is how #321 happened.
These live in one version-controlled place — THRESHOLDS in
src/omind/audit.py — and this table mirrors it
(tests/test_audit.py fails if the two drift). They are anchored to omind's own
declared budgets and to targets set in the issues that found each problem, not
to whatever the numbers were on the day the audit was written. Inferring them from
current data would grade on a curve: yesterday's regression becomes today's
baseline. Change one only in a commit that says why.
| key | what is measured | limit | min samples | where the limit comes from |
|---|---|---|---|---|
preflight.median_chars |
preflight recall: per-turn push, median | <= 1,200 chars | 20 | hint mode is capped at guard.PREFLIGHT_HINT_CHARS (500); 1,200 leaves room for hard-rule notes, which keep their full excerpt |
preflight.p99_chars |
preflight recall: per-turn push, p99 | <= 6,000 chars | 20 | #321 was a tail problem (p99 6,809 / max 16,406 chars), invisible in totals |
preflight.precision_pct |
preflight recall: injection precision | >= 70 % | 5 | the 70% target #321 set; measured by omind bench --precision |
session.p99_tokens |
preflight recall: omind push per session, p99 | <= 15,000 tokens | 10 | guard.SESSION_INJECTION_BUDGET_CHARS is 60,000 chars = 15,000 tokens; since #387 it limits push only (priming, preflight, name hints) |
namehint.p99_chars |
tool-output name hints: per-call push, p99 | <= 1,200 chars | 20 | #388: at most 3 one-line, titles-only hints per tool call, titles cut at 140 chars |
namehint.session_p99_chars |
tool-output name hints: per-session push, p99 | <= 8,000 chars | 10 | namehints.SESSION_BUDGET_CHARS (8,000 chars, a thirtieth of the push budget) |
priming.median_tokens |
SessionStart priming: capsule size, median | <= 2,000 tokens | 10 | the default balanced capsule is budgeted at 8,000 chars = 2,000 tokens |
priming.p99_tokens |
SessionStart priming: capsule size, p99 | <= 4,000 tokens | 10 | twice the balanced budget; the full profile is an explicit opt-in |
mcp.median_chars |
MCP tool responses: response size, median | <= 6,000 chars | 20 | recall-note defaults to 4,000 chars; a median above 6,000 means the expensive tools are the habitual ones |
mcp.p99_chars |
MCP tool responses: response size, p99 | <= 20,000 chars | 20 | read-note's default body cap is 20,000 chars (invariant 8) |
session.pull_p99_tokens |
MCP tool responses: agent's own reads per session, p99 | <= 30,000 tokens | 10 | #387: twice the push budget; pull is deliberate, so only habitual whole-note reading should trip it |
gate.offtopic_per_100 |
consult gate: off-topic verdicts per 100 OMI reads | <= 25 per 100 | 20 | #326: 'if most forced consults are scored off-topic, the gate is spending attention for nothing'; 1 in 4 is already generous |
gate.ceremony_pct |
consult gate: denies that were pure ceremony | <= 80 % | 20 | a guard whose denies are >80% satisfy-and-retry is mostly friction |
gate.auto_clear_pct |
consult gate: turns cleared with nothing injected | <= 90 % | 20 | #296 measured 362 auto-cleared turns carrying 2,037 tool calls |
rules.never_fired_pct |
learned + note rules: rules that have never fired | <= 50 % | 3 | #326: 'how many exist vs. how many have ever fired' |
verifier.p99_chars |
verifier: prompt size, p99 | <= 6,000 chars | 20 | the verifier judges one consult against one task; 6,000 chars is ample |
schema.tokens |
MCP tool schema: fixed per-session cost | <= 6,000 tokens | 1 | measured by omind bench; #177/#181 exist because this only grows |
Two questions from #326 are
reported as unmeasured on every run, deliberately:
- Which priming capsules are consulted later in the session. Priming events carry no session id and no note names, so a capsule nothing ever reads back is indistinguishable from a useful one.
- Which MCP payloads the agent acts on versus discards. The ledger stores sizes only — no tool name, no payload. That is by design (it is privacy-safe), and it means per-tool waste cannot be attributed.
Answering either needs new telemetry. They are listed in the verdict table so the gap is part of the report rather than an omission.
Nothing new is collected. The audit reads:
- the AI-usage ledger (
omind ai usagereads the same file) — preflight, priming, tool-output name-hint, MCP-response and verifier sizes, and per-session accumulation; - the compliance log — off-topic verdicts, denies, and gate continuity;
omind bench --precision— injection precision on the labelled query set;- the live MCP server definition — the fixed per-session schema cost;
- learned rules (
policy.json) and compiled note rules, against every rule id the compliance log has ever recorded.
It is read-only: no access statistics, consults, compliance events or usage
rows are written. Precision goes through bench.run_precision, which reads notes
through OmiStore rather than recall-note for exactly this reason — measuring
retrieval through the normal path would write the act of measurement into the
access statistics that feed ranking, and the numbers would drift toward looking
fine.
One broken instrument never hides the others: a measurement that raises is
reported unmeasured with its error, and the rest are still judged.
The acceptance criterion for #326 was that the first honest run must be able to fail. On the author's vault (30-day window, omind 9.4.0 installed) it reported 6 failing, 6 ok, 4 unmeasured:
| row | value | limit |
|---|---|---|
| per-turn push, median | 2,133 chars | <= 1,200 |
| per-turn push, p99 | 7,330 chars | <= 6,000 |
| omind context per session, p99 | 70,714 tokens | <= 15,000 |
| injection precision | 69.6 % | >= 70 |
| MCP response size, median | 8,481 chars | <= 6,000 |
| denies that were pure ceremony | 94.4 % | <= 80 |
The precision figure matches what omind bench --precision reported when 9.2.0
shipped (69.6%, one notch under the target #321 set) — shipped saying so rather
than tuned until it passed. None of the limits were adjusted after seeing these.
Proudly Made in Nebraska. Go Big Red! 🌽 https://xkcd.com/2347/