Add conformance/negative/: vectors, failure codes, and runner (closes the suite gap for the enforcement layer)

[avp9-nexus]

Implements the proposal of #29 and begins closing the conformance-suite gap of #19.

What is in here:

• 18 framework-agnostic vectors in JSON across the six categories of Proposal: negative conformance vectors for the enforcement layer (valid-but-unauthorized, replayed, and malformed agent actions) #29: 12 must-reject inputs and 6 positive controls, one per category
• a proposed failure code enumeration. Its backbone is one distinction: REJECT (evidence measured and found wrong) is a different verdict from UNMEASURABLE (evidence could not be measured; the action must still be refused, under a distinct code). Collapsing the two either lets unmeasurable inputs fail open, or sends operators hunting for frauds that never happened
• runner.py, which fails the suite on any verdict mismatch, code mismatch, missing reason substring, or any category lacking its positive control. That last rule is structural: a suite made only of must-reject inputs cannot distinguish an enforcement layer that works from one that rejects everything
• reference_adapter.py, a minimal passing implementation, so the suite is executable end to end as shipped: python conformance/negative/runner.py --adapter conformance/negative/reference_adapter.py

Category 3 (expired or revoked mandates) is covered for expiry only. Revocation vectors are a declared gap: the contributing implementation has no revocation mechanism, by documented decision, and vectors for a path never exercised in production would be design fiction. Contributions from implementations that exercise revocation are the way to close it.

Every failure code in the enumeration was produced by a real refusal in a production enforcement layer before it was named here. Vector wording is deliberately implementation-neutral; naming, directory layout and the enumeration itself are all open to review.

---

Type of change

• Specification change (schema, hooks, events, AgBOM)
• Documentation
• Tooling or CI
• Governance (licensing, security policy, contributor docs)

(Nothing under specification/ or docs/spec/ is touched; the new directory sits at conformance/negative/.)

Checklist

• Commits are signed off with git commit -s (required by the DCO)
• Prose follows STYLE.md
• uv run mkdocs build --strict passes (not run locally; conformance/ sits outside the docs tree, happy to add the check if maintainers want it wired into CI)
• No secrets, tokens, or internal URLs in the diff

Security

• This change has no security impact

(It adds test vectors and a runner; it changes no normative text and no schema.)

[astrogilda]

I cloned this at 18314b7 and ran it before writing anything, so everything below comes from the files themselves. The eighteen vectors live as a single array inside vectors/negative_vectors.json, which is the right call, and it is worth a line in the README, because the PR title reads as eighteen separate files. Separately, runner.py's load_adapter execs the adapter by path without putting the adapter's own directory on sys.path, so an adapter that spans more than one file cannot import its siblings. One line fixes the import.

The two-verdict rule is the most valuable thing in the PR, and it belongs in the normative text. Today it sits in the README as a design rule. I reached the same split independently while building a verification core for an in-toto attestation predicate, and it is load-bearing there for the reason your README gives. In that core, a record evaluation carries two distinct failure fields: one code for a record of the right class whose constraints were measured and not met, and a separate code for a record whose class the verifying implementation cannot read at all. The comment I wrote beside those two code definitions says what yours says, from the other end. Reporting both as unknown "tells the first producer to upgrade its verifier, which will not help." Both refuse. Neither is allowed to be a pass. What differs is what an operator does next, and that is the whole argument for spending a second verdict on it. ACS will have implementers, so put the two-verdict requirement in the specification text with the codes as a registry, and conformance stops resting on a suite an implementation can decline to run.

Now the part I think needs to change before the PR merges. Your structural rule is right and your stated reason for it is right: a suite made only of must-reject inputs cannot tell an enforcement layer that works from one that rejects everything. As implemented, though, the rule certifies the adapter, and the adapter is not the enforcement layer. I wrote a four-line adapter that reads the answer key and enforces nothing at all. It hands back expected["verdict"] and expected["code"] untouched and joins reason_must_mention into reason, and runner.py reports 18/18 vectors conform (6 positive controls across 6 categories) and exits 0, byte for byte what your own reference_adapter.py prints. The positive controls miss it, because a tautological adapter passes the must-pass inputs as easily as it passes the must-reject ones. Every discriminating property you built is real, and all of it is a property of the adapter a conforming implementation writes for the suite, and nothing in the suite requires that adapter to route through the code path that runs in production.

runner.py:51 is the second defect, and it is a single line. It reads cats[v["category"]]["pos" if v.get("positive_control") else "neg"] += 1, which counts a declared boolean and does not check that the vector so declared actually expects PASS. I deleted pos-cat1-001 and set positive_control: true on neg-cat1-001, a must-reject vector, and the runner printed 17/17 vectors conform (6 positive controls across 6 categories) and exited 0 with category 1 holding no must-pass input at all. The structural gate that exists to stop a suite of pure rejections is satisfied by a pure rejection. The fix is two lines: count a control only when v["expected"]["verdict"] == "PASS", and while you are in there, runner.py:69 skips the code and reason checks whenever the expected verdict is PASS, so a positive control's code field goes uncompared.

A positive control that runs only inside a test harness establishes that the harness discriminates. The fix I would actually argue for is larger, and it is the notch the suite is missing. What an operator needs established is that the check was alive in the run that produced the evidence they are holding, in the window that evidence covers. The construction I use puts the stimulus into the corpus of records: an author commits in advance to what a given attack looks like on the wire between two components, the enforcement substrate signs what it actually committed to during the run, the row asserts the two are comparable, and a run-end seal signed by a party that does not control the carried record set names what was attributed. A channel counts as demonstrated only when all of those line up, the verdict is computed per channel and is not rolled into one boolean for the run, and a channel where nothing was planted reports as unprobed, which is neither a pass nor a failure. The asymmetry that makes it worth the trouble: a detector that quietly died and a boundary nobody tested produce the same report otherwise, and no amount of reading the report harder separates them. If it is useful, the write-up is at docs/DETECTOR-LIVENESS.md in the aee-conformance corpus, along with the part I would press on hardest, which is that the liveness probe is itself adversarially tested. A probe hard-wired to answer "demonstrated" would clear every vector in the corpus and report a live detector for a dead one, which is the whole problem one level up, and the same shape as the answer-key adapter above.

The cheap version for ACS needs no new construction at all. It does not get you in-window liveness, and it does get you the property your rule reaches for, which is that the thing being certified is the enforcement layer and not the test double. Add a required field to the vector schema naming which production entry point the adapter dispatched through for that vector, and make the runner refuse a suite whose positive controls resolved to a different entry point than the negative vectors did. At that point the answer-key adapter stops passing.

You state the revocation gap plainly and ask for contributions from implementations that exercise revocation, so here are seven vectors in your exact schema. I checked what happens without them first: your reference adapter returns PASS on all five of the refusals below, which makes the gap a live fail-open, and that is the strongest argument for merging them. The implementation I am drawing from carries three mechanisms that bear on it. A run-end seal whose stillArmed boolean must be true for the seal to cover anything, so a withdrawal mid-run leaves every clean row uncovered. A mandate chain in which a re-issued record at sequence greater than one must name its predecessor by digest and fails closed when it does not. And a normative rule that a key's validity window is evaluated against an authority-signed instant and never against a producer-asserted timestamp, because a window evaluated against the latter rehabilitates, by back-dating alone, every record the revoked key ever signed. Each of the seven vectors comes from one of those three.

Two failure codes are added and neither collapses into one you already have. MANDATE_REVOKED is a REJECT: the withdrawal was read and found. REVOCATION_UNCHECKABLE is UNMEASURABLE: the withdrawal state could not be read, and the action being judged must still be refused. Reusing STALE_DECISION for either is the exact collapse your rule 1 forbids, since a mandate revoked three minutes into a fifteen-minute freshness window is not old, and telling an operator it is stale sends them to look at clocks.

The five refusals are: a mandate withdrawn well inside its freshness bound, which a freshness check alone passes; a revocation registry that cannot be reached; a registry that is reachable but has not been re-signed within its own declared publication interval, so it has stopped being able to answer; a revocation whose window would be evaluated against a party-controlled timestamp; and a superseding mandate that names no predecessor, which is unmeasurable and not merely unsatisfied, because a legitimate replacement and a re-labelled revoked original cannot be told apart without the predecessor link. The two positive controls make the five discriminating: a reachable registry that does not list the mandate must pass, and a re-issued mandate that correctly names the revoked predecessor must pass, or an implementation that refuses on sight of a revocation registry scores full marks and an operator has no way to recover from any revocation it enforces.

They validate against vector.schema.json unmodified except for the closed code enum, which rejects the two new codes in exactly five places and nowhere else; every other field, the id pattern, the category bound, and the input shape are unchanged. So the schema delta is two enum entries. Merged into your file the suite is 25 vectors, ids do not collide, and cat3 carries three positive controls; with a combined adapter it runs 25/25 with 8 positive controls across 6 categories, and with your reference adapter alone it runs 20/25, with all five failures being a PASS where a refusal was required. I have included a small reference adapter for the revocation path alongside the vectors, on a principle your own adapter states: a runner nobody can execute end to end is a promise.

One integration question I do not want to answer unilaterally. Merging these turns your reference adapter red, because it correctly has no revocation mechanism, and a red reference adapter in CI is worse than an unfilled gap. Your README handles declared gaps in prose, and the runner has no way to express "this vector belongs to a capability an implementation may legitimately not have." I would add a requires array to the vector schema and a --profile flag to the runner, so an implementation declares its capabilities, the runner refuses only on vectors inside them, and a declared capability with no positive control still fails loudly. That turns declared gaps from prose into something machine-readable, and it generalizes past revocation to every capability the next contributor brings. Happy to send that as a separate PR if you want it split, or to fold it in here. Tell me which and I will match whatever you prefer.

Sankalp Gilda

[avp9-nexus]

Reproduced all three before replying, same order as your review. Your answer-key adapter: 18/18 vectors conform (6 positive controls across 6 categories), exit 0, byte for byte the reference adapter's output. Your mutation at runner.py:51: deleted pos-cat1-001, flagged neg-cat1-001, got 17/17 vectors conform (6 positive controls across 6 categories), exit 0, with category 1 holding no must-pass input. And runner.py:69 skips code and reason whenever the expected verdict is PASS, confirmed at the line. All three stand.

The gate failed a rule we publish ourselves, which is the useful part of this. We mutation-test our controls on the principle that a bench green on first try proves nothing, and the gate that guards this suite is the one control here that was never mutated. Your answer-key adapter is the same lesson one level up, and it is the same lesson a reviewer taught us on a neighbouring thread: a suite written from the same assumption as the control cannot fail on that assumption. Finding both in one pass, with reproductions, is the most useful review this PR could have received.

Fixes I am preparing on this branch: the sys.path line in load_adapter, counting a positive control only when expected.verdict == "PASS", comparing code and reason on PASS vectors too, and your cheap version of the entry-point check - I would have the adapter report which entry point it dispatched through and the runner refuse a suite whose positive controls resolved differently from its negative ones, which keeps the schema untouched; say so if you meant the field to live in the vector schema instead. On the larger liveness construction, the in-window property is beyond what this PR claims, and I want to read DETECTOR-LIVENESS.md properly before saying anything about it that is worth reading.

On your integration question: separate PR, please, under your name. Three reasons. The seven vectors and the requires/--profile mechanism are your work and should carry your authorship. Repairing what this PR already claims and extending it to a capability it declared out of scope are different review objects, and keeping them separate keeps both reviewable. And one practical point: the vectors and the revocation adapter are not attached anywhere I can fetch - no review comments carry them, there is no fork, and I checked aee-conformance - so the separate PR is also simply the place where they get to exist. The requires/--profile design answers the right problem, because you are correct that a red reference adapter in CI is worse than an unfilled gap, and a declared capability with no positive control failing loudly is exactly the rule this suite already applies one level down.

On the two-verdict rule moving into normative text: agreed on substance, and that call belongs to the maintainers. Your two codes are the right shape by the rule's own test - a mandate revoked inside its freshness window is not stale, and an unreadable revocation registry is not a found withdrawal, so neither collapses into an existing code. Your five refusals plus two positive controls read like the category 3 contribution the README asked for, and I would rather review them as files than as prose.