Skip to content

fix(judge): classify varied refusal wording - #5

Merged
royalpinto007 merged 1 commit into
AgentPostmortem:mainfrom
vsolano9:fix-4-leak-verdict-boundary
Sep 4, 2026
Merged

fix(judge): classify varied refusal wording#5
royalpinto007 merged 1 commit into
AgentPostmortem:mainfrom
vsolano9:fix-4-leak-verdict-boundary

Conversation

@vsolano9

Copy link
Copy Markdown
Contributor

Closes #4

Owner-truth clarification

Current main already has the issue’s safer option-2 scoring architecture: only secret/canary evidence sets cracked: true; both refused and no-leak are held attempts worth zero points. This PR preserves that boundary and fixes the remaining player-message classification gap.

What changed

  • recognize the issue’s unusual refusal forms plus typographic-apostrophe variants
  • keep phrase matching downstream of all secret/canary checks and document that invariant
  • cover nine refusal forms, an ordinary non-refusal no-leak, and refusal wording that also contains a real secret leak

Validation

  • exact Node 20.20.2 / npm 10.8.2: 7 files, 47 tests passed
  • focused judge suite: 17 tests, repeated 20 times
  • npm run typecheck
  • npm run lint — no warnings or errors
  • npm run build — production Next.js build passed
  • changed files pass Prettier; full-repository Prettier finding set is identical to owner main

No dependency or lockfile changes. The fresh install reports the existing 32-advisory baseline (24 production).

Keep secret and canary evidence as the sole crack boundary while recognizing additional common refusal phrasings for the held-result message. Cover straight and typographic apostrophes, generic no-leak output, and leak precedence.

Closes AgentPostmortem#4
@royalpinto007
royalpinto007 merged commit 25914ee into AgentPostmortem:main Sep 4, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Refusal detection is five English regexes, and a polite refusal in any other wording reads as a crack attempt

2 participants