Skip to content

Add opt-in Jev Auto Mode for hard-to-undo tool calls - #77

Merged
linted merged 1 commit into
worktree-update-deps-2026-11from
auto-mode-middleware
Nov 17, 2026
Merged

linted merged 1 commit into
worktree-update-deps-2026-11from
auto-mode-middleware

Conversation

@linted

@linted linted commented Nov 17, 2026

Copy link
Copy Markdown
Owner

langchain 1.5 added AutoModeMiddleware: a decision model scores each guarded tool call for risk and, at P(risky) >= 0.5, returns an error result instead of running it. AUTO_MODE=1 (with TYPESAFE_API_KEY) wires a Jev-backed subclass into the coordinator and every sub-agent, right after argument validation so a rejected call costs no Jev request.

The subclass changes three things about the stock class:

  • State: the stock class sends the last 30 messages verbatim, which on a real session overruns Jev's 32k-token budget and would 400 nearly every check. Ours sends the shape context pruning already measured: task and latest request (head and tail), recent reasoning, the clipped call.
  • Instructions: the stock wording lets only explicit user messages authorize a call; ours treats a sub-agent's delegated task as authorization and asks about irreversible changes the task does not call for.
  • Errors: a classifier failure blocks that call with an explanatory result (fail closed) instead of propagating and failing the run.

Only hard-to-undo tools are guarded by default: patch_bytes, assemble_code, scripts, project_files, open/close_program, import_file, export_program, analysis_options/control, and sandbox execute/delete. AUTO_MODE_TOOLS replaces the set.

On a seven-case live probe, ours classified all seven correctly; the stock instructions put the requested patch at 0.44 and let an unrequested analysis-settings change through at 0.33 (ours: 0.52, close to the line).

Extract make_jev_classifier from JevJudge so both share construction.

langchain 1.5 added AutoModeMiddleware: a decision model scores each
guarded tool call for risk and, at P(risky) >= 0.5, returns an error
result instead of running it. AUTO_MODE=1 (with TYPESAFE_API_KEY) wires a
Jev-backed subclass into the coordinator and every sub-agent, right after
argument validation so a rejected call costs no Jev request.

The subclass changes three things about the stock class:

- State: the stock class sends the last 30 messages verbatim, which on a
  real session overruns Jev's 32k-token budget and would 400 nearly every
  check. Ours sends the shape context pruning already measured: task and
  latest request (head and tail), recent reasoning, the clipped call.
- Instructions: the stock wording lets only explicit user messages
  authorize a call; ours treats a sub-agent's delegated task as
  authorization and asks about irreversible changes the task does not
  call for.
- Errors: a classifier failure blocks that call with an explanatory
  result (fail closed) instead of propagating and failing the run.

Only hard-to-undo tools are guarded by default: patch_bytes,
assemble_code, scripts, project_files, open/close_program,
import_file, export_program, analysis_options/control, and sandbox
execute/delete. AUTO_MODE_TOOLS replaces the set.

On a seven-case live probe, ours classified all seven correctly; the
stock instructions put the requested patch at 0.44 and let an unrequested
analysis-settings change through at 0.33 (ours: 0.52, close to the line).

Extract make_jev_classifier from JevJudge so both share construction.
@linted
linted added this pull request to stack #78 November 17, 2026 23:06
@linted
linted merged commit 06aa2a9 into main Nov 17, 2026
2 checks passed
@linted
linted deleted the auto-mode-middleware branch November 18, 2026 01:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant