docs(design): plan exact MCP client-tool continuation

docs/design/agent-workflows/projects/mcp-client-tool-continuation/README.md

@@ -0,0 +1,45 @@
+# Exact client-tool continuation
+
+Claude currently ends an Agenta client-tool turn by closing the internal MCP request without a
+result. When the browser returns the result, the runner starts or loads a session and Claude asks
+for the tool again. That path is safe, but it cannot preserve the original call id or arguments.
+
+This project studied a faster exact path for local Claude sessions: keep the original MCP
+request and harness prompt open, park the session, and write the browser result to that same
+JSON-RPC request. After a Codex review on 2026-07-11, that warm path is **deferred behind two
+measurement gates**: Claude must demonstrably hold the request open for a useful interval, and
+cold replay must demonstrably harm users. What ships now is the measurement (WP0) and the
+independent hardening of the loopback MCP endpoint (WP1). The design for the warm path stays
+in this workspace as a revised note so the build can start quickly if the gates pass.
+
+## Status
+
+Design only. Implementation has not started. The warm-path deferral is the top review
+question for the owner; see [status.md](status.md) and [open-questions.md](open-questions.md).
+
+The scope is local Claude only. Pi approval parking is already implemented through the ACP
+permission plane. Non-Pi Daytona runs cannot receive Agenta tools through the current internal
+MCP endpoint because the endpoint binds to the runner's loopback interface.
+
+## Documents
+
+- [context.md](context.md) explains the current behavior, goals, scope, and user-visible result.
+- [research.md](research.md) records the code findings and the effect of PRs #5153, #5185, and
+  #5197.
+- [interface.md](interface.md) is the design note for the deferred warm path: the
+  pending-operation contract, the slimmed delivery port, and the pool-owned placement.
+- [plan.md](plan.md) defines what ships now (WP0 expanded, WP1), the unlock gates, and the
+  revised shape of the deferred warm path.
+- [qa.md](qa.md) defines the verification layers; the layers beyond WP0 and WP1 apply only if
+  the warm path unlocks.
+- [open-questions.md](open-questions.md) lists the decisions that still need review, led by
+  the deferral itself.
+- [status.md](status.md) is the live source of truth for progress and dependencies.
+
+## Proposed sequence
+
+1. Measure Claude's MCP request timeout and the cold path's real user cost (WP0).
+2. Authenticate and test the existing internal loopback MCP endpoint (WP1), independently.
+3. Score the measurements against the unlock gates in [plan.md](plan.md).
+4. Only if both gates pass, and only in owner-routed deployments: build the smallest
+   pool-owned hold-open path per the revised design note, after PR #5197 merges.

docs/design/agent-workflows/projects/mcp-client-tool-continuation/context.md

@@ -0,0 +1,120 @@
+# Context
+
+## The two tool paths called MCP
+
+User-configured MCP servers and Agenta client tools are different paths.
+
+- A user MCP server is supplied by the user. Claude calls that server directly.
+- An Agenta tool is resolved by the platform. For local Claude, the runner exposes those tools
+  through a small internal MCP HTTP server in `tool-mcp-http.ts`.
+- A client tool is an Agenta tool whose result comes from the browser. `request_connection` is one
+  example. The runner must pause while the user completes the browser interaction.
+
+This project changes only the third path. It does not change user MCP servers, tool resolution, Pi
+tool delivery, or the public `/run` request shape.
+
+## Current client-tool flow
+
+The current local Claude flow is:
+
+1. Claude sends `tools/call` to the runner's loopback MCP server.
+2. The runner validates the arguments, correlates the call with Claude's ACP tool-call id, emits a
+   `client_tool` interaction, and pauses the turn.
+3. `tool-mcp-http.ts` returns the internal `MCP_PAUSED` sentinel. The HTTP listener calls
+   `res.destroy()` without writing a JSON-RPC result.
+4. The runner tears down the prompt or session unless another parkable gate owns it.
+5. The browser adds a `tool_result` to a new `/run` request.
+6. The cold path indexes that result by tool name and canonical arguments. Claude reissues the
+   client tool, and the runner returns the stored browser output.
+
+The cold path is fail-safe. If the model changes the arguments, the stored result does not match
+and the user sees the interaction again. The cost is that the original MCP request is gone, the
+new call has a new id, and the model has made another decision.
+
+## Desired warm flow
+
+This flow is the deferred goal, not the first deliverable; [plan.md](plan.md) gates it on
+two measurements. The exact path changes steps 3 through 6:
+
+1. The runner leaves the original JSON-RPC response open.
+2. The session pool parks the live harness in `awaiting_client_tool` with the original prompt
+   promise and a transport-neutral pending-operation handle.
+3. The browser result arrives in a new `/run` request and matches the original ACP tool-call id.
+4. The runner atomically claims the pending operation and writes the result to the held JSON-RPC
+   response.
+5. Claude settles the original tool call and continues the original prompt. The new `/run` owns
+   streaming and tracing for the continuation.
+
+There is no new model request between the browser result and continuation of the original prompt.
+
+## How PR 5197 changes the fallback
+
+PR #5197 adds native harness session continuity, durable harness-session ids, reconnectable Daytona
+sandboxes, and `session/load`. Those changes improve the path after a live park is lost. They do
+not answer an MCP request that was already destroyed.
+
+After PR #5197:
+
+- A live pending operation is still the only exact path.
+- If the live operation is gone, `session/load` restores structured harness history when eligible.
+- Claude settles the interrupted pending call and reissues it with a new id. The stored browser
+  output answers that new call through the existing cold decision store.
+- Plain cold replay remains the final fallback when native session load is unavailable or stale.
+
+The pending socket must not be stored in `session_states.data`. A Node `ServerResponse` belongs to
+one process and cannot survive a process restart. PR #5197's durable state is context recovery, not
+a pending-operation ledger.
+
+## Goals
+
+- Continue the original local Claude MCP call after a browser result arrives.
+- Preserve the original harness tool-call id, tool name, arguments, and prompt.
+- Keep cold replay as a correct fallback for every failure and unsupported case.
+- Authenticate the internal MCP endpoint before increasing its lifetime.
+- Bound pending sessions, sockets, output size, and wait time.
+- Make duplicate, late, expired, and disconnected results deterministic.
+- Expose metrics that distinguish exact continuation from cold fallback.
+- Define an interface that an in-memory runner implementation and a future MCP gateway can both
+  implement.
+
+## Non-goals
+
+- Building or selecting a production MCP gateway.
+- Making the runner-loopback MCP endpoint reachable from Daytona.
+- Adding cross-replica forwarding or changing load-balancer routing.
+- Guaranteeing end-to-end exactly-once delivery across a runner crash.
+- Changing the public `/run` wire shape in the first implementation.
+- Replacing the existing cold name-and-arguments matching path.
+- Supporting MCP batches that contain a client tool.
+- Supporting more than one pending client tool per session in the first release.
+- Changing Pi. Pi approval gates already use the ACP permission plane.
+
+## Invariants
+
+1. A browser result can complete only a pending operation in the same project and session and with
+   the same harness tool-call id.
+2. One pending operation has at most one successful claimant and one terminal outcome.
+3. No tool arguments, browser outputs, bearer tokens, or credentials appear in logs or metrics.
+4. A transport failure never discards the inbound browser result before cold fallback can read it.
+5. Expiry, shutdown, client disconnect, pool eviction, and duplicate completion close the held
+   response and reclaim the session.
+6. Unsupported cases use today's cold path. They do not fail the run only because exact
+   continuation is unavailable.
+7. The default remains off until timeout, security, race, and resource tests pass.
+
+## Scope of the first warm release, if it unlocks
+
+A warm release requires all of the following:
+
+- Claude or another non-Pi harness using the internal MCP channel.
+- A local sandbox, because the internal MCP endpoint is runner-loopback only.
+- Session keepalive enabled.
+- The client-tool continuation kill switch enabled.
+- A single JSON-RPC call, not a batch.
+- No other pending client tool in the same session.
+- The resume request reaches the runner replica that owns the live operation.
+
+If any condition is false, the runner closes the original request as it does today and uses cold
+replay. A multi-replica deployment can therefore remain correct without new routing, although its
+exact-continuation hit rate may be lower.
+

docs/design/agent-workflows/projects/mcp-client-tool-continuation/interface.md

@@ -0,0 +1,228 @@
+# Gateway-neutral continuation interface
+
+Status: design note for the deferred warm path (see [plan.md](plan.md)). Nothing here is
+scheduled for implementation until the unlock gates pass. Revised 2026-07-11 after the
+Codex review: the delivery port shrank to delivery plus disposal, transport liveness became
+an optional adapter-owned signal, the standalone registry was replaced by pool-owned
+placement, and `harnessToolCallId` was renamed.
+
+## Design rule
+
+The contract describes a pending client-tool operation. It does not describe Node HTTP
+objects or the runner's current pool implementation.
+
+Each field is grouped by semantic role:
+
+- `identity` identifies the operation and browser interaction.
+- `scope` identifies the tenant boundary that may complete it.
+- `routing` says where the live owner can be reached.
+- `tool` describes the call without storing raw input.
+- `lifecycle` controls expiry and terminal state.
+
+Credentials stay under the MCP connection that they authenticate. They do not appear in
+operation metadata. Adapter-specific protocol context (the JSON-RPC `requestId`, the
+transport kind) lives with the adapter that needs it, not in the neutral operation.
+
+## Operation shape
+
+The following TypeScript is a design contract, not a required final spelling:
+
+```ts
+type PendingClientToolState =
+  | "pending"
+  | "delivering"
+  | "delivered"
+  | "cancelled"
+  | "expired";
+
+interface PendingClientToolOperation {
+  identity: {
+    operationId: string;
+    interactionId: string;
+    toolCallId: string;
+  };
+  scope: {
+    projectId: string;
+    sessionId: string;
+  };
+  routing: {
+    runnerInstanceId: string;
+    environmentId: string;
+  };
+  tool: {
+    name: string;
+    inputDigest: string;
+  };
+  lifecycle: {
+    state: PendingClientToolState;
+    createdAt: number;
+    expiresAt: number;
+  };
+}
+```
+
+**Provenance of `toolCallId`, stated as an invariant:** it is the harness-correlated ACP
+tool-call id, the exact key a browser result carries back. Current correlation is
+best-effort and falls back to the MCP-generated random id when matching fails; that
+fallback id must never enter this field. Warm registration is allowed only when the
+correlation index returned a real harness tool-call id; otherwise the runner destroys the
+MCP response and uses cold replay. The registration gate is what proves the provenance; the
+earlier name `harnessToolCallId` asserted more than the code guaranteed.
+
+`inputDigest` detects internal mismatches without retaining or logging raw arguments. It is
+not an authorization decision and is not the live browser-result key.
+
+## Delivery port
+
+The port carries exactly two capabilities: deliver a result, release adapter-owned
+resources. The HTTP implementation can close over a Node `ServerResponse`, but only the
+adapter can see it:
+
+```ts
+interface ClientToolResultDelivery {
+  deliver(output: unknown): Promise<
+    | { kind: "accepted" }
+    | { kind: "unavailable"; reason: string }
+  >;
+
+  dispose(reason: string): Promise<void>;
+}
+```
+
+- `deliver` attempts to hand the browser output to the waiting client. `accepted` means the
+  transport took the response; `unavailable` means it cannot (closed socket, dead handle)
+  and the caller falls back cold with the preserved result.
+- `dispose` means only "release adapter-owned resources". It makes no claim about what the
+  remote MCP client observed. For a future relay-backed adapter, whether abandoning a call
+  writes a JSON-RPC error or writes nothing is a protocol decision that adapter documents;
+  it is not hidden inside a generic `cancel`.
+
+The earlier port also required `cancel(reason)` and `onClosed(listener)`. Both modeled an
+HTTP response handle and were cut: a relay-backed adapter has no per-request abort and no
+close signal, and a port is not neutral because its awkward methods "may do nothing".
+
+**Transport liveness is optional adapter input, not part of the port.** The HTTP adapter
+may own a closed signal (the socket close event) and use it to trigger early cleanup.
+Correctness must never depend on it: lease expiry and environment teardown are the
+guaranteed exits for every adapter.
+
+## Placement instead of a registry
+
+A standalone `ClientToolContinuationRegistry` is premature. The session pool already owns
+capacity, TTL, atomic checkout, and environment teardown, and the first release would allow
+exactly one pending client tool per environment. If the warm path is built:
+
+- `ParkedClientTool` sits beside `ParkedApproval` in the sandbox-agent session model,
+  holding the neutral operation, the delivery port, and the original prompt promise.
+- `awaiting_client_tool` and its dedicated checkout live in `session-pool.ts`. Checkout is
+  the atomic claim: a losing request never sees the delivery port.
+- The claim behaves as a lease: a checkout that fails to deliver must finalize (deliver,
+  dispose, or re-park) within a bounded time, finalization is idempotent, and expiry
+  applies to a claimed operation the same as to a pending one.
+- The exact current-turn result extraction sits beside the existing responder extractors.
+- `McpHttpResultDelivery` sits under `tools/` as transport code.
+
+A durable pending-operation registry appears only at a future gateway boundary, where a
+second implementation exists to constrain the abstraction.
+
+## State machine
+
+| From | Event | To | Required action |
+| --- | --- | --- | --- |
+| None | Valid single client-tool call with proven correlation | `pending` | Register before emitting the pause; on failure, commit the cold-pause form. |
+| `pending` | Exact browser result arrives | `delivering` | One atomic claimant receives the port. |
+| `delivering` | `deliver` returns `accepted` | `delivered` | Continue the original prompt and resolve the interaction. |
+| `delivering` | `deliver` returns `unavailable`, or the claimant fails to finalize within the lease | `cancelled` | Dispose, evict the live session, fall back cold with the preserved result. |
+| `pending` or `delivering` | Session teardown or shutdown | `cancelled` | Dispose and destroy the environment. |
+| `pending` | TTL expires | `expired` | Dispose and destroy the environment. |
+| `pending` | Adapter-owned closed signal fires (optional) | `cancelled` | Early cleanup only; expiry and teardown remain the guaranteed exits. |
+| Any terminal state | Late or duplicate result | Unchanged | Do not deliver again; use or retain cold fallback. |
+
+Terminal entries are retained for the length of the session turn so a late duplicate reads
+as `duplicate` rather than `missing`; after that, cleanup may drop them.
+
+The local implementation provides at-most-once completion of a live handle. It cannot
+provide end-to-end exactly-once delivery across a process crash. The browser-side action
+has already completed before the result reaches the runner, and the cold path can return
+that stored output without repeating the browser action.
+
+## Exact resume input
+
+The first implementation can derive the live resume input from the existing `/run`
+messages:
+
+```ts
+interface ClientToolResumeInput {
+  projectId: string;
+  sessionId: string;
+  toolCallId: string;
+  output: unknown;
+}
+```
+
+The extractor accepts exactly one current-turn `tool_result` for the parked `toolCallId`.
+Missing or duplicate results do not claim the operation. The existing cold store still
+indexes the same result by name and arguments if live delivery cannot proceed.
+
+## Loopback authentication
+
+This section backs WP1, which ships now regardless of the warm path.
+
+Each internal MCP server receives a random per-environment bearer token. The ACP MCP
+connection places it under the standard `authorization` header:
+
+```ts
+{
+  type: "http",
+  name: "agenta-tools",
+  url,
+  headers: [{ name: "authorization", value: `Bearer ${token}` }],
+}
+```
+
+The server validates the token, from the request headers and with a timing-safe
+comparison, before `initialize`, `tools/list`, or `tools/call`. Agenta keeps the token only
+for the live environment, never places it in operation metadata, and never logs it. The
+harness receives it as connection credentials and may retain its own session
+configuration, so the token has no value after the environment closes its dedicated
+server.
+
+## Future in-sandbox stdio mapping
+
+The in-sandbox tool MCP project
+([../in-sandbox-tool-mcp/](../in-sandbox-tool-mcp/README.md), PR #5234) commits Daytona
+delivery to a harness-spawned stdio shim that writes relay request files. If exact
+continuation ever extends to that path, a relay-backed adapter implements the port as
+follows:
+
+| Port method | Relay-backed implementation |
+| --- | --- |
+| `deliver(output)` | Write a late `<id>.res.json` that the shim's response wait picks up and answers on the original JSON-RPC id. `unavailable` when the relay dir or shim is gone. |
+| `dispose(reason)` | Adapter decision, documented there: write an error response, or write nothing and let the shim-side wait time out. |
+| Closed signal | None. TTL and teardown are the only exits, which the slimmed port already assumes. |
+
+Three prerequisites do not exist yet and belong to a separate bridge design, not to this
+project: the relay response protocol must gain a park shape (today a paused client tool
+writes no response file and the shim's wait times out at 60 seconds), the shim's per-call
+wait must learn to outlive that timeout for parked calls, and the held browser wait must
+survive or refuse the five-minute Daytona auto-stop that kills the shim on
+park-to-stopped. The recommendation to open one owned workspace for that bridge is recorded
+in
+[../mcp-delivery-architecture/orchestration.md](../mcp-delivery-architecture/orchestration.md).
+
+## Future gateway mapping
+
+A real MCP gateway can implement the same roles with different ownership:
+
+| Local implementation | Future gateway implementation |
+| --- | --- |
+| Pool-owned `ParkedClientTool` | Durable pending-operation store with compare-and-set claim. |
+| `runnerInstanceId` lookup | Gateway route to the owning runner or session worker. |
+| Per-environment loopback bearer | Authenticated harness-to-gateway credential. |
+| `McpHttpResultDelivery` | Gateway response stream or result-delivery endpoint. |
+| Pool TTL timer | Gateway lease and retention policy. |
+| Process-local metrics | Fleet-wide audit and quota metrics. |
+
+The gateway will still need authenticated actor context, encrypted result retention,
+revocation, and acknowledgement semantics. Those concerns are not hidden inside `metadata`
+and are not pretended to exist in the local implementation.