[adithya-s-k]

Summary

Two fixes for running your own Harbor agent through OpenEnv.

1. --harness module:Class now works. The rollout CLI help and the Harbor docs offer it, but seams.get raised no seam for '<path>' for any import path.
2. Host-side agents are now given the proxy's loopback URL instead of the public forward. This affects terminus-2, computer-1 and your own BaseAgent. Through a gradio share link, a long completion was cut with a 502 while the model was still generating. The agent retried, and the capture recorded the same prompt again as a new root.
3. The "no native step-limit setting" warning is said once per harness, not once per rollout. A custom host-side agent never has a native step limit, so a training run printed the same line for every rollout. The capture proxy still enforces the call budget.

I found both while serving an LLM-judged proof environment (QED-Nano's training problems as Harbor tasks) with a custom host-side agent, through openenv harbor serve on Daytona and HF Inference Providers.

Type of Change

• Bug fix
• New feature
• Breaking change
• Documentation
• New environment
• Refactoring

What was wrong

module:Class. rollout.build_trial_config already passes seam.import_path to Harbor, but seams.get only knows the names in SEAMS:

FAILED (KeyError)  reward=None  turns=0
  "no seam for 'qed_proof_agent.agent:QedProofAgent'. Known: ['acp', 'antigravity-cli', ...]"

Loopback. runner.run_batch and HarborEnvironment._run_rollout pass intercept_url=public_url to every harness. Host-side agents make their model calls from the server process, yet they too went out through the forward and back. The docs already say terminus-2 "reaches the proxy on localhost". The capture proxy calls the upstream without streaming, so a long completion holds the agent's request open for its whole duration, and the gradio share link answered 502 Bad Gateway after about 73 seconds. The same task with a 2-call agent:

expose	result
gradio, retries off	FAILED (ApiInternalServerError): 502 Bad Gateway for url 'https://….gradio.live/v1/chat/completions' after 73 s
gradio, retries on	4 turns, 3 roots: the first prompt captured three times (two cut replies of 18k and 16k chars, then an empty one)
direct (loopback)	2 turns, 1 root, graded

Fix

• seams.custom(import_path) is what seams.get returns for an unknown name containing :. It resolves by what the class is:

  • a subclass of an agent with a seam (cls.name() in SEAMS) is wired like that agent;
  • a host-side agent (a plain BaseAgent) gets api_base (the proxy's /v1 URL) and api_key (the session id) as constructor kwargs, terminus-2's convention. Nothing goes into the sandbox's or the server's environment, so nothing can reach the verifier;
  • any other agent installed in the sandbox is refused with a message, rather than given guessed wiring that would end as a rollout with zero model calls.

  The status is untested in every case.

• Seam.host_side is set for terminus-2, computer-1 and custom host-side agents. capabilities now uses it as well as HOST_SIDE_AGENTS.

• run_rollout(local_intercept_url=...) is used only when the seam is host_side. The rollout CLI passes http://127.0.0.1:<port>. serve passes HarborService.local_url, which is empty when the proxy is mounted on a Space, so a Space keeps its public URL. Callers that don't pass it, such as TRL's runner, behave exactly as before.

• Docs: a "Your own agent" section explains what each kind of class receives.

Alignment Checklist

• I have read .claude/docs/PRINCIPLES.md and this PR aligns with our principles
• I have checked .claude/docs/INVARIANTS.md and no invariants are violated. The agent gets only the capture session id, never a credential, and no simulation control.
• I have run usort check, ruff format --check and ruff check on the changed files, plus the Harbor tests

RFC Status

• Not required (bug fix, docs, minor refactoring)
• RFC exists: #___
• RFC needed (will create before merge)

Test Plan

New unit tests:

• test_harbor_seams.py: a host-side agent gets api_base/api_key and no env; an opencode subclass is wired like opencode; an unknown installed agent is refused; a bad import path says why.
• test_harbor_rollout_contract.py: terminus-2 takes the loopback URL and opencode keeps the public one; host_side is set on the right seams.
• test_harbor_seams.py: the step-limit warning appears once per harness across repeated rollouts.

Harbor test suite: PYTHONPATH=src:envs pytest tests/envs/test_harbor*.py gives 611 passed and 1 failed. The failure, test_start_refuses_a_port_held_by_a_non_capture_listener, also fails on unmodified main on macOS, where binding 0.0.0.0 succeeds while 127.0.0.1 on the same port is held.

End to end:

• Setup: openenv harbor serve on Daytona, with Qwen/Qwen3.5-9B:deepinfra through HF Inference Providers (eval tier; the :provider suffix needs Harbor: make endpoint validation work with hosted providers (probe retry/eval fallback, router provider suffixes) #1565). A host-side custom agent (--harness qed_proof_agent.agent:QedProofAgent) ran 64 tasks, 12 at a time, graded by an LLM judge in Harbor's separate verifier environment.

• Results:

  	graded	mean reward	full marks
  through openenv harbor serve	64/64	0.667	34
  the same agent, model and tasks with plain harbor run	64/64	0.717	40

  The paired difference is −0.049 per task with a standard error of 0.046. 13 tasks were better through OpenEnv, 15 through harbor run, and 36 were identical.

• Every rollout made exactly 2 model calls on both sides: 128 POSTs to the proxy, no retries, no 502s, no empty proofs.

• terminus-2 through serve (8 of the same tasks): 8/8 graded, mean 0.714, one root per rollout, no 502s or retries. Its empty_completion warnings come from the provider: deepinfra currently ignores enable_thinking: false for Qwen3.5-9B (it honours reasoning_effort: "none", which terminus-2 doesn't send), so some turns spend the whole budget reasoning. That is unrelated to this PR.

Claude Code Review

N/A

🤖 Generated with Claude Code

---

Note

Medium Risk
Harbor rollout and serve paths change URL selection and allow dynamic imports of user-specified modules; misconfiguration or overly broad --allow-harness could widen server attack surface, though custom paths are gated by an explicit allowlist.

Overview
Enables custom Harbor agents via --harness module:Class by adding seams.custom() wiring (known subclasses inherit seam config, host-side BaseAgent gets api_base/api_key, sandbox-installed agents are rejected) and openenv harbor serve --allow-harness so servers only import operator-approved import paths.

Host-side agents (including custom ones) now use the capture proxy’s loopback URL when local_intercept_url is set, avoiding long-completion failures through public forwards; Seam.host_side replaces the old HOST_SIDE_AGENTS set, and step-limit warnings log once per harness.

openenv harbor info --harness preflights construction with dummy capture credentials (JSON harness block, exit 1 on failure). Docs and tutorials describe the custom-agent contract and the new flags.

.claude/hooks/lint.sh was rewritten to mirror CI (usort + ruff on src/ and tests/) using a temp git tree so checks never overwrite uncommitted work; tests were extended accordingly.

^{Reviewed by Cursor Bugbot for commit a0068bf. Bugbot is set up for automated code reviews on this repo. Configure here.}

[burtenshaw]

Needs changes before merge: the branch conflicts with main, the Python 3.11 CI job fails on the new tests, and a subclass of seven known agents is wired for an OpenEnv class it does not inherit from. The design itself looks sound, and the loopback change is a clear fix.

Please merge main first. One file conflicts:

• tests/envs/test_harbor_rollout_contract.py

#1567 added test_ambiguous_rewards_fail_training_but_not_a_rollout_to_look_at at the same place as your test_only_a_host_side_agent_takes_the_loopback_url and test_host_side_seams. Keep all three tests. With that resolution, rollout.py and runner.py merge on their own and stay consistent: run_rollout takes both local_intercept_url and require_reward, and the runner passes both.

Blocking

1. CI test (3.11) fails on the four new tests in tests/envs/test_harbor_seams.py with ModuleNotFoundError: No module named 'harbor.agents'. Harbor is installed only on Python 3.12+ (pyproject.toml:75), and in the full CI run pytest.importorskip("harbor") (test_harbor_seams.py:127 and :186) finds a harbor module without harbor.agents, so it does not skip. Skip on the module the tests use, as the other Harbor tests do: pytest.importorskip("harbor.agents.installed.base"). I reproduced the CI failure pattern with a bare harbor package on the path, and with that change the four tests skip there and still pass with Harbor 0.25 installed.
2. seams.custom wires a class like its parent whenever cls.name() is a known seam (src/openenv/harbor/seams.py:988-995). Seven seams (pi, cline-cli, kimi-cli, openclaw, swe-agent, openhands, gemini-cli) do not build Harbor's class. They build an OpenEnv subclass from import_path (InterceptPi, InterceptOpenHands and so on), and that subclass carries part of the wiring or an install fix. A subclass of Harbor's own Pi receives intercept_config, which only InterceptPi reads, and Harbor refuses it at construction (reproduction below). A subclass of Harbor's OpenHands is accepted and gets the env vars, but it runs Harbor's own install instead of the pinned one, which the openhands seam's comment says fails inside the sandbox. When the parent seam has an import_path, require issubclass(cls, <that class>) and refuse otherwise with a message that names the class to subclass. The docs row "a subclass of an agent above ... that agent's wiring" (envs/harbor_env/README.md:580) needs the same caveat.

Smaller, in scope

3. src/openenv/harbor/ui_runs.py:433-439: rollouts started from the serve web page still pass only intercept_url=service.public_url, so terminus-2 there keeps the public hop and the 502 you measured. Pass local_intercept_url=service.local_url there too.
4. src/openenv/harbor/environment.py:313: getattr(service, "local_url", "") can be service.local_url. HarborService has the property now, and the Harbor tests that build fake services pass with the direct attribute. CLAUDE.md asks to avoid getattr for cases like this.
5. src/openenv/harbor/capabilities.py:291-293: list_harnesses iterates only over SEAMS, where host_side is true for terminus-2 and computer-1, and both are already in HOST_SIDE_AGENTS. The added or seam.host_side never changes the output. Use seam.host_side alone and delete HOST_SIDE_AGENTS (nothing else reads it), so there is one source of truth, or leave this file unchanged.
6. src/openenv/harbor/rollout.py:443-447: the seam is resolved here and again at line 472. Move seam = seams.get(...) above config_kwargs and reuse it. For a module:Class harness every call imports and classifies the class again.
7. seams.custom returns a host-side seam for any class that is not a BaseInstalledAgent, including classes that are not agents: seams.get("collections:OrderedDict") succeeds. Check issubclass(cls, BaseAgent) and refuse otherwise, so the error comes from OpenEnv with a clear message.
8. Docs (envs/harbor_env/README.md:575 and the synced page): in serve and from a trainer, the server process imports the class, so the module has to be installed where the server runs (in the image, for a hosted env). Please say so. A trainer that passes a module only it has gets cannot import harness.
9. Tests: test_a_host_side_agent_of_your_own_gets_the_proxy_as_kwargs does not assert seam.host_side, which is what gives a custom agent the loopback URL. A two-line test of HarborService.local_url (empty when mounted, loopback otherwise) would cover the Space case.

Design questions I checked

• --harness module:Class adds no registry or plugin layer. custom imports the class the caller names, classifies it and returns a Seam. Nothing is registered and SEAMS does not change. Main already documented the flag (the rollout --harness help, the HarborEnv.run_rollout docstring and the Harbor docs) and then raised no seam, so this makes documented behavior work.
• Loopback exposes nothing new. The capture proxy already binds 0.0.0.0 on main, and this PR opens no port. Only host-side harness code in the server process receives the 127.0.0.1 URL. The proxy checks the same session key on that URL, and the admin routes check the admin key, never the caller's address. On a Space (mounted), local_url is empty and the public URL is kept. Sandboxed agents, and callers that do not pass local_intercept_url, behave as before.
• The dual API boundary holds. A host-side agent gets the capture proxy's /v1 URL and its session id, the same data plane every harness gets. It never sees the env server's WebSocket or MCP endpoints, and the loopback URL points at the capture port, not the env server's port.

What I verified

• On this branch, PYTHONPATH=src:envs pytest tests/envs/ -k harbor gives 618 passed and 1 failed. The failure is test_start_refuses_a_port_held_by_a_non_capture_listener, which fails on main on macOS too.
• With main merged in and the conflict resolved as above: 648 passed, the same single failure.
• usort, ruff format, ruff check and scripts/sync_env_docs.py --check pass.

A subclass of Harbor's Pi named by import path

import sys, tempfile, types
from pathlib import Path

from harbor.agents.installed.pi import Pi
from openenv.harbor import seams


class MyPi(Pi):  # a user's own pi variant, subclassing Harbor's Pi
    pass


mod = types.ModuleType("my_agents")
mod.MyPi = MyPi
sys.modules["my_agents"] = mod

seam = seams.get("my_agents:MyPi")
model, kwargs, _, _ = seam.resolve(base_url="http://127.0.0.1:8100", session="s", model="m")
print(seam.notes, kwargs)
MyPi(logs_dir=Path(tempfile.mkdtemp()), model_name=model, **kwargs)

my_agents:MyPi is a pi subclass, wired like pi. {'intercept_config': {'base_url': 'http://127.0.0.1:8100', 'api_key': 's', 'model': 'm'}}
ValueError: Invalid kwargs for agent 'pi':
  Unknown option 'intercept_config'.
See harbor agent schema pi for available options.

[burtenshaw]

I pushed three commits on top of this branch. The commit messages have the details.

• A merge of main. The only conflict was two tests added at the same place in test_harbor_rollout_contract.py, and both are kept.
• importorskip("harbor.agents.base"), which fixes the test (3.11) failure. Without Harbor, examples/harbor imports as harbor once another test puts examples/ on sys.path.
• serve --allow-harness, a refusal for subclasses that would skip an Intercept* class, and two small cleanups.

CI's test command passes on 3.11 and 3.12 with these changes (run in an HF Sandbox).

One design question before this merges. Harbor 0.25.0, the version in our lock, already has a connection override: AgentConfig.base_url, model_api_key_name and model_api_format. Agents that declare MODEL_CONNECTION honor it, including claude-code, codex, goose, gemini-cli, cline, computer-1 and dspy-rlm. Agents that don't declare it raise does not support inference connection overrides.

Could custom() wire your own agents through those fields instead of new api_base/api_key kwargs? Two things would change:

1. Agents installed in the sandbox that declare MODEL_CONNECTION would work instead of being refused.
2. A host-side agent that doesn't declare api_base/api_key would fail loudly. Today Harbor's parse_options drops undeclared kwargs when a class has no options model, so the agent builds without error and never reaches the proxy.

Could the session key travel in AgentConfig.env, with model_api_key_name naming it, so it stays out of the verifier? I haven't run a rollout this way, and it may need a Harbor floor above >=0.22.0. If it doesn't fit, a check in custom() that the class declares both kwargs would cover point 2.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit bfe8b9d. Configure here.}

[cursor]

UI bypasses harness allow-list

High Severity

seams.get now imports any module:Class harness, but OPENENV_HARBOR_ALLOW_HARNESS is enforced only in HarborEnvironment._run_rollout. The Harbor UI calls run_rollout directly, so a client-supplied import path still loads that module on the server. Before this change unknown names failed without importing.

Additional Locations (1)

• src/openenv/harbor/environment.py#L251-L262

 

^{Reviewed by Cursor Bugbot for commit bfe8b9d. Configure here.}

[cursor]

UI omits host-side loopback URL

Medium Severity

local_intercept_url is passed from the CLI runner and the MCP serve path, but the Harbor UI still calls run_rollout with only service.public_url. Host-side agents started from the UI, including terminus-2, keep going through the public forward, so long completions can still be cut with a 502 and retried as extra roots.

Additional Locations (1)

• src/openenv/harbor/runner.py#L149-L156

 

^{Reviewed by Cursor Bugbot for commit bfe8b9d. Configure here.}