[sergiopaniego]

Summary

With --harness opencode, most rollouts came back as FAILED (NonZeroAgentExitCodeError) even when the agent finished normally and the verifier scored them. A trial exception now stays a [WARN] finding when it is only a non-zero exit code and the verifier produced a reward. Real agent errors and ungraded runs still fail.

Type of Change

• Bug fix
• New feature
• Breaking change
• Documentation
• New environment
• Refactoring

What was wrong

In a 500-rollout run with OpenCode, 306 rollouts were reported as FAILED (NonZeroAgentExitCodeError), solved and unsolved alike, with reward=1.00, atif=match and full turns. Since openenv harbor rollout exits with 0 if all(r.ok ...) else 1, scripts saw good batches as failures.

Root cause

1. opencode run --format=json exits 1 on runs that finished. In hf-sandbox trials (opencode 1.19.5) every completed run exited 1: the stream ends on step_finish with reason: "stop", there is no error event and stderr is empty. Harbor runs the command under set -o pipefail (harbor/agents/installed/base.py:1033), turns a non-zero status into NonZeroAgentExitCodeError (base.py:960-996), records it on the trial and still runs the verifier. In opencode's run command (packages/opencode/src/cli/cmd/run.ts, src/index.ts) exit code 1 is only set on a session.error event, a failed prompt call or an uncaught error, and none of those shows up in these logs, so the exact source inside opencode is still unclear and no flag changes it.
2. run_rollout treated any exception_info on the trial result as a failed rollout (src/openenv/harbor/rollout.py:575-579), even when the verifier had already produced the headline reward.

Fix

In run_rollout, a trial exception is kept as a [WARN] finding instead of failing the rollout when all of these hold:

• the exception type is exactly NonZeroAgentExitCodeError (subclasses such as ApiError and ApiRateLimitError still fail)
• the message is Harbor's plain Command failed (exit N) (Harbor's OpenCode emitted error event(s) still fails)
• the captured output carries no opencode "type":"error" event (a run cut short by the provider still fails)
• the verifier produced a usable headline reward

The capture gates that run afterwards are unchanged, so a graded rollout with an empty or invalid capture still fails, and a rollout the verifier never graded still fails with NonZeroAgentExitCodeError.

RFC Status

• Not required (bug fix, docs, minor refactoring)

Test Plan

New tests/envs/test_harbor_agent_exit_code.py (8 tests) drives run_rollout with a fake Trial and a fake capture:

• a graded exit 1 with reward 1.0 and with 0.0: ok=True, one [WARN] finding, no error or exception_type
• exit 1 with no verifier result: still ok=False with NonZeroAgentExitCodeError
• ApiRateLimitError, OpenCode emitted error event(s), an exit with an opencode error event, and AgentTimeoutError: still fail even with a reward
• a graded exit 1 with an empty capture: still fails on the capture gate

3 of the 8 fail on main and pass with the fix. Across tests/envs/test_harbor_*.py the only failure (test_harbor_capture_server.py::test_start_refuses_a_port_held_by_a_non_capture_listener) also fails on main. ruff format, ruff check and usort are clean.

Claude Code Review

N/A

---

Note

Medium Risk
Changes rollout success semantics for Harbor trials with exceptions, which could mask some edge-case failures if the heuristic misclassifies; mitigated by narrow string/type checks and new tests.

Overview
Harbor rollouts stop failing solely on a spurious non-zero agent exit when the verifier already produced a headline reward.

run_rollout now routes that case through new helpers that recognize plain NonZeroAgentExitCodeError messages without an opencode "type":"error" event, keep ok=True, preserve the reward, and add a single [WARN] agent exited non-zero after a graded run finding with a trimmed first-line summary. Ungraded trials, API/provider errors, streams that include error events, and subclasses of the exit-code error still mark the rollout failed exactly as before; capture validation and trainability gates remain unchanged.

Adds focused pytest coverage that monkeypatches Trial, capture export, and trace reconciliation to assert the warning path for graded exits, continued failures without rewards or on real errors, and failure when the capture is empty even if graded.

^{Reviewed by Cursor Bugbot for commit 9966d7f. Bugbot is set up for automated code reviews on this repo. Configure here.}