[burtenshaw]

Summary

Add runtime.seed_control, runtime.episode_determinism and runtime.trajectory_record on #1320.

primary episode → fresh-session/container replay → changed-seed control

Compare subject-emitted records with wire evidence and compare deterministic replays using exact, type-aware JSON values. Seed control establishes observed forwarding to reset; empirical determinism is a separate result. Judged rewards require 20 completed samples and per-step population variance in reward-squared units.

Evidence

• Before: float decoding could collapse distinct precise or underflowing values into equal replay/trajectory evidence.
• After: exact comparisons retain those distinctions; bounded exact variance preserves supported violations as FAIL even when another column exceeds arithmetic support. Unsupported arithmetic remains explicit SKIP.
• Explicit UNKNOWN_TYPE produces a named session_telemetry SKIP. Authorization refusals and malformed replies remain FAIL. Replays retain shared time/byte budgets, completed prefixes and cleanup evidence.

Test Plan

The refreshed full follow-on stack at clean a96f68e1cc860068156813df81f4e20d85a06189 passed 1,759 source/CLI tests and 138 fresh non-editable-wheel protocol tests. All artifact hashes verified and 185 installed Python files matched source. These macOS arm64 results cover the combined stack, not this PR alone or Linux/Docker/HF qualification.

Independent replay-budget review passed 338 focused tests and 12 controls, with nine meaningful parent failures. Unsupported-prefix review passed 236 tests and 32 independent controls, with 30 meaningful parent failures. Exact reward/trajectory comparisons, cleanup/error precedence and the 300-second / 32-MiB replay bounds remain intact. Unsupported replay prefixes and their named JSON-limit diagnostics are retained; unsupported comparisons remain SKIP.

Current-head required checks and Linux acceptance: PR checks.

RFC Status

RFC 008 / #778 remains In Review. Depends on #1320 after #1524; adds three runtime implementations, moving 9/16 to 12/16. Fresh-container checks require provider identity and cleanup evidence; unavailable capabilities remain named SKIPs. Implementation coverage does not establish full RFC qualification.

Merge Danger

Door: Two-way.

Blast Radius: Replay.

Adds bounded replay work and grading to applicable runtime validation.

[bot-ci-comment]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[cursor]

Alignment Review Report

Automated Checks

• Lint: PASS — usort check src/ tests/, ruff format src/ tests/ envs/ --check, and ruff check src/ tests/ all pass for every file this PR touches (graders/runtime/repeatability.py, runtime/replay.py, runtime/artifacts.py, runtime/contracts.py, providers/docker.py, runner.py, types.py, and the test/fixture files). The only findings are pre-existing and unrelated to this diff: usort flags tests/envs/test_grid_world.py and tests/envs/test_julia_env.py (documented as pre-existing in AGENTS.md), and ruff format flags ~58 files under envs/ plus src/openenv/core/harness/README.md — none are part of this PR's changed-file set.
• Debug code: CLEAN — no print/breakpoint/debugger statements or new TODO/FIXME were introduced by this diff. check-debug.sh's whole-repo scan still surfaces older print/TODO lines elsewhere, but all are in files untouched by this PR.
• Tests: Ran the targeted suites (test_runtime_repeatability.py, test_runtime_replay.py, test_runtime_artifacts.py, test_runtime_execution.py, integration/test_runtime_cli.py, integration/test_runtime_process.py): 166 passed, 16 skipped (skips are the expected Docker-only cases in this non-Docker review environment). Full suite: 3618 passed, 2 failed, 169 skipped — the 2 failures are in tests/envs/test_websockets.py (echo_env state step_count assertions); confirmed via git diff that these files are untouched by this PR and the failures reproduce in isolation, so they're pre-existing/environmental, not caused by this change.

Open RFCs Context

• RFC 008 (Environment Auto-Validation) — Status: In Review, author @zkwentz. This PR is squarely in RFC 008's territory: it implements the "Runtime replay evidence (PR5)" slice (seed-control, episode-determinism, and trajectory-record graders; the fresh-session/fresh-container replay collector; replays.json artifact) that follows PR4's session-telemetry protocol (base branch of this PR). The PR also edits rfcs/008-environment-auto-validation.md itself to replace the "future protocol slice must authorize access..." hedge with "The protocol authorizes access..." and to flesh out the previously-stubbed "Later runtime evidence contracts" section into concrete, shipped behavior (300s/32MiB budgets, container-identity invariants, null-reward handling, judged-sample counts). I cross-checked every claim added to the RFC text against the implementation (budgets in replay.py, container/image identity checks, the empty volatile-exclusion set, null-vs-terminal reward handling in repeatability.py, replays.json contents in artifacts.py) and found them faithfully matched — this reads as planned, sequential RFC execution rather than a surprise deviation.
• Other Draft/In-Review RFCs (000, 001, 002, 003, 005, 006, 010, 011, 012) do not appear touched by this diff — this PR stays inside src/openenv/validation/ and its tests, with no changes to the Gym-like API, MCP surface, client/server boundary, or reward computation.

Tier 1: Fixes Required

None. No mechanical/lint/debug issues found in the diff.

Tier 2: Alignment Discussion

Principle Conflicts

None identified. The new code is validation/grading logic that reads already-computed rewards and telemetry to check determinism and tolerance bounds declared in the manifest — it does not compute rewards itself, so "rewards inside environment" is not implicated. Session telemetry stays orchestrator-only, is carried over the same validation WebSocket connection, and is never exposed as an MCP tool, consistent with the "agent isolation" invariant. No client code imports server internals or vice versa.

RFC Conflicts

ALIGNMENT FLAG: PR amends RFC 008's own prose while the RFC is still "In Review"

• Principle/RFC at stake: RFC 008 (Environment Auto-Validation), status In Review
• The concern: This PR changes RFC 008's text from forward-looking/hedged language ("A future protocol slice must authorize access...") to present-tense, implemented-fact language, and expands the "Later runtime evidence contracts" section into fully specified behavior. The implementation matches the new text in every detail I checked, so this isn't a technical conflict — but since the RFC is still formally "In Review" under a different primary author, it's worth the RFC owner explicitly signing off that this PR's wording is the intended canonical description of PR5 (and that no further changes to this contract are expected before PR6+).
• Suggested reviewer: @zkwentz (RFC 008 author)

Summary

• 0 mechanical issues to fix
• 1 alignment point for human review (RFC-text sign-off, not a code concern)
• 0 RFC conflicts to discuss (implementation matches the RFC text it adds)

  

_{Sent by Cursor Automation: Pre-review}

[rycerzes]

Reviewed at 88232843 and ran the pinned lab (Python 3.12.13, uv 0.9.3, --require-complete) on Docker 29.7.2, linux/amd64. fast, protocol and docker (20/20) pass.

I also validated maze_env built from its own Dockerfile with this stack. state_contract and episode_determinism correctly caught maze ignoring episode_id. One comment inline about how subjects on an older core are graded.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

Autofix Details

Bugbot Autofix prepared a fix for 1 of the 2 issues found in the latest run.

• ✅ Fixed: Isolation consumes replay deadline
  • The replay deadline now excludes elapsed isolation and oracle probing time while retaining the primary episode's use of the shared budget.

Or push these changes by commenting:

@cursor push 62e6c76197

Preview (62e6c76197)

diff --git a/src/openenv/validation/runner.py b/src/openenv/validation/runner.py
--- a/src/openenv/validation/runner.py
+++ b/src/openenv/validation/runner.py
@@ -344,6 +344,7 @@
                 started=started,
             )
         if result.status is CheckStatus.PASS:
+            isolation_started = time.monotonic()
             not_started = RuntimeEvidence(
                 failure_phase="not_started",
                 failure_reason="validation interrupted before the fresh episode",
@@ -382,6 +383,7 @@
                     subject.root,
                     manifest.capabilities.oracle.location,
                 )
+            replay_deadline += time.monotonic() - isolation_started
         evidence = collect_replays(
             provider,
             running,

diff --git a/tests/test_validation/test_runtime_execution.py b/tests/test_validation/test_runtime_execution.py
--- a/tests/test_validation/test_runtime_execution.py
+++ b/tests/test_validation/test_runtime_execution.py
@@ -189,6 +189,33 @@
     ]
 
 
+def test_isolation_does_not_consume_replay_deadline(package, monkeypatch):
+    clock = [0.0]
+    monkeypatch.setattr(
+        "openenv.validation.runner.time",
+        SimpleNamespace(monotonic=lambda: clock[0]),
+    )
+    calls = []
+
+    def collect(*args, **kwargs):
+        calls.append(kwargs)
+        clock[0] += 10.0 if len(calls) == 1 else 100.0
+        return measured_episode()
+
+    remaining = []
+
+    def replays(provider, running, spec, plan, original, **kwargs):
+        remaining.append(kwargs["deadline"] - clock[0])
+        return original
+
+    monkeypatch.setattr("openenv.validation.runner.collect_runtime_evidence", collect)
+    monkeypatch.setattr("openenv.validation.runner.collect_replays", replays)
+    monkeypatch.setattr("openenv.validation.runner._oracle_probes", lambda *a: ())
+    run_validation(package, max_level=Level.RUNTIME, provider=FakeRuntimeProvider())
+
+    assert remaining == [REPLAY_BUDGET_SECONDS - 10.0]
+
+
 @pytest.mark.parametrize("capability", ["json_depth", "json_numeric_range"])
 def test_initial_unsupported_json_skips_and_keeps_the_received_schema(
     package, monkeypatch, tmp_path, capability

_{You can send follow-ups to the cloud agent here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: Replay drops unsupported JSON evidence
  • Replay collection now retains unsupported JSON evidence for session and container paths and preserves the named JSON capability failure.

Or push these changes by commenting:

@cursor push 3b6b949194

_{You can send follow-ups to the cloud agent here.}

^{Reviewed by Cursor Bugbot for commit bf50009. Configure here.}