Restore runtime repeatability

docs/source/reference/cli.md

@@ -63,12 +63,16 @@ openenv validate ./my_env --level runtime --local --output report.json
 Runtime validation requires Docker and a `validation.execution` declaration in
 `openenv.yaml`. The declaration points to a bounded JSON replay plan containing
 one reset and a sequence of actions. The validator builds an immutable image,
-opens one WebSocket session, measures reward values, observation schemas and
-state continuity, and removes the container even if a check fails.
-
-The runtime level implements startup, reward, observation, state, network
-policy, host containment, resource bounds, episode isolation and oracle
-containment checks. Other applicable Level 2 checks appear explicitly as `SKIP`; they make
+collects the original episode, repeats the plan in a fresh WebSocket session and
+with a different seed, then replays it in a fresh container when the provider
+supports that capability. Docker validation uses two containers concurrently
+during the fresh-container replay. Both use the same immutable image and are
+removed even if a check fails.
+
+The runtime level implements startup, reward, observation, state, network policy,
+host containment, resource bounds, episode isolation, oracle containment, seed
+forwarding, determinism and emitted trajectory record checks. Other applicable
+Level 2 checks appear explicitly as `SKIP`; they make
 the result `WARN`, which exits zero and does not mean Level 2 is complete.
 `FAIL` exits 1, unsupported package formats exit 2, and internal or policy errors
 exit 3. `--level semantic`
@@ -139,11 +143,19 @@ these measured responses. It does not test arbitrary MCP tool reads, partial or
 transformed disclosures, or writes to reward artifacts; it does not certify the
 full agent boundary.
 `--local` explicitly selects this default package mode and rejects a remote URL.
-The declared `episode_timeout_s` bounds collection; a reset or judged step may use
-its remaining budget. Collection failures appear once under `runtime.startup`,
-with dependent contract checks skipped and the completed trace retained.
+Explicitly unsupported session telemetry yields a `SKIP` naming `session_telemetry`;
+authorization refusals and malformed telemetry remain failures. Seed control
+reports observed forwarding to reset, separately from empirical determinism.
+The declared `episode_timeout_s` bounds each episode; a reset or judged step may use
+its remaining budget. Original and replay collection share a 300-second deadline
+and a 32 MiB evidence budget. `llm_judged` requests 20 identical-input episodes,
+including the original, plus the different-seed episode. Validation repeats the
+actions and any external service calls they make, so budget for the additional
+runtime and cost. Missing replay samples cannot pass determinism. Primary episode
+collection failures appear under `runtime.startup`, with dependent contract checks
+skipped and the completed trace retained.
 Validator JSON numeric-range and depth limits report named capability `SKIP`s,
-preserving the received evidence.
+preserving the received evidence when it can be retained without validation credentials.
 
 Credential delivery is deferred for this release. Set
 `validation.execution.requires_credentials: true` when the environment needs an

rfcs/008-environment-auto-validation.md

@@ -606,7 +606,7 @@ limitations. Alternate DNS, unmatched addresses and unsupported address families
 must fail closed. This requires a separate reviewed enforcement implementation;
 parsing these declarations does not claim they are enforced.
 
-### Later runtime evidence contracts
+### Runtime replay evidence (PR5)
 
 Seed acceptance and empirical determinism are separate findings. A reset that
 silently drops its seed does not establish seed control, while a deterministic
@@ -616,11 +616,24 @@ policy-owned volatile metadata can be excluded. Authors cannot exclude fields.
 For `llm_judged`, the bounded variance path uses 20 completed identical-input fresh
 replays and population reward variance in reward-squared units, compared to the
 declared bound. The total run budget bounds all samples; fewer than 20 is incomplete.
+An unscored non-terminal step contributes no reward variance. Its null position must
+agree across replays; a null/numeric mismatch is a divergence. Variance is measured
+separately at each numeric step position, and an entirely unscored episode is
+incomplete rather than passing. Terminal null rewards remain invalid.
 This procedure is a runtime check, not a statistical confidence claim.
 
+The initial implementation compares the baseline against a new session and an
+independently inspected new container, and separately requests a different seed.
+The judged sample count includes the completed baseline. Container identity must
+change while image identity remains fixed. There are no volatile-field exclusions
+in this version. Collection shares a 300-second deadline and retains at most
+32 MiB across baseline and replay evidence. Missing samples, missing provider
+capabilities and failed cleanup cannot produce a passing determinism finding.
+`replays.json` preserves completed traces, telemetry, identity and cleanup outcomes.
+
 Session telemetry for seed handling, named rubric/configuration, child attribution
-and subject-emitted record references is orchestrator-only. A future protocol
-slice must authorize access with an opt-in, random per-run/per-session capability
+and subject-emitted record references is orchestrator-only. The protocol
+authorizes access with an opt-in, random per-run/per-session capability
 attached to the **same** replay connection, reject unauthorized/cross-session
 reads and never expose telemetry as agent MCP tools. A second WebSocket creates
 another environment and cannot supply evidence for the measured instance.