Run Level 2 validation probes

docs/source/reference/cli.md

@@ -35,6 +35,71 @@ openenv import path/to/source --name my_env --output-dir ./envs --env-class MyEn
 
 ## `openenv validate`
 
+Run static declaration checks without Docker, or build and probe a package locally:
+
+```bash
+openenv validate ./my_env --level static
+openenv validate ./my_env --level runtime --local --output report.json
+```
+
+Runtime validation requires Docker and a `validation.execution` declaration in
+`openenv.yaml`. The declaration points to a bounded JSON replay plan containing
+one reset and a sequence of actions. The validator builds an immutable image,
+opens one WebSocket session, measures reward values, observation schemas and
+state continuity, and removes the container even if a check fails.
+
+This first runtime slice implements startup, reward, observation and state
+checks. Other applicable Level 2 checks appear explicitly as `SKIP`; they make
+the result `WARN`, which exits zero and does not mean Level 2 is complete.
+`FAIL` exits 1, unsupported package formats exit 2, and internal or policy errors
+exit 3. `--level semantic`
+includes the available lower-level checks but does not claim semantic execution.
+`--skip-build` runs declaration checks and skips runtime execution entirely.
+The Docker provider currently supports CPU workloads and `public` network mode;
+unsupported network or GPU requirements skip runtime before building.
+`--local` explicitly selects this default package mode and rejects a remote URL.
+The declared `episode_timeout_s` bounds collection; a reset or judged step may use
+its remaining budget. Collection failures appear once under `runtime.startup`,
+with dependent contract checks skipped and the completed trace retained.
+
+Credential delivery is deferred for this release. Set
+`validation.execution.requires_credentials: true` when the environment needs an
+externally supplied credential, such as a judge API key. This boolean declaration
+causes a named `credential_delivery` SKIP before build or launch; dependent runtime
+checks also SKIP. The resulting WARN does not establish complete Level 2 coverage.
+The declaration accepts no secret values and defaults to false. Validation never
+inherits host credentials or forwards API keys. Self-contained LLM judges can run
+without this requirement; `llm_judged` alone does not imply a need for credentials.
+An undeclared startup or reset failure still fails validation.
+
+Observation validation uses the `reset_observation` field from `/schema` for resets
+and `observation` for steps. Servers can declare `reset_observation_cls` when
+reset returns a different observation type; it defaults to the step observation
+class. Older servers without a reset schema use the step schema for both. Reset
+validation is always required; the validator does not silently substitute a base
+schema. These are output schemas, separate from any reset input declaration.
+The Level 2 profile permits null rewards only on reset. Every step, including a
+nonterminal step, requires a finite numeric reward within the declared range;
+boolean rewards are invalid. This certification requirement is stricter than the
+backward-compatible core `Observation.reward` type. Emit zero only when the intended
+reward is zero; the validator never converts missing or null step rewards to zero.
+
+Cleanup removes run-owned containers. Built images remain in Docker's local cache
+for reuse; the report records their immutable image IDs. Remove an unwanted image
+with `docker image rm <image-id>` after its validation runs have finished.
+
+Runtime reports use schema version 2 and severity policy v2. Static reports for
+v1 manifests retain schema version 1 and policy v1. An explicit v1 policy with a
+runtime ceiling is rejected. With `--output report.json`, a sibling
+`report.artifacts/` directory contains the replay plan, bounded redacted evidence,
+coverage inventory, provider settings, cleanup outcome and checksums. Treat these
+as author-validation evidence; this command does not issue certification.
+
+For a pinned, installed-wheel reproduction of the shared fixture and its fault
+cases, follow [the runtime lab](../../../tests/validation_runtime/README.md).
+The implementation and remaining check inventory are tracked in
+[the Level 2 umbrella](https://github.com/huggingface/OpenEnv/issues/1177).
+
 [[autodoc]] openenv.cli.commands.validate.validate
 
 ## `openenv push`

envs/coding_tools_env/server/app.py

@@ -13,12 +13,15 @@
 
 from openenv.core.env_server.http_server import create_app
 from openenv.core.env_server.mcp_types import CallToolAction, CallToolObservation
+from openenv.core.env_server.types import Observation
 
 try:
     from .coding_tools_env_environment import CodingToolsEnvironment
     from .gradio_ui import coding_tools_ui_builder
 except ImportError:  # pragma: no cover
-    from server.coding_tools_env_environment import CodingToolsEnvironment  # type: ignore
+    from server.coding_tools_env_environment import (  # type: ignore
+        CodingToolsEnvironment,
+    )
     from server.gradio_ui import coding_tools_ui_builder  # type: ignore
 
 
@@ -44,6 +47,7 @@ def _load_env_file() -> None:
     CodingToolsEnvironment,
     CallToolAction,
     CallToolObservation,
+    reset_observation_cls=Observation,
     env_name="coding_tools_env",
     max_concurrent_envs=int(os.getenv("MAX_CONCURRENT_ENVS", "4")),
     gradio_builder=coding_tools_ui_builder,

envs/echo_env/server/app.py

@@ -23,6 +23,8 @@
 
 import os
 
+from openenv.core.env_server.types import Observation
+
 # Support both in-repo and standalone imports
 try:
     # In-repo imports (when running from OpenEnv repository)
@@ -45,6 +47,7 @@
     EchoEnvironment,
     CallToolAction,
     CallToolObservation,
+    reset_observation_cls=Observation,
     env_name="echo_env",
     max_concurrent_envs=max_concurrent,
 )

envs/finqa_env/server/app.py

@@ -14,6 +14,7 @@
 
 from openenv.core.env_server.http_server import create_app
 from openenv.core.env_server.mcp_types import CallToolAction, CallToolObservation
+from openenv.core.env_server.types import Observation
 from pydantic import field_validator
 
 from .finqa_environment import FinQAEnvironment
@@ -44,7 +45,11 @@ def parse_arguments(cls, v: Any) -> Dict[str, Any]:
 
 
 app = create_app(
-    _env_factory, FinQACallToolAction, CallToolObservation, env_name="finqa_env"
+    _env_factory,
+    FinQACallToolAction,
+    CallToolObservation,
+    reset_observation_cls=Observation,
+    env_name="finqa_env",
 )
 
 

envs/jupyter_env/server/app.py

@@ -13,6 +13,7 @@
 
 from openenv.core.env_server.http_server import create_app
 from openenv.core.env_server.mcp_types import CallToolAction, CallToolObservation
+from openenv.core.env_server.types import Observation
 
 try:
     from .gradio_ui import jupyter_ui_builder
@@ -47,6 +48,7 @@ def _load_env_file() -> None:
     CallToolAction,
     CallToolObservation,
     env_name="jupyter_env",
+    reset_observation_cls=Observation,
     max_concurrent_envs=int(os.getenv("MAX_CONCURRENT_ENVS", "4")),
     gradio_builder=jupyter_ui_builder,
 )

envs/opencode_env/server/app.py

@@ -29,6 +29,8 @@
 import os
 from pathlib import Path
 
+from openenv.core.env_server.types import Observation
+
 
 def _load_env_file() -> None:
     """Lightweight ``.env`` loader (no python-dotenv dep).
@@ -56,19 +58,13 @@ def _load_env_file() -> None:
 
 try:
     from openenv.core.env_server.http_server import create_app
-    from openenv.core.env_server.mcp_types import (
-        CallToolAction,
-        CallToolObservation,
-    )
+    from openenv.core.env_server.mcp_types import CallToolAction, CallToolObservation
 
     from .gradio_ui import opencode_gradio_builder
     from .opencode_environment import OpenCodeEnvironment
 except ImportError:  # pragma: no cover
     from openenv.core.env_server.http_server import create_app
-    from openenv.core.env_server.mcp_types import (
-        CallToolAction,
-        CallToolObservation,
-    )
+    from openenv.core.env_server.mcp_types import CallToolAction, CallToolObservation
     from server.gradio_ui import opencode_gradio_builder  # type: ignore
     from server.opencode_environment import OpenCodeEnvironment  # type: ignore
 
@@ -102,6 +98,7 @@ def _custom_gradio_builder(
     CallToolAction,
     CallToolObservation,
     env_name="opencode_env",
+    reset_observation_cls=Observation,
     max_concurrent_envs=int(os.getenv("MAX_CONCURRENT_ENVS", "4")),
     gradio_builder=_custom_gradio_builder,
 )

envs/pi_env/server/app.py

@@ -29,6 +29,8 @@
 import os
 from pathlib import Path
 
+from openenv.core.env_server.types import Observation
+
 
 def _load_env_file() -> None:
     """Lightweight ``.env`` loader (no python-dotenv dep).
@@ -56,19 +58,13 @@ def _load_env_file() -> None:
 
 try:
     from openenv.core.env_server.http_server import create_app
-    from openenv.core.env_server.mcp_types import (
-        CallToolAction,
-        CallToolObservation,
-    )
+    from openenv.core.env_server.mcp_types import CallToolAction, CallToolObservation
 
     from .gradio_ui import pi_gradio_builder
     from .pi_environment import PiEnvironment
 except ImportError:  # pragma: no cover
     from openenv.core.env_server.http_server import create_app
-    from openenv.core.env_server.mcp_types import (
-        CallToolAction,
-        CallToolObservation,
-    )
+    from openenv.core.env_server.mcp_types import CallToolAction, CallToolObservation
     from server.gradio_ui import pi_gradio_builder  # type: ignore
     from server.pi_environment import PiEnvironment  # type: ignore
 
@@ -102,6 +98,7 @@ def _custom_gradio_builder(
     CallToolAction,
     CallToolObservation,
     env_name="pi_env",
+    reset_observation_cls=Observation,
     max_concurrent_envs=int(os.getenv("MAX_CONCURRENT_ENVS", "4")),
     gradio_builder=_custom_gradio_builder,
 )

envs/terminus_env/server/app.py

@@ -13,6 +13,7 @@
 
 from openenv.core.env_server.http_server import create_app
 from openenv.core.env_server.mcp_types import CallToolAction, CallToolObservation
+from openenv.core.env_server.types import Observation
 
 try:
     from .gradio_ui import terminus_ui_builder
@@ -45,6 +46,7 @@ def _load_env_file() -> None:
     CallToolAction,
     CallToolObservation,
     env_name="terminus_env",
+    reset_observation_cls=Observation,
     max_concurrent_envs=int(os.getenv("MAX_CONCURRENT_ENVS", "4")),
     gradio_builder=terminus_ui_builder,
 )

pyproject.toml

@@ -34,6 +34,7 @@ dependencies = [
     # Web UI dependencies
     "gradio>=4.0.0",
     "httpx>=0.28.1",
+    "jsonschema>=4.20.0",
 ]
 
 [project.optional-dependencies]

rfcs/008-environment-auto-validation.md

@@ -482,6 +482,7 @@ validation:
     dockerfile: Dockerfile
     context: .
     agent_boundary: api
+    requires_credentials: false
 ```
 
 The other manifest sections remain authoritative for capabilities, resources,
@@ -518,12 +519,31 @@ privileged oracle inputs nor host callbacks may be substituted for public action
 One collector owns reset, ordered actions until termination, and state reads on
 **one** WebSocket session. It records immutable raw request/response strings before
 client defaults or Pydantic coercion can hide malformed responses. Graders consume
-that evidence and cannot mutate the measured episode. The advertised observation
-schema is recorded alongside the transcript; reconstruct observation plus the
-separate reward/done envelope before validating it. Reset reward may be null; every
-step reward must be a finite JSON number, excluding booleans, within the declared
-range. State must retain the requested episode identity, reset its step count to
-zero and advance it coherently for successful steps.
+that evidence and cannot mutate the measured episode. `/schema.observation`
+describes step observations. The additive `/schema.reset_observation` field
+describes reset observations; core servers publish it from an explicit optional
+`reset_observation_cls`, defaulting to `observation_cls`. Environments whose reset
+returns a distinct model must declare it. For legacy servers that omit the field,
+the validator applies the step schema to reset as well. An explicitly malformed
+reset schema is a failure, never a reason to fall back. Both schemas are recorded
+alongside the transcript; reconstruct observation plus the separate reward/done
+envelope before validating against the operation's schema. Neither schema may
+retrieve non-local references.
+
+Reset validation is required even when reset and step return different models.
+An explicit reset output schema describes that difference; the validator never
+silently substitutes the base `Observation` schema or skips reset. Reset input
+declarations are a separate contract from these observation output schemas.
+
+Reset reward may be null. Every step, including a nonterminal step, must emit a
+finite JSON number within the declared range. Every numeric reset reward is also
+checked against that range; booleans are invalid. These are Level Two validation
+requirements, stricter than the core API: the core `Observation.reward` type
+continues to accept `bool | int | float | None` for compatibility. A valid core
+environment can therefore fail Level Two validation. Authors should emit numeric
+zero when a step's intended reward is zero; the validator never converts an absent
+or null reward to zero. State must retain the requested episode identity, reset
+its step count to zero and advance it coherently for successful steps.
 
 ### Startup, policy and provider supervision
 
@@ -542,6 +562,20 @@ The provider owns build, readiness, bounded exec/logs, inspected settings and
 idempotent cleanup on success, failure, timeout and cancellation. Cleanup evidence
 must establish that no run-owned subject remains. Core provider ABCs are unchanged.
 
+Credential delivery is deferred for this release. An environment that needs an
+externally supplied credential declares `validation.execution.requires_credentials:
+true` (a strict boolean, default `false`). This records a prerequisite only; no
+credential names, values, sources or injection mechanism are accepted. After
+validating the public plan, the runner reports `runtime.startup` as SKIP with the
+missing `credential_delivery` capability named, before building or launching the
+subject. Dependent checks remain SKIP and the report cannot establish complete
+Level Two conformance. Missing credentials are never inferred from exception text
+or from `llm_judged`: a self-contained judge may run without external credentials.
+An undeclared startup or reset failure remains FAIL. Host tokens are never a
+fallback, and credentials must not be embedded in manifests, plans or images.
+Supporting external judge credentials requires a later explicit contract for
+delivery, access isolation, lifetime, network access and evidence redaction.
+
 Launches use no privilege, host namespaces, host-directory mounts, Docker socket
 or forwarded credentials. They drop Linux capabilities, enable no-new-privileges,
 use a read-only image and explicitly bounded writable roots, and expose only a