Skip to content

feat: add tensorlake sandbox - #1555

Open
cooltx7eel wants to merge 4 commits into
openenv:mainfrom
tensorlakeai:feat/tensorlake-sandbox
Open

cooltx7eel wants to merge 4 commits into
openenv:mainfrom
tensorlakeai:feat/tensorlake-sandbox

Conversation

@cooltx7eel

@cooltx7eel cooltx7eel commented Nov 3, 2026 •

Copy link
Copy Markdown

Summary

Adds TensorlakeProvider, a ContainerProvider that runs an environment server in a Tensorlake sandbox and returns a public HTTPS base_url for EnvClient. It is an optional extra (pip install openenv[tensorlake]) with a lazy SDK import, and follows the same keep-alive-then-launch approach as the Modal, Daytona and Novita providers. Also adds tests, an echo-env example, and docs entries.

  • The sandbox URL is public, so the caller must pass public=True to accept that.
  • Egress to the cloud metadata range 169.254.0.0/16 is always blocked. allow_out and deny_out control the rest.
  • Server output is withheld from errors unless the caller passes surface_server_logs=True.
  • Import from openenv.core.containers.runtime.tensorlake_provider. It is not re-exported, same as KubernetesProvider.

Type of Change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation
  • New environment
  • Refactoring

Alignment Checklist

Before submitting, verify:

  • I have read .claude/docs/PRINCIPLES.md and this PR aligns with our principles
  • I have checked .claude/docs/INVARIANTS.md and no invariants are violated
  • I have run /pre-submit-pr (or bash .claude/hooks/lint.sh and tests) and addressed all issues

RFC Status

  • Not required (bug fix, docs, minor refactoring)
  • RFC exists: #___
  • RFC needed (will create before merge)

Test Plan

  • PYTHONPATH=src:envs uv run pytest tests/test_core/test_tensorlake_provider.py: 24 passed, no Tensorlake account needed. The tests use a fake SDK and cover the public URL contract, the public=True opt-in, metadata egress blocking, empty allowlist, cleanup after a failed start, cmd override and openenv.yaml discovery, idempotent and retried stop, env var merging, HTTPS enforcement, context-manager exit, and wait_for_ready on dead, healthy, foreign and flaky servers.
  • PYTHONPATH=src:envs uv run pytest tests/test_core/: 586 passed, 20 skipped.
  • ruff check, ruff format --check and usort check on the changed files. The repo-wide usort check and ruff format --check report tests/envs/test_julia_env.py, tests/envs/test_grid_world.py and src/openenv/core/harness/README.md, which are untouched by this PR and already fail on main.
  • Live run against Tensorlake:
    tl sbx image import ghcr.io/huggingface/openenv-echo-env:latest --registered-name echo-env
    TENSORLAKE_API_KEY=... PYTHONPATH=src:envs uv run python examples/tensorlake_echo_env.py
    Expected: the sandbox starts, /health returns 200, echo_message round-trips "Hello, World!", and the sandbox is terminated at exit.
    

Claude Code Review


Note

Medium Risk
New cloud runtime exposes publicly reachable, unauthenticated env servers and configures egress; mitigated by explicit public=True, HTTPS enforcement, metadata blocking, and secret-safe error handling.

Overview
Adds TensorlakeProvider, an optional cloud runtime (pip install openenv[tensorlake]) that boots a registered Tensorlake sandbox image, replaces the image CMD with a keep-alive, launches the env server on port 8000 (from /app/env/openenv.yaml or cmd=), and returns a public HTTPS base_url for clients.

Callers must pass public=True to acknowledge unauthenticated ingress. Egress always blocks 169.254.0.0/16; allow_out / deny_out tune the rest. Failed starts delete sandboxes; SDK details and sandbox URLs are withheld from errors unless surface_server_logs=True (with redaction).

Also adds the tensorlake extra in pyproject.toml, docs/runtime-provider table entries, autodoc, an examples/tensorlake_echo_env.py script, and extensive fake-SDK unit tests.

Reviewed by Cursor Bugbot for commit e3c85bb. Bugbot is set up for automated code reviews on this repo. Configure here.

@burtenshaw burtenshaw left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes.

  1. [P1] Keep the allocated sandbox ID when SDK creation fails. src/openenv/core/containers/runtime/tensorlake_provider.py:223 assigns the result of Sandbox.create() before entering the cleanup block at line 236. That SDK call allocates and then connects. In both downloaded SDKs, a running allocation without routing information raises SandboxNotRoutableError after allocation. Neither SDK deletes it on that path. In 0.5.161, the default cancel_on_timeout=False also raises SandboxPending with an allocated ID. The provider retains neither the handle nor the ID, close() deletes nothing, and another start is permitted. This violates RFC 002 cloud-provider invariant 5.

    Reproduction: pr1555_repro.py::test_allocation_is_deleted_when_real_sdk_connect_raises (below) fails on 0.5.135 and 0.5.161 with delete.call_count == 0 and provider state (None, None). test_real_sdk_pending_timeout_is_cancelled fails on 0.5.161 for the same reason. These execute the real SDK creation and connection logic with allocation responses supplied offline.

    Smallest fix: allocate through SandboxClient, record the ID before connecting, and include allocation, waiting, and connection in the existing cleanup block. Change the lazy import at line 216 to SandboxClient, sandbox_url_from_ingress_endpoint. Replace lines 223 through 237 with this block. Keep the following command discovery and launch code inside the same try.

    self._secret_values = [v for v in effective_env_vars.values() if v]
    try:
        client = SandboxClient(api_key=self._api_key)
        info = client.create(
            image=effective_image,
            cpus=self._cpus,
            memory_mb=self._memory_mb,
            timeout_secs=self._timeout_secs,
            entrypoint=_KEEPALIVE_ENTRYPOINT,
            allow_internet_access=self._allow_out != [],
            allow_out=self._allow_out,
            deny_out=self._deny_out,
        )
        self._sandbox_id = info.sandbox_id
        deadline = time.monotonic() + 300.0
        while info.status == "pending":
            remaining = deadline - time.monotonic()
            if remaining <= 0:
                raise TimeoutError("Tensorlake sandbox startup timed out.")
            time.sleep(min(0.5, remaining))
            info = client.get(self._sandbox_id)
        if info.status != "running":
            raise RuntimeError("Tensorlake sandbox failed to start.")
        self._sandbox = client.connect(self._sandbox_id)

    The existing ID-based stop path can then delete an allocation even when connection fails. Keep the ID if that delete fails. Moving Sandbox.create() inside try alone does not recover its lost ID. Passing cancel_on_timeout=True alone also leaves the connection-failure leak and is unsupported by 0.5.135.

  2. [P1] Withhold SDK exception details as well as server output. src/openenv/core/containers/runtime/tensorlake_provider.py:260 logs the cleanup traceback with exc_info=True. Line 265 propagates raw launch errors, line 275 chains file-read errors, and line 323 propagates termination errors. surface_server_logs=False does not protect these paths. The SDK preserves error messages. Supplying a real SandboxError containing a URL and an injected token exposes both in the default traceback and cleanup log. At line 300, the debug formatter also leaves the returned ingress URL intact because it only scrubs environment values. These violate INVARIANTS.md's credential rule and RFC 002 S2/S4.

    Reproduction: test_default_error_traceback_withholds_sdk_secrets, test_failed_cleanup_log_withholds_sdk_secrets, and test_debug_output_redacts_known_ingress_url in the reproduction file all fail on both SDK versions. The first two inject SDK errors at the file-read and delete boundaries; they do not claim a live service returned those messages. The third supplies server output containing the exact returned URL and observes that URL in the raised error.

    Smallest fix, applied after finding 1: include the API key among known redaction values, replace the startup exception handler with a constant error and no exception chain, and remove the sandbox ID and traceback from the cleanup warning.

    self._secret_values = [
        v for v in (self._api_key, *effective_env_vars.values()) if v
    ]
    # ... existing protected allocation and launch ...
    except Exception:
        try:
            self.stop_container()
        except Exception:
            logger.warning(
                "Could not terminate Tensorlake sandbox after a failed start. "
                "Retry close()."
            )
        raise RuntimeError(
            "Tensorlake sandbox startup failed. SDK details are withheld."
        ) from None

    Replace the URL return at line 250 with an assignment to base_url, then append the following before returning it. Redact longer values first at line 300 so an environment value cannot partially replace a URL before the URL itself is scrubbed.

    self._secret_values.extend(
        [base_url, base_url.replace("https://", "wss://", 1)]
    )
    return base_url
    # In _server_exited_message:
    for value in sorted(self._secret_values, key=len, reverse=True):
        log = log.replace(value, "***")

    In stop_container(), preserve the existing SandboxNotFoundError case and add this handler before clearing the saved ID:

    except Exception:
        raise RuntimeError(
            "Tensorlake sandbox termination failed. Retry close(). "
            "SDK details are withheld."
        ) from None
  3. [P2] Bound the readiness probe and reject success after its deadline. src/openenv/core/containers/runtime/tensorlake_provider.py:366 calls get_process() using the creation handle. Both SDK versions configure that handle with a 300-second request timeout. The readiness default is 120 seconds. The deadline is checked only before the SDK call, so line 375 can report success after the budget has expired.

    Reproduction: test_readiness_cannot_succeed_after_its_deadline advances the clock by the SDK's 300-second request budget during the process probe, then supplies a health response of 200. wait_for_ready(timeout_s=120) returns successfully. The expected TimeoutError assertion fails on both SDK versions.

    Smallest fix: use a separate, bounded SDK proxy handle for status probes and check a monotonic deadline before accepting HTTP success. Add self._probe_config: Dict[str, Any] = {} in the constructor. After the update() response at line 245, save its management endpoint and routing hint:

    self._probe_config = {
        "identifier": self._sandbox_id,
        "proxy_url": info.sandbox_url,
        "routing_hint": info.routing_hint,
        "api_key": self._api_key,
    }

    Keep the readiness docstring and replace its implementation after import requests with the following. Directly constructed SDK handles close their connection on context exit without deleting the sandbox in both inspected releases.

    deadline = time.monotonic() + timeout_s
    while time.monotonic() < deadline:
        remaining = deadline - time.monotonic()
        if self._sandbox is not None and self._pid is not None:
            from tensorlake.sandbox import Sandbox, SandboxError
    
            try:
                with Sandbox(
                    **self._probe_config,
                    request_timeout=min(5.0, remaining),
                ) as probe:
                    status = probe.get_process(self._pid).status
            except SandboxError:
                status = "running"
            if status != "running":
                raise RuntimeError(self._server_exited_message())
    
        remaining = deadline - time.monotonic()
        if remaining <= 0:
            break
        try:
            response = requests.get(
                f"{base_url}/health", timeout=min(5.0, remaining)
            )
            if response.status_code == 200 and time.monotonic() < deadline:
                return
        except requests.RequestException:
            pass
        remaining = deadline - time.monotonic()
        if remaining > 0:
            time.sleep(min(1.0, remaining))
    raise TimeoutError(
        f"Tensorlake sandbox did not become ready within {timeout_s}s"
    )

Suggestion: correct the lifetime wording at src/openenv/core/containers/runtime/tensorlake_provider.py:25 and :129, and document URL lifetime in docs/source/guides/runtime-providers.md:343. Tensorlake defines timeout_secs as an idle threshold; ongoing proxy traffic keeps the sandbox running. It does not impose a one-hour wall-clock lifetime. Tensorlake lifecycle documentation.

-# Bound the sandbox lifetime (RFC 002 S6). Same default as NovitaProvider.
+# Idle timeout; ongoing proxy traffic keeps the sandbox running.
@@
-                Sandbox lifetime in seconds.
+                Idle timeout in seconds. Ongoing proxy traffic keeps the
+                sandbox running; this is not a wall-clock lifetime.

Add after the public-ingress paragraph in the guide:

The URL remains usable while this ephemeral sandbox runs. `timeout_secs`
defaults to 3600 seconds of proxy inactivity. Ongoing proxy traffic keeps
the sandbox running, so this is not a wall-clock lifetime. A successful
stop ends access. Starting again creates a new sandbox and URL; reconnect
the client to that URL.

Verification at c301d0ea5897978c047b2c93e85bcd026d488ad2: uv sync --all-extras completed. The provider suite passed 24 tests with Python socket connections and DNS resolution forbidden. The core suite passed 611 tests and skipped 19. Changed-file ruff, formatting, and usort checks passed. The repository lint hook reported 47 pre-existing formatting failures and two pre-existing import-sort failures; none of its reported files are changed here. The debug hook reported no new-provider findings. All 13 GitHub checks in the initial snapshot were successful. GitHub returned no review threads, reviews, or issue comments, so there are no unresolved Bugbot threads or author replies to confirm or refute.

Downloaded the 0.5.135 and 0.5.161 wheels and bound every provider SDK call against both real signatures: Sandbox.create, start_process, update, read_file, get_process, get_output, terminate, SandboxClient(api_key=...), delete, and sandbox_url_from_ingress_endpoint. All exist with the supplied arguments. Both SDKs allow websockets>=14,<18, which overlaps core's websockets>=15.0.1; the lower bound is installable. Import and construction without the SDK succeeded, and starting produced the advertised extra-install error.

The existing ID-based retry after a failed stop retains the ID and blocks restart. The confirmed lifecycle leak occurs before that ID is recorded. The public-ingress opt-in and HTTPS rejection tests pass. Tensorlake imports only parse_openenv_app_field from _server_config.py, quotes the app with shlex.quote, and does not use or change parse_dockerfile_cmd; the Dockerfile CMD parser is outside this PR's execution path.

The egress controls map to a platform firewall, not an in-sandbox command. The minimum SDK documents host-side enforcement and deny precedence; its creation request serializes the metadata CIDR into the network policy. The platform describes enforcement outside the workload in its firewall design. The empty allowlist also sets allow_internet_access=False, matching the documented network matrix. This review did not perform a live metadata-denial test. Platform documentation states that ingress supports WebSocket upgrades, but no live /ws round trip was verified.

The regressions are in the collapsed block below. Extract a tensorlake wheel (0.5.135 or 0.5.161), point PR1555_SDK_ROOT at it, and run from the repository root:

PR1555_SDK_ROOT=<extracted wheel> PYTHONPATH=src:envs uv run pytest pr1555_repro.py -q --tb=short

The results are five failed regression assertions and one version-specific skip on 0.5.135, and six failed assertions on 0.5.161. They cover the three finding groups above. The proposed patches have not been applied to the branch; the fake-SDK tests need to model the allocation and bounded-probe paths when the fixes are implemented.

Separately, the branch now conflicts with main in docs/source/getting-started.md: #1526 added the Kubernetes example to the same runtime-provider line. Keep both entries when you merge main. Maintainer edits are disabled on this fork, so we cannot resolve it for you.

Three questions remain for maintainers under the proposed RFC 002 Cloud Sandbox Providers amendment, and are not changes requested from you: whether core carries another vendor provider, whether public-only ingress behind an explicit public=True is acceptable, and whether a live EnvClient WebSocket round trip and metadata-denial check must be recorded before merge. We will follow up here on those.

pr1555_repro.py
"""Offline regression checks for PR 1555, using the downloaded real SDK.

Run with PR1555_SDK_ROOT pointing at an extracted SDK wheel. These assertions
describe the required behavior and fail against c301d0ea. No cloud calls run.
"""

import os
import socket
import sys
import traceback
from types import SimpleNamespace
from unittest.mock import MagicMock, patch

import pytest

sys.path.insert(0, os.environ["PR1555_SDK_ROOT"])

from tensorlake.sandbox import (
    Sandbox,
    SandboxClient,
    SandboxError,
    SandboxNotRoutableError,
)
from tensorlake.sandbox.models import SandboxStatus

from openenv.core.containers.runtime.tensorlake_provider import TensorlakeProvider

PUBLIC_URL = "https://8000-sbx-review.sandbox.tensorlake.ai"
SECRET = "review-secret-12345"


@pytest.fixture(autouse=True)
def forbid_python_network(monkeypatch):
    def fail(*args, **kwargs):
        raise AssertionError("Unexpected network call")

    monkeypatch.setattr(socket.socket, "connect", fail)
    monkeypatch.setattr(socket, "create_connection", fail)
    monkeypatch.setattr(socket, "getaddrinfo", fail)


def offline_client_init(self, *args, **kwargs):
    self._request_timeout = kwargs.get("request_timeout", 300.0)
    self._api_key = "review-api-key"
    self._organization_id = None
    self._project_id = None
    self._rust_client = MagicMock()


def make_provider_sandbox():
    sandbox = MagicMock(spec=Sandbox)
    sandbox.sandbox_id = "sbx-review"
    sandbox.read_file.return_value = SimpleNamespace(value=b"app: server.app:app\n")
    sandbox.start_process.return_value = SimpleNamespace(pid=42)
    sandbox.update.return_value = SimpleNamespace(
        ingress_endpoint="https://sandbox.tensorlake.ai", sandbox_id="sbx-review"
    )
    sandbox.get_process.return_value = SimpleNamespace(status="running")
    return sandbox


def test_allocation_is_deleted_when_real_sdk_connect_raises():
    # Run real Sandbox.create, SandboxClient.create_and_connect, and connect.
    # The simulated control plane allocated a running sandbox but omitted its
    # route. The real SDK raises SandboxNotRoutableError after allocation.
    result = SimpleNamespace(
        sandbox_id="allocated-review-1",
        status=SandboxStatus.RUNNING,
        routing_hint=None,
        sandbox_url=None,
        ingress_endpoint=None,
        name=None,
        trace_id="review-trace",
    )
    allocation_method = (
        "_create_and_wait" if hasattr(SandboxClient, "_create_and_wait") else "create"
    )
    provider = TensorlakeProvider(public=True, image="echo-env", api_key="review-key")
    with (
        patch.object(SandboxClient, "__init__", offline_client_init),
        patch.object(SandboxClient, allocation_method, return_value=result) as allocate,
        patch.object(SandboxClient, "delete") as delete,
    ):
        with pytest.raises(SandboxNotRoutableError):
            provider.start_container()
        provider.close()
        assert allocate.call_count == 1
        assert delete.call_count == 1, (
            "allocated-review-1 was allocated but never deleted; "
            f"provider state is {(provider._sandbox, provider._sandbox_id)}"
        )


def test_real_sdk_pending_timeout_is_cancelled():
    if not hasattr(SandboxClient, "_create_and_wait"):
        pytest.skip("SDK 0.5.135 has no SandboxPending path")
    from tensorlake.sandbox import SandboxPending

    result = SimpleNamespace(
        sandbox_id="pending-review-1",
        status=SandboxStatus.PENDING,
        pending_reason="waiting_for_capacity",
        trace_id="review-trace",
    )
    provider = TensorlakeProvider(public=True, image="echo-env", api_key="review-key")
    with (
        patch.object(SandboxClient, "__init__", offline_client_init),
        patch.object(SandboxClient, "_create_and_wait", return_value=result),
        patch.object(SandboxClient, "_wait_for_sandbox", return_value=result),
        patch.object(SandboxClient, "delete") as delete,
    ):
        with pytest.raises(SandboxPending):
            provider.start_container()
        provider.close()
        assert delete.call_count == 1, (
            "pending-review-1 remains allocated after timeout and close; "
            f"provider state is {(provider._sandbox, provider._sandbox_id)}"
        )


def test_default_error_traceback_withholds_sdk_secrets():
    sandbox = make_provider_sandbox()
    sandbox.read_file.side_effect = SandboxError(f"request failed: {PUBLIC_URL} {SECRET}")
    provider = TensorlakeProvider(public=True, image="echo-env")
    with patch.object(Sandbox, "create", return_value=sandbox):
        with pytest.raises(ValueError) as error:
            provider.start_container(env_vars={"TOKEN": SECRET})
    rendered = "".join(traceback.format_exception(error.value))
    assert SECRET not in rendered and PUBLIC_URL not in rendered, rendered


def test_failed_cleanup_log_withholds_sdk_secrets(caplog):
    sandbox = make_provider_sandbox()
    sandbox.start_process.side_effect = SandboxError("launch failed")
    sandbox.terminate.side_effect = SandboxError(f"delete failed: {PUBLIC_URL} {SECRET}")
    provider = TensorlakeProvider(public=True, image="echo-env")
    with patch.object(Sandbox, "create", return_value=sandbox):
        with pytest.raises(SandboxError):
            provider.start_container(env_vars={"TOKEN": SECRET})
    assert SECRET not in caplog.text and PUBLIC_URL not in caplog.text, caplog.text


def test_debug_output_redacts_known_ingress_url():
    sandbox = make_provider_sandbox()
    provider = TensorlakeProvider(public=True, image="echo-env", surface_server_logs=True)
    with patch.object(Sandbox, "create", return_value=sandbox):
        url = provider.start_container()
    sandbox.get_process.return_value = SimpleNamespace(status="exited")
    sandbox.get_output.return_value = SimpleNamespace(lines=[f"Listening at {url}"])
    with pytest.raises(RuntimeError) as error:
        provider.wait_for_ready(url)
    assert url not in str(error.value), str(error.value)


def test_readiness_cannot_succeed_after_its_deadline():
    sandbox = make_provider_sandbox()
    provider = TensorlakeProvider(public=True, image="echo-env")
    with patch.object(Sandbox, "create", return_value=sandbox):
        url = provider.start_container()
    clock = SimpleNamespace(now=0.0)

    def slow_process_probe(pid):
        # Both real SDK versions inherit a 300 second request timeout here.
        clock.now += 300.0
        return SimpleNamespace(status="running")

    sandbox.get_process.side_effect = slow_process_probe
    with (
        patch("time.time", side_effect=lambda: clock.now),
        patch("time.monotonic", side_effect=lambda: clock.now),
        patch("requests.get", return_value=SimpleNamespace(status_code=200)),
    ):
        with pytest.raises(TimeoutError):
            provider.wait_for_ready(url, timeout_s=120)

@cooltx7eel
cooltx7eel force-pushed the feat/tensorlake-sandbox branch from c301d0e to 6087512 Compare November 5, 2026 06:48
shanshan wang and others added 2 commits November 5, 2026 00:51
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@cooltx7eel
cooltx7eel force-pushed the feat/tensorlake-sandbox branch from 6087512 to 7191717 Compare November 5, 2026 06:54
@cooltx7eel

Copy link
Copy Markdown
Author

@burtenshaw, thanks for your thorough review. I fixed all the issues you raised, rebased on main, and squashed the fixes into one commit.

  1. Sandbox ID lost when creation fails. The provider now allocates through SandboxClient, records the sandbox ID before it waits or connects, and keeps allocation, pending-wait and connect inside the cleanup block. A failed connect or pending timeout now deletes the allocation, and the ID stays if that delete fails so close() can retry.

  2. SDK details leaking into errors and logs. Startup, termination and cleanup failures now raise a constant message with from None, and the cleanup warning no longer logs the traceback or sandbox ID. The API key and the returned ingress URL (https and wss) join the redaction list, and redaction runs longest value first.

  3. Readiness probe not bounded. Status probes now use a separate SDK handle with request_timeout=min(5, remaining), built from the update() response. A 200 from /health is only accepted while the monotonic deadline has not passed.

  4. Lifetime wording. The constant comment and docstring now describe timeout_secs as an idle timeout, and the runtime-providers guide documents URL lifetime with the text you proposed.

The fake-SDK tests now cover the allocation path, pending timeout, failed connect, withheld SDK errors, failed-stop retry and the bounded probe deadline (30 tests, up from 24). The branch is rebased on main, so the getting-started.md conflict is gone and both the Kubernetes and Tensorlake entries are kept.

@burtenshaw burtenshaw left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes for one remaining item. Allocation cleanup, the readiness deadline and the docs conflict are fixed.

Reviewed head: 7191717622b46d89dbc97800a37907664b36f21e. The provider and its tests are identical to 6087512f, where these checks ran; the rebase only adds main's version bump.

Verification of the previous blockers:

  • PASS, allocation cleanup. src/openenv/core/containers/runtime/tensorlake_provider.py:255 records the allocated ID before connecting. The real SDK creation, connection, and deletion paths delete the unroutable allocation on both 0.5.135 and 0.5.161. Pending-timeout cleanup also passes on both. A failed delete retains the ID and blocks restart until close succeeds.
  • FAIL, complete SDK-error secrecy. The three original secrecy regressions pass, as do launch-error and termination-error checks. The new readiness-handle constructor at src/openenv/core/containers/runtime/tensorlake_provider.py:416 remains outside the SDK-error handler and exposes injected URL and token values on both SDK versions. See the remaining finding below.
  • PASS, readiness deadline. src/openenv/core/containers/runtime/tensorlake_provider.py:418 bounds the SDK probe timeout. Lines 428 and 439 recompute the remaining budget and reject late success. Both the original slow-process-probe regression and a late HTTP 200 check pass on both SDK versions.

[P1] Withhold SDK details when constructing the readiness handle. The try at src/openenv/core/containers/runtime/tensorlake_provider.py:420 starts after Sandbox(...) has been constructed. Both inspected SDKs convert native-client construction errors into SandboxError subclasses while retaining their messages. Those errors escape wait_for_ready() without sanitization, including with surface_server_logs=False. This leaves the credential-exposure blocker open.

The following regression fails on both SDK 0.5.135 and 0.5.161. It executes the real SDK constructor and exception conversion, injecting a real native SDK error at the transport-construction boundary. This is an offline injection; no live Tensorlake request ran. Set PR1555_SDK_ROOT to either extracted wheel and use PYTHONPATH=src:envs when running this test.

import importlib
import os
import sys
import traceback
from unittest.mock import patch

import pytest

sys.path.insert(0, os.environ["PR1555_SDK_ROOT"])
from openenv.core.containers.runtime.tensorlake_provider import TensorlakeProvider

sdk = importlib.import_module("tensorlake.sandbox.sandbox")
URL = "https://8000-sbx-review.sandbox.tensorlake.ai"
TOKEN = "review-secret-12345"


def test_readiness_constructor_error_withholds_sdk_secrets():
    provider = TensorlakeProvider(public=True, image="echo-env", api_key=TOKEN)
    provider._pid = 42
    provider._probe_config = {
        "identifier": "sbx-review",
        "proxy_url": "https://sbx-review.sandbox.tensorlake.ai",
        "routing_hint": "hint",
        "api_key": TOKEN,
    }
    native_error = sdk.RustCloudSandboxClientError(
        "connection", None, f"proxy setup failed: {URL} {TOKEN}"
    )
    with (
        patch.object(sdk, "RustCloudSandboxProxyClient", side_effect=native_error),
        pytest.raises(Exception) as error,
    ):
        provider.wait_for_ready(URL, timeout_s=0.01)
    rendered = "".join(traceback.format_exception(error.value))
    assert TOKEN not in rendered and URL not in rendered, rendered

Both failures include this exception line:

tensorlake.sandbox.exceptions.SandboxConnectionError: Connection error: proxy setup failed: https://8000-sbx-review.sandbox.tensorlake.ai review-secret-12345

Put SDK-error handling around readiness-handle construction and context exit as well as get_process(). Retry within the existing deadline or raise a constant error with from None. Add the constructor-error regression to the provider suite.

The original reproduction was adapted for SandboxClient.create/get/delete, the separate readiness handle, and the sanitized startup exception type. The provider now owns pending polling on both SDKs, so the pending-timeout test also runs on 0.5.135. Each original assertion group passes on both versions:

  • PASS: test_allocation_is_deleted_when_real_sdk_connect_raises.
  • PASS: test_real_sdk_pending_timeout_is_cancelled.
  • PASS: test_default_error_traceback_withholds_sdk_secrets.
  • PASS: test_failed_cleanup_log_withholds_sdk_secrets.
  • PASS: test_debug_output_redacts_known_ingress_url.
  • PASS: test_readiness_cannot_succeed_after_its_deadline.

Each SDK run reports 6 passed. The additional checks report 1 failed and 3 passed per SDK: constructor secrecy fails; retained cleanup state, termination-error secrecy, and rejection of a late health response pass. SDK regression tests forbid Python socket connections and DNS resolution.

uv sync --all-extras completed. tests/test_core/test_tensorlake_provider.py reports 32 passed. tests/test_core reports 632 passed, 19 skipped, and 23 subtests passed. Changed Python files pass usort, ruff formatting, and ruff rules. pyproject.toml parses and git diff --check passes. The repository lint hook reports 47 existing formatting failures and two existing import-sort failures; every reported file is identical to origin/main. The debug hook reports no Tensorlake-provider findings.

The docs conflict is resolved. git merge-tree --write-tree origin/main HEAD exits 0 against main at e5a9268b373478c6e8a6f1cbe43fe1a7845fb78a. docs/source/getting-started.md:158 retains both Tensorlake and Kubernetes. docs/source/guides/runtime-providers.md:349 documents the idle timeout, URL lifetime, and reconnect behavior.

Whether core carries another vendor provider, whether public-only ingress is acceptable, and whether live WebSocket and metadata-denial evidence is required remain open maintainer questions.

Shanshan Wang and others added 2 commits November 5, 2026 01:11
Sandbox(...) and its context exit were outside the SandboxError handler in
wait_for_ready(), so a constructor error could surface the sandbox URL and
the API key. Move construction, get_process() and exit inside the handler
and retry until the deadline. The dead-server message is built inside the
handle and raised after it closes.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The old floor, 0.5.135, was the newest release when the extra was added.
Its comment cited the websockets<14 pin, but that was lifted at 0.5.117.
0.5.161 is the oldest release the provider suite and the review
regressions run against.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@burtenshaw

Copy link
Copy Markdown
Collaborator

Rechecked at e3c85bbb. The readiness-handle fix holds: the constructor-error regression from our review passes on SDK 0.5.161 (it failed before), and tests/test_core/test_tensorlake_provider.py passes (36 tests). Raising the floor to 0.5.161 is fine. With allocation cleanup, error withholding and the readiness deadline verified earlier, there are no open review findings on the code.

What remains are the maintainer questions under the proposed RFC 002 Cloud Sandbox Providers amendment: whether core carries another vendor provider, whether public-only ingress behind public=True is acceptable, and whether a live round trip and metadata-denial check must be recorded first. We will follow up here once those are decided.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants