Skip to content

TESTING.md

Latest commit

 

History

History
265 lines (202 loc) · 8.97 KB

TESTING.md

File metadata and controls

265 lines (202 loc) · 8.97 KB
Raw

Testing

Running Tests

mise run test          # Rust + Python unit tests
mise run e2e           # End-to-end tests (starts a Docker-backed gateway)
mise run ci            # Everything: lint, compile checks, and tests

Test Layout

crates/*/src/          # Inline #[cfg(test)] modules
crates/*/tests/        # Rust integration tests
python/openshell/      # Python unit tests (*_test.py suffix)
e2e/python/            # Python E2E tests (test_*.py prefix)
e2e/rust/              # Rust CLI E2E tests

Rust Tests

Unit tests live inline with #[cfg(test)] mod tests blocks. Integration tests go in crates/*/tests/ and are named *_integration.rs.

Use #[tokio::test] for anything async:

#[cfg(test)]
mod tests {
    use super::*;

    #[tokio::test]
    async fn store_round_trip() {
        let store = Store::connect("sqlite::memory:").await.unwrap();
        store.put("sandbox", "abc", "my-sandbox", b"payload").await.unwrap();
        let record = store.get("sandbox", "abc").await.unwrap().unwrap();
        assert_eq!(record.payload, b"payload");
    }
}

Run Rust tests only:

mise run test:rust     # cargo test --workspace

Python Unit Tests

Python unit tests use the *_test.py suffix convention (not test_* prefix) and live alongside the source in python/openshell/. They use mock-based patterns with fake gRPC stubs:

def test_exec_python_serializes_callable_payload() -> None:
    stub = _FakeStub()
    client = _client_with_fake_stub(stub)

    def add(a: int, b: int) -> int:
        return a + b

    result = client.exec_python("sandbox-1", add, args=(2, 3))
    assert result.exit_code == 0

Run Python unit tests only:

mise run test:python   # uv run pytest python/

E2E Tests

E2E tests run against a live gateway. By default, mise run e2e starts an ephemeral standalone gateway with the Docker compute driver, runs the suite, and cleans it up afterward. To run the suite against an existing plaintext gateway, set OPENSHELL_GATEWAY_ENDPOINT:

OPENSHELL_GATEWAY_ENDPOINT=http://127.0.0.1:18080 mise run e2e

Raw endpoint mode is HTTP-only. Use a named gateway config when a gateway requires mTLS.

Python E2E (e2e/python/)

Tests use the sandbox fixture from conftest.py to create real sandboxes:

def test_exec_returns_stdout(sandbox):
    with sandbox(delete_on_exit=True) as sb:
        result = sb.exec(["echo", "hello"])
        assert result.exit_code == 0
        assert "hello" in result.stdout

Sandbox.exec_python

exec_python serializes a Python callable with cloudpickle, sends it to the sandbox, and returns the result. Because cloudpickle serializes module-level functions by reference (which fails inside the sandbox), use one of these patterns:

Closures from factory functions:

def _make_adder():
    def add(a, b):
        return a + b
    return add

def test_addition(sandbox):
    with sandbox(delete_on_exit=True) as sb:
        result = sb.exec_python(_make_adder(), args=(2, 3))
        assert result.stdout.strip() == "5"

Bound methods on local classes:

def test_multiply(sandbox):
    class Calculator:
        def multiply(self, a, b):
            return a * b

    with sandbox(delete_on_exit=True) as sb:
        result = sb.exec_python(Calculator().multiply, args=(6, 7))
        assert result.stdout.strip() == "42"

Shared Fixtures (e2e/python/conftest.py)

Fixture Scope Purpose
sandbox_client session gRPC client connected to the active gateway
sandbox function Factory returning a Sandbox context manager
inference_client session Client for managing inference routes
mock_inference_route session Creates a mock OpenAI-protocol route for tests

Rust CLI E2E (e2e/rust/)

Rust-based e2e tests that exercise the openshell CLI binary as a subprocess. They live in the openshell-e2e crate and use a shared harness for sandbox lifecycle management, output parsing, and cleanup.

Suites:

  • Common suite (--features e2e) - driver-neutral CLI behavior, sandbox lifecycle, sync, port forwarding, policy, and provider tests.
  • CLI conformance (openshell-conformance) - the portable deployment smoke scenario plus focused tests for its reusable command runner.
  • Driver suites (--features e2e-docker, e2e-podman, e2e-kubernetes, or e2e-vm) - CLI conformance plus the common and driver-specific coverage for the selected deployment.
  • Docker suite (--features e2e-docker) - includes Docker-only coverage such as Dockerfile image builds, Docker preflight checks, and managed Docker gateway start.
  • Docker GPU suite (--features e2e-docker-gpu) - Docker suite plus GPU sandbox smoke coverage.
  • VM suite (--features e2e-vm) - runs e2e tests on a VM.
  • Kubernetes credential-driver suite (--features e2e-kubernetes-credential-drivers) - targeted Kubernetes Secrets and Vault provider credential storage coverage.

GPU device-selection tests compare OpenShell sandboxes against a plain Docker or Podman container that requests --device nvidia.com/gpu=all. The probe image defaults to the image used by the gateway stage in deploy/docker/Dockerfile.images; set OPENSHELL_E2E_GPU_PROBE_IMAGE to override it. Per-device checks run only for NVIDIA CDI device IDs reported by the runtime's discovered devices list, so WSL2 hosts that expose only nvidia.com/gpu=all skip the index-based cases. Exact CDI device selection is passed through --driver-config-json with the active Docker or Podman driver key.

Run the Docker-backed Rust CLI e2e suite:

mise run e2e:docker

Run the minimal portable CLI conformance profile against the gateway selected in your OpenShell CLI configuration:

mise run e2e:cli-conformance

The gateway must already be installed, reachable, and selected before the task starts. The task does not provision a gateway or select a compute driver. Set OPENSHELL_BIN to test a prebuilt CLI; otherwise, the task builds the CLI from the current checkout.

The phase-1 scenario verifies the complete CLI-to-gateway-to-driver path without depending on how the gateway was installed or which driver is configured. It requires machine-readable gRPC status, creates a uniquely named detached sandbox with --from base, verifies the sandbox is Ready by finding its unique name in paginated JSON list output, executes echo with a run-specific marker, deletes the sandbox, and verifies that its name no longer appears. Driver suites enable the same profile instead of maintaining a separate smoke implementation. Sandbox lifecycle, label matrices, VM overlay, and TLS-key permission assertions remain regular E2E coverage.

Each invocation prints a ten-character run ID before creating resources. Conformance sandboxes use names such as ct-<run-id>-01. The runner tracks the exact name and uses it for cleanup; phase 1 does not add ownership labels.

The runner deletes owned resources after both success and failure. If the test process is interrupted before cleanup, locate leftovers without touching unrelated gateway state:

openshell sandbox list --output json
openshell sandbox delete <sandbox-name>

Gateway-backed Rust E2E tasks build the standalone conformance CLI, run its registered scenarios against the configured gateway, then run any lane-specific Rust tests that still apply. Run the Podman-backed Rust CLI e2e suite:

mise run e2e:podman

Run the VM-backed Rust CLI e2e suite:

mise run e2e:vm

Run the targeted Kubernetes credential-driver e2e suite. This deploys an OpenBao fixture for the Vault-compatible driver path and validates Kubernetes Secrets and Vault storage backends one at a time:

mise run e2e:kubernetes:credential-drivers

Run a single test directly with cargo:

cargo test --manifest-path e2e/rust/Cargo.toml --features e2e --test sync

Run a single Docker-only test directly with cargo:

cargo test --manifest-path e2e/rust/Cargo.toml --features e2e-docker --test custom_image

The harness (e2e/rust/src/harness/) provides:

Module Purpose
binary Builds and resolves the openshell binary from the workspace
container Container-engine selection and support containers for proxy tests
gateway Managed gateway restart controls for gateway-owned e2e runs
sandbox SandboxGuard RAII type — creates sandboxes and deletes them on drop
output ANSI stripping and field extraction from CLI output
port wait_for_port() and find_free_port() for TCP testing

Environment Variables

Variable Purpose
OPENSHELL_GATEWAY Override active gateway name for E2E tests
OPENSHELL_GATEWAY_ENDPOINT Run E2E tests against an existing plaintext HTTP gateway endpoint
OPENSHELL_E2E_DRIVER Driver name exported by the e2e gateway wrapper (docker, podman, or vm)
OPENSHELL_E2E_CREDENTIAL_DRIVERS Enables the Kubernetes credential-driver fixture path in e2e/with-kube-gateway.sh