Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
df2de7c
refactor(config): normalize compute driver field names
jhjaggars Aug 19, 2026
c4e6f76
refactor(config): introduce canonical gateway fields
jhjaggars Aug 20, 2026
2250ff6
refactor(config): enforce gateway schema version 2
jhjaggars Sep 1, 2026
1d6a325
fix(config): preserve compute driver runtime guarantees
jhjaggars Sep 1, 2026
912951d
fix(config): address schema v2 review regressions
jhjaggars Sep 1, 2026
c2f2875
fix(config): complete schema v2 migration safeguards
jhjaggars Sep 2, 2026
fef7275
test(config): expand schema v2 regression coverage
jhjaggars Sep 2, 2026
d2d496e
test(config): add schema v2 parity manifest
jhjaggars Sep 2, 2026
dd636f7
fix(config): correct parity manifest inventory
jhjaggars Sep 2, 2026
de067b2
docs(config): record schema v2 intentional changes
jhjaggars Sep 2, 2026
3d1ff83
docs(config): disposition schema v2 parity gaps
jhjaggars Sep 2, 2026
d876531
test(e2e): add dual schema parity harness
jhjaggars Sep 2, 2026
1f8e937
test(e2e): establish compute lifecycle parity baseline
jhjaggars Sep 3, 2026
0ea6041
fix(config): preserve gateway option compatibility
jhjaggars Sep 3, 2026
eb5dfad
test(e2e): record gateway option parity
jhjaggars Sep 3, 2026
e49fc3a
docs(config): close gateway-wide parity gaps
jhjaggars Sep 3, 2026
4af9e5b
fix(podman): apply configured pids limit
jhjaggars Sep 3, 2026
d03ee98
test(e2e): validate Podman option parity
jhjaggars Sep 3, 2026
44ef387
test(e2e): add Kubernetes option parity harness
jhjaggars Sep 3, 2026
d6b70a2
test(e2e): record Kubernetes option parity
jhjaggars Sep 3, 2026
51f69d8
test(e2e): disposition VM parity lanes
jhjaggars Sep 3, 2026
31322df
test(e2e): add external driver parity lane
jhjaggars Sep 3, 2026
a7de50e
fix(e2e): preserve external driver pull policy
jhjaggars Sep 4, 2026
251cf44
test(e2e): attest parity artifacts and launches
jhjaggars Sep 4, 2026
51fe98a
test(e2e): require clean parity build sources
jhjaggars Sep 4, 2026
8711fbf
test(e2e): bind parity runtime artifacts
jhjaggars Sep 4, 2026
6896aec
fix(e2e): use isolated supervisor tags
jhjaggars Sep 4, 2026
1db9ba3
fix(e2e): qualify parity image tags
jhjaggars Sep 4, 2026
8ecfb3d
fix(e2e): serve parity supervisor locally
jhjaggars Sep 4, 2026
7e4655a
test(e2e): isolate parity podman services
jhjaggars Sep 4, 2026
0bc049c
test(e2e): harden parity evidence provenance
jhjaggars Sep 4, 2026
b829783
test(e2e): pin parity sandbox artifacts
jhjaggars Sep 4, 2026
7e40c2f
test(e2e): attest parity runtime inputs
jhjaggars Sep 4, 2026
4688e84
test(e2e): bind parity runtime evidence
jhjaggars Sep 4, 2026
d904600
test(e2e): record compute boundary parity
jhjaggars Sep 4, 2026
c44ec6a
test(e2e): disposition cross-cutting parity lanes
jhjaggars Sep 4, 2026
ae51165
fix(packaging): preflight gateway config upgrades
jhjaggars Sep 4, 2026
5486551
fix(config): preserve rebase integration guarantees
jhjaggars Sep 4, 2026
6e8b2b4
test(ci): isolate temporary git signing config
jhjaggars Sep 4, 2026
18e17c1
fix(config): update remaining schema v2 consumers
jhjaggars Sep 4, 2026
6c91583
fix(ci): provide e2fs tools to VM tests
jhjaggars Sep 4, 2026
e5ff58b
fix(config): align preflight with gateway startup
jhjaggars Sep 5, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion .agents/skills/test-release-canary/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,10 @@ does not contribute to product usage metrics.

`install.sh` defaults to the *latest tagged* release — the canary is therefore checking that the most recent public release still installs, not the just-published `dev` build. The `kubernetes` job is the exception: it pins to `0.0.0-dev` chart + `:dev` images.

The host-package jobs exercise fresh installs, not upgrades from a persisted
schema-v1 gateway config. Validate Homebrew and RPM exact-default migration with
the release-tooling and package lifecycle tests before relying on the canary.

The canary does not install or import `@nvidia/openshell-sdk`. TypeScript SDK
validation lives in the `TypeScript SDK` branch check, including a publish
dry-run. The tagged release workflow publishes the package to GitHub Packages;
Expand Down Expand Up @@ -131,7 +135,7 @@ Loopback registration auto-derives the gateway name to `openshell` if `--name` i
| Symptom | Likely cause | Where to look |
|---|---|---|
| `macos`/`ubuntu`/`fedora` job fails on `install.sh` | Latest tagged release missing an asset, checksum mismatch, or `install.sh` regression on this branch. | Job log around the `curl … install.sh \| sh` step. |
| `macos`/`ubuntu`/`fedora` job fails on `openshell status` | Local gateway service did not start (systemd/brew/podman). Often a driver issue. | Service logs in the job log; `OPENSHELL_DRIVERS` env in the "Ensure …" step. |
| `macos`/`ubuntu`/`fedora` job fails on `openshell status` | Local gateway service did not start (systemd/brew/podman). Often a driver issue. | Service logs in the job log; `OPENSHELL_COMPUTE_DRIVER` env in the "Ensure …" step. |
| `ubuntu-snap` fails after interface connection | The gateway did not recover after Docker became available, or did not become reachable within the 30-second bound. | Failure diagnostics dump Snap service/connection/change state, gateway and snapd journals, Snap logs, and port 17670 listeners. |
| `kubernetes` job fails on `helm install --wait` | Chart did not deploy in 5 min — usually image pull failure or readiness probe failing. | "Diagnostics on failure" step dumps `helm status`, manifest, pod describe, pod logs. |
| `kubernetes` job fails on `kubectl wait` | Gateway pod stuck `CrashLoopBackOff` or `ImagePullBackOff`. | Diagnostics dump; check `:dev` image existence at `ghcr.io/nvidia/openshell/gateway`. |
Expand Down
5 changes: 5 additions & 0 deletions .github/workflows/branch-checks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -231,6 +231,11 @@ jobs:
- name: Test
run: mise run test:python

- name: Test local gateway configuration helpers
run: |
bash tasks/scripts/test-gateway-pull-policy.sh
bash tasks/scripts/test-gateway-config.sh

go:
name: Go SDK
needs: pr_metadata
Expand Down
8 changes: 4 additions & 4 deletions .github/workflows/release-canary.yml
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ jobs:
steps:
- name: Ensure VM driver
run: |
launchctl setenv OPENSHELL_DRIVERS vm
launchctl setenv OPENSHELL_COMPUTE_DRIVER vm
launchctl setenv OPENSHELL_TELEMETRY_ENABLED "$OPENSHELL_TELEMETRY_ENABLED"

- name: Install and check status
Expand All @@ -53,7 +53,7 @@ jobs:
fi
sudo systemctl start docker || sudo service docker start
mkdir -p "${HOME}/.config/openshell"
printf 'OPENSHELL_DRIVERS=docker\nOPENSHELL_TELEMETRY_ENABLED=%s\n' \
printf 'OPENSHELL_COMPUTE_DRIVER=docker\nOPENSHELL_TELEMETRY_ENABLED=%s\n' \
"$OPENSHELL_TELEMETRY_ENABLED" > "${HOME}/.config/openshell/gateway.env"
docker info

Expand Down Expand Up @@ -145,7 +145,7 @@ jobs:
bash -s <<'EOF'
set -euo pipefail
mkdir -p "${HOME}/.config/openshell"
printf 'OPENSHELL_DRIVERS=podman\nOPENSHELL_TELEMETRY_ENABLED=%s\n' \
printf 'OPENSHELL_COMPUTE_DRIVER=podman\nOPENSHELL_TELEMETRY_ENABLED=%s\n' \
"$OPENSHELL_TELEMETRY_ENABLED" > "${HOME}/.config/openshell/gateway.env"
podman info
curl -LsSf "${INSTALL_SH_URL}" | sh
Expand Down Expand Up @@ -301,7 +301,7 @@ jobs:
run: |
set -euo pipefail
mkdir -p "${HOME}/.config/openshell"
printf 'OPENSHELL_DRIVERS=docker\n' > "${HOME}/.config/openshell/gateway.env"
printf 'OPENSHELL_COMPUTE_DRIVER=docker\n' > "${HOME}/.config/openshell/gateway.env"
curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/${{ github.event.workflow_run.head_sha || github.sha }}/install.sh | sh

- name: Register kind gateway and check status
Expand Down
2 changes: 2 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

51 changes: 37 additions & 14 deletions architecture/compute-runtimes.md
Original file line number Diff line number Diff line change
Expand Up @@ -127,11 +127,13 @@ defines the available implementation set, while the runtime consumes a generic
registry. Adding or removing a compiled driver therefore changes registration
rather than the server's selection flow. Alternate gateway binaries can install
their own `ComputeDriverFactory` registrations and hand the completed registry
to `run_cli_with_compute_drivers`; factories receive merged driver config and
return either an in-process driver or a gateway-managed remote endpoint. The
server constructs the common runtime adapter and snapshots `GetCapabilities`
for either result. A configured UDS endpoint still takes precedence over a
compiled registration with the same name.
to `run_cli_with_compute_drivers`. Factories expose the same side-effect-free
configuration validation to package preflight and runtime startup, receive only
the selected `[openshell.drivers.<name>]` table, and return either an in-process
driver or a gateway-managed remote endpoint when built. Preflight never builds a
driver or connects to its transport. The server constructs the common runtime
adapter and snapshots `GetCapabilities` for either result. A configured UDS
endpoint still takes precedence over a compiled registration with the same name.

The `openshell-gateway` composition crate groups first-party registrations
behind the `in-tree-compute-drivers` feature. `openshell-server` has no compute
Expand Down Expand Up @@ -251,7 +253,7 @@ delete, reconciliation removes the row; otherwise it can remain `Deleting`.
| Podman | Rootless or single-machine deployments. | Container plus nested sandbox namespace. | Uses the Podman REST API and CDI GPU devices when available. Delivers the supervisor via OCI image volume by default; falls back to extracting the binary to a host-side cache and bind-mounting it when `userns` is configured (overlay does not support idmapped mounts). Advertises the combined-supervisor policy-DNS and transparent-TCP substrate. |
| Kubernetes | Cluster deployment through Helm. | Pod plus nested sandbox namespace. | Uses Kubernetes API objects, service accounts, secrets, PVC-backed workspace storage, and GPU resources. |
| VM | Experimental microVM isolation. | Per-sandbox libkrun VM. | Managed endpoint-backed driver. The gateway spawns `openshell-driver-vm`, waits for its Unix socket, and then consumes it through the same remote `compute_driver.proto` path used by unmanaged endpoint drivers. The VM driver boots a cached bootstrap `rootfs.ext4`, prepares requested OCI images inside a bootstrap VM with `umoci`, attaches the prepared image disk read-only, and gives each sandbox a writable `overlay.ext4` for merged-root changes and runtime material. The driver persists each accepted launch request beside the overlay and restarts those VMs on driver startup without recreating the overlay. |
| Extension | Out-of-tree drivers operated alongside the gateway. | Whatever boundary the driver implements. | Selected by a custom `compute_drivers = ["<name>"]` entry with `[openshell.drivers.<name>].socket_path`, or at launch time by pairing `--drivers <name>` with `--compute-driver-socket=<path>`. A launch-time endpoint may use a canonical built-in name to preserve its driver-config key while replacing in-process construction. The gateway connects to an operator-provisioned UDS, snapshots `GetCapabilities`, and dispatches all sandbox lifecycle calls through `compute_driver.proto`. The driver process and socket lifecycle are operator-owned; the gateway does not spawn, supervise, or remove unmanaged extension drivers. The trust boundary is the socket's filesystem permissions: the operator must ensure only the gateway uid can read/write it. |
| Extension | Out-of-tree drivers operated alongside the gateway. | Whatever boundary the driver implements. | Selected by a custom `compute_driver = "<name>"` entry with `[openshell.drivers.<name>].socket_path`, or at launch time by pairing `--compute-driver <name>` with `--compute-driver-socket=<path>`. A launch-time endpoint may use a canonical built-in name to preserve its driver-config key while replacing in-process construction. The gateway connects to an operator-provisioned UDS, snapshots `GetCapabilities`, and dispatches all sandbox lifecycle calls through `compute_driver.proto`. The driver process and socket lifecycle are operator-owned; the gateway does not spawn, supervise, or remove unmanaged extension drivers. The trust boundary is the socket's filesystem permissions: the operator must ensure only the gateway uid can read/write it. |

Per-sandbox CPU and memory values currently enter the driver layer through
template resource limits. Docker and Podman apply them as runtime limits.
Expand Down Expand Up @@ -288,10 +290,27 @@ pinned dialing, relay behavior, and OCSF decisions. Docker and Podman advertise
they implement and validate the same complete contract. The capability marker
is driver-owned supervisor input and is removed from workload environments.

Kubernetes deployments may set an AppArmor profile on sandbox agent containers
through the driver configuration. The Helm chart defaults sandbox agents to
`Unconfined` so runtime/default AppArmor profiles do not block supervisor
network namespace setup on AppArmor-enabled nodes.
Kubernetes, Docker, and Podman share one AppArmor configuration model:
`RuntimeDefault`, `Unconfined`, or `Localhost/<profile>`. Each driver translates
that model to its native API and rejects an explicitly requested confined
profile when its backend reports AppArmor unavailable. Docker keeps its
historical explicit `Unconfined` default. Podman sends no override when the
field is omitted, preserving the runtime-selected profile; development paths
that require the supervisor's namespace mount setup opt into `Unconfined`
explicitly. The Helm chart independently uses `Unconfined` for Kubernetes.

Corporate proxy settings are driver-owned supervisor inputs. Docker, Podman,
and VM propagate `https_proxy`, `no_proxy`, an optional root-only auth file,
and the explicit cleartext-Basic-auth acknowledgement without allowing
workload environment to override them. Podman and VM can also project an
operator CA bundle for an HTTPS or TLS-intercepting proxy. The VM driver validates and
stages its credential and CA bundle under fixed guest paths, then forwards
those paths through the protected supervisor argument file rather than the
guest environment. Local containers project provider SPIFFE
through a dedicated host UNIX-socket parent mount. A VM cannot safely expose
that host socket: it accepts only a separately operated, concrete TCP listener
when `provider_spiffe_allow_guest_tcp = true` explicitly acknowledges guest
access. Host-only sockets are never implicitly forwarded to VM guests.

The Kubernetes deployment packaging has two ownership boundaries. The gateway
chart owns the gateway workload, configuration, Services, PKI, and
Expand All @@ -309,10 +328,14 @@ can request a specific number of GPUs or the driver-specific default behaviour.
For all in-tree drivers, this is equivalent to selecting a single GPU.

VM runtime state paths are derived only from driver-validated sandbox IDs
matching `[A-Za-z0-9._-]{1,128}`. The gateway-owned VM driver socket uses a
private `run/` directory plus Unix peer UID/PID checks. Standalone
unauthenticated TCP mode is disabled unless explicitly enabled for local
development.
matching `[A-Za-z0-9._-]{1,128}`. Each writable overlay records its effective
sandbox UID/GID so later rootfs cache changes cannot rewrite persisted file
ownership. Unmarked pre-migration overlays recover identity from concrete
overlay or prepared-rootfs state, an explicit operator override, or the current
image account. The driver never assumes `10001:10001`; it preserves that legacy
identity only when persisted state reports it. The gateway-owned VM driver
socket uses a private `run/` directory plus Unix peer UID/PID checks. Standalone unauthenticated TCP
mode is disabled unless explicitly enabled for local development.

Runtime-specific implementation notes belong in the driver crate README:

Expand Down
64 changes: 38 additions & 26 deletions architecture/gateway.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,20 @@ immediately without a grace period. Finalization is persisted separately from
the exit result; the gateway deletes an ephemeral sandbox only after the
finalized supervisor session disconnects.

## Configuration Boundary

The gateway accepts exactly schema version 2. Missing, legacy, and future
versions fail before runtime construction, and driver settings belong only to
`[openshell.drivers.<name>]`. The process does not migrate legacy files.
Package lifecycle code may replace an exact package-generated v1 default, but
it preserves edited configurations for explicit operator migration.

Gateway listener TLS and sandbox callback TLS are separate inputs. A selected
local Docker, Podman, or VM driver requires a complete guest bundle whenever
the gateway listener uses TLS; package-managed local TLS can supply that bundle.
Kubernetes instead projects guest credentials through its configured Secret.
The gateway validates this requirement before constructing the selected driver.

## Protocol and Auth

The gateway listens on one service port and multiplexes gRPC and HTTP traffic.
Expand Down Expand Up @@ -246,10 +260,10 @@ controllers and `agents.x-k8s.io/v1alpha1` ownerReferences from existing
deployments. Supervisors renew gateway JWTs in memory before expiry only while
the sandbox record still exists. Older tokens are not server-revoked; shared
deployments bound replay exposure with short `gateway_jwt.ttl_secs` lifetimes.
The config default is
`gateway_jwt.ttl_secs = 0` for local single-player Docker, Podman, and VM
gateways; those tokens carry `exp = 0` and do not expire. Kubernetes and other
shared deployments should set a positive TTL.
Omitting `gateway_jwt.ttl_secs` selects non-expiring tokens for local
single-player Docker, Podman, and VM gateways; those tokens carry `exp = 0`.
Kubernetes and other shared deployments should set a positive TTL. Explicit
zero is rejected.

Gateway JWT signing-key rotation is currently an offline operator action. The
runtime loads one active signing key and one matching public verification key
Expand Down Expand Up @@ -690,9 +704,10 @@ Gateway CLI flag > gateway OPENSHELL_* env var > TOML file > built-in defa
```

The TOML file is opt-in via `--config <PATH>` / `OPENSHELL_GATEWAY_CONFIG`.
Driver implementation settings live in the TOML driver tables. See
`docs/reference/gateway-config.mdx` for worked per-driver examples and RFC
0003 for the full schema.
Driver implementation settings live exclusively in TOML driver tables. The
selector is the singular `[openshell.gateway] compute_driver`; legacy
`compute_drivers` lists are rejected. See `docs/reference/gateway-config.mdx`
for worked per-driver examples and RFC 0003 for the full schema.

Each installation has an operator-assigned gateway name. Configure it with
`[openshell.gateway].name`, `--name`, or `OPENSHELL_GATEWAY_NAME`.
Expand All @@ -706,26 +721,20 @@ aliases, network names, and the sandbox JWT issuer.
`database_url` is env-only and rejected when present in the file
(`OPENSHELL_DB_URL` / `--db-url`).

### Driver inheritance

`[openshell.gateway]` carries a small set of values (`sandbox_namespace`,
`default_image`,
`supervisor_image`, `guest_tls_ca/cert/key`, `client_tls_secret_name`,
`host_gateway_ip`, `enable_user_namespaces`) that are inherited into each
driver's `[openshell.drivers.<name>]` table when the driver-specific table
does not override them. The allowlist is per-driver so a gateway-wide
default cannot land in a driver that does not understand it (e.g.
`client_tls_secret_name` is K8s-only).
### Driver ownership

`image_pull_policy` is intentionally **not** inheritable: Kubernetes uses
`Always | IfNotPresent | Never` (passed verbatim to the K8s API) while
Podman uses the lowercase enum `always | missing | never | newer`. No
value means the same thing in both, so the key lives only under each
driver's own table.
`[openshell.gateway]` contains gateway process settings only. Each selected
driver reads its own configuration exclusively from
`[openshell.drivers.<name>]`; values are never inherited from gateway scope.
Kubernetes owns `namespace`, `default_image`, `supervisor_image`,
`client_tls_secret_name`, `service_account_name`, `host_gateway_ip`,
`enable_user_namespaces`, and `sa_token_ttl_secs`. Docker uses
`sandbox_label` instead of the legacy `sandbox_namespace` name. Podman and VM
likewise own their image, endpoint, and runtime settings in their tables.

Driver-specific values that are not part of the inheritance allowlist
(e.g. Podman `socket_path`, VM `vcpus`) only come from the driver's own
table.
`image_pull_policy` uses the shared canonical vocabulary
`always | if_not_present | never | newer`. Drivers translate it to their runtime
APIs; `newer` is supported only by Podman and rejected by Docker and Kubernetes.

### OTLP export

Expand Down Expand Up @@ -799,7 +808,10 @@ system entry instead of pretending to delete package-manager owned state.
- Gateway TLS and client certificate distribution are deployment concerns owned
by the operator or packaging layer.
- Compute runtimes own the mechanics of starting workloads and injecting
callback configuration.
callback configuration. Local Docker, Podman, and VM callback endpoints can
be derived from their fixed host aliases. Kubernetes requires an explicit
endpoint from deployment topology; Helm renders it from the gateway Service
name and namespace rather than inferring it from sandbox placement.
- Docker-backed local gateways use Docker's `host-gateway` callback alias on
macOS and Docker Desktop-style runtimes. They request IPv4 loopback callback
reachability and add a listener only when the primary does not cover it.
Expand Down
Loading
Loading