jgarciao · jgarciao · Aug 26, 2026
diff --git a/.agents/skills/debug-openshell-cluster/SKILL.md b/.agents/skills/debug-openshell-cluster/SKILL.md
@@ -71,6 +71,7 @@ Use gateway metadata, deployment values, or the user's setup notes to identify t
 | Docker | Gateway process logs, Docker daemon health, sandbox containers, image pulls. |
 | Podman | Podman socket, rootless networking, sandbox containers, image pulls. |
 | Kubernetes | Helm release, gateway workload, service, secrets, sandbox pods, events. |
+| OpenShift | Same as Kubernetes, plus SecurityContextConstraints (SCCs) and, for external access, an OpenShift `Route`. Detect OpenShift by the presence of the `route.openshift.io` API group (`oc api-resources --api-group=route.openshift.io`). |
 | VM | VM driver logs, rootfs availability, host virtualization support. |
 | Extension | External driver process, Unix socket ownership/mode, configured driver name, capability handshake, gateway logs. |
 
@@ -609,6 +610,8 @@ openshell logs <sandbox-name>
 | Kubernetes gateway pod pending | PVC unbound, taint, selector, or insufficient resources | `kubectl -n openshell describe pod <pod>` |
 | Kubernetes sandbox pod stuck pending, workspace PVC unbound | Cluster has no default `StorageClass` and OpenShell does not set `storageClassName` on the workspace PVC (clusters with a default `StorageClass` bind fine without it) | `kubectl -n openshell describe pvc`; set `server.workspaceStorageClass` (gateway config `workspace_storage_class`) to a valid `StorageClass` |
 | Kubernetes gateway pod crash loops | Missing secret, bad DB URL, bad TLS config | `kubectl -n openshell logs deployment/openshell -c openshell-gateway` or `kubectl -n openshell logs statefulset/openshell -c openshell-gateway` |
+| OpenShift gateway pod fails to start with an SCC/`runAsUser` error (e.g. `unable to validate against any security context constraint`) | Chart's default `podSecurityContext`/`securityContext` hardcodes `runAsUser`/`fsGroup`, which the restricted-v2 SCC rejects; it must instead inject the namespace-assigned UID/GID range | `oc -n openshell describe pod <pod>`; deploy with `podSecurityContext: null` and clear `securityContext.runAsUser` (see `deploy/helm/openshell/ci/values-openshift-scc.yaml`) |
+| OpenShift sandbox pod fails to start (`unable to validate against any security context constraint`) | The `openshell-sandbox` service account lacks the privileged SCC it needs | `oc adm policy add-scc-to-user privileged -z openshell-sandbox -n openshell`; remove with `remove-scc-from-user` when done |
 | CLI TLS error | Local mTLS bundle does not match server cert/CA | Check `~/.config/openshell/gateways/<name>/mtls/` |
 | Edge or OIDC gateway returns `Unauthenticated` | Stored login expired, audience/scopes mismatch, or gateway auth configuration changed | `openshell gateway info`, `openshell gateway login <name>`, gateway auth logs |
 | Gateway fails before serving health after enabling an interceptor | Interceptor endpoint unavailable or manifest/binding validation failed | Gateway and interceptor logs; interceptor socket; `binding_policy`, phases, and failure policy |

diff --git a/TESTING.md b/TESTING.md
@@ -189,6 +189,116 @@ Secrets and Vault storage backends one at a time:
 mise run e2e:kubernetes:credential-drivers
 ```
 
+### Kubernetes E2E (`e2e/rust/e2e-kubernetes.sh`)
+
+Kubernetes e2e tests deploy an OpenShell gateway into a real Kubernetes cluster
+via Helm, port-forward the gateway, and run the Rust e2e suite against it.
+
+Run with an ephemeral k3d cluster (macOS; created and torn down automatically):
+
+```shell
+mise run e2e:kubernetes
+```
+
+Target an existing cluster (kind, k3d, or OpenShift):
+
+```shell
+OPENSHELL_E2E_KUBE_CONTEXT=my-context mise run e2e:kubernetes
+```
+
+Scope to a single test for local debugging:
+
+```shell
+OPENSHELL_E2E_KUBE_TEST=smoke mise run e2e:kubernetes
+```
+
+**OpenShift**: when the target cluster exposes the `route.openshift.io` API
+group, the harness automatically applies SCC-compatible Helm overrides and
+grants the required SCCs. No extra flags or steps are needed.
+
+On a **remote** cluster, drop the `e2e-host-gateway` feature. Those tests rely
+on the sandbox-side `host.openshell.internal` alias reaching the machine running
+the tests, which is unreachable from pods on a remote cluster, so they fail.
+Left enabled, the `host_gateway_alias` suite fails because
+`host.openshell.internal` does not resolve inside the pod, so the gateway
+SSRF-denies the request (`DNS resolution failed` / `ssrf_denied`) — a networking
+property of remote pods, not a gateway or transport fault. Override
+`OPENSHELL_E2E_KUBERNETES_FEATURES` to exclude it:
+
+```shell
+OPENSHELL_E2E_KUBE_CONTEXT=$(oc config current-context) \
+  OPENSHELL_E2E_KUBERNETES_FEATURES="e2e,e2e-kubernetes" \
+  mise run e2e:kubernetes
+```
+
+On an existing cluster the harness builds the CLI from your branch but pulls the
+**published** gateway/supervisor image (default tag `latest`). The CLI and the
+image can therefore be different versions. If tests fail because of this version
+difference — for example, sandbox tests fail with `Pod exists with phase: Failed`
+or connect-based tests stall because the deployed image predates a feature your
+branch CLI needs — set `IMAGE_TAG` to an image that matches your branch.
+
+The `latest` tag lags to the last semver release, so it is often older than
+`main`. Two better choices:
+
+- `IMAGE_TAG=dev` — a floating tag that tracks the latest `main` build. Good for
+  an ad-hoc run when your branch is close to `main` HEAD. Because it floats, two
+  runs on different days can pull different images, so it is not reproducible.
+- **Pin the exact commit your branch is based on** — deterministic and immune to
+  a floating tag moving. Published tags are the full 40-char git SHA (semver tags
+  without a `v` prefix also exist but only for released versions):
+
+```shell
+OPENSHELL_E2E_KUBE_CONTEXT=$(oc config current-context) \
+  OPENSHELL_E2E_KUBERNETES_FEATURES="e2e,e2e-kubernetes" \
+  IMAGE_TAG=$(git rev-parse "$(git merge-base HEAD upstream/main)") \
+  mise run e2e:kubernetes
+```
+
+To pin a specific released version, use its semver tag without a `v` prefix
+(`0.0.115`, not `v0.0.115`):
+
+```shell
+OPENSHELL_E2E_KUBE_CONTEXT=$(oc config current-context) \
+  OPENSHELL_E2E_KUBERNETES_FEATURES="e2e,e2e-kubernetes" \
+  IMAGE_TAG=0.0.115 \
+  mise run e2e:kubernetes
+```
+
+A semver tag matches a released commit, which may be behind `main`; if your
+branch CLI needs a newer feature, pin the SHA of your branch's base instead.
+
+Confirm a tag exists before relying on it:
+`skopeo inspect docker://ghcr.io/nvidia/openshell/gateway:<tag>`.
+
+`IMAGE_TAG` sets only the gateway/supervisor image; the CLI under test is always
+built from your branch. To validate against images from your exact commit
+instead, build and push them and point `OPENSHELL_REGISTRY`/`IMAGE_TAG` at them.
+
+Available task variants:
+
+| Task | Purpose |
+|---|---|
+| `e2e:kubernetes` | Default Rust e2e against Helm-deployed gateway |
+| `e2e:kubernetes:db` | All database backend scenarios (SQLite + external PostgreSQL) |
+| `e2e:kubernetes:sidecar` | Supervisor sidecar topology overlay |
+| `e2e:kubernetes:credential-drivers` | Kubernetes Secrets and Vault credential storage |
+| `e2e:kubernetes:workspace-managed` | Managed workspace mode (auto-created namespaces) |
+| `e2e:kubernetes:workspace-operator` | Operator workspace mode (pre-provisioned namespaces) |
+| `e2e:kubernetes:v1alpha1` | Agent Sandbox v1alpha1 compatibility |
+| `e2e:kubernetes:external-driver` | External Kubernetes driver sidecar |
+
+Kubernetes e2e environment variables:
+
+| Variable | Purpose |
+|---|---|
+| `OPENSHELL_E2E_KUBE_CONTEXT` | kubectl context for an existing cluster (skips k3d creation) |
+| `OPENSHELL_E2E_KUBE_TEST` | Scope to a single test (e.g. `smoke`) |
+| `OPENSHELL_E2E_KUBE_EXTRA_VALUES` | Colon-separated additional Helm values files |
+| `OPENSHELL_E2E_KUBERNETES_FEATURES` | Cargo feature flags (default: `e2e,e2e-host-gateway,e2e-kubernetes`) |
+| `IMAGE_TAG` | Gateway/supervisor image tag (default: `latest` for existing clusters) |
+| `OPENSHELL_REGISTRY` | Image registry prefix (default: `ghcr.io/nvidia/openshell`) |
+
 Run a single test directly with cargo:
 
 ```shell
@@ -220,3 +330,5 @@ The harness (`e2e/rust/src/harness/`) provides:
 | `OPENSHELL_GATEWAY_ENDPOINT` | Run E2E tests against an existing plaintext HTTP gateway endpoint |
 | `OPENSHELL_E2E_DRIVER` | Driver name exported by the e2e gateway wrapper (`docker`, `podman`, or `vm`) |
 | `OPENSHELL_E2E_CREDENTIAL_DRIVERS` | Enables the Kubernetes credential-driver fixture path in `e2e/with-kube-gateway.sh` |
+| `OPENSHELL_E2E_KUBE_CONTEXT` | kubectl context for Kubernetes e2e (skips ephemeral k3d) |
+| `OPENSHELL_E2E_KUBE_TEST` | Scope Kubernetes e2e to a single test by name |
diff --git a/deploy/helm/openshell/ci/values-openshift-e2e.yaml b/deploy/helm/openshell/ci/values-openshift-e2e.yaml
@@ -0,0 +1,45 @@
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+# OpenShift overlay for the Kubernetes e2e harness.
+#
+# Bundles the OpenShift-specific settings the harness needs: the Route/mTLS
+# transport, and an image pull policy that avoids stale cached images.
+#
+# Route/mTLS transport: on OpenShift, `kubectl port-forward` stalls the SSH-relay
+# `sandbox connect` path (round-trip-heavy SSH over SPDY), so the harness drives
+# the gateway through a passthrough OpenShift Route with mTLS instead. This
+# overlay turns TLS back on (values-skaffold.yaml disables it), enables the Route,
+# and promotes the cert-verified caller to a dev principal.
+#
+# Image pull policy: force `Always` so runs against the `latest` upstream image
+# actually use it, instead of a stale copy cached on the cluster nodes.
+#
+# Layered by e2e/with-kube-gateway.sh AFTER ci/values-skaffold.yaml and
+# ci/values-openshift-scc.yaml when an OpenShift cluster is detected. The harness
+# supplies `openshiftRoute.host` and `pkiInitJob.serverDnsNames[0]` via --set at
+# install time (both are the cluster-derived Route hostname).
+#
+# Security: this is NOT an open gateway. `server.tls.clientCaSecretName` defaults
+# to `openshell-server-client-ca` and there is no OIDC, so `require_client_auth`
+# is true and mTLS is MANDATORY at the TLS handshake — a caller with only the
+# Route URL and no client certificate is rejected before any RPC. The passthrough
+# Route terminates TLS at the gateway pod, so this holds end-to-end.
+# `allowUnauthenticatedUsers` only promotes the already cert-verified caller to a
+# dev principal at the app layer (mtls_auth is unsupported with the Kubernetes
+# driver). Both are required together; the client certificate is the access gate.
+image:
+  pullPolicy: Always
+
+supervisor:
+  image:
+    pullPolicy: Always
+
+server:
+  disableTls: false
+  auth:
+    allowUnauthenticatedUsers: true
+
+openshiftRoute:
+  enabled: true
+  # host is supplied via --set at install time (cluster-derived Route hostname).
diff --git a/deploy/helm/openshell/ci/values-openshift-scc.yaml b/deploy/helm/openshell/ci/values-openshift-scc.yaml
@@ -0,0 +1,20 @@
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+# OpenShift SCC compatibility overlay. Removes the hardcoded runAsUser and
+# fsGroup so that OpenShift's restricted-v2 SCC can inject the namespace-
+# assigned UID/GID range. Layer after values.yaml:
+#   helm install openshell deploy/helm/openshell -f ci/values-openshift-scc.yaml
+#
+# The e2e Kubernetes harness applies this automatically when it detects an
+# OpenShift cluster (route.openshift.io API present).
+
+podSecurityContext: null
+
+securityContext:
+  runAsNonRoot: true
+  runAsUser: null
+  allowPrivilegeEscalation: false
+  capabilities:
+    drop:
+      - ALL
diff --git a/deploy/helm/openshell/templates/_gateway-workload.tpl b/deploy/helm/openshell/templates/_gateway-workload.tpl
@@ -34,8 +34,10 @@ spec:
         - host.docker.internal
         - host.openshell.internal
   {{- end }}
+  {{- with .Values.podSecurityContext }}
   securityContext:
-    {{- toYaml .Values.podSecurityContext | nindent 4 }}
+    {{- toYaml . | nindent 4 }}
+  {{- end }}
   containers:
     - name: openshell-gateway
       securityContext: