refactor(config): normalize and enforce gateway schema v2

[jhjaggars]

Summary

Normalize the gateway configuration surface and enforce strict schema version 2 for gateway, compute-driver, Helm, and package-managed configurations. The change gives each setting one canonical name and owner, preserves current runtime and security guarantees, documents the breaking migration, and adds deterministic plus paired live capability-parity validation against the frozen schema-v1 baseline.

Related Issue

Closes #2792

Changes

Schema and ownership

• Require explicit [openshell] version = 2; reject omitted, version 1, and unsupported future schemas with migration-focused diagnostics.
• Replace plural driver selection with compute_driver, --compute-driver, and OPENSHELL_COMPUTE_DRIVER.
• Retain narrow OPENSHELL_DRIVERS compatibility for one normalized driver when the canonical selector is absent; reject comma-delimited values and conflicting selectors. Continue to reject legacy --driver and --drivers CLI flags.
• Remove gateway-to-driver field inheritance. Driver-owned settings now live under [openshell.drivers.<name>]; gateway-owned guest TLS paths remain under [openshell.gateway] and are injected only into the selected local driver.
• Rename Docker sandbox_namespace to sandbox_label, Podman sandbox_ssh_socket_path to ssh_socket_path, and VM openshell_endpoint to grpc_endpoint.
• Replace sentinel-zero configuration with typed optional values where omission has distinct semantics. Explicit zero remains invalid for PID and health-check limits.
• Centralize shared image-pull, AppArmor, proxy, and local-driver configuration models while preserving backend-specific validation and translation.
• Keep server configuration backend-independent and construct built-in drivers in openshell-gateway through the current registration architecture.

Runtime and security behavior

• Preserve the 2048-process default guard when Docker or Podman PID limits are omitted.
• Derive default sandbox callback endpoints by runtime topology while retaining explicit driver overrides.
• Preserve VM image-account metadata, record persisted overlay UID/GID identity, and recover older overlay identity from persisted evidence instead of applying an unsafe blanket fallback.
• Add Docker and VM corporate-proxy propagation, including validated credential handling and VM proxy CA-bundle staging through protected guest overlay paths rather than workload-controlled environment variables.
• Add Docker-compatible read-only provider-SPIFFE socket projection and require explicit acknowledgement before exposing a guest-reachable SPIFFE TCP listener to VMs.
• Preserve QEMU proxy reachability rejection, VM registry retries, protected proxy delivery, and supervisor callback readiness behavior from current main.
• Add Docker, Podman, and Kubernetes AppArmor configuration with backend capability checks and fail-closed handling.
• Keep Podman health checks optional without weakening public readiness: sandboxes remain Provisioning with SupervisorNotConnected until the supervisor control session connects.

Packaging and migration

• Add openshell-gateway config preflight [--path PATH] for read-only, source-free validation before package-managed startup.
• Add Debian systemd ExecStartPre validation and fail-closed Snap configuration selection while preserving CLI --config and OPENSHELL_GATEWAY_CONFIG precedence.
• Preserve operator-edited configuration during package upgrades. Legacy or malformed Debian and Snap files are not rewritten without trustworthy package-default provenance.
• Keep the frozen RPM schema-v1 default and its migration contract unchanged.
• Update Helm rendering, packaged defaults, deployment scripts, architecture documentation, driver READMEs, diagnostics skills, and the published gateway configuration reference.

Capability-parity validation

• Add a paired baseline/candidate harness with isolated state, ports, databases, PKI, sockets, networks, container stores, and immutable source/artifact attestations.
• Require digest-pinned images, exact-source staged binaries, clean build sources, independent external-driver artifacts, and SHA-256-bound evidence.
• Record separate capability inventory, intentional-change ledger, gap dispositions, raw evidence, normalized comparisons, and final results.
• Record 30 final capability dispositions: 4 pass, 2 intentional changes, 24 platform-blocked, and 0 regressions.
  • Pass: rootless Podman lifecycle, gateway-wide options and singleton-selector compatibility, Kubernetes core options, and in-tree/external-UDS compute-driver boundary behavior.
  • Intentional changes: derived gateway TLS client-auth policy and normalized Podman option semantics.
  • Platform-blocked results remain explicit for Docker, qualified Podman/Kubernetes security lanes, VM, MXC, cross-cutting auth/observability/provider fixtures, and real RPM/Debian/Snap/Homebrew upgrade lanes. Deterministic checks are not promoted to live parity.
• Preserve immutable execution SHAs after rebasing; successful result IDs are explicitly pinned to the original evidence-producing commits rather than incorrectly requiring ancestry from the rewritten tip.

The migration procedure and schema-v2 examples are in docs/reference/gateway-config.mdx. The complete validation campaign report is available at https://gist.github.com/jhjaggars/d890ce0d377d8fc5187f61aa390bbf5c.

Breaking migration notes

• Existing files must add [openshell] version = 2 and use the singular compute_driver selector.
• Move driver settings out of [openshell.gateway] and into the selected [openshell.drivers.<name>] table. Keep only guest_tls_ca, guest_tls_cert, and guest_tls_key at gateway scope.
• Apply the Docker, Podman, and VM field renames listed above; stale aliases are rejected by the strict schema.
• Raw gateway TOML uses pull policies always, if_not_present, never, or Podman-only newer. Existing Helm values may retain Kubernetes spellings because the chart normalizes them.
• Omit JWT TTL to disable expiry, omit Docker or Podman PID limits to use the 2048-process default, and omit Podman health-check intervals to disable backend probes.
• Run openshell-gateway config preflight --path <gateway.toml> before restarting a package-managed gateway. The command validates but never rewrites the file.

Testing

• mise run pre-commit passed after the final rebase integration fixes.
• mise run test passed after rebasing onto current origin/main.
• Unit, strict-schema, migration, Helm, packaging-asset, and parity-verifier tests added or updated.
• Focused VM driver library tests passed (177 tests), gateway VM tests passed (22 tests), and the VM corporate-proxy E2E target compiled.
• Rootless Podman paired lifecycle and option validation completed.
• Guarded disposable Kubernetes core-option validation completed and campaign-owned resources were removed.
• In-tree and external-UDS compute-boundary validation completed with artifact provenance checks.
• Final live-results and temporary-Git-fixture tests passed (21 tests).
• Terra security and architecture review found no actionable findings after the rebase integration fixes.
• Docker live validation requires a Docker CLI and daemon; the Docker driver correctly rejects the available Libpod socket.
• Qualified Kubernetes, VM, MXC, cross-cutting service, and real package-upgrade lanes require their assigned infrastructure and fixtures; each remains platform_blocked rather than being reported as parity.

Checklist

• Follows Conventional Commits
• All 41 commits are signed and include DCO sign-offs
• Architecture docs updated
• User-facing and deployment documentation updated
• Breaking changes and migration steps documented
• Frozen migration artifacts preserved

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[jhjaggars]

Review concern dispositions and completed follow-up

The hard schema-v2 cut remains intentional: OpenShell does not silently accept v1 configuration syntax. Compatibility is preserved at explicit operational boundaries where an upgrade must not break runtime behavior, persisted state, or stable Helm values.

Concern	Final disposition	Resolution
Existing Homebrew and RPM v1 files fail after upgrade	Intentional compatibility break	Keep strict schema-v2 validation and migration-focused startup errors rather than adding a general v1 migration shim.
Docker provider-SPIFFE bind uses unsupported rbind	Fixed	Project the socket's parent directory with Docker's supported read-only bind syntax and assert the exact bind string in tests.
Podman AppArmor capability is silently read as false	Fixed	Deserialize Podman's camelCase apparmorEnabled field and test confined profiles against both available and unavailable capability reports.
Removed OPENSHELL_DRIVERS is silently ignored	Fixed	Reject the variable whenever it is present, including an empty value, and direct users to OPENSHELL_COMPUTE_DRIVER.
Omitted PID limit removes the 2048 fork-bomb guard	Fixed	Omission retains OpenShell's 2048-process default for Docker and Podman; documentation no longer calls this the container runtime default.
Podman readiness can bypass or hang when health checks are omitted	No readiness code change required; documentation fixed	Backend health checks remain optional. Gateway composition keeps the public sandbox phase in Provisioning with SupervisorNotConnected until a supervisor session registers, so omission cannot make a sandbox publicly Ready early.
VM legacy UID 10001 and image-provided accounts are not preserved	Fixed	Store exact UID/GID in a v2 overlay marker, recover unmarked state from the persisted prepared rootfs when possible, use explicit configuration before the legacy fallback, and preserve image-owned account metadata while reconciling IDs.
Kubernetes/Helm pull-policy capitalization is rejected	Fixed at the Helm boundary	Raw schema-v2 TOML stays canonical. Helm accepts stable legacy values Always, IfNotPresent, and Never, translates them to canonical values, and fails rendering for unsupported policies.
Guest TLS documentation points at rejected driver tables	Fixed	Current-schema guidance keeps guest TLS fields under [openshell.gateway].
Kubernetes full example omits required callback configuration	Fixed	The raw TOML example now includes required grpc_endpoint and explains that Helm derives it from Service topology.
VM legacy-field rejection test can pass for the wrong reason	Fixed	Keep valid grpc_endpoint input while injecting openshell_endpoint, then assert the error names that unknown legacy field.
Broad prerelease release-script changes appear in the branch history	No action	Those changes come from mainline commit c8f13205; this PR's release-related edits are limited to schema-v2 packaging/config snippets.

Verification completed with targeted driver/server tests, Helm tests, mise run pre-commit, mise run test, and mise run ci. Podman E2E passed all applicable suites; only the environment-specific keep-id case was blocked because UID 16446 is not present in the host passwd database. Docker and VM runtime E2E were unavailable because this host has no Docker daemon or /dev/kvm.

[jhjaggars]

Schema-v2 follow-up completed

I implemented the remaining review findings while preserving the strict schema-v2 boundary.

Area	Resolution
Current main	Merged current main, resolved the openshell-gateway crate move, and retained the new factory/config ownership boundaries.
Homebrew upgrades	Formula startup rewrites only the exact empty or IPv6 schema-v1 config generated by prior formulas. Edited prefix and user configs remain untouched.
RPM upgrades	The user service now invokes a packaged migrator before startup. It seeds missing config, replaces only the byte-for-byte historical v1 default, rejects non-regular destinations, and preserves edited files.
Podman AppArmor	Omission now sends no override and preserves Podman's runtime-selected profile. Docker retains its explicit Unconfined default. The Podman development script opts into Unconfined explicitly.
Development pull policies	Docker, Podman, and generic gateway scripts normalize supported legacy/case variants such as IfNotPresent at the script boundary; schema-v2 TOML remains strict.
Guest TLS	Complete gateway-owned guest bundles are validated and injected before constructing local Docker, Podman, or VM drivers. TLS-enabled local drivers fail early without a bundle. Kubernetes continues to project TLS through its Secret and Helm does not render host guest-key paths.
JWT lifetime	SandboxJwtIssuer now carries Option<Duration> directly. None is non-expiring, Some(Duration::ZERO) is rejected, and extension credentials always use a finite bounded TTL.
VM gateway identity	[openshell.drivers.vm].sandbox_uid and sandbox_gid validate at the gateway and are forwarded to the VM subprocess.
VM persisted identity	Fresh identity markers are atomically installed before overlay creation. Restore prefers a validated marker, overlay upper-layer passwd, persisted prepared rootfs, explicit config, then the current image. Missing evidence fails safely; the driver never guesses 10001:10001, and ownership failures no longer fall back silently.
Documentation/RFC	Updated package docs, public references, architecture, driver READMEs, debugging/release skills, and RFC 0003 to match implemented schema-v2 behavior.

Verification

• mise run pre-commit
• mise run test
• mise run ci
• mise run helm:test — 131 tests passed
• mise run test:packaging-assets
• Homebrew/RPM migration pytest suite — 8 tests passed
• cargo test -p openshell-driver-podman — 221 tests passed
• cargo test -p openshell-driver-vm — 173 tests passed
• Focused gateway TLS/VM, server TLS/JWT/auth, Docker AppArmor, pull-policy, and shell syntax checks

The branch is now synchronized and pushed at a4acfb1c.

[johntmyers]

gator-agent

PR Review Status

Thanks @jhjaggars. I checked the schema-v2 follow-up and the compatibility boundaries you described. The core migration work is well covered, but two repository-owned operator paths remain inconsistent with the implemented runtime contract.

Action required: update the test-guest configuration producer and correct the compute-driver compatibility guidance, then push a new head for a focused follow-up review.

Blocking findings:

• GATOR-6e8b2b41-01: the supported test-guest provisioner still emits a gateway config rejected by schema v2
• GATOR-6e8b2b41-02: the debugging skill says a legacy environment alias is rejected even though this head still honors it

Carried findings:

• None

Gator metadata

• Validation: Project-valid implementation of accepted issue #2792, including the documented schema-v2 migration and compute-driver normalization.
• Docs: Fern documentation is updated; the installable debugging skill has one blocking runtime-contract contradiction.
• Checks: DCO passes; Branch Checks and Helm Lint are waiting for the current-head copy-pr mirror.
• E2E: test:e2e is required for the gateway, driver, policy, proxy, credential, and lifecycle changes, but dispatch waits until review blockers are resolved.
• Head SHA: 6e8b2b41bd0cf5b09f8da0e51863aeb9fff6029b
• Base SHA: 48a8a4bf09824e9d832ac844c76ce5dfcb8c9b0b
• Merge base SHA: c93b2fa7da3a40a6dd6c077973cb13d2755f7e50
• Patch ID: dc73b48cc8186a992c56477c27fd03e3c8deb984
• Gator payload: 8
• Review mode: initial
• Previous reviewed SHA: none
• Review budget exhausted: no
• Maintainer decision required: no
• Next state: gator:in-review

[jhjaggars]

Review follow-up

Addressed the current-head findings in 18e17c10:

• GATOR-6e8b2b41-01: updated the rootless-Podman test-guest producer to schema v2 (version = 2, singular compute_driver, omitted JWT TTL) and added deterministic coverage for the generated TOML.
• GATOR-6e8b2b41-02: corrected debug-openshell-cluster to document the deprecated singleton OPENSHELL_DRIVERS compatibility alias and its fail-closed invalid/conflicting cases.
• Zizmor unpinned-images: removed the newly added containerized gateway-config job and moved its checks into the existing Python job, so this PR no longer introduces another floating CI image reference.

Verification:

• uv run pytest -q python/openshell/gateway_config_fixture_test.py — 5 passed
• bash tasks/scripts/test-gateway-pull-policy.sh — passed
• bash tasks/scripts/test-gateway-config.sh — passed
• Rendered test-guest TOML passed openshell-gateway config preflight
• Actionlint passed
• mise run pre-commit passed

[github-actions]

Label test:e2e applied, but pull-request/2814 does not exist yet. A maintainer needs to comment /ok to test 18e17c101c64d69e51048fd5234104665dd800dd to mirror this PR. Once the mirror exists, re-apply the label or re-run Branch E2E Checks from the Actions tab.

[johntmyers]

/ok to test 18e17c1

[github-actions]

Label test:e2e applied for 18e17c1. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

[johntmyers]

gator-agent

Re-check After Author Update

Thanks @jhjaggars. I reviewed current head 18e17c101c64d69e51048fd5234104665dd800dd after your September 4 follow-up explaining the test-guest schema-v2 fix, the corrected OPENSHELL_DRIVERS guidance, and the CI-job consolidation.

What I checked: the focused delta from the prior reviewed head, both durable Gator findings, the new gateway-config fixture coverage, and the current-head workflow dispatch. Both findings are resolved, and the independent follow-up review found no new blockers.

Disposition: resolved. No blocking items remain, and current-head Branch Checks, Helm Lint, and E2E are now queued or running.

Blocking findings:

• No blocking findings remain

Carried findings:

• None

Gator metadata

• Validation: Project-valid implementation of accepted issue refactor: clean up gateway.toml config schema inconsistencies #2792 and its documented schema-v2 migration.
• Docs: Fern and operator guidance are updated; the corrected debugging skill now matches the retained compatibility behavior.
• Checks: Current-head Branch Checks and Helm Lint are queued/running; an earlier DCO job failed on a GitHub repository-contents HTTP 500 and will be monitored as infrastructure rather than treated as an author defect.
• E2E: test:e2e applied; /ok to test 18e17c101c64d69e51048fd5234104665dd800dd created the exact-head mirror, and Branch E2E is running.
• Head SHA: 18e17c101c64d69e51048fd5234104665dd800dd
• Base SHA: 48a8a4bf09824e9d832ac844c76ce5dfcb8c9b0b
• Merge base SHA: c93b2fa7da3a40a6dd6c077973cb13d2755f7e50
• Patch ID: c79604dae6bd59e3da2e81422317bbc40819231f
• Gator payload: 8
• Review mode: follow_up
• Previous reviewed SHA: 6e8b2b41bd0cf5b09f8da0e51863aeb9fff6029b
• Review budget exhausted: no
• Maintainer decision required: no
• Next state: gator:watch-pipeline

[jhjaggars]

Darwin ARM test follow-up

Fixed the aarch64-Darwin VM identity-test failure in 6c91583c by adding e2fsprogs to the shared Nix development shell. The five failing tests create ext4 fixtures and then inspect /etc/passwd with debugfs; the Darwin shell had a formatter available but did not expose debugfs.

This keeps the identity recovery and fail-closed behavior unchanged while making mke2fs, mkfs.ext4, and debugfs explicit test dependencies on every CI architecture.

Local verification:

• cargo test -p openshell-driver-vm --lib — 177 passed
• mise run pre-commit — passed
• Terra security/architecture review — no findings

The aarch64-Darwin CI lane remains the authoritative platform validation.