diff --git a/specs/index.spec.md b/specs/index.spec.md index b68a66af..b7703721 100644 --- a/specs/index.spec.md +++ b/specs/index.spec.md @@ -47,6 +47,7 @@ Machine-readable index for autonomous reconciliation (`/reconcile` skill). | `security/rbac-enforcement.spec.md` | security | User, Role, RoleBinding, RBAC middleware | API | data-model | | `standards/security/security.spec.md` | standards | - | ALL | - | | `platform/local-development.spec.md` | platform | Kind cluster, images, Make targets | ALL | cross-cutting, security | +| `platform/openshell-branch-build.spec.md` | platform | OpenShell branch builds, dev gateway provisioning | ALL | local-development, openshell-gateway | | `platform/oidc-integration.spec.md` | platform | API JWT validation, BFF OIDC session, IdP client config, Kind opt-in | API, WEB, CP | local-development, openshell-gateway-oidc, web-console/architecture | | `platform/e2e-testing.spec.md` | platform | Infra drivers, e2e test suite, CI workflow, deploy overlays | ALL | local-development, control-plane, openshell-gateway-routing | | `platform/api-server-observability.spec.md` | platform | API OTel SDK bootstrap, HTTP/gRPC server spans, W3C trace continuation, request metrics | API | web-console/tracing, security, local-development, e2e-testing | diff --git a/specs/platform/data-model.spec.md b/specs/platform/data-model.spec.md index 2130d776..dab2e9e1 100644 --- a/specs/platform/data-model.spec.md +++ b/specs/platform/data-model.spec.md @@ -179,12 +179,15 @@ A Gateway SHALL include provisioning configuration fields that the control plane |---|---|---| | `image` | string | Gateway container image reference (e.g., `ghcr.io/nvidia/openshell/gateway:21da343c9f838bd9ac85dc61bf44889de1a72873`) | | `supervisor_image` | string | Supervisor sidecar container image (default: `ghcr.io/nvidia/openshell/supervisor:0.0.109`) | +| `sandbox_image` | string | Sandbox base image the gateway uses when launching sandboxes (default: `ghcr.io/nvidia/openshell-community/sandboxes/base:latest`). See [`openshell-gateway.spec.md`](./openshell-gateway.spec.md) | | `server_dns_names` | string[] | DNS names for TLS certificate SANs | | `oidc` | JSONB | OIDC authentication config: `{issuer, audience, jwks_ttl, roles_claim, admin_role, user_role, scopes_claim}` | | `route` | JSONB | Route exposure config for GRPCRoute provisioning: `{host}` | | `route_address` | text | Read-only external address populated by the control plane (e.g., `grpcs://hostname:443`) | | `database` | JSONB | Database backend config: `{storageSize, image, externalSecretRef}` | | `credential_driver` | JSONB | Credential storage driver config: `{type, kubernetes_secrets, vault}`. See [`openshell-gateway-credentials.spec.md`](./openshell-gateway-credentials.spec.md) | +| `dev_build` | boolean | Marks this Gateway as a dev/branch build (default: false). Control plane copies to `hypershell.redhat.io/openshell-dev-build` label on K8s resources. See [`openshell-branch-build.spec.md`](./openshell-branch-build.spec.md) | +| `dev_build_metadata` | JSONB | Dev build provenance: `{ref, sha, repo}`. Control plane copies to annotations on K8s resources. See [`openshell-branch-build.spec.md`](./openshell-branch-build.spec.md) | See [`openshell-gateway.spec.md`](./openshell-gateway.spec.md) and its sub-specs for full provisioning details. diff --git a/specs/platform/local-development.spec.md b/specs/platform/local-development.spec.md index 9602c509..2fb966e6 100644 --- a/specs/platform/local-development.spec.md +++ b/specs/platform/local-development.spec.md @@ -783,6 +783,9 @@ The system SHALL deploy a Jaeger all-in-one instance in the local environment an | `CLOUD_PROVIDER_KIND_REPO` | (pinned in Makefile) | Git repository URL for cloud-provider-kind fork (BackendTLSPolicy + ALPN h2 support) | | `CLOUD_PROVIDER_KIND_REF` | (pinned in Makefile) | Exact commit SHA of the cloud-provider-kind fork to build (deterministic; idempotent-by-SHA rebuild) | | `CLOUD_PROVIDER_KIND_BRANCH` | (unset) | Optional testing override: build from a branch tip or arbitrary git ref instead of the pinned SHA; always rebuilds when set | +| `OPENSHELL_REPO` | (canonical upstream OpenShell repo) | Git repository URL to build OpenShell from for `make kind-openshell-up`; override to target a fork | +| `OPENSHELL_BRANCH` | (unset) | OpenShell git ref (branch, tag, or commit) to build gateway/supervisor/sandbox images from for `make kind-openshell-up` | +| `OPENSHELL_PR` | (unset) | Convenience for `make kind-openshell-up`: OpenShell pull request number, resolved to its head ref | | `KIND_RESTART_CPK` | (unset) | Set to `true` to force `make kind-up` to restart cloud-provider-kind (republishes ephemeral LB ports; otherwise the running instance is reused to keep ports stable) | | `CERT_MANAGER_VERSION` | `v1.21.1` | cert-manager release version | | `CNPG_VERSION` | `v1.30.0` | CloudNativePG operator release version | @@ -801,6 +804,7 @@ All targets operate on `KIND_NAMESPACE` (default: `hypershell-system`). | `make kind-teardown` | Destroy the Kind cluster + stop cloud-provider-kind + stop CoreDNS + flush port forwarding rules + revert resolver | | `make kind-status` | Show cluster info, pods, services, hostnames, DNS status, port forwarding status, and active component swaps | | `make kind-fix-ports` | Re-establish host port forwarding (443 + 8080) after a cloud-provider-kind restart; re-discovers ephemeral ports and re-runs the stop-then-start flush | +| `make kind-openshell-up` | Build OpenShell (gateway + supervisor + sandbox base) from `OPENSHELL_BRANCH`/`OPENSHELL_PR` + load into cluster (creating it if needed) + seed a dev-labeled gateway running those images. See [`openshell-branch-build.spec.md`](./openshell-branch-build.spec.md) | | `make kind-api-server-up` | Build api-server from working tree + load + replace deployment + wait (cluster must exist; idempotent - rebuilds and replaces on every call) | | `make kind-api-server-down` | Revert api-server to baseline image + restart + wait | | `make kind-control-plane-up` | Build control-plane from working tree + load + replace deployment + wait (cluster must exist; idempotent - rebuilds and replaces on every call) | diff --git a/specs/platform/openshell-branch-build.spec.md b/specs/platform/openshell-branch-build.spec.md new file mode 100644 index 00000000..a3715ca6 --- /dev/null +++ b/specs/platform/openshell-branch-build.spec.md @@ -0,0 +1,303 @@ +# OpenShell Branch Build Specification + +## Purpose + +HyperShell deploys and manages OpenShell gateways, but its local development and +test workflows always run against a pinned, published OpenShell release. This +spec defines a workflow that lets a developer stand up a Kind environment whose +gateways are built from an arbitrary OpenShell branch or pull request, so new +OpenShell changes can be validated end-to-end inside HyperShell before they are +released. A single entry point, `make kind-openshell-up`, checks out the +requested OpenShell source, builds gateway and supervisor images from it, loads +them into the Kind cluster, and provisions a distinctly named gateway +(`openshell-dev-gateway`) that runs those images. This gateway coexists with the +standard `dev-gateway` created by `make kind-up`. + +This feature is a local-development and testing capability. It reuses the +existing Kind environment ([`local-development.spec.md`](./local-development.spec.md)) +and gateway provisioning flow ([`openshell-gateway.spec.md`](./openshell-gateway.spec.md)); +it does not change how production gateways are provisioned. + +## Requirements + +### Requirement: Branch Build Entry Point + +The platform SHALL provide a `make kind-openshell-up` target that brings up a +local Kind environment whose OpenShell gateways are built from a caller-specified +OpenShell source ref instead of the pinned default images. + +The target SHALL accept the OpenShell source through configuration variables and +SHALL NOT require code changes to select a different branch, PR, or repository: + +- `OPENSHELL_BRANCH` - the git ref (branch name, tag, or commit) to build from. +- `OPENSHELL_PR` - a pull request number that resolves to `refs/pull//head`. + This ref exists only in the repository that received the PR (not in forks), + so `OPENSHELL_PR` SHALL only be used against the repository hosting the pull + request. When `OPENSHELL_REPO` points to a fork and `OPENSHELL_PR` is set, + the fetch SHALL fail with an actionable message directing the user to either + use `OPENSHELL_BRANCH` with the PR's head branch name or fetch from the + canonical upstream. +- `OPENSHELL_REPO` - the OpenShell repository URL, defaulting to + `https://github.com/NVIDIA/OpenShell.git`, so that forks and alternate + sources can be targeted. + +The target SHALL require exactly one source ref. When both `OPENSHELL_BRANCH` +and `OPENSHELL_PR` are set, `OPENSHELL_BRANCH` SHALL take precedence and +`OPENSHELL_PR` SHALL be ignored. When neither is set, the target SHALL fail +with an actionable message explaining how to supply a branch or PR. + +#### Scenario: Bring up a Kind environment from an OpenShell branch + +- GIVEN a developer with a valid `OPENSHELL_BRANCH` value +- WHEN they run `OPENSHELL_BRANCH=my-feature make kind-openshell-up` +- THEN the platform SHALL build OpenShell from `my-feature` +- AND SHALL bring up (or reuse) a Kind cluster running the branch-built gateway + +#### Scenario: Target an OpenShell pull request + +- GIVEN a developer who wants to validate OpenShell PR #123 +- WHEN they run `OPENSHELL_PR=123 make kind-openshell-up` +- THEN the platform SHALL resolve the PR head ref and build OpenShell from it + +#### Scenario: Missing source ref is rejected + +- GIVEN neither `OPENSHELL_BRANCH` nor `OPENSHELL_PR` is set +- WHEN the developer runs `make kind-openshell-up` +- THEN the target SHALL fail before building or mutating the cluster +- AND SHALL print how to supply a branch or PR + +#### Scenario: Both source refs are set + +- GIVEN both `OPENSHELL_BRANCH` and `OPENSHELL_PR` are set +- WHEN the developer runs `make kind-openshell-up` +- THEN the platform SHALL use `OPENSHELL_BRANCH` and ignore `OPENSHELL_PR` + +#### Scenario: PR ref against a fork is rejected + +- GIVEN `OPENSHELL_REPO` pointing to a fork and `OPENSHELL_PR` set +- WHEN the developer runs `make kind-openshell-up` +- THEN the target SHALL fail before fetching +- AND SHALL explain that PR refs only exist in the repository hosting the pull request + +#### Scenario: Override the OpenShell repository + +- GIVEN a developer building from a fork +- WHEN they run `OPENSHELL_REPO=https://github.com/acme/openshell.git OPENSHELL_BRANCH=wip make kind-openshell-up` +- THEN the platform SHALL fetch the source from the specified repository + +### Requirement: Deterministic Source Checkout + +The platform SHALL fetch the OpenShell source into an isolated, disposable +location and check out the requested ref without mutating the developer's working +tree, mirroring the external-source build pattern already used for +`cloud-provider-kind`. + +The platform SHALL resolve the requested ref to a concrete commit SHA, SHALL +record the resolved SHA, and SHALL rebuild whenever the requested ref resolves to +a commit different from the one currently deployed, so that re-running the target +against a moved branch tip picks up new commits. + +#### Scenario: Branch tip advances between runs + +- GIVEN a prior `kind-openshell-up` built commit `abc123` from branch `my-feature` +- AND new commits have since landed on `my-feature` +- WHEN the developer re-runs `OPENSHELL_BRANCH=my-feature make kind-openshell-up` +- THEN the platform SHALL fetch the current tip, resolve its SHA, and rebuild +- AND SHALL redeploy the gateway with the newly built images + +#### Scenario: Resolved commit is recorded + +- GIVEN a successful branch build +- WHEN the build completes +- THEN the platform SHALL record the concrete commit SHA it built from +- AND SHALL surface that SHA to the developer + +### Requirement: OpenShell Image Set Built From Source + +The platform SHALL build the gateway and supervisor images from the checked-out +OpenShell source using the repository's `docker-build-image.sh` script, which +produces the `gateway` and `supervisor` targets. The sandbox base image SHALL +use the published community image (`ghcr.io/nvidia/openshell-community/sandboxes/base:latest`) +since it is maintained in a separate repository (NVIDIA/OpenShell-Community) not +covered by this checkout. + +Each branch-built image SHALL be tagged with a dev tag that encodes the resolved +commit SHA (e.g., `gateway:dev-abc123`, `supervisor:dev-abc123`) so branch builds +are distinguishable from pinned releases and from one another. The platform SHALL +load the branch-built gateway and supervisor images into the Kind cluster. The +sandbox base image SHALL either be pre-pulled into Kind or pulled on first sandbox +launch. + +The image references written into the seeded Gateway resource SHALL be: +- `image`: `gateway:dev-` +- `supervisor_image`: `supervisor:dev-` +- `sandbox_image`: `ghcr.io/nvidia/openshell-community/sandboxes/base:latest` + +#### Scenario: Gateway and supervisor images are built and loaded + +- GIVEN a resolved OpenShell source checkout +- WHEN the platform builds OpenShell via `docker-build-image.sh` +- THEN it SHALL produce a gateway image and a supervisor image from that source +- AND SHALL tag each as `gateway:dev-` and `supervisor:dev-` +- AND SHALL load both into the Kind cluster so no registry pull is required + +#### Scenario: Sandbox base uses published community image + +- GIVEN a branch-built gateway +- WHEN the gateway is provisioned +- THEN its `sandbox_image` field SHALL reference the published community base image +- AND sandboxes launched by the gateway SHALL use that published image + +### Requirement: Branch-Built Images Wired Into Provisioning + +The seeded dev gateway SHALL be provisioned with the branch-built images rather +than the pinned defaults. The branch-built gateway and supervisor images SHALL +be set via the Gateway's `image` and `supervisor_image` fields directly. The +Gateway MUST NOT have a `release_id` set, because per `data-model.spec.md`, +when both `release_id` and `image` are present, `release_id` takes precedence +and the direct image references would be ignored. The `sandbox_image` field +SHALL reference the published community base image. + +The control plane SHALL reconcile the gateway workload so that the running +gateway container and supervisor sidecar use the branch-built images, and launched +sandboxes use the community base image. + +#### Scenario: Dev gateway runs branch-built images + +- GIVEN a completed OpenShell branch build with gateway and supervisor images + loaded into Kind +- WHEN the dev gateway is seeded and reconciled +- THEN the Gateway resource SHALL have no `release_id` field set +- AND the gateway container SHALL run `gateway:dev-` +- AND the supervisor sidecar SHALL run `supervisor:dev-` +- AND sandboxes launched by the gateway SHALL use the published community base + image + +### Requirement: Sandbox Image Configuration + +The Gateway's `sandbox_image` field (defined in [`openshell-gateway.spec.md`](./openshell-gateway.spec.md)) +SHALL be set to the published community base image for branch-built gateways, +since the sandbox base is maintained in a separate repository and cannot be built +from the OpenShell checkout. + +### Requirement: Dev Gateway Identity + +The gateway seeded by `kind-openshell-up` SHALL be identifiable as a dev/branch +build, distinguishing it from gateways created by the standard `kind-up` flow. + +The seeded Gateway resource SHALL be named `openshell-dev-gateway` (a stable, +DNS-valid name that allows update-or-create semantics on subsequent runs). The +control plane SHALL copy Gateway-level metadata onto the Kubernetes workload +resources it creates. The seeding flow SHALL set two new Gateway fields to mark +branch builds: + +- `dev_build`: boolean field set to `true` for branch-built gateways, unset + (or `false`) otherwise. The control plane SHALL copy this to a label + `hypershell.redhat.io/openshell-dev-build: "true"` on Deployment, Service, + and other K8s resources it provisions. +- `dev_build_metadata`: JSONB field carrying provenance: + `{ref: "", sha: "", repo: ""}`. The control plane + SHALL copy this to annotations `hypershell.redhat.io/openshell-dev-build-ref`, + `hypershell.redhat.io/openshell-dev-build-sha`, and + `hypershell.redhat.io/openshell-dev-build-repo` on the K8s resources. + +This survives reconcile because the control plane reads the Gateway fields and +re-applies them on every reconciliation. Gateways created by the standard +`kind-up` flow SHALL NOT have `dev_build` set. + +#### Scenario: Dev gateway has a stable name + +- GIVEN any branch build +- WHEN the dev gateway is seeded +- THEN its Gateway name SHALL be `openshell-dev-gateway` +- AND subsequent runs SHALL update the existing Gateway rather than creating duplicates + +#### Scenario: Dev-build label is propagated to K8s resources + +- GIVEN a Gateway with `dev_build: true` +- WHEN the control plane reconciles it +- THEN the Deployment, Service, and other K8s resources SHALL carry the label + `hypershell.redhat.io/openshell-dev-build: "true"` + +#### Scenario: Dev-build label is selectable + +- GIVEN a running branch-built gateway +- WHEN an operator queries workloads with the selector + `hypershell.redhat.io/openshell-dev-build=true` +- THEN the gateway's Kubernetes resources SHALL be returned + +#### Scenario: Provenance is recorded on the workload + +- GIVEN a branch-built gateway built from branch `my-feature` at commit `abc123` + from `https://github.com/NVIDIA/OpenShell.git` +- WHEN the operator inspects the deployed Deployment +- THEN annotations SHALL record: + - `hypershell.redhat.io/openshell-dev-build-ref: "my-feature"` + - `hypershell.redhat.io/openshell-dev-build-sha: "abc123"` + - `hypershell.redhat.io/openshell-dev-build-repo: "https://github.com/NVIDIA/OpenShell.git"` + +#### Scenario: Standard gateways are not labeled as dev builds + +- GIVEN a gateway created by the standard `make kind-up` flow (which does not set `dev_build`) +- WHEN an operator queries with `hypershell.redhat.io/openshell-dev-build=true` +- THEN that gateway's resources SHALL NOT be returned + +### Requirement: Coexistence With Standard Local Development + +`kind-openshell-up` SHALL integrate with the existing Kind environment: it SHALL +reuse a running cluster when present (including the full stack: API server, control +plane, Keycloak, and the standard `dev-gateway`), and otherwise create the complete +Kind environment before provisioning the branch-built gateway. The branch-built +gateway (`openshell-dev-gateway`) SHALL coexist with the standard `dev-gateway`, +allowing both to run simultaneously. Standard teardown (`kind-down`, `kind-teardown`) +SHALL remove both gateways. + +The standard `make kind-up` target SHALL remain unchanged and SHALL NOT invoke +branch build logic. + +#### Scenario: Reuse an existing cluster with full stack + +- GIVEN a Kind cluster already created by `make kind-up` with API server, control + plane, Keycloak, and `dev-gateway` running +- WHEN the developer runs `OPENSHELL_BRANCH=my-feature make kind-openshell-up` +- THEN the platform SHALL reuse the existing cluster and stack +- AND SHALL provision or update the `openshell-dev-gateway` alongside `dev-gateway` + +#### Scenario: Create full stack when cluster does not exist + +- GIVEN no Kind cluster exists +- WHEN the developer runs `OPENSHELL_BRANCH=my-feature make kind-openshell-up` +- THEN the platform SHALL create the Kind cluster with the full stack (API server, + control plane, Keycloak, `dev-gateway`) +- AND SHALL additionally provision the `openshell-dev-gateway` with branch-built images + +#### Scenario: Standard flow is unaffected + +- GIVEN a developer who runs `make kind-up` +- WHEN the environment comes up +- THEN only `dev-gateway` SHALL be created with the pinned default OpenShell images +- AND `openshell-dev-gateway` SHALL NOT be created +- AND no `dev_build` field SHALL be set on any Gateway + +### Requirement: Enablement for PR and Version Validation + +The branch build workflow SHALL enable HyperShell's dev and E2E test workflows to +exercise an unreleased OpenShell version. Tests SHALL be able to target the +branch-built gateway via its stable name (`openshell-dev-gateway`) or by querying +for Gateways with `dev_build: true`. The workflow SHALL surface a non-zero exit +status when the OpenShell source cannot be fetched or built so that failures are +not silently masked. + +#### Scenario: E2E run against a branch-built gateway + +- GIVEN a branch-built gateway named `openshell-dev-gateway` is running in the Kind cluster +- WHEN an E2E or dev workflow targets the gateway by name or queries for gateways + with `dev_build: true` +- THEN it SHALL exercise the branch-built OpenShell version + +#### Scenario: Build failure is surfaced + +- GIVEN an OpenShell ref that fails to fetch or build +- WHEN `make kind-openshell-up` runs +- THEN the target SHALL exit non-zero with an actionable error +- AND SHALL NOT seed a dev gateway that references missing images diff --git a/specs/platform/openshell-gateway.spec.md b/specs/platform/openshell-gateway.spec.md index f9b91924..4a1fa90c 100644 --- a/specs/platform/openshell-gateway.spec.md +++ b/specs/platform/openshell-gateway.spec.md @@ -373,7 +373,8 @@ The GatewayReconciler SHALL validate Gateway resource fields before applying K8s #### Scenario: Invalid image reference -- GIVEN a Gateway with an image reference containing invalid characters +- GIVEN a Gateway with an `image`, `supervisor_image`, or `sandbox_image` reference + containing invalid characters - WHEN the GatewayReconciler validates the configuration - THEN validation SHALL fail with a descriptive error - AND the Gateway SHALL not be reconciled until the configuration is corrected @@ -634,8 +635,8 @@ health_bind_address = "0.0.0.0:8081" metrics_bind_address = "0.0.0.0:9090" log_level = "info" sandbox_namespace = "" -default_image = "" -supervisor_image = "" +default_image = "" +supervisor_image = "" client_tls_secret_name = "openshell-client-tls" enable_loopback_service_http = true policy_validation_failure_mode = "fail_closed" @@ -788,6 +789,7 @@ Control Plane | `namespace` | No | API assigned | Read-only Kubernetes namespace derived from the Gateway identifier | | `image` | No | `ghcr.io/nvidia/openshell/gateway:0.0.109` | Gateway container image reference | | `supervisor_image` | No | `ghcr.io/nvidia/openshell/supervisor:0.0.109` | Supervisor sidecar container image | +| `sandbox_image` | No | `ghcr.io/nvidia/openshell-community/sandboxes/base:latest` | Sandbox base image the gateway uses when launching sandboxes | | `serverDnsNames` | Yes | - | DNS names for TLS certificate generation | | `oidc` | No | - | OIDC authentication configuration (see OIDC spec) | | `oidc.issuer` | Yes (to enable OIDC) | `""` | OIDC issuer URL; empty disables OIDC | @@ -800,6 +802,8 @@ Control Plane | `route` | No | - | Route configuration for external exposure | | `route.host` | No | auto-derived | Hostname for the GRPCRoute | | `routeAddress` | - | - | Read-only. External address populated by the control plane | +| `dev_build` | No | `false` | Marks this as a dev/branch build. Control plane applies `hypershell.redhat.io/openshell-dev-build` label to K8s resources | +| `dev_build_metadata` | No | - | Dev build provenance (JSONB): `{ref, sha, repo}`. Control plane copies to annotations on K8s resources | > **Database provisioning:** Gateway databases are provisioned automatically by the control plane using the CloudNativePG operator. The gateway's `database_id` field references a ManagedDatabase resource (provider=cnpg) that determines which CNPG Cluster hosts the gateway's logical database. When `database_id` is blank at creation time and the fleet has exactly one ManagedDatabase, the API server auto-assigns it. See [`openshell-gateway-database.spec.md`](./openshell-gateway-database.spec.md).