From 5f0ed59eaa98c45a53d48be4e72d516ddf392b25 Mon Sep 17 00:00:00 2001 From: Kyle Squizzato Date: Fri, 4 Sep 2026 16:27:01 -0700 Subject: [PATCH] docs(specs): Add ephemeral PR env, update e2e and dev specs Signed-off-by: Kyle Squizzato --- specs/platform/e2e-testing.spec.md | 68 +- .../ephemeral-pr-environments.spec.md | 803 ++++++++++++++++++ specs/platform/openshift-development.spec.md | 412 +++------ 3 files changed, 953 insertions(+), 330 deletions(-) create mode 100644 specs/platform/ephemeral-pr-environments.spec.md diff --git a/specs/platform/e2e-testing.spec.md b/specs/platform/e2e-testing.spec.md index 7297ebae..2a910a1c 100644 --- a/specs/platform/e2e-testing.spec.md +++ b/specs/platform/e2e-testing.spec.md @@ -4,7 +4,8 @@ **Status:** Draft **Jira:** HYPERSHELL-18 **Related:** `local-development.spec.md` -- Kind cluster setup and the shared `scripts/cluster/` lifecycle dispatcher; - `openshift-development.spec.md` (HYPERSHELL-44) -- `make openshift-*` lifecycle, blessed `deploy/openshift/` overlay, cluster infrastructure bootstrap, OpenShift CI (this spec owns the e2e driver interface and the OpenShift e2e driver; that spec owns bring-up); + `openshift-development.spec.md` (HYPERSHELL-44) -- `make openshift-*` lifecycle, blessed `deploy/openshift/` overlay, cluster infrastructure bootstrap (this spec owns the e2e driver interface and the OpenShift e2e driver; that spec owns bring-up); + `ephemeral-pr-environments.spec.md` (HYPERSHELL-240) -- automated OpenShift pull-request CI, GitHub-brokered grant path, and `e2e-openshell.sh` deprecation; `control-plane.spec.md` -- reconciler behavior; `openshell-gateway-routing.spec.md` -- GRPCRoute provisioning; `openshell-gateway-namespace-gc.spec.md` -- gateway deletion + namespace GC; @@ -12,7 +13,7 @@ ## Purpose -HyperShell requires infrastructure-agnostic end-to-end testing that validates the full provisioning path: API creation of a Gateway, control plane reconciliation, gateway pod readiness, route connectivity, and sandbox lifecycle. The same test suite SHALL run against Kind (local development, CI) and OpenShift (manual, on-demand runs against any OpenShift cluster) with infrastructure-specific logic isolated into driver scripts. A CI workflow SHALL execute these tests automatically on pull requests that modify e2e-relevant components. +HyperShell requires infrastructure-agnostic end-to-end testing that validates the full provisioning path: API creation of a Gateway, control plane reconciliation, gateway pod readiness, route connectivity, and sandbox lifecycle. The same test suite SHALL run against Kind (local development, CI, and the merge-queue gate) and OpenShift (manual on-demand runs, and origin pull-request environments specified in `ephemeral-pr-environments.spec.md`) with infrastructure-specific logic isolated into driver scripts. A Kind CI workflow SHALL execute these tests automatically on pull requests that modify e2e-relevant components. The existing e2e test (`components/pr-test/e2e-openshell.sh`) validates 6 areas -- gateway provisioning, infrastructure verification, route discovery, connectivity, sandbox lifecycle, and sandbox interaction -- but is hardcoded for OpenShift. This spec defines the driver abstraction, CI workflow, and deploy restructuring required to run the same tests across multiple infrastructure targets. @@ -22,9 +23,9 @@ HyperShell also needs a **performance test**. The performance test measures how This spec covers the **e2e driver interface contract** (for all targets), the **Kind driver**, the **OpenShift e2e driver**, the **Kind-based CI workflow**, and the **infra-agnostic performance test**. -This spec owns the driver interface contract and the **OpenShift e2e driver** (`tests/e2e/drivers/openshift.sh`) so a user can run `make e2e` and `make e2e-performance` **manually** against any OpenShift cluster the user is already logged in to (via `oc login`) -- the target environment for scale and performance testing. Bring-up is a precondition: `make openshift-up` (specified in `openshift-development.spec.md`) deploys the blessed `deploy/openshift/` overlay into the current `oc` project (`OPENSHIFT_NAMESPACE` overrides), companion `${OPENSHIFT_NAMESPACE}-keycloak`, and the per-environment `${OPENSHIFT_NAMESPACE}-dev-*` cluster-scoped RBAC. This spec does not duplicate that lifecycle. An automated OpenShift CI job and the consolidation of `components/pr-test/e2e-openshell.sh` remain in HYPERSHELL-44. +This spec owns the driver interface contract and the **OpenShift e2e driver** (`tests/e2e/drivers/openshift.sh`) so a user can run `make e2e` and `make e2e-performance` **manually** against any OpenShift cluster the user is already logged in to (via `oc login`) -- the target environment for scale and performance testing. Bring-up is a precondition: `make openshift-up` (specified in `openshift-development.spec.md`) deploys the blessed `deploy/openshift/` overlay into the current `oc` project (`OPENSHIFT_NAMESPACE` overrides), companion `${OPENSHIFT_NAMESPACE}-keycloak`, and the per-environment `${OPENSHIFT_NAMESPACE}-dev-*` cluster-scoped RBAC. This spec does not duplicate that lifecycle. Automated OpenShift pull-request CI is specified in `ephemeral-pr-environments.spec.md` (HYPERSHELL-240). The deprecation window for `components/pr-test/e2e-openshell.sh` is specified there as well. -An automated OpenShift CI job is **out of scope** here; it belongs to HYPERSHELL-44. In this spec, OpenShift runs are manual and on-demand, and only the Kind e2e workflow runs in CI. +Manual OpenShift e2e and performance runs remain in scope here. Kind CI, including the merge-queue gate, remains in scope here. The OpenShift pull-request environment job is not this spec. ## Architecture @@ -53,11 +54,11 @@ Each driver exports shell functions that abstract infrastructure-specific operat | `get_cluster_domain` | Get the base domain for constructing gateway DNS names | `gw.localhost` (static, matching `GATEWAY_API_BASE_DOMAIN` in `deploy/kind/`) | Gateway base domain derived from the shared Gateway listener hostname -- the same value `make openshift-up` sets on the control plane. Not a developer-supplied `GATEWAY_API_BASE_DOMAIN`, and not the cluster apps domain | | `get_cli_binary` | Return the Kubernetes CLI binary path | `kubectl` | `oc` | | `wait_for_gateway_route` | Block until the gateway is externally reachable | Check Gateway API Gateway status conditions and GRPCRoute parent status | Check Gateway `Programmed=True` and GRPCRoute parent `Accepted=True` | -| `acquire_oidc_token` | Obtain an OIDC access token for a given user, stored in `_OIDC_ACCESS_TOKEN` for `api_curl` to use | Resource-owner password grant against Keycloak at `keycloak.hypershell.localhost`, trusting the Kind self-signed CA (`curl -k`) | Resource-owner password grant against the HyperShell Keycloak at its Route in the `${OPENSHIFT_NAMESPACE}-keycloak` namespace, in the `hypershell` realm, trusting the cluster CA | +| `acquire_oidc_token` | Obtain an OIDC access token for a given user, stored in `_OIDC_ACCESS_TOKEN` for `api_curl` to use | Resource-owner password grant against Keycloak at `keycloak.hypershell.localhost`, trusting the Kind self-signed CA (`curl -k`) | Grant-agnostic against the HyperShell Keycloak at its Route in the `${OPENSHIFT_NAMESPACE}-keycloak` namespace, in the `hypershell` realm, trusting the cluster CA. Manual OpenShift defaults to the resource-owner password grant against seeded users. GitHub-brokered pull-request environments set `E2E_OIDC_GRANT=client_credentials` as `ephemeral-pr-environments.spec.md` defines | | `api_curl` | Issue an authenticated HTTP request to the HyperShell API, adding the bearer token from `acquire_oidc_token` | `curl` with the bearer header against the discovered API host, trusting the Kind CA | `curl` with the bearer header against the API Route host, trusting the cluster CA | | `assign_gateway_client_role` | Grant a user a role on a gateway's per-gateway OIDC client (mirrors the `gateway:viewer` RoleBinding); idempotent | Keycloak admin API assigns the client role in the `hypershell` realm | HyperShell Keycloak admin API (at its Route in the `${OPENSHIFT_NAMESPACE}-keycloak` namespace) assigns the client role in the `hypershell` realm | | `assign_realm_role` | Grant a user a platform-wide realm role, for example `platform:admin`; idempotent | Keycloak admin API assigns the realm role | HyperShell Keycloak admin API (at its Route in the `${OPENSHIFT_NAMESPACE}-keycloak` namespace) assigns the realm role in the `hypershell` realm | -| `acquire_gateway_token_with_role` | Acquire a per-gateway OIDC token and block until the named role lands in it (roles reconcile asynchronously after gateway create); sets `_OIDC_ACCESS_TOKEN` | Password grant against the per-gateway client, polling until the role appears | Password grant against the per-gateway client on the HyperShell Keycloak at its Route, polling until the role appears | +| `acquire_gateway_token_with_role` | Acquire a per-gateway OIDC token and block until the named role lands in it (roles reconcile asynchronously after gateway create); sets `_OIDC_ACCESS_TOKEN` | Password grant against the per-gateway client, polling until the role appears | Grant-agnostic against the per-gateway client on the HyperShell Keycloak at its Route, polling until the role appears. Manual OpenShift defaults to the password grant. GitHub-brokered pull-request environments obtain the token by token-exchange impersonation of the seeded developer principal targeting that gateway client (`ephemeral-pr-environments.spec.md`) | ### CI Pipeline @@ -179,6 +180,13 @@ The script SHALL auto-detect the driver from the current KUBECONFIG context rath Each driver script SHALL export the following shell functions. The main test script SHALL call only these functions for infrastructure-specific operations. A driver that does not implement all required functions SHALL cause the test script to exit with an error at startup. This spec defines the contract for all drivers and covers the Kind driver implementation. The OpenShift implementation of this contract (the `oc` commands, Route discovery, gateway-base-domain lookup from the shared Gateway listener, and OIDC issuer derivation from the Keycloak Route) is specified in `openshift-development.spec.md` (HYPERSHELL-44); the table below is the contract it implements. +`acquire_oidc_token` and `acquire_gateway_token_with_role` SHALL keep those names, signatures, and suite call sites. The token grant they use SHALL be selected by `E2E_OIDC_GRANT`: + +- unset or `password` -- resource-owner password grant against the supplied (or default seeded) username and password. This is the Kind path and the default for manual OpenShift runs. +- `client_credentials` -- the GitHub-brokered pull-request path. Admin tokens SHALL use the `hypershell-e2e` client-credentials grant. Developer HyperShell API tokens and per-gateway `openshell-user` tokens SHALL use token-exchange impersonation of the seeded developer principal, as `ephemeral-pr-environments.spec.md` defines. + +The pull-request workflow SHALL set `E2E_OIDC_GRANT=client_credentials`. Kind CI SHALL leave it unset or set `password`. + #### Scenario: API Host Discovery -- Kind - GIVEN the `kind` driver is active @@ -214,7 +222,23 @@ Each driver script SHALL export the following shell functions. The main test scr - AND verify the corresponding GRPCRoute's parent status reports `Accepted=True` - AND return success when both conditions are met or fail after `E2E_PROVISION_TIMEOUT` seconds -The OpenShift driver implements the same ten functions with OpenShift constructs (Route host for `discover_api_host`, GRPCRoute hostname via the shared Gateway with `Programmed=True` for `discover_gateway_endpoint`, the gateway base domain `make openshift-up` derived from the shared Gateway listener hostname for `get_cluster_domain`, `oc` for `get_cli_binary`, Gateway `Programmed=True` plus GRPCRoute parent `Accepted=True` for `wait_for_gateway_route`, the HyperShell Keycloak reached at its Route in the `${OPENSHIFT_NAMESPACE}-keycloak` namespace for `acquire_oidc_token` and `api_curl`, and that same Keycloak's admin API for the `assign_gateway_client_role`, `assign_realm_role`, and `acquire_gateway_token_with_role` role helpers), as the interface table above shows. `openshift-development.spec.md` owns bring-up (`make openshift-up`, the overlay, cluster bootstrap); this spec owns the driver the suite calls after that environment exists. Automated OpenShift CI remains in HYPERSHELL-44. +#### Scenario: Kind and manual OpenShift use the password grant + +- GIVEN `E2E_OIDC_GRANT` is unset or set to `password` +- WHEN the suite calls `acquire_oidc_token` or `acquire_gateway_token_with_role` +- THEN the driver SHALL use a resource-owner password grant against the seeded user +- AND the function name and arguments SHALL be unchanged + +#### Scenario: Pull-request OpenShift uses client credentials and token exchange + +- GIVEN `E2E_OIDC_GRANT=client_credentials` +- WHEN the suite calls `acquire_oidc_token` for the admin path +- THEN the driver SHALL use the `hypershell-e2e` client-credentials grant +- AND when the suite calls `acquire_gateway_token_with_role` for the seeded developer principal +- THEN the driver SHALL use token-exchange impersonation targeting that gateway client +- AND neither call SHALL use a password grant + +The OpenShift driver implements the same ten functions with OpenShift constructs (Route host for `discover_api_host`, GRPCRoute hostname via the shared Gateway with `Programmed=True` for `discover_gateway_endpoint`, the gateway base domain `make openshift-up` derived from the shared Gateway listener hostname for `get_cluster_domain`, `oc` for `get_cli_binary`, Gateway `Programmed=True` plus GRPCRoute parent `Accepted=True` for `wait_for_gateway_route`, the HyperShell Keycloak reached at its Route in the `${OPENSHIFT_NAMESPACE}-keycloak` namespace for `acquire_oidc_token` and `api_curl`, and that same Keycloak's admin API for the `assign_gateway_client_role`, `assign_realm_role`, and `acquire_gateway_token_with_role` role helpers), as the interface table above shows. `openshift-development.spec.md` owns bring-up (`make openshift-up`, the overlay, cluster bootstrap); this spec owns the driver the suite calls after that environment exists. Automated OpenShift pull-request CI is specified in `ephemeral-pr-environments.spec.md` (HYPERSHELL-240). ### Requirement: Custom OpenShift Runs @@ -226,7 +250,7 @@ Each target SHALL auto-detect the driver from the current KUBECONFIG context (se **Namespace GC timing.** Area 11 exercises the periodic namespace reaper. To make it pass on OpenShift without waiting the production GC defaults (5m sweep / 10m grace), the OpenShift deployment SHOULD set shortened `GATEWAY_NAMESPACE_GC_INTERVAL` and `GATEWAY_NAMESPACE_GC_GRACE_PERIOD` (as the Kind overlay does), or the user SHOULD raise `E2E_ORPHAN_GC_TIMEOUT` and `E2E_GC_TIMEOUT` to fit the cluster's configured timing. -**Not in CI.** OpenShift e2e and performance runs SHALL NOT be wired into CI in this iteration (see [Design Decisions](#design-decisions)). CI runs the Kind e2e workflow only. +**Not in this spec's CI.** OpenShift performance runs SHALL NOT be wired into CI. Kind e2e, including the merge-queue gate, SHALL remain the CI job this spec defines. Origin-repository OpenShift pull-request environments are specified in `ephemeral-pr-environments.spec.md` and SHALL NOT be restated here. #### Scenario: e2e Against OpenShift @@ -252,12 +276,21 @@ Each target SHALL auto-detect the driver from the current KUBECONFIG context (se - THEN the suite SHALL fail with a clear error about the missing environment - AND it SHALL NOT report false passes -#### Scenario: OpenShift Not in CI +#### Scenario: Kind remains this spec's CI job + +- GIVEN the CI configuration this spec owns +- WHEN a pull request runs the Kind e2e workflow +- THEN it SHALL run the Kind e2e job as this spec defines +- AND it SHALL NOT run the performance test against OpenShift +- AND OpenShift pull-request environments SHALL be the job + `ephemeral-pr-environments.spec.md` defines, not a second copy of this spec -- GIVEN the CI configuration for this iteration -- WHEN a pull request runs the e2e workflow -- THEN it SHALL run only the Kind e2e job -- AND it SHALL NOT run e2e or performance tests against OpenShift +#### Scenario: Merge-queue stays on Kind + +- GIVEN a pull request enters the GitHub merge queue +- WHEN CI evaluates which e2e jobs to run +- THEN the Kind e2e job SHALL run +- AND the OpenShift pull-request environment workflow SHALL NOT run ### Requirement: E2E Test Suite Coverage @@ -708,7 +741,7 @@ deploy/ e2e.yml -- CI e2e workflow ``` -`components/pr-test/e2e-openshell.sh` SHALL be deprecated in a follow-up once all tests are migrated to `tests/e2e/`. +`components/pr-test/e2e-openshell.sh` SHALL be deprecated as `ephemeral-pr-environments.spec.md` specifies. Removal is deferred until manual usage migrates; the ROKS variant is out of that deprecation. ## Environment Variables @@ -725,7 +758,8 @@ deploy/ | `E2E_ORPHAN_GC_TIMEOUT` | `90` | Seconds from orphan namespace seed time for the periodic reaper to delete the synthetic orphan (validated in step 11) | | `E2E_SKIP_CLEANUP` | `0` | Set to `1` to keep test resources after run | | `E2E_OIDC_USERNAME` | `admin` | Admin OIDC user (member of `hypershell-admins` + `hypershell-users`) used for areas 1--8 and 11 | -| `E2E_OIDC_PASSWORD` | `admin` | Password for the admin OIDC user (local dev only) | +| `E2E_OIDC_PASSWORD` | `admin` | Password for the admin OIDC user (local dev only; unused when `E2E_OIDC_GRANT=client_credentials`) | +| `E2E_OIDC_GRANT` | `password` | Token grant for `acquire_oidc_token` and `acquire_gateway_token_with_role`: `password` (Kind and manual OpenShift) or `client_credentials` (GitHub-brokered pull-request environments, see `ephemeral-pr-environments.spec.md`) | | `E2E_SEED_CLUSTER_NAME` | `local-kind` on kind; unset otherwise | Pin seed discovery to this managed-cluster name. Unset means the first list item | | `E2E_SEED_RELEASE_NAME` | `dev-release` on kind; unset otherwise | Pin seed discovery to this gateway-release name. Unset means the first list item | | `E2E_DEV_USERNAME` | `developer` | Standard OIDC user (`openshell-user` tier) used for the RBAC boundary assertions | @@ -1310,7 +1344,7 @@ On failure, the harness SHALL collect diagnostics that explain resource pressure |----------|-----------| | Shell-based drivers as starting point | The e2e test is a shell script; shell functions provide the simplest driver abstraction without adding a new language or build step. Each driver is a single file implementing a known function interface. If the test suite grows in complexity -- structured assertions, parallel execution, direct Kubernetes API client usage -- migrating to a Go-based e2e framework (e.g., `go test` with client-go) is a natural follow-up. The driver interface contract is function-shape-agnostic, so the same logical abstraction applies in either language | | `E2E_INFRA_DRIVER` is auto-detected from the KUBECONFIG context, with an explicit override | `route.openshift.io` is a reliable, cheap signal for OpenShift, so a developer running against whichever cluster their context selects does not need to remember to set a flag. CI still sets `E2E_INFRA_DRIVER=kind` explicitly so the invocation stays self-documenting and does not depend on the runner's kubeconfig | -| Tests live in `tests/e2e/`, not `components/pr-test/` | A top-level `tests/` tree is the natural home for e2e tests and their drivers. `components/pr-test/` will be deprecated in a follow-up once migration is complete | +| Tests live in `tests/e2e/`, not `components/pr-test/` | A top-level `tests/` tree is the natural home for e2e tests and their drivers. `components/pr-test/e2e-openshell.sh` is deprecated per `ephemeral-pr-environments.spec.md`; the ROKS variant and the `pr_test` component stay until that spec's removal window closes | | Shared test utilities in `tests/e2e/lib.sh` | Pass/fail tracking, color output, and retry helpers are currently inline in `e2e-openshell.sh`. Extracting them into `lib.sh` makes them reusable across future test scripts without duplicating code | | CI pulls Konflux-built images, not rebuild | Images are built by Konflux (the existing build pipeline). The e2e workflow gates on those builds and pulls images by digest, avoiding duplicate builds and ensuring CI tests the exact images that ship. This is expected to cover HYPERSHELL-16 | | Diagnostic artifacts only on failure | Uploading pod logs, events, and describes on every run wastes GitHub Actions storage. Conditional upload on failure provides debugging information when needed | @@ -1318,7 +1352,7 @@ On failure, the harness SHALL collect diagnostics that explain resource pressure | e2e workflow skips for irrelevant changes | SDK-only or docs-only PRs do not affect the e2e path. Skipping avoids CI time and Konflux build overhead. The `detect-components.sh` infrastructure tracks `api_server`, `control_plane`, `pr_test`, and `e2e` component paths for "should we re-run e2e" decisions. Separately, Konflux image builds only trigger on changes under `components//` source paths -- the workflow checks the actual diff to distinguish e2e-relevant infrastructure changes (which use baseline images) from source changes (which require Konflux-built images) | | `make kind-up` accepts image overrides | Passing `IMAGE_TAG=` or per-component image variables to `make kind-up` allows CI to deploy Konflux-built images directly without a separate load step. Developers can also use this to test specific image versions locally | | Backward-compatible migration | The refactoring does not change `make kind-up`. `scripts/kind/up.sh` can be migrated to use `kustomize build deploy/kind/` incrementally. The spec defines the target state; the migration path is incremental | -| OpenShift e2e runs use `make openshift-up` as the environment | This spec owns the driver the suite calls. `openshift-development.spec.md` owns bring-up: `make openshift-up`, the `deploy/openshift/` overlay (Routes, Keycloak NetworkPolicy, SCC), namespace rewrite, `${OPENSHIFT_NAMESPACE}-dev-*` cluster RBAC, and cluster bootstrap. Automated OpenShift CI and `components/pr-test/` consolidation stay in HYPERSHELL-44 and are not duplicated here | +| OpenShift e2e runs use `make openshift-up` as the environment | This spec owns the driver the suite calls. `openshift-development.spec.md` owns bring-up: `make openshift-up`, the `deploy/openshift/` overlay (Routes, Keycloak NetworkPolicy, SCC), namespace rewrite, `${OPENSHIFT_NAMESPACE}-dev-*` cluster RBAC, and cluster bootstrap. Automated OpenShift pull-request CI and the `e2e-openshell.sh` deprecation window live in `ephemeral-pr-environments.spec.md` (HYPERSHELL-240) and are not duplicated here | | Env vars renamed with `E2E_` prefix | The existing `e2e-openshell.sh` uses `SANDBOX_TIMEOUT`, `PROVISION_TIMEOUT`, `SKIP_CLEANUP`, and `GATEWAY_NAMESPACE`. These are renamed to `E2E_SANDBOX_TIMEOUT`, `E2E_PROVISION_TIMEOUT`, `E2E_SKIP_CLEANUP`, and `E2E_NAMESPACE` to avoid namespace collisions with non-e2e configuration and make the e2e origin of these variables explicit | | CI uses `make kind-up`, not raw `kind create cluster` | Reuses the same cluster setup path developers use locally. Ensures the CI environment is identical to local development. Avoids a second "create a Kind cluster" implementation that could drift | | Performance harness reuses the e2e driver interface | The performance test needs the same cross-infrastructure portability as the e2e suite: run on Kind locally, run on any OpenShift cluster for on-demand load tests. Reusing the driver interface means the harness holds no infra-specific code and a new target needs only a new driver file. It also keeps one abstraction to maintain, not two | diff --git a/specs/platform/ephemeral-pr-environments.spec.md b/specs/platform/ephemeral-pr-environments.spec.md new file mode 100644 index 00000000..305f859f --- /dev/null +++ b/specs/platform/ephemeral-pr-environments.spec.md @@ -0,0 +1,803 @@ +# Ephemeral Pull-Request Environments Specification + +**Date:** 2026-09-04 +**Status:** Draft +**Jira:** HYPERSHELL-240 +**Related:** `openshift-development.spec.md` (HYPERSHELL-44) -- `make openshift-up` + lifecycle, the blessed `deploy/openshift/` overlay, ephemeral-namespace + isolation, cluster-scoped RBAC, and the OpenShift e2e driver + (this spec owns pull-request CI on top of that lifecycle); + `e2e-testing.spec.md` (HYPERSHELL-18) -- the infra-agnostic e2e suite, + the grant-agnostic driver interface contract, Kind CI and the + merge-queue gate, and Konflux image consumption; + `local-development.spec.md` -- the Keycloak realm/client model this + spec builds on; + `oidc-integration.spec.md` -- platform OIDC and RBAC role mapping; + `../security/rbac-enforcement.spec.md` -- `platform:admin` vs + `gateway:creator` + +## Purpose + +HyperShell delivers a live OpenShift environment for every open pull request and +keeps that environment in continuous deployment for the life of the pull request. +When a pull request opens, CI deploys the full stack into a per-PR ephemeral +namespace group on a shared target OpenShift cluster, waits for Konflux to build +the pull request's component images, swaps those images into the environment, +runs the OpenShift e2e suite against it, and posts a pull-request comment telling +the developer how to log in. When a later commit is pushed to the same pull +request, CI does not create a second environment: it reuses the existing one, +waits for Konflux to rebuild the changed images, swaps them in, reruns the e2e +suite, and updates the comment to say the environment now runs that commit. The +environment lives independently of any single CI run so a developer can use it as +a live debug and development target, and it is reaped after a fixed timebox so an +abandoned pull request cannot hold cluster resources. + +This spec owns the automated OpenShift pull-request CI workflow. The +per-namespace deployment, reconcile, image swap-by-digest, and OpenShift e2e +driver remain defined in `openshift-development.spec.md`; the infra-agnostic e2e +suite, Kind CI (including the merge-queue gate), and Konflux image gating remain +defined in `e2e-testing.spec.md`. This spec adds the pull-request-scoped +concerns those specs leave open: the deterministic per-PR namespace naming, the +continuous-deployment triggers across the pull request lifecycle, the +deploy-serialization rule, the origin-only trust boundary, the fixed timebox and +external reaping, the per-commit pull-request comment, and a Keycloak +authentication model that brokers to GitHub (restricted to the +`openshift-online` organization plus an allowlist) instead of Red Hat SSO. +GitHub brokering lets an allowlisted outside contributor log in to an +already-deployed origin-repo environment; it does not deploy fork pull requests +onto the shared cluster. + +### Scope + +This spec covers: + +- the per-pull-request ephemeral environment naming and identity, +- the continuous-deployment lifecycle triggered by pull-request open, synchronize + (push), reopen, and close/merge on the origin repository, +- the origin-only trust boundary (fork pull requests do not receive cluster + credentials), +- the per-pull-request deploy serialization rule, +- the fixed-duration timebox and the out-of-band reaper, +- the per-commit pull-request comment and secure credential handoff, +- the GitHub-brokered Keycloak authentication model for these environments, + including the GitHub OAuth App, the stable callback, admin authorization, and + developer-tier testing by impersonation, and +- the deprecation of the legacy `components/pr-test/e2e-openshell.sh` script in + favor of the shared e2e harness and this workflow (the ROKS variant is out of + scope). + +This spec does not redefine `make openshift-up`, the `deploy/openshift/` overlay, +the ephemeral-namespace isolation rules, the cluster-scoped RBAC handling, or +the Konflux build pipeline. It depends on all of them. It amends the OpenShift +`acquire_oidc_token` / `acquire_gateway_token_with_role` grant in +`e2e-testing.spec.md` so those functions are grant-agnostic; it does not change +their signatures or the suite's call sites. Kind merge-queue e2e stays in +`e2e-testing.spec.md` and is not this workflow. It is a behavior contract, not +the CI YAML or the Keycloak realm export. + +### Reserved Terms + +This spec adds no new domain kinds. It refers to the existing kinds (Gateway, +GatewayNetwork, GatewayRelease, ManagedCluster, ManagedDatabase) only where a +scenario provisions one. "Environment" here means the per-pull-request namespace +group (the platform namespace and its companion `-keycloak` namespace) that +`openshift-development.spec.md` defines. + +## Requirements + +### Requirement: Per-Pull-Request Environment Identity + +Each pull request SHALL map to exactly one ephemeral environment on the shared +target cluster, and that mapping SHALL be deterministic from the pull-request +number so that every CI run for the same pull request resolves to the same +environment without storing external state. The CI workflow SHALL set +`OPENSHIFT_NAMESPACE` to `hypershell-ci-pr-` (for example +`hypershell-ci-pr-232`), and the companion Keycloak namespace SHALL therefore be +`hypershell-ci-pr--keycloak`, following the namespace-group derivation +in `openshift-development.spec.md`. The platform namespace name SHALL remain a +valid RFC 1123 DNS label within the 54-character bound that keeps the derived +`-keycloak` name under the 63-character limit; because pull-request numbers are +short, the `hypershell-ci-pr-` prefix leaves ample room. + +The workflow SHALL stamp both namespaces with the same ownership labels the +OpenShift lifecycle driver uses, so the reaper and `make openshift-status` can +attribute every namespace to its pull request: + +- `hypershell.redhat.io/owned=true` +- `hypershell.redhat.io/environment=pr-` (for example `pr-232`) +- `app.kubernetes.io/managed-by=hypershell-lifecycle` +- `app.kubernetes.io/part-of=hypershell` + +The `pr-` prefix on the environment identifier distinguishes pull-request +environments from local `make openshift-up` environments, whose identifier is +an opaque per-deployment value, not `pr-*`. After `make openshift-up`, the +workflow SHALL set `hypershell.redhat.io/environment=pr-` on both +namespaces, overwriting the identifier that command assigned, so later +reconciles recover `pr-` from the namespace and the reaper can match +it. The CI service account SHALL be able to patch namespace objects; +if labeling either namespace fails, the workflow SHALL fail the job and SHALL +NOT continue. The local-dev "warn and continue" path in +`openshift-development.spec.md` does not apply to this workflow. The workflow +SHALL NOT derive the namespace from the branch name, the commit SHA, or the +developer identity, because those are not stable across the life of one pull +request. + +#### Scenario: Namespace derives from the pull-request number + +- GIVEN a pull request numbered 232 runs the workflow +- WHEN the workflow resolves the target environment +- THEN it SHALL set `OPENSHIFT_NAMESPACE=hypershell-ci-pr-232` +- AND the Keycloak namespace SHALL be `hypershell-ci-pr-232-keycloak` +- AND both namespaces SHALL carry `hypershell.redhat.io/owned=true` and + `hypershell.redhat.io/environment=pr-232` + +#### Scenario: Labeling failure fails the job + +- GIVEN the CI service account cannot patch namespace objects +- WHEN the workflow attempts to stamp the namespace group +- THEN the job SHALL fail +- AND it SHALL NOT leave an unlabeled pull-request environment + +#### Scenario: Same pull request always resolves the same environment + +- GIVEN pull request 232 already has an environment from an earlier run +- WHEN any later run for pull request 232 starts +- THEN it SHALL resolve `hypershell-ci-pr-232` without consulting external state +- AND it SHALL NOT create a second environment for the same pull request + +### Requirement: Continuous Deployment Across the Pull-Request Lifecycle + +The workflow SHALL keep the pull request's environment continuously deployed to +the pull request's current head commit for the life of the pull request. It SHALL +trigger on origin-repository pull-request `opened`, `reopened`, and `synchronize` +(a new commit pushed to the pull-request branch), and it SHALL trigger on +`closed` (which covers both merge and close) to release the environment (see the +Timebox and Reaping requirement). It SHALL NOT trigger on `merge_group`. Kind +e2e, as `e2e-testing.spec.md` defines, remains the merge-queue gate; this +workflow does not share a namespace with a merge-queue SHA. + +The workflow SHALL run only for pull requests targeting the origin repository. +Fork pull requests SHALL NOT receive cluster credentials and SHALL NOT get an +environment (see Pull-Request Trust Boundary). The workflow SHALL NOT use +`pull_request_target`. + +On every deploying trigger (`opened`, `reopened`, `synchronize`), the workflow +SHALL run `make openshift-up` unconditionally, whether or not the environment +already exists. Because `make openshift-up` is idempotent and reconciling +(`openshift-development.spec.md`), one code path both creates the environment on +first run and reconciles it to the current overlay on later runs; the workflow +SHALL NOT branch on a "does the environment exist" check before deciding whether +to deploy. After a successful `make openshift-up`, the workflow SHALL refresh +the environment's timebox (see the Timebox and Reaping requirement), so an +active pull request is continuously renewed while an abandoned one expires. +`make openshift-up` itself SHALL NOT stamp or refresh that timebox; local +environments are not time-boxed by this spec. + +Deploying runs for the same pull request SHALL serialize on a per-pull-request +concurrency group. A newer run SHALL cancel or queue an older in-flight run for +that pull request so two swaps cannot leave a mixed digest set. The access +comment's commit SHA SHALL be the commit whose digest swap completed, not a +cancelled run's head. + +The reconcile SHALL bring the environment to the current desired state, including +pruning resources the overlay no longer declares, so the long-lived environment +does not drift across the many deployments a pull request accumulates. The +reconcile SHALL preserve any active per-namespace component swap the same way +`openshift-development.spec.md` specifies. + +#### Scenario: Pull request opens + +- GIVEN a pull request is opened and has no environment yet +- WHEN the workflow runs +- THEN it SHALL run `make openshift-up` with `OPENSHIFT_NAMESPACE=hypershell-ci-pr-` +- AND the environment SHALL be created and deployed +- AND the workflow SHALL post the initial access comment (see Pull-Request Comment) + +#### Scenario: Commit pushed to an existing pull request + +- GIVEN pull request 232 already has a running environment +- WHEN a new commit is pushed and the `synchronize` trigger fires +- THEN the workflow SHALL reuse `hypershell-ci-pr-232` and SHALL NOT create a new environment +- AND it SHALL run `make openshift-up` to reconcile the environment +- AND it SHALL wait for Konflux to build the new commit's images and swap them in + by digest (see Image Gating and Swap) +- AND it SHALL rerun the e2e suite against the environment +- AND it SHALL update the access comment to reflect the new head commit (see + Pull-Request Comment) + +#### Scenario: Deploy runs unconditionally + +- GIVEN a deploying trigger for a pull request +- WHEN the workflow reaches the deploy step +- THEN it SHALL invoke `make openshift-up` regardless of whether the environment + already exists +- AND it SHALL NOT skip deployment based on a prior-existence check + +#### Scenario: Overlapping runs serialize per pull request + +- GIVEN a deploying run for pull request 232 is already in flight +- WHEN a later `synchronize` run for pull request 232 starts +- THEN the workflow SHALL cancel or queue the older run +- AND at most one deploying run SHALL swap images into `hypershell-ci-pr-232` at + a time +- AND the access comment SHALL name the commit whose digest swap completed + +#### Scenario: Merge-queue does not use this environment + +- GIVEN a pull request enters the GitHub merge queue +- WHEN CI evaluates which jobs to run +- THEN this workflow SHALL NOT run +- AND the Kind e2e job SHALL remain the merge-queue gate as + `e2e-testing.spec.md` defines + +### Requirement: Image Gating and Swap + +The workflow SHALL NOT build component images itself. It SHALL gate on the +Konflux builds for the pull request's head commit and swap the built images into +the environment by digest, reusing the same digest-injection mechanism the Kind +e2e job uses (`scripts/kind/set-component-images.sh`) as +`openshift-development.spec.md` and `e2e-testing.spec.md` define. The workflow +SHALL overlap environment bring-up with the Konflux builds: it MAY run +`make openshift-up` with baseline images while Konflux builds are still in +flight, then wait for each changed component's build to conclude and swap that +component's image by digest, so cluster reconcile time is hidden behind build +time. Unchanged components SHALL keep baseline registry images. The workflow +SHALL determine which components to wait for using the shared change-detection and +Konflux-trigger-mirroring rules that `e2e-testing.spec.md` defines, so it never +falls back to a baseline image while Konflux is building an image the pull request +produced. + +The workflow SHALL NOT consume an untrusted, mutable image tag when an immutable +digest is available. Once a component's Konflux build has concluded, the workflow +SHALL resolve that build's output to its manifest digest and inject the image into +the environment by `@sha256:`, not by a floating tag such as +`on-pr-`, so the environment runs exactly the artifact CI verified and a +tag that is later re-pushed cannot silently change what the environment runs. +Baseline images for unchanged components SHALL likewise be pinned by digest when +the registry exposes one; a tag SHALL be used only as a last resort when no +digest is available, and that fallback SHALL be recorded in the run output rather +than passed silently. This reinforces the cross-cutting image-reference +consistency convention: image references SHALL resolve to the same immutable +artifact across the stack. + +#### Scenario: Swap the pull request's images by digest + +- GIVEN a pull request changed `components/api-server/` +- AND Konflux built the API server image for the head commit +- WHEN the workflow deploys the environment +- THEN it SHALL wait for that build to conclude +- AND it SHALL swap the API server image into the environment by digest +- AND the control plane and web console SHALL keep baseline registry images + +#### Scenario: New commit rebuilds and re-swaps + +- GIVEN a new commit changes `components/control-plane/` +- WHEN the `synchronize` trigger deploys the environment +- THEN the workflow SHALL wait for the control plane's Konflux build for the new + head commit +- AND it SHALL swap the new control plane image into the environment by digest + before running e2e + +#### Scenario: Immutable digest is preferred over a mutable tag + +- GIVEN a component's Konflux build has concluded and exposes both a floating + `on-pr-` tag and a manifest digest +- WHEN the workflow injects that image into the environment +- THEN it SHALL pin the image by `@sha256:` +- AND it SHALL NOT deploy the image by its mutable tag +- AND when no digest is available it SHALL fall back to the tag and record that + fallback in the run output + +### Requirement: E2E Execution Against the Environment + +After the environment is deployed and the pull request's images are swapped in, +the workflow SHALL run the OpenShift e2e suite against it, exactly as +`e2e-testing.spec.md` and `openshift-development.spec.md` define: it SHALL run +`E2E_INFRA_DRIVER=openshift E2E_OIDC_GRANT=client_credentials bash tests/e2e/e2e-openshell.sh` against a KUBECONFIG +context pointed at the environment, exercising the same test areas the Kind suite +exercises. The suite SHALL run on the pull request's first deployment and on every +later deployment for that pull request, so each commit is validated against a live +environment the same way the Kind e2e job validates each commit today. On failure +the workflow SHALL collect the diagnostics `e2e-testing.spec.md` defines. Whether +the suite passes or fails, the environment SHALL survive (see Timebox and +Reaping), so a developer can inspect a failing run on the live environment. + +The e2e suite's authentication SHALL set `E2E_OIDC_GRANT=client_credentials` and +use the non-interactive path this spec defines (see Automated E2E Authentication), +because the environment's interactive login is GitHub-brokered and brokered users +have no password grant. + +#### Scenario: E2E runs on every deployment + +- GIVEN the environment is deployed and the pull request's images are swapped in +- WHEN the workflow reaches the test step +- THEN it SHALL run the OpenShift e2e suite against the environment +- AND it SHALL run the suite again on each later commit's deployment + +#### Scenario: Environment survives a failing run + +- GIVEN the e2e suite fails +- WHEN the workflow finishes +- THEN it SHALL collect the failure diagnostics +- AND the environment SHALL remain deployed for developer inspection + +### Requirement: Timebox and Reaping + +Each pull-request environment SHALL be time-boxed to three days and SHALL be +reaped out-of-band, independently of any CI run, so a stale or abandoned pull +request cannot hold cluster resources. After each successful `make openshift-up`, +the CI workflow SHALL stamp both namespaces in the group with the annotation +`hypershell.redhat.io/expires-at` set to an RFC 3339 timestamp three days in the +future (or a reservation duration of three days when the environment is +provisioned through a reservation mechanism such as the +ephemeral-namespace-operator). `make openshift-up` SHALL NOT write that +annotation; local environments this command creates are not time-boxed by this +spec. Because every deploying trigger refreshes the annotation after bring-up, +an actively worked pull request is continuously renewed and never reaped +mid-flight, while a pull request with no activity for three days falls past its +expiry and is reclaimed. The three-day timebox SHALL be configurable through a +single documented workflow setting rather than hardcoded in the workflow logic. + +Reaping SHALL be performed by an out-of-band mechanism -- a scheduled reaper job +or the reservation mechanism's own duration -- not by the pull-request workflow's +own teardown step, so that an environment expires even when no further CI runs for +that pull request. The reaper SHALL delete a namespace group whose +`hypershell.redhat.io/expires-at` has passed, and SHALL identify HyperShell-owned +pull-request environments only when all of these match: + +- `hypershell.redhat.io/owned=true` +- `hypershell.redhat.io/environment` equal to `pr-` +- namespace name prefixed with `hypershell-ci-pr-` + +The reaper SHALL NOT delete namespaces that fail that match, including local +`make openshift-up` environments and any other HyperShell-owned namespace whose +environment identifier is not `pr-*`. It SHALL refuse reserved names +(`default`, `kube-*`, `openshift-*`). + +On pull-request `closed` (merge or close), the workflow SHALL release the +environment as the primary path by removing the namespace group the same way +`make openshift-down` does. The timebox SHALL remain the backstop for the case +where the close event does not fire or its release cannot be confirmed; when the +release step cannot confirm the release, the workflow SHALL report the failure so +an operator can free the environment. + +#### Scenario: Deploying run refreshes the expiry + +- GIVEN a pull-request environment exists +- WHEN a new commit's deploying run finishes `make openshift-up` +- THEN both namespaces SHALL have `hypershell.redhat.io/expires-at` reset to + three days from that run +- AND `make openshift-up` SHALL NOT have written that annotation +- AND the environment SHALL NOT be reaped while the pull request stays active + +#### Scenario: Abandoned environment is reaped by the timebox + +- GIVEN a pull-request environment has had no deploying run for three days +- AND the pull request was neither merged nor closed +- WHEN the out-of-band reaper evaluates environments +- THEN it SHALL delete the expired namespace group +- AND it SHALL delete only namespaces matching the pull-request ownership labels + and `pr-*` environment identifier + +#### Scenario: Local environments are not reaped + +- GIVEN a developer ran `make openshift-up` into a namespace that is not + `hypershell-ci-pr-*` +- WHEN the out-of-band reaper evaluates environments +- THEN it SHALL NOT delete that namespace group + +#### Scenario: Close releases the environment; timebox backstops + +- GIVEN a pull-request environment exists +- WHEN the pull request merges or closes +- THEN the workflow SHALL remove the environment's namespace group as the primary path +- AND when the close event does not fire, the timebox SHALL reclaim the environment +- AND when release cannot be confirmed, the workflow SHALL report the failure + +#### Scenario: Idempotent re-run after reaping + +- GIVEN a pull request's environment was reaped after its timebox +- WHEN a new commit is pushed to that still-open pull request +- THEN `make openshift-up` SHALL recreate the environment under the same + `hypershell-ci-pr-` name +- AND the workflow SHALL stamp a fresh `hypershell.redhat.io/expires-at` +- AND the workflow SHALL proceed with image swap, e2e, and comment as on first open + +### Requirement: Pull-Request Comment and Access Handoff + +The workflow SHALL communicate the live environment to the developer through a +pull-request comment, and SHALL keep exactly one such comment current for the +pull request rather than post a new comment per run, so the pull request shows the +live environment's current state. The comment SHALL contain a stable hidden HTML +marker (``) so later runs can find and update +that comment rather than any other comment on the pull request. The comment SHALL +contain the same non-secret access facts `openshift-development.spec.md` defines +-- the environment namespaces, the OpenShift console URL for the platform +namespace, the API Route URL, and the web-console Route URL -- presented as the +same login guidance `make openshift-up` prints at the end of a successful +bring-up, so the comment and the command agree. + +On the pull request's first deployment, the workflow SHALL post the initial +comment once the environment is ready. On each later deployment for the same pull +request, the workflow SHALL update the marked comment to state that the +environment has been updated to commit `` and SHALL refresh the same login +details. The `` in the comment SHALL be the commit whose digest swap +completed, so the comment never claims a commit the swap did not deploy. + +The comment SHALL NOT contain any credential. It MAY include an `oc login` +template with the credential redacted (for example +`oc login --server= --token=`). The credential itself SHALL be +delivered only through a channel that only an authorized developer can read, and +SHALL be short-lived and namespace-scoped, as `openshift-development.spec.md` +requires. No kubeconfig, token, or password SHALL appear in the comment, the job +logs, or a public artifact. + +#### Scenario: Initial comment on pull-request open + +- GIVEN a pull request is opened and its environment becomes ready +- WHEN the workflow finishes deploying +- THEN it SHALL post one pull-request comment with the namespaces, console URL, + API Route URL, and web-console Route URL +- AND the comment SHALL contain the hidden marker `` +- AND the comment SHALL present the same login details `make openshift-up` prints +- AND the comment SHALL NOT contain a credential + +#### Scenario: Comment updated on each new commit + +- GIVEN a pull request already has an access comment that carries the marker +- WHEN a new commit's digest swap completes +- THEN the workflow SHALL update that marked comment to say the environment was + updated to commit `` +- AND `` SHALL be the commit whose digest swap completed +- AND the workflow SHALL NOT post a second access comment +- AND the comment SHALL refresh the login details + +#### Scenario: Credentials never leak + +- GIVEN the workflow delivers access details +- WHEN a reader inspects the comment, the job logs, and public artifacts +- THEN no kubeconfig, token, or password appears in any of them +- AND the credential is available only through a secure channel + +### Requirement: Pull-Request Trust Boundary + +The workflow SHALL run only on `pull_request` events from the origin repository +(the repository that holds the workflow and the cluster credentials). It SHALL +NOT use `pull_request_target`. Fork pull requests SHALL NOT receive the cluster +kubeconfig, the GitHub OAuth client secret, or any other secret this workflow +needs, and SHALL NOT get an environment. The GitHub organization gate and +allowlist (see GitHub-Brokered Keycloak Authentication) govern interactive +login to an already-deployed origin-repo environment; they SHALL NOT be used as +a reason to deploy untrusted pull-request trees with cluster credentials. + +#### Scenario: Origin pull request is deployed + +- GIVEN a pull request opened against the origin repository by a repository + collaborator +- WHEN the workflow runs +- THEN it SHALL deploy the environment with the cluster credentials + +#### Scenario: Fork pull request is not deployed + +- GIVEN a pull request whose head branch lives in a fork +- WHEN GitHub evaluates this workflow +- THEN the workflow SHALL NOT receive cluster credentials +- AND it SHALL NOT create or update a `hypershell-ci-pr-*` environment + +### Requirement: GitHub-Brokered Keycloak Authentication + +The per-pull-request environment's Keycloak SHALL broker interactive +authentication to GitHub, and SHALL NOT broker to Red Hat SSO, so an allowlisted +outside contributor can log in to an already-deployed origin-repo environment. +This is the one intentional divergence from the Keycloak model in +`local-development.spec.md` and the downstream Red Hat SSO brokering described +there: the realm structure (realm `hypershell`, the `hypershell-frontend`, +`hypershell-cli`, and `hypershell-provisioner` clients, and the per-gateway +client model) SHALL otherwise match, so the rest of the platform and the e2e +suite behave identically. This spec adds a dedicated confidential client +`hypershell-e2e` for CI (see Automated E2E Authentication). + +The target cluster SHALL provide one GitHub OAuth App (or GitHub App used as +the OAuth client) for these environments. The App's client id, client secret, +and a single stable callback URL SHALL come from configuration, not from the +overlay and not from a per-pull-request Route host. GitHub does not accept +wildcard redirect URIs and limits registered callback URLs, so GitHub SHALL +redirect only to that stable callback. The callback is cluster infrastructure, +in the same class as the shared Gateway: it receives GitHub's redirect and +completes the broker login against the Keycloak that the OAuth `state` +identifies (the pull-request number). GitHub SHALL NOT redirect to a per-PR +Keycloak Route host. If the client id, client secret, or stable callback URL is +unset, the workflow SHALL fail before the access comment is posted, rather than +leave an environment nobody can log into. + +The Keycloak SHALL configure a GitHub identity provider using that OAuth App +and the OAuth `read:org` scope so it can read the authenticating user's +organization membership. The realm SHALL restrict which GitHub identities may +complete authentication: + +- **Organization membership is the default gate.** A GitHub user who is a member + of the `openshift-online` organization SHALL be allowed to authenticate. +- **An allowlist admits extra usernames outside the organization.** The realm + SHALL support an allowlist of individual GitHub usernames that MAY authenticate + even when they are not members of `openshift-online`, so an outside contributor + can log in without being added to the organization. The allowlist is + additive: it widens login beyond the organization gate, never narrows it, and + it does not grant the listed user a CI deploy. +- **Everyone else is denied.** A GitHub user who is neither an `openshift-online` + member nor on the allowlist SHALL be denied at authentication; the environment + SHALL NOT create a HyperShell session for them. + +The organization gate and the allowlist SHALL be enforced during authentication +(for example through a first-broker-login flow step or an equivalent authenticator +that checks `read:org` membership and the configured allowlist), not merely by +post-hoc role assignment, so a denied user never obtains a token. The organization +name, the allowlist, the GitHub OAuth client id and secret, and the stable +callback URL SHALL come from configuration, not code, so a different +organization, allowlist, or OAuth App does not require an overlay edit. + +#### Scenario: Organization member authenticates + +- GIVEN a GitHub user who is a member of `openshift-online` +- WHEN they log in to a pull-request environment through GitHub +- THEN Keycloak SHALL allow the authentication +- AND SHALL create their HyperShell session + +#### Scenario: Allowlisted non-member authenticates + +- GIVEN a GitHub user who is not a member of `openshift-online` +- AND that username is on the environment's allowlist +- AND an origin-repo pull request has already deployed the environment +- WHEN they log in through GitHub +- THEN Keycloak SHALL allow the authentication +- AND that allowlist entry SHALL NOT have caused CI to deploy a fork pull request + +#### Scenario: Non-member, non-allowlisted user is denied + +- GIVEN a GitHub user who is neither an `openshift-online` member nor allowlisted +- WHEN they attempt to log in through GitHub +- THEN Keycloak SHALL deny the authentication +- AND SHALL NOT issue a token or create a session + +#### Scenario: GitHub redirects to the stable callback + +- GIVEN a pull-request environment's Keycloak brokers to GitHub +- AND the target cluster provides the configured stable callback URL +- WHEN a user completes GitHub authentication for pull request 232 +- THEN GitHub SHALL redirect to that stable callback URL +- AND GitHub SHALL NOT redirect to the `hypershell-ci-pr-232-keycloak` Route host +- AND the callback SHALL complete the broker login against that pull request's + Keycloak + +#### Scenario: Missing GitHub OAuth configuration fails bring-up + +- GIVEN the GitHub OAuth client id, client secret, or stable callback URL is unset +- WHEN the workflow deploys the environment +- THEN the job SHALL fail before posting an access comment +- AND it SHALL NOT leave an environment that nobody can log into + +#### Scenario: No Red Hat SSO brokering + +- GIVEN a pull-request environment's Keycloak +- WHEN a maintainer inspects its identity providers +- THEN GitHub SHALL be the brokered identity provider +- AND Red Hat SSO SHALL NOT be configured as an identity provider + +### Requirement: Admin Authorization and Developer-Tier Testing by Impersonation + +Every GitHub identity that authenticates to a pull-request environment (whether +by organization membership or by allowlist) SHALL be granted the realm roles +needed to fully drive the environment: `platform:admin` and `gateway:creator`, +as `../security/rbac-enforcement.spec.md` defines those roles. +`platform:admin` grants global gateway view and delete; it does not grant +gateway create. `gateway:creator` grants gateway create. The realm SHALL assign +both roles to brokered GitHub users on first broker login, so no manual role +assignment is required after login. + +To let an admin verify the developer permission boundary with the same GitHub +login, the environment SHALL support developer-tier testing by impersonation +rather than by a second interactive account. Keycloak federates one GitHub +identity to exactly one Keycloak user, so a GitHub user cannot "pick" between an +admin and a developer account; instead the environment SHALL seed a +developer-tier principal -- a Keycloak user that holds `gateway:viewer` (mapped +to `openshell-user` on a reachable gateway, per `e2e-testing.spec.md` area 9) +and holds neither `platform:admin` nor `gateway:creator` -- and SHALL enable +impersonation so an admin can obtain tokens for that principal. Impersonation +SHALL be available both interactively (an admin impersonates the developer +principal through the Keycloak account/admin console) and programmatically +through Keycloak token exchange, so the e2e suite can acquire developer-scoped +tokens without an interactive login (see Automated E2E Authentication). + +#### Scenario: Authenticated GitHub user can fully drive the environment + +- GIVEN a GitHub user authenticates to a pull-request environment +- WHEN their HyperShell session is created +- THEN they SHALL hold `platform:admin` and `gateway:creator` +- AND creating a gateway via the HyperShell API SHALL succeed because they hold + `gateway:creator` +- AND viewing or deleting any gateway SHALL succeed because they hold + `platform:admin` + +#### Scenario: Admin tests the developer boundary by impersonation + +- GIVEN an admin is logged in to a pull-request environment +- AND the environment has seeded a developer-tier principal with `gateway:viewer` + / `openshell-user` and without `platform:admin` or `gateway:creator` +- WHEN the admin obtains developer-scoped tokens by impersonating that principal +- THEN operations allowed to that tier (creating a sandbox on a reachable + gateway) SHALL succeed +- AND creating a gateway via the HyperShell API SHALL return `403 Forbidden` + +### Requirement: Automated E2E Authentication + +Because the pull-request environment's interactive login is GitHub-brokered and +brokered users have no resource-owner password grant, the e2e suite SHALL NOT +authenticate with a username/password grant against a brokered user. The +OpenShift driver's `acquire_oidc_token` and `acquire_gateway_token_with_role` +keep the same signatures and call sites `e2e-testing.spec.md` defines; they +SHALL become grant-agnostic. Kind and manual OpenShift runs default to the +password grant against seeded users. This workflow SHALL set `E2E_OIDC_GRANT` to +the service-account and token-exchange path so the suite still calls those +functions. + +The realm SHALL include a dedicated confidential client `hypershell-e2e` whose +service account holds `platform:admin` and `gateway:creator`. The workflow SHALL +NOT reuse `hypershell-provisioner` for e2e (that client holds `manage-clients` +and `manage-users`). After `make openshift-up`, CI SHALL read the `hypershell-e2e` +client secret from the deployed Keycloak namespace (a Kubernetes Secret in +`hypershell-ci-pr--keycloak`) and SHALL NOT take it from a repo secret +that cannot match a per-PR realm. The e2e suite's admin `acquire_oidc_token` +path SHALL obtain its HyperShell API token through that client's client- +credentials grant, so the CI run authenticates without a human GitHub login. + +Area 9 needs both a HyperShell API token (to 403 on gateway create) and a +per-gateway `openshell-user` token (to create a sandbox on a reachable gateway). +Both SHALL come from Keycloak token exchange impersonating the seeded developer +principal: `acquire_oidc_token` for the HyperShell API audience, and +`acquire_gateway_token_with_role` targeting that gateway's Keycloak client for +the gateway audience. Neither call SHALL use a password grant on these +environments. Area 10 SHALL acquire the platform-admin token through the +`hypershell-e2e` service account, not through a passworded `admin` user. + +The `hypershell-e2e` client secret and any impersonation credential SHALL NOT +appear in logs, the pull-request comment, or public artifacts. + +#### Scenario: CI acquires the admin token without a GitHub login + +- GIVEN a pull-request environment whose interactive login is GitHub-brokered +- WHEN the e2e suite acquires its admin token in CI +- THEN `acquire_oidc_token` SHALL use the `hypershell-e2e` client-credentials + grant, not a password grant +- AND the token SHALL carry `platform:admin` and `gateway:creator` +- AND CI SHALL have read that client secret from the Keycloak namespace after + `make openshift-up` + +#### Scenario: CI acquires the developer HyperShell API token by impersonation + +- GIVEN the seeded developer-tier principal exists in the realm +- WHEN the e2e suite needs a developer-scoped HyperShell API token +- THEN `acquire_oidc_token` SHALL obtain it by impersonating that principal + through token exchange +- AND the token SHALL NOT carry `platform:admin` or `gateway:creator` + +#### Scenario: CI acquires the developer gateway token by impersonation + +- GIVEN the seeded developer-tier principal exists in the realm +- AND a gateway with a per-gateway Keycloak client is running +- WHEN area 9 needs an `openshell-user` token for that gateway +- THEN `acquire_gateway_token_with_role` SHALL obtain it by token exchange + targeting that gateway client for the same principal +- AND it SHALL NOT use a password grant +- AND the token SHALL carry `openshell-user` for that gateway + +#### Scenario: Automation credentials do not leak + +- GIVEN CI holds the `hypershell-e2e` client secret +- WHEN a reader inspects the job logs, the pull-request comment, and public artifacts +- THEN that secret does not appear in any of them + +### Requirement: Legacy pr-test Deprecation + +This spec's workflow SHALL be the canonical pull-request e2e path, superseding the +legacy `components/pr-test/e2e-openshell.sh` script. That script is a hardcoded +OpenShift pull-request e2e script that predates the infra-agnostic suite and does +the job this workflow now owns. Some team members still run it directly, so it +SHALL NOT be removed yet; it SHALL be marked deprecated and kept working, and it +SHALL be removed in the future once that manual usage has migrated to the shared +harness. `openshift-development.spec.md` and `e2e-testing.spec.md` now defer this +deprecation window to this spec: removal of `e2e-openshell.sh` remains the +eventual goal; this spec is the living document that times it. During the +window those specs SHALL NOT require the script or the `pr_test` component to +already be gone. + +The ROKS variant `components/pr-test/e2e-openshell-roks.sh` is OUT OF SCOPE for +this spec. It targets IBM ROKS, which this pull-request ephemeral-environment +workflow does not cover, so this spec neither supersedes nor deprecates it. The +`pr_test` component and its CI wiring (`.github/component-paths.json` `pr_test` +entry, the `lint-pr-test` job, and component detection) SHALL remain in place -- +both because the deprecated `e2e-openshell.sh` is retained and because the ROKS +script continues to live under the same component. + +Deprecation of `e2e-openshell.sh` means it stays in place and runnable and every +authoritative reference marks it deprecated in favor of the shared harness and this +workflow. The script SHALL carry a deprecation notice at the top of the file that +names the canonical replacement (`tests/e2e/e2e-openshell.sh` with +`E2E_INFRA_DRIVER=openshift`, driven for pull requests by this workflow), and the +prose that documents it -- `CLAUDE.md`, `DEVELOPMENT.md`, and the skills that +mention `components/pr-test` (for example the deploy-cluster skills and +`skills/RECONCILE.md`) -- SHALL mark the pull-request OpenShift e2e usage +deprecated and point at the replacement, while leaving ROKS guidance unchanged. + +New work SHALL NOT depend on `components/pr-test/e2e-openshell.sh`. The +pull-request e2e path, new CI jobs, and new documentation SHALL use the shared +harness and this workflow, not the legacy script. The deprecated script SHALL NOT +be extended with new test areas; area coverage grows in the shared harness +(`tests/e2e/`) so the two paths do not diverge further. During the deprecation window, +this spec does NOT require consolidating its logic into +`tests/e2e/drivers/openshift.sh`. Removal is deferred, not cancelled: once manual +usage has migrated, a later change SHALL remove `e2e-openshell.sh`, and it SHALL +remove the `pr_test` component and its CI wiring only once the ROKS variant is also +retired or rehomed (the ROKS script is the other reason the component still +exists). + +#### Scenario: Legacy OpenShift script remains runnable but deprecated + +- GIVEN a team member still runs `components/pr-test/e2e-openshell.sh` directly +- WHEN this spec is in effect +- THEN the script SHALL remain present and runnable +- AND it SHALL carry a deprecation notice naming the canonical replacement +- AND the `pr_test` CI wiring SHALL remain intact so the script does not rot + +#### Scenario: ROKS script is untouched + +- GIVEN `components/pr-test/e2e-openshell-roks.sh` targets IBM ROKS +- WHEN this spec is in effect +- THEN the ROKS script SHALL be neither deprecated nor removed by this spec +- AND its documentation and usage guidance SHALL remain unchanged + +#### Scenario: Documentation marks the OpenShift pr-test script deprecated + +- GIVEN `CLAUDE.md`, `DEVELOPMENT.md`, and the skills reference + `components/pr-test/e2e-openshell.sh` +- WHEN a reader consults them +- THEN those references SHALL mark that OpenShift pull-request script deprecated +- AND SHALL point at the shared harness and this workflow as the canonical + pull-request e2e path +- AND ROKS references to `e2e-openshell-roks.sh` SHALL remain unchanged +- AND the `pr_test` component SHALL NOT be marked deprecated while the ROKS + script still lives there + +#### Scenario: Removal is deferred, not cancelled + +- GIVEN `e2e-openshell.sh` is deprecated but still in manual use +- WHEN the deprecation window is in effect +- THEN the script SHALL remain until that usage migrates to the shared harness +- AND removal SHALL remain the eventual goal, consistent with + `openshift-development.spec.md` and `e2e-testing.spec.md` + +#### Scenario: New work uses the canonical path + +- GIVEN a change adds a pull-request e2e test area or CI job +- WHEN the change is made +- THEN it SHALL use `tests/e2e/` and this spec's workflow +- AND it SHALL NOT extend `components/pr-test/e2e-openshell.sh` with new coverage + +## Design Decisions + +| Decision | Rationale | +|----------|-----------| +| Namespace name from the pull-request number (`hypershell-ci-pr-`) | A short, stable, collision-free identifier that every run for a pull request derives without external state; fits well within the DNS-label bound that keeps `-keycloak` under 63 characters. Branch names and commit SHAs are not stable for the life of one pull request | +| Same lifecycle labels as `make openshift-up`, with `pr-` as the environment id | Reuses `hypershell.redhat.io/owned` and `hypershell.redhat.io/environment` so status and cleanup tooling stay one selector set; the `pr-` prefix lets the reaper ignore local environments. CI must be able to patch namespaces; failing closed beats an unlabeled environment the reaper cannot see | +| `make openshift-up` on every deploying trigger, unconditionally | The command is already idempotent and reconciling, so one code path creates on first run and reconciles on later runs; branching on "does it exist" would duplicate logic and risk drift | +| CI stamps `hypershell.redhat.io/expires-at`; `make openshift-up` does not | The timebox is a pull-request cost bound, not a local-dev contract. Stamping from the workflow after bring-up refreshes active PRs without time-boxing developer namespaces | +| Origin `pull_request` only; Kind remains the merge-queue gate | `merge_group` has no stable pull-request number the way this namespace is keyed, and would race a `synchronize` swap on the same namespace. Fork PRs must not receive cluster credentials; the allowlist is login, not deploy | +| Per-PR concurrency group | Two in-flight swaps on one namespace can leave mixed digests; cancelling or queuing the older run keeps the comment SHA honest | +| One GitHub OAuth App and one stable callback | GitHub does not allow wildcard redirect URIs and limits callback URLs, so per-PR Keycloak Routes cannot be registered as GitHub callbacks. A cluster-scoped callback, like the shared Gateway, is the identity infrastructure this workflow depends on | +| Hidden HTML comment marker | Later runs have to find "the" access comment; a stable marker avoids editing an unrelated comment or posting duplicates | +| Immutable digests over untrusted tags | The environment runs exactly the artifact CI verified; pinning by `@sha256:` means a tag that is later re-pushed cannot silently change what the environment runs. A tag is a last-resort fallback only when no digest exists, and the fallback is recorded rather than silent | +| Close releases as primary path, timebox as backstop | The merge/close event frees the environment promptly in the common case; the timebox covers the case where the event does not fire or release cannot be confirmed | +| One updated comment per pull request, carrying the completed-swap commit SHA | The pull request shows the live environment's current state instead of a growing list of stale comments; pinning the SHA whose digest swap completed prevents claiming a commit the swap did not deploy | +| GitHub brokering, not Red Hat SSO | These are developer/debug environments; GitHub identity plus an organization gate and allowlist lets an outside contributor log in to an origin-repo environment, where Red Hat SSO would tie the environment to production identity | +| Organization gate by default, allowlist for extras | Organization membership is the common case; the additive allowlist admits outside contributors to login without adding them to the organization. Enforcing both during authentication (not by post-hoc roles) means a denied user never gets a token | +| Authenticated users get `platform:admin` and `gateway:creator`; developer tier by impersonation | `platform:admin` is view and delete only; create requires `gateway:creator`. A single GitHub identity federates to one Keycloak user, so there is no admin-or-developer account picker. A seeded `gateway:viewer` / `openshell-user` principal plus impersonation lets an admin still verify the developer boundary with the same login | +| Dedicated `hypershell-e2e` client; secret read from the deployed Keycloak | Brokered GitHub users have no password grant. A per-PR realm cannot share a repo-held provisioner secret, and `hypershell-provisioner` is too privileged (`manage-clients` / `manage-users`). Token exchange onto the HyperShell API client and onto the per-gateway client covers area 9 without a password grant. `E2E_OIDC_GRANT` keeps Kind and manual OpenShift on the password grant | +| Deprecate `e2e-openshell.sh` now, remove it later; leave ROKS alone | This workflow is the canonical pull-request OpenShift e2e path, so the legacy `e2e-openshell.sh` is superseded. Team members still run it, so it is deprecated first (notice + docs pointing at the shared harness) and removed later once that usage migrates. New coverage lands only in `tests/e2e/`. The ROKS variant is out of scope; the `pr_test` component stays until both scripts are gone | diff --git a/specs/platform/openshift-development.spec.md b/specs/platform/openshift-development.spec.md index 90bdac2f..22e88cbf 100644 --- a/specs/platform/openshift-development.spec.md +++ b/specs/platform/openshift-development.spec.md @@ -4,7 +4,10 @@ **Status:** Draft **Jira:** HYPERSHELL-44 **Related:** `local-development.spec.md` -- Kind lifecycle and component swap; - `e2e-testing.spec.md` -- driver interface contract and CI pipeline; + `e2e-testing.spec.md` -- driver interface contract and Kind CI; + `ephemeral-pr-environments.spec.md` (HYPERSHELL-240) -- pull-request + OpenShift CI, timebox, GitHub-brokered Keycloak, and `e2e-openshell.sh` + deprecation; `control-plane.spec.md` -- reconciler behavior; `openshell-gateway-routing.spec.md` -- GRPCRoute provisioning @@ -17,27 +20,26 @@ deploys a complete HyperShell environment to an ephemeral namespace on an OpenShift cluster. The developer can swap one component at a time from the working tree, exactly as `make kind--up` does today. -This spec also defines automated end-to-end testing on OpenShift. This spec owns the -OpenShift lifecycle, the `deploy/openshift/` overlay, the cluster bootstrap, and the -OpenShift CI workflow. `e2e-testing.spec.md` owns the e2e driver interface contract -and the `tests/e2e/drivers/openshift.sh` driver file that implements the OpenShift -side of that contract. It defines a CI workflow that deploys HyperShell to an ephemeral -namespace for each pull request, runs the e2e suite, gives the environment access -details to the developer, keeps the environment alive for the life of the pull -request, and releases the environment when the pull request merges or closes. It -defines the consolidation of the legacy `components/pr-test/e2e-openshell.sh` -script into the shared e2e harness. +This spec owns the OpenShift lifecycle, the `deploy/openshift/` overlay, the +cluster bootstrap, and the OpenShift side of the e2e driver contract. +`e2e-testing.spec.md` owns the e2e driver interface and the +`tests/e2e/drivers/openshift.sh` file that implements the OpenShift side of that +contract. Automated pull-request CI on OpenShift -- namespace naming, continuous +deployment, timebox, access comment, GitHub-brokered Keycloak, and the +deprecation of `components/pr-test/e2e-openshell.sh` -- is owned by +`ephemeral-pr-environments.spec.md` (HYPERSHELL-240). This spec supplies the +lifecycle that workflow runs (`make openshift-up` / `make openshift-down`, the +overlay, and the OpenShift e2e driver) so a local deployment and a CI +deployment cannot drift. HyperShell uses one ephemerality model -- an ephemeral namespace on an existing OpenShift cluster -- in two contexts: - **Local development** -- The developer supplies a target OpenShift cluster. The `make openshift-up` command deploys into an isolated namespace on that cluster. -- **Pull request CI** -- The pipeline deploys into an ephemeral namespace on a - shared target environment that has capacity for several simultaneous - pull-request namespaces. The namespace lives for the life of the pull request, - so it also serves as a live environment to debug e2e failures and to do other - work. +- **Pull request CI** -- Specified in `ephemeral-pr-environments.spec.md`. The + pipeline deploys into an ephemeral namespace on a shared target environment + using the same `make openshift-up` command this spec defines. HyperShell does not provision full clusters for CI. A full cluster (for example a ROSA cluster) adds 15-20 minutes of provisioning time per run; an ephemeral @@ -52,10 +54,11 @@ deployment cannot drift. ### Scope -This spec covers the OpenShift lifecycle driver, the OpenShift e2e driver, the -ephemeral-namespace CI workflow, the environment-access handoff to the developer, -and the reconciliation of the OpenShift deploy overlay against a production -reference. +This spec covers the OpenShift lifecycle driver, the OpenShift e2e driver, and +the reconciliation of the OpenShift deploy overlay against a production +reference. Pull-request CI, the access handoff comment, the timebox, and the +`e2e-openshell.sh` deprecation window are specified in +`ephemeral-pr-environments.spec.md`. This spec does not change the Kind driver, the Kind lifecycle scripts, or the Kind CI job. This spec does not redesign the e2e driver interface contract that `e2e-testing.spec.md` defines; it implements the OpenShift side of that contract. @@ -725,296 +728,75 @@ suite validates both infrastructure targets. ### Requirement: E2E Script Consolidation -The legacy `components/pr-test/e2e-openshell.sh` script SHALL be consolidated into -the shared e2e harness. The OpenShift-specific logic in that script SHALL move into -`tests/e2e/drivers/openshift.sh`. The infrastructure-agnostic test logic SHALL use -the shared `tests/e2e/e2e-openshell.sh` suite. After the consolidation, no -OpenShift e2e logic SHALL remain hardcoded outside the driver model. - -The `components/pr-test/` component SHALL be removed after the consolidation. The -removal SHALL update every reference to `components/pr-test/`, including the -`pr_test` entry in `.github/component-paths.json` and any CI workflow that runs the -legacy script, so that no reference points to a removed path. - -#### Scenario: Legacy script is consolidated - -- GIVEN `components/pr-test/e2e-openshell.sh` runs OpenShift e2e tests today -- WHEN the consolidation is complete -- THEN the OpenShift-specific logic lives in `tests/e2e/drivers/openshift.sh` -- AND the test areas run through `tests/e2e/e2e-openshell.sh` -- AND `components/pr-test/` is removed - -#### Scenario: No dangling references remain - -- GIVEN `components/pr-test/` is removed +The legacy `components/pr-test/e2e-openshell.sh` script SHALL be superseded by the +shared e2e harness and the pull-request workflow in +`ephemeral-pr-environments.spec.md`. That spec owns the deprecation window: +the script SHALL stay present and runnable, SHALL carry a deprecation notice, +and SHALL be removed only after manual usage has migrated. The ROKS variant +`components/pr-test/e2e-openshell-roks.sh` is out of scope there and SHALL remain. +The `pr_test` component and its CI wiring SHALL remain until both scripts are +gone. + +The eventual end state is unchanged: OpenShift-specific logic lives in +`tests/e2e/drivers/openshift.sh`, infrastructure-agnostic tests live in +`tests/e2e/e2e-openshell.sh`, and no OpenShift e2e logic remains hardcoded +outside the driver model. This spec SHALL NOT require the script or the +`pr_test` component to already be removed. + +#### Scenario: Legacy script is deprecated, not yet removed + +- GIVEN `components/pr-test/e2e-openshell.sh` still has manual users +- WHEN this spec is in effect +- THEN the script remains present and runnable +- AND `ephemeral-pr-environments.spec.md` is the contract for when it is removed +- AND `components/pr-test/` is not removed while the ROKS variant still lives there + +#### Scenario: No dangling references after eventual removal + +- GIVEN `e2e-openshell.sh` has been removed after the deprecation window +- AND the ROKS variant has been retired or rehomed - WHEN a maintainer inspects CI configuration and component registration - THEN no workflow references the removed path - AND `.github/component-paths.json` no longer contains a `pr_test` entry that - points to the removed path - -### Requirement: Ephemeral CI Environment Provisioning - -The CI workflow SHALL deploy HyperShell to an ephemeral environment on a shared -target OpenShift cluster for each pull request that runs the e2e suite. An -ephemeral environment is the namespace group that the Keycloak Namespace -requirement defines. The workflow SHALL NOT provision a full cluster. -The target cluster SHALL have capacity for several simultaneous pull-request -environments. The workflow MAY provision the environment through the -ephemeral-namespace-operator or an equivalent mechanism. - -The workflow SHALL key the ephemeral environment to the pull request, so that every -run for one pull request uses the same environment. On the first run for a pull -request, the workflow SHALL create the environment and deploy HyperShell. On a -later run for the same pull request, the workflow SHALL reuse the existing -environment and redeploy with the same `make openshift-up` reconcile that the first -run uses. The redeploy SHALL deploy the images that Konflux built for the pull -request, injecting those image references by digest into the deployment the same way -the Kind job does through `scripts/kind/set-component-images.sh`, rather than rebuild -images from the working tree, so that the PR e2e job tests the images that ship. The -redeploy SHALL let the overlay reconcile bring the running environment to the new -desired state, including pruning resources that the overlay no longer declares, so -that the environment does not drift from the overlay and so that the environment -serves as a live development and debug environment across the life of the pull -request. - -The workflow SHALL keep the ephemeral environment alive after the e2e suite -completes, whether the suite passes or fails, so that the developer can inspect the -live environment. The workflow SHALL release the environment when the pull request -merges or closes. The workflow SHALL NOT release the environment at the end of a -single CI run. - -The environment SHALL be cost-bounded even though it lives across the pull request. -When the workflow provisions the environment through a reservation mechanism with a -duration, the workflow SHALL renew or set the duration to cover the pull request, -and SHALL rely on that duration as a backstop, so that an abandoned pull request -does not hold the environment forever. The workflow SHALL release the environment -on pull-request close as the primary path, and SHALL let the duration reclaim the -environment when the close event does not fire. - -The workflow SHALL NOT leave an orphaned environment. If the release step cannot -confirm the release, the workflow SHALL report the failure so that an operator can -free the environment. - -#### Scenario: First run creates the pull-request environment - -- GIVEN a pull request runs the e2e suite for the first time -- WHEN the CI workflow runs -- THEN the workflow creates an ephemeral environment keyed to the pull request -- AND the workflow deploys HyperShell into that environment - -#### Scenario: Later run reuses the environment - -- GIVEN an ephemeral environment already exists for a pull request -- WHEN a later push triggers the CI workflow for the same pull request -- THEN the workflow reuses the existing environment -- AND the workflow reruns `make openshift-up` to reconcile the full overlay -- AND the workflow deploys the Konflux-built images by digest, without rebuilding from the working tree -- AND the reconcile prunes resources that the overlay no longer declares - -#### Scenario: Environment survives after tests complete - -- GIVEN the e2e suite completes for a pull request, whether it passes or fails -- WHEN the CI run finishes -- THEN the ephemeral environment stays alive -- AND the developer can access the live environment - -#### Scenario: Environment releases on pull-request merge - -- GIVEN an ephemeral environment exists for a pull request -- WHEN the pull request merges or closes -- THEN the workflow releases the environment -- AND a reservation duration reclaims the environment when the close event does not - fire - -### Requirement: Environment Access Handoff - -The CI workflow SHALL give the developer the access details for the ephemeral -environment, which lives for the life of the pull request, so that the developer -can inspect a failing test and do live work on the environment. The access details -have two parts: non-secret facts and a credential. The non-secret facts are the -environment namespaces, the OpenShift console URL for the platform namespace, the -API Route URL, and the web-console Route URL. The credential is a short-lived, -namespace-scoped token or kubeconfig that grants access to the environment. - -The workflow SHALL deliver the non-secret facts through a pull request comment, and -SHALL keep the comment current across runs, so that the comment reflects the live -environment for the pull request. The comment MAY include an `oc login` command -template, but the template SHALL show the credential as redacted, for example -`oc login --server= --token=`. The comment SHALL NOT contain the -credential itself. - -The workflow SHALL deliver the credential only through a channel that only an -authorized developer can read, such as a masked secret or a restricted artifact. -The workflow SHALL NOT print a kubeconfig, a token, or a password into a pull -request comment, into the job logs, or into a public artifact. The credential SHALL -be short-lived and namespace-scoped, so that a leak has a bounded blast radius. - -#### Scenario: Developer receives environment links in a pull request comment - -- GIVEN the CI workflow deploys the ephemeral environment -- WHEN the deployment is ready -- THEN the workflow posts a pull request comment with the environment namespaces, - the console URL, the API Route URL, and the web-console Route URL -- AND the comment shows any `oc login` template with the credential redacted -- AND the workflow delivers the credential through a secure channel, not the comment - -#### Scenario: Credentials do not leak - -- GIVEN the workflow delivers environment access details -- WHEN a reader inspects the pull request comment, the job logs, and the public - artifacts -- THEN no kubeconfig, token, or password appears in the comment, the logs, or the - public artifacts -- AND the credential is available only through a secure channel - -### Requirement: Blessed OpenShift Overlay - -The `deploy/openshift/` overlay SHALL derive from `deploy/base/` and SHALL NOT -duplicate the base resources. The `deploy/base/` overlay is the independent baseline -for drift validation: both `deploy/openshift/` and the production `deploy/hub/` -overlay derive from it, and it is not derived from either, so a change in one overlay -cannot flow into the reference before the check runs. The drift check SHALL NOT use -`deploy/hub/` as the reference for `deploy/openshift/`, because `deploy/hub/` derives -from `deploy/openshift/` and a change would reach the reference before comparison. - -The overlay SHALL parameterize the namespace, so that a deployment into an ephemeral -namespace does not require an overlay edit. The overlay SHALL NOT hardcode -`hypershell-system`; the deployment SHALL set the namespace from configuration, the -same way `make openshift-up` maps the platform namespace to `OPENSHIFT_NAMESPACE`. - -Each overlay SHALL differ from `deploy/base/` only within a declared allowlist. For -`deploy/openshift/`, the allowed differences are the OpenShift-specific additions the -overlay layers on the base (the Route, the SecurityContextConstraints binding, the -certificates, and the network policies) together with the namespace, the name prefix, -the image references, the gateway base domain `GATEWAY_API_BASE_DOMAIN`, and the SSO -configuration. Ephemeral OpenShift and hub intentionally differ: ephemeral OpenShift -bundles a per-environment Keycloak and the bundled CNPG `Cluster` that the base -provides, while `deploy/hub/` (production) shares one Keycloak per cluster and uses a -managed database, so `deploy/hub/` deletes the bundled Keycloak unit and the bundled -CNPG `Cluster`. The `deploy/hub/` allowlist SHALL therefore additionally permit those -deletions. These declared differences are intentional, not drift. - -The overlay SHALL replace its placeholder values with values that a real -environment supplies through configuration, not through code. The known placeholder -to reconcile is the gateway base domain -`GATEWAY_API_BASE_DOMAIN=openshell.stage.example.com`, which SHALL come from -configuration, so that a deployment to a different cluster does not require an -overlay edit. - -A drift check SHALL compare each overlay against `deploy/base/` after a defined -normalization step, and SHALL fail when an overlay differs from the base outside its -declared allowlist. The drift check SHALL run in CI, so that a pull request that -changes an overlay cannot merge an unintended drift. Because both overlays are -checked against the same independent base, `deploy/openshift/` and `deploy/hub/` -cannot silently diverge on any resource that neither allowlist covers. - -#### Scenario: Base domain comes from configuration - -- GIVEN the overlay needs a gateway base domain -- WHEN a developer deploys to a cluster with a different base domain -- THEN the deployment reads the base domain from configuration -- AND the developer does not edit the overlay to change the base domain - -#### Scenario: Drift check fails on unintended drift - -- GIVEN `deploy/base/` is the independent baseline for the overlays -- WHEN a pull request changes `deploy/openshift/` outside its declared allowlist -- THEN the drift check fails in CI -- AND the pull request cannot merge until the drift is resolved - -## Deploy Directory Structure - -The repo root `deploy/` directory contains all kustomize overlays for the platform. It is the **single source of truth** for desired state across all deployment modes (development, testing, and production). The directory structure reflects the layering relationship: - -``` -deploy/ -├── base/ # Foundation: all platform resources (namespace, API, controller, DB, web console, Keycloak) -│ ├── kustomization.yaml -│ ├── namespace.yaml # hypershell-system -│ ├── api-server.yaml # Deployment + Service -│ ├── controller.yaml # Deployment + Service (includes GATEWAY_IMAGE/GATEWAY_SUPERVISOR_IMAGE env vars) -│ ├── controller-rbac.yaml # ClusterRole + ClusterRoleBinding for tenant reconciliation -│ ├── web-console.yaml # Deployment + Service -│ ├── hypershell-db-cluster.yaml # CNPG Cluster resource -│ ├── keycloak/ # Shared Keycloak (base config, realm import) -│ │ └── ... -│ └── ... -├── kind/ # Kind-specific: bundles Keycloak + shared Gateway; disables OIDC by default -├── openshift/ # OpenShift development: bundles Keycloak; adds Routes, TLS, RBAC, NetworkPolicies -├── ibm/ # IBM ROKS: Route mode; image mirroring and cluster-wide RBAC -├── hub/ # Production (multi-cluster); shared Keycloak per cluster; managed external DB -├── cloud-hub-ingress-bootstrap/ # Cloud Hub bootstrap: shared Gateway API + wildcard DNS/TLS (AWS/functional clusters) -├── keycloak/ # OpenShift Keycloak overlay: adds Route + domain patching -└── components/ # (Future) per-component overlays for flexibility -``` - -Each overlay builds on `base/` and adds only its specific differences: - -- **`kind/`**: Keycloak with kind-local domain; disables OIDC by default -- **`openshift/`**: Keycloak with Route; adds cert-manager Issuers/Certificates, per-tenant PKI, SecurityContextConstraints, NetworkPolicies -- **`ibm/`**: Extends `openshift/` with image refs for internal registry, cluster-wide RBAC, namespace mapping -- **`hub/`** (production): Removes bundled Keycloak (shares cluster-level instance); removes CNPG `Cluster` (uses external managed DB) - -The `keycloak/` overlay (separate from `base/keycloak/`) is used in production to customize the shared Keycloak instance (Route, domain patching). - -### Known Limitations in `deploy/openshift/` - -The current `deploy/openshift/` overlay has **unresolved runtime dependencies** documented below. These are not integration gaps but known limitations that the bootstrap workflow (e.g. `skills/deploy/deploy-cluster/SKILL.md`) addresses manually: - -1. **Missing `hypershell-api-config` Secret**: The API server and controller Deployments reference a Secret (`hypershell-api-config`) with keys `api-service.{issuerUrl,clientId,clientSecret,jwkCertUrl}`. This Secret is **not defined in the repo**; it must be created manually from the Keycloak realm's client credentials and the Keycloak Route host. Without it, pods fail with `CreateContainerConfigError`. The bootstrap workflow creates this Secret in Step 3 after Keycloak is deployed. - -2. **Keycloak bundled but without external Route**: The overlay bundles Keycloak with `KC_HOSTNAME=https://keycloak.hypershell.localhost` (a Kind-only hostname). On real OpenShift clusters, this hostname is unreachable, so tokens minted by the bundled Keycloak carry a bogus, externally-unreachable issuer. The bootstrap workflow deploys Keycloak via the `deploy/keycloak/` overlay (which adds a Route and patches `KC_HOSTNAME`) and creates the `hypershell-api-config` Secret from the actual Route host. - -3. **Hardcoded `GATEWAY_API_BASE_DOMAIN` placeholder**: The overlay hardcodes `GATEWAY_API_BASE_DOMAIN=openshell.stage.example.com`, which is a placeholder. This value is only used if Gateway API ingress mode is enabled (via setting `GATEWAY_INGRESS_MODE=gateway-api`); Route mode (the default) does not require it. The bootstrap workflow or operator automation must parameterize this value when switching to Gateway API mode. - -### Requirement: OpenShift CI Workflow Shape - -The OpenShift e2e CI job SHALL extend the existing e2e workflow structure that -`.github/workflows/e2e.yml` defines, so that both drivers share the same gating and -the same summary pattern. The job SHALL gate on the Konflux image builds the same -way the Kind job does, so that the images tested on OpenShift are the images that -ship. The OpenShift e2e job SHALL run whenever the Kind e2e job runs, so that a -change that triggers the Kind e2e suite also runs the OpenShift e2e suite. A change -under `deploy/openshift/` or the OpenShift lifecycle scripts SHALL also trigger the -job. - -The job SHALL run these steps in order: gate on Konflux images; reuse or create the -ephemeral environment for the pull request; deploy with `make openshift-up`, which -reconciles the full overlay, then inject the Konflux-built image references by digest -(the same digest-injection pattern the Kind job uses through -`scripts/kind/set-component-images.sh`) rather than rebuild from the working tree; run -`E2E_INFRA_DRIVER=openshift bash tests/e2e/e2e-openshell.sh`; collect diagnostics on -failure; and post or update the pull request comment with the environment access -details. The job SHALL keep the -environment alive after the run. A separate step, triggered on pull-request merge -or close, SHALL release the environment. The CI summary gate SHALL include the -OpenShift job result, so that the gate reflects both drivers. - -The `.github/component-paths.json` registration SHALL include the OpenShift paths in -the e2e component so that the OpenShift e2e job triggers on the same changes as the -Kind e2e job. The registration SHALL add `deploy/openshift/**` and the OpenShift -lifecycle script paths to the e2e component paths. - -#### Scenario: OpenShift job gates on Konflux images - -- GIVEN a pull request runs the e2e suite -- WHEN the OpenShift CI job runs -- THEN the job waits for the Konflux image builds to complete -- AND the job deploys the images that Konflux built - -#### Scenario: OpenShift e2e runs whenever Kind e2e runs - -- GIVEN a change triggers the Kind e2e job -- WHEN CI evaluates which e2e jobs to run -- THEN the OpenShift e2e job runs as well -- AND a change under `deploy/openshift/` also triggers the OpenShift e2e job - -#### Scenario: Summary gate includes the OpenShift result - -- GIVEN the CI workflow runs both the Kind job and the OpenShift job -- WHEN the summary gate evaluates the results -- THEN the gate includes the OpenShift job result -- AND the gate fails when the OpenShift job fails + points to a removed path + +### Requirement: Pull-Request CI Uses This Lifecycle + +Automated OpenShift pull-request CI is specified in +`ephemeral-pr-environments.spec.md` (HYPERSHELL-240). That spec owns namespace +naming, deploy triggers, origin-only trust boundary, timebox and reaping, the +access comment, GitHub-brokered Keycloak, e2e grant selection, and the +`e2e-openshell.sh` deprecation window. This spec SHALL NOT restate those +requirements. + +The pull-request workflow SHALL deploy and release with the same lifecycle this +spec defines: `make openshift-up` and `make openshift-down`, the +`deploy/openshift/` overlay, the namespace-group derivation, the ownership +labels `hypershell.redhat.io/owned` and `hypershell.redhat.io/environment`, and +the OpenShift e2e driver. A local `make openshift-up` and a CI deploy SHALL NOT +drift. `make openshift-up` SHALL NOT stamp a pull-request timebox; expiry is a +CI annotation defined in `ephemeral-pr-environments.spec.md`. + +Kind e2e, including the merge-queue gate, stays in `e2e-testing.spec.md`. The +OpenShift pull-request job SHALL run for origin `pull_request` events that +trigger Kind e2e, and for changes under `deploy/openshift/` or the OpenShift +lifecycle scripts. It SHALL NOT run on `merge_group`. + +#### Scenario: CI reuses the local-dev lifecycle + +- GIVEN an origin pull request that triggers this workflow +- WHEN the workflow deploys the environment +- THEN it SHALL run `make openshift-up` with `OPENSHIFT_NAMESPACE` set as + `ephemeral-pr-environments.spec.md` defines +- AND it SHALL run the OpenShift e2e driver this spec defines +- AND it SHALL NOT use a second OpenShift bring-up path + +#### Scenario: Merge-queue stays on Kind + +- GIVEN a pull request enters the GitHub merge queue +- WHEN CI evaluates which jobs to run +- THEN the Kind e2e job SHALL run as `e2e-testing.spec.md` defines +- AND the OpenShift pull-request environment workflow SHALL NOT run ### Requirement: Cluster Infrastructure Prerequisites @@ -1029,12 +811,15 @@ administrator provisions this infrastructure once per cluster, as `infrastructure/GATEWAY-SETUP.md` describes. Both local development and pull-request CI depend on this precondition, because an ephemeral namespace grants namespace-scoped access and does not grant permission to create cluster -infrastructure. +infrastructure. The GitHub OAuth App and stable callback that +`ephemeral-pr-environments.spec.md` requires are additional cluster infrastructure +for pull-request CI only; local `make openshift-up` does not depend on them. -The `make openshift-up` command SHALL check for the required infrastructure, and -SHALL report a clear error when the infrastructure is missing, rather than deploy a -broken environment. The CI workflow SHALL run against a target cluster that already -provides this infrastructure. +The `make openshift-up` command SHALL check for the required Gateway +infrastructure, and SHALL report a clear error when it is missing, rather than +deploy a broken environment. The pull-request workflow in +`ephemeral-pr-environments.spec.md` SHALL run against a target cluster that +already provides the Gateway infrastructure and the GitHub OAuth callback. #### Scenario: Missing infrastructure fails fast @@ -1048,7 +833,8 @@ provides this infrastructure. - GIVEN the CI workflow deploys into an ephemeral environment - WHEN the workflow prepares the environment for deployment - THEN the workflow relies on the target cluster for the shared Gateway, the - GatewayClass, and the certificate issuer + GatewayClass, the certificate issuer, and the GitHub OAuth callback + `ephemeral-pr-environments.spec.md` requires - AND the workflow does not attempt to create that cluster infrastructure from the ephemeral namespaces