feat(helm): add BackendTLSPolicy support

[bsquizz]

Add BackendTLSPolicy support for end-to-end TLS between Gateway proxies and the OpenShell gateway pod, with automated backend CA ConfigMap creation, single-stage install with cert-manager, configuration validation, and explicit mTLS control.

Summary

This PR enables end-to-end TLS for Gateway API deployments by adding BackendTLSPolicy support. The Gateway proxy terminates client-facing TLS and re-encrypts when connecting to the backend, validating the gateway pod's certificate against an auto-created CA ConfigMap. A new server.tls.enableMtls flag provides explicit control over mTLS client certificate authentication, which must be disabled when using BackendTLSPolicy because ingress proxies cannot present client certificates to backends.

The certgen hook now polls for cert-manager certificates, and cert-manager Certificate resources run as pre-install hooks (weight -30) before certgen (weight -20), ensuring proper ordering. This eliminates the two-stage install requirement - a single helm install succeeds when cert-manager issues certificates within the configurable timeout. By default, the install fails immediately if the timeout is reached (pkiInitJob.failOnTimeout=true), providing clear feedback that BackendTLSPolicy is non-functional rather than silently succeeding with an incomplete configuration.

Configuration validation prevents invalid combinations - the chart fails at install time if both server.tls.enableMtls=true and grpcRoute.backendTLSPolicy.enabled=true are set.

Traffic flow: client → HTTPS → Gateway (terminate) → TLS (re-encrypt) → gateway pod

Related Issue

No issue required: adds optional deployment path for OpenShift 4.22+ and other platforms with BackendTLSPolicy support.

Changes

• BackendTLSPolicy resource: New grpcRoute.backendTLSPolicy.* values to optionally create a BackendTLSPolicy that configures the Gateway proxy to validate the backend's TLS certificate
• Auto-created backend CA: The certgen hook now automatically creates the backend CA ConfigMap (<fullname>-backend-ca) from the server certificate CA when backendTLSPolicy.enabled=true
  • With pkiInitJob (default): created during install/upgrade
  • With cert-manager: Certificate resources run as pre-install hooks (weight -30), then certgen hook (weight -20) polls for certificate issuance and creates ConfigMap — single-stage install
• Fixed hook ordering: Certificate and Issuer resources are now pre-install hooks with weight -30, running before certgen (weight -20). This fixes the chicken-and-egg problem where certgen was waiting for Secrets from Certificates that hadn't been created yet.
• Configuration validation: Chart fails at install time if both server.tls.enableMtls=true and grpcRoute.backendTLSPolicy.enabled=true are set, preventing invalid configurations from being deployed
• Configurable timeout: New pkiInitJob.timeoutSeconds value (default 120) controls the polling duration for cert-manager certificate issuance
  • The value represents the actual polling time users experience
  • Job deadline is automatically set to (timeoutSeconds + 30) to allow buffer time for ConfigMap creation
  • Example: --set pkiInitJob.timeoutSeconds=180 polls for exactly 180 seconds
• Fail-fast by default: New pkiInitJob.failOnTimeout value (default true) controls hook behavior on timeout
  • When true (default): hook fails immediately, install fails, clear feedback that BackendTLSPolicy is incomplete
  • When false: hook succeeds with warning, install completes, BackendTLSPolicy non-functional until helm upgrade
• Troubleshooting documentation: Added detailed troubleshooting section for the error TLS error: Secret is not supplied by SDS with step-by-step resolution when the backend CA ConfigMap is missing
• Explicit mTLS control: New server.tls.enableMtls flag (default true) to control mTLS client certificate authentication independently of BackendTLSPolicy
  • Users must set to false when using BackendTLSPolicy
  • Chart validates this requirement and fails at install time if violated
• Documentation: Updated OpenShift and ingress docs with end-to-end TLS setup instructions, hook ordering explanation, timeout configuration, failure behavior options, validation behavior, and troubleshooting guidance
• Tests: Added Helm unit tests for BackendTLSPolicy rendering and mTLS flag behavior

Testing

• mise run pre-commit passes
• Helm unit tests added for BackendTLSPolicy and enableMtls flag (106 tests passing)
• Code compiles and passes clippy
• Fixed hook ordering verified with Helm template rendering
• Validation tested: fails with enableMtls=true + backendTLSPolicy=true, succeeds with enableMtls=false
• Documented troubleshooting for "TLS error: Secret is not supplied by SDS"
• E2E tests with cert-manager (manual validation on OpenShift 4.22+ or Envoy Gateway)

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (Helm README, ingress.mdx, openshift.mdx)
• Breaking change documented in PR description and commit message
• Hook ordering fix for cert-manager resources
• Configuration validation for incompatible settings

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[bsquizz]

Starting to test E2E in both k8s (kind) and OpenShift.

[bsquizz]

Tested sandbox connectivity on kind with a pre-existing network Gateway, with TLS enabled. Verified BackendTLSPolicy exists.

  helm upgrade --install openshell deploy/helm/openshell \
    --namespace openshell \
    --create-namespace \
    --set image.repository=localhost/openshell/gateway \
    --set image.tag=dev \
    --set image.pullPolicy=Never \
    --set supervisor.image.repository=localhost/openshell/supervisor \
    --set supervisor.image.tag=dev \
    --set supervisor.image.pullPolicy=Never \
    --set grpcRoute.enabled=true \
    --set grpcRoute.backendTLSPolicy.enabled=true \
    --set grpcRoute.gateway.create=false \
    --set grpcRoute.gateway.name=gw \
    --set grpcRoute.gateway.namespace=gw-namespace \
    --set server.tls.enableMtls=false \
    --set server.oidc.issuer='' \
    --set server.auth.allowUnauthenticatedUsers=true \
    --set 'grpcRoute.hostnames[0]=openshell-gateway.gw.localhost' \

both with / without --set certManager.enabled=true (with cert-manager, a second 'upgrade' is required after cert-manager issues the certificate in order to create the ConfigMap for the BackendTLSPolicy to consume. This is documented and expected)

[bsquizz]

Changed the way this works so that a follow-up 'helm upgrade' is not required. Now, the certgen hook will wait for cert-manager to issue a certificate. Timeout is configurable, and failing upon timeout is also configurable (default: true)

Tested on OpenShift 4.22.9 when using cert-manager with:

  helm install openshell deploy/helm/openshell \
    --namespace bsquizza-openshell-test \
    --set image.repository=openshell-gateway \
    --set image.tag=9d0f8cc \
    --set supervisor.image.repository=openshell-supervisor \
    --set supervisor.image.tag=9d0f8cc \
    --set grpcRoute.enabled=true \
    --set grpcRoute.backendTLSPolicy.enabled=true \
    --set grpcRoute.gateway.create=false \
    --set grpcRoute.gateway.name=openshell-grpc-gateway \
    --set grpcRoute.gateway.namespace=openshift-ingress \
    --set server.tls.enableMtls=false \
    --set server.oidc.issuer='' \
    --set server.auth.allowUnauthenticatedUsers=true \
    --set 'grpcRoute.hostnames[0]=testgw.mygw.example.com' \
    --set podSecurityContext.fsGroup=null \
    --set securityContext.runAsUser=null \
    --set certManager.enabled=true \
    --set pkiInitJob.timeoutSeconds=180

Also tested without cert-manager by removing:

    --set certManager.enabled=true \
    --set pkiInitJob.timeoutSeconds=180

[jhjaggars]

Review findings

1. Enabling BackendTLSPolicy on an existing built-in-PKI release creates the wrong trust anchor. The hook generates a new in-memory PKI bundle on every run, but run_kubernetes skips the already-existing server/client/JWT Secrets. It then writes the newly generated bundle's unrelated CA to the backend ConfigMap. On an upgrade that first enables this feature, the Gateway proxy will reject the existing server certificate. The backend CA should come from the authoritative server Secret (as it does in cert-manager mode), or run_kubernetes should report whether it created the current bundle. Details: crates/openshell-server/src/certgen.rs:126-130, :214-245, :360-362.

2. The backend CA ConfigMap is never reconciled. Once the ConfigMap exists, certgen returns without comparing or updating ca.crt. A later CA rotation or server Secret replacement therefore leaves the ingress proxy trusting stale material and can break external access. Please reconcile the ConfigMap from the authoritative CA on every hook run rather than treating existence as sufficient. Details: crates/openshell-server/src/certgen.rs:345-357.

3. The cert-manager Issuers and Certificates now escape Helm's release lifecycle. These long-lived resources were previously normal release objects. Marking all of them as hooks means Helm will not manage them as part of uninstall or removal through certManager.enabled=false; they can remain active and continue reconciling/renewing Secrets. Please preserve lifecycle cleanup or use a different ordering mechanism. Details: deploy/helm/openshell/templates/cert-manager-pki.yaml:11-26, :78-80, :156-158.

4. Required companion references and deployment skills were not updated. The canonical PKI architecture still describes only Secret outputs and the old idempotency contract; the Kubernetes gateway-config example does not explain conditional client_ca_path; and the cluster/Helm skills have no BackendTLSPolicy, backend CA ConfigMap, timeout, or enableMtls diagnostics. The gateway-deployment maintenance map explicitly routes this kind of change to debug-openshell-cluster and helm-dev-environment. Details: architecture/gateway.md:628-659, docs/reference/gateway-config.mdx:421-440, .agents/skills/debug-openshell-cluster/SKILL.md:320-374, .agents/skills/helm-dev-environment/SKILL.md:180-193, .agents/skills/sync-agent-infra/SKILL.md:45.

[bsquizz]

Implemented feedback and re-tested successfully in our e2e setup

[mrunalp]

/ok to test cb4d05f