ORCHORDS · ORCHORDS · Sep 5, 2026 · Sep 5, 2026
@@ -0,0 +1,76 @@
+# Envoy Listener Hot Reload Playbook
+
+## Purpose
+
+Define the operational procedure for safely reloading Envoy listener configuration without dropping in-flight connections. The procedure covers both LDS (Listener Discovery Service) hot reload and full restart scenarios.
+
+## Audience
+
+Platform engineers, SREs, and gateway operators.
+
+## Pre-conditions
+
+- Envoy 1.25+ (per `ENVOY_VERSION_GOVERNANCE.md`).
+- The Envoy instance is part of an xDS-managed fleet OR a self-managed fleet.
+- Hot restart is enabled.
+
+## Procedure
+
+### Step 1 — Detect
+
+1. Confirm the listener config drift: `curl localhost:9901/config_dump | jq '.configs[1].dynamic_active_clusters'`.
+2. Confirm the listener is accepting connections: `curl localhost:9901/ready`.
+
+### Step 2 — Author the new config
+
+3. Edit the bootstrap or LDS source.
+4. Validate locally: `envoy --mode validate -c <config-path>`.
+5. If using SOTW (State of the World) xDS, prepare a new snapshot.
+
+### Step 3 — Hot reload via LDS
+
+6. Send the new Listener via xDS:
+   - `curl -X POST localhost:9901/listeners/<name>` (rare).
+   - Or push via the control plane: `envoy-xds <snapshot>`.
+7. Confirm the new listener is `warming`.
+8. Confirm Envoy drains old connections via `admin.early_header_conn_end`.
+
+### Step 4 — Hot reload via full restart
+
+9. If hot reload is not feasible, perform a hot restart:
+   - `systemctl reload envoy` (sends SIGHUP, hot restart).
+   - Confirm the parent and child processes are running.
+   - Confirm the old listener drains over `drain_timeout`.
+
+### Step 5 — Verify
+
+10. Confirm new connections are accepted.
+11. Confirm in-flight connections continue.
+12. Confirm metrics show the listener is healthy.
+
+### Step 6 — Validate
+
+13. Send a synthetic HTTP request.
+14. Confirm response matches the new config.
+15. Confirm headers / routes / filters behave as expected.
+
+### Step 7 — Monitor
+
+16. Confirm 5xx rate is at baseline.
+17. Confirm connection error rate is at baseline.
+18. Confirm listener accept rate matches expected traffic.
+
+## Rollback
+
+If the new listener config breaks traffic:
+
+1. Hot-restart Envoy to the previous config.
+2. If a full restart, restore the previous binary and config.
+3. Validate traffic recovery.
+
+## References
+
+- `ENVOY_VERSION_GOVERNANCE.md`
+- `GITOPS_SYNC_FAILURE_RECOVERY_PLAYBOOK.md`
+- Envoy hot restart: `https://www.envoyproxy.io/docs/envoy/latest/operations/hot_restarter`
+- Envoy admin: `https://www.envoyproxy.io/docs/envoy/latest/operations/admin`
@@ -0,0 +1,87 @@
+# NATS Cluster Failover Playbook
+
+## Purpose
+
+Define the operational procedure for recovering a NATS cluster when one or more nodes fail. The procedure covers core NATS (no JetStream), JetStream persistence, and leafnode edges.
+
+## Audience
+
+Platform engineers, SREs, and on-call responders.
+
+## Pre-conditions
+
+- NATS 2.10+ (per `NATS_VERSION_GOVERNANCE.md`).
+- The cluster has at least 3 nodes.
+- JetStream is enabled if persistence is required.
+- Monitoring emits cluster health events.
+
+## Procedure
+
+### Step 1 — Detect
+
+1. Confirm cluster health: `nats server check connection -s nats://<host>:4222`.
+2. Confirm peer list: `nats server list -s nats://<host>:4222`.
+3. Confirm JetStream status (if enabled): `nats str list -s nats://<host>:4222`.
+
+### Step 2 — Diagnose
+
+4. Identify the failed node(s) via monitoring.
+5. Inspect the failed node's logs (`stderr.log` or `journalctl`).
+6. Determine root cause:
+   - Process crash.
+   - OOM.
+   - Disk full.
+   - Network partition.
+   - Disk corruption (JetStream).
+
+### Step 3 — Recover a single node
+
+7. If the failed node is recoverable, restart it: `systemctl restart nats`.
+8. Confirm the node rejoins the cluster.
+9. Confirm quorum is restored.
+
+### Step 4 — Recover a partition
+
+10. If the cluster is partitioned, restore network access.
+11. Confirm all nodes rejoin via gossip.
+12. Confirm the leader is elected.
+
+### Step 5 — Recover a JetStream node
+
+13. If a JetStream node has disk corruption, quarantine the node.
+14. Replace the disk.
+15. Initialize the JetStream node with the same name.
+16. Confirm the stream metadata rebalances.
+
+### Step 6 — Recover from total loss
+
+17. If the entire cluster is lost, restore from backup:
+    - Streams are persisted under `${store_dir}/jetstream`.
+    - `$JSZ` API or `nats str info` confirms restoration.
+18. Reapply configuration from GitOps (per `GITOPS_SYNC_FAILURE_RECOVERY_PLAYBOOK.md`).
+
+### Step 7 — Verify
+
+19. Confirm all subjects are reachable.
+20. Confirm JetStream consumers can resume.
+21. Confirm cluster metrics return to baseline.
+
+### Step 8 — Postmortem
+
+22. File a postmortem per `INCIDENT_POSTMORTEM_REVIEW_PLAYBOOK.md`.
+23. Identify the failure origin.
+24. Add regression tests or guard rails.
+
+## Rollback
+
+If recovery introduces regressions:
+
+1. Restore from backup.
+2. Verify against pre-incident snapshots.
+
+## References
+
+- `NATS_VERSION_GOVERNANCE.md`
+- `INCIDENT_POSTMORTEM_REVIEW_PLAYBOOK.md`
+- NATS operations: `https://docs.nats.io/running-a-nats-service/nats_admin`
+- JetStream disaster recovery: `https://docs.nats.io/nats-concepts/jetstream/disaster_recovery`
@@ -0,0 +1,76 @@
+# NGINX TLS Certificate Renewal Playbook
+
+## Purpose
+
+Define the operational procedure for renewing TLS certificates served by NGINX or the NGINX Ingress Controller without service disruption. The procedure covers both ACME / Let's Encrypt and private CA scenarios.
+
+## Audience
+
+Platform engineers, SREs, and security engineers.
+
+## Pre-conditions
+
+- NGINX 1.22+ (per `NGINX_VERSION_GOVERNANCE.md`).
+- Certificates are managed via cert-manager, acme.sh, or `lego`.
+- The renewal job is automated.
+- Monitoring alerts on certificate expiry < 30 days.
+
+## Procedure
+
+### Step 1 — Detect
+
+1. Confirm the certificate is approaching expiry.
+2. `echo | openssl s_client -servername <host> -connect <host>:443 2>/dev/null | openssl x509 -noout -dates`.
+3. Alert threshold: 30 days, 14 days, 7 days.
+
+### Step 2 — Renew
+
+4. Trigger the renewal job:
+   - `cert-manager`: annotation on the Ingress (`cert-manager.io/renew-before`).
+   - `acme.sh`: `acme.sh --renew -d <domain>`.
+   - `lego`: `lego renew --domain <domain>`.
+5. Confirm the new certificate is issued.
+
+### Step 3 — Reload NGINX
+
+6. Reload NGINX with zero downtime:
+   - `nginx -s reload`.
+   - For ingress-nginx: `kubectl exec -n ingress-nginx <pod> -- nginx -s reload`.
+7. Confirm the new certificate is in memory.
+
+### Step 4 — Verify
+
+8. Re-run the `openssl s_client` check.
+9. Confirm the new `notAfter` date.
+10. Confirm the chain is valid (intermediate + root).
+
+### Step 5 — Validate against CT logs
+
+11. Submit the certificate to CT logs (Let's Encrypt does this automatically).
+12. Verify via `crt.sh` or `certspotter`.
+
+### Step 6 — Cleanup
+
+13. Remove the old certificate from the secret store.
+14. Remove the old certificate file from disk.
+
+### Step 7 — Monitor
+
+15. Confirm monitoring does not alert after renewal.
+16. Update the expiry tracker.
+
+## Rollback
+
+If a renewal breaks a domain:
+
+1. Restore the previous certificate from the secret store.
+2. Reload NGINX.
+3. Investigate the renewal issue.
+
+## References
+
+- `NGINX_VERSION_GOVERNANCE.md`
+- `TLS_RFC_8446_VERSION_GOVERNANCE.md`
+- cert-manager: `https://cert-manager.io/docs/`
+- acme.sh: `https://github.com/acmesh-official/acme.sh`
+- Let's Encrypt: `https://letsencrypt.org/docs/`
@@ -271,3 +271,9 @@ This family contains repeatable procedures for development, deployment, incident
 - [Protobuf Schema Deprecation Playbook](PROTOBUF_SCHEMA_DEPRECATION_PLAYBOOK.md)
 - [OPA Bundle Rotation Playbook](OPA_BUNDLE_ROTATION_PLAYBOOK.md)
 - [Vault Audit Log Shipping Playbook](VAULT_AUDIT_LOG_SHIPPING_PLAYBOOK.md)
+
+## 2026-09-05 NATS failover, NGINX TLS renewal, and Envoy hot-reload playbooks (Batch 90)
+
+- [NATS Cluster Failover Playbook](NATS_CLUSTER_FAILOVER_PLAYBOOK.md)
+- [NGINX TLS Certificate Renewal Playbook](NGINX_TLS_CERT_RENEWAL_PLAYBOOK.md)
+- [Envoy Listener Hot Reload Playbook](ENVOY_LISTENER_HOT_RELOAD_PLAYBOOK.md)
@@ -0,0 +1,102 @@
+---
+title: Envoy Proxy Version Governance
+owner: Knowledge Engineering
+status: approved
+classification: public
+last-reviewed: 2026-09-05
+review-cycle: 180 days
+next-review: 2027-03-04
+source: Envoy documentation; CNCF Envoy project; envoyproxy.io
+---
+
+# Envoy Proxy Version Governance
+
+## Scope
+
+This card governs how `orchords-docs` evaluates the Envoy proxy across versions, xDS APIs, and integration patterns (Ingress, Gateway API, Service Mesh).
+
+## Why this card exists
+
+Envoy is the canonical cloud-native L4/L7 proxy, the data plane of Istio, and the reference implementation for the Gateway API. Without an explicit card, the KB cites Envoy practices that ignore the rapid 6-week release cadence and the v3 xDS protocol.
+
+## Versions
+
+| Version | Status |
+|---|---|
+| 1.20–1.24 | legacy |
+| 1.25–1.30 | current stable |
+| 1.31–1.33 | current |
+
+References: `https://github.com/envoyproxy/envoy/releases`.
+
+Envoy has a 6-week minor release cadence. Patch releases are monthly.
+
+## xDS APIs
+
+| API | Use |
+|---|---|
+| `envoy.config.listener.v3.Listener` | listener config |
+| `envoy.config.route.v3.RouteConfiguration` | HTTP routes |
+| `envoy.config.cluster.v3.Cluster` | upstream clusters |
+| `envoy.extensions.transport_sockets.tls.v3` | TLS context |
+| `envoy.config.bootstrap.v3.Bootstrap` | bootstrap config |
+
+The v3 API is mandatory since 1.18; v2 is removed.
+
+References: `https://www.envoyproxy.io/docs/envoy/latest/configuration/configuration`.
+
+## Gateway API
+
+Envoy is a reference implementation of the Kubernetes Gateway API:
+
+- `GatewayClass`, `Gateway`, `HTTPRoute`, `TCPRoute`.
+- Gateway API ≥ v1.0 (since 2023).
+- Compatible with `envoy-gateway`, `contour`, `istio`, etc.
+
+References: `https://gateway-api.sigs.k8s.io/`.
+
+## Filter chain
+
+| Filter | Use |
+|---|---|
+| `envoy.filters.network.http_connection_manager` | HTTP handling |
+| `envoy.filters.http.router` | routing |
+| `envoy.filters.http.lua` | scripting |
+| `envoy.filters.http.ratelimit` | rate limiting |
+| `envoy.filters.http.ext_authz` | external authorization |
+| `envoy.filters.http.jwt_authn` | JWT validation |
+| `envoy.filters.network.tcp_proxy` | L4 proxy |
+
+## TLS
+
+- `transport_socket` with `envoy.transport_sockets.tls` context.
+- `common_tls_context.tls_certificate_sds_secret_configs` for SDS.
+- HTTP/2 ALPN: `h2`.
+- HTTP/1.1 fallback.
+
+## Observability
+
+| Feature | Use |
+|---|---|
+| Access logs | JSON, configurable |
+| Stats | Prometheus integration |
+| Tracing | OpenTelemetry, Zipkin |
+| Tap | on-the-wire capture |
+
+References: `https://www.envoyproxy.io/docs/envoy/latest/operations/`.
+
+## Cross-reference
+
+| Domain | Card |
+|---|---|
+| Istio | `ISTIO_VERSION_GOVERNANCE.md` |
+| Kubernetes | `KUBERNETES_VERSION_GOVERNANCE.md` |
+| HTTP/3 | `HTTP_3_RFC_9114_VERSION_GOVERNANCE.md` |
+| OPA | `OPA_VERSION_GOVERNANCE.md` |
+
+## Sources
+
+- Envoy documentation: `https://www.envoyproxy.io/docs/envoy/latest/`
+- Envoy releases: `https://github.com/envoyproxy/envoy/releases`
+- Envoy quickstart: `https://www.envoyproxy.io/docs/envoy/latest/start/quick-start/`
+- Gateway API: `https://gateway-api.sigs.k8s.io/`