ORCHORDS · ORCHORDS · Sep 5, 2026 · Sep 5, 2026
@@ -0,0 +1,81 @@
+# GitOps Sync Failure Recovery Playbook
+
+## Purpose
+
+Define the operational procedure for recovering from a GitOps controller (Argo CD / Flux CD) that has fallen out of sync, when the drift is between the desired state in Git and the actual cluster state. The procedure ensures that recovery is automated where possible and that manual overrides are auditable.
+
+## Audience
+
+Platform engineers, SREs, and on-call responders.
+
+## Pre-conditions
+
+- The cluster is managed by Argo CD 2.4+ or Flux CD 2.0+ (per `GITOPS_VERSION_GOVERNANCE.md`).
+- The cluster is reconciled by at least one `Application` (Argo) or `Kustomization` / `HelmRelease` (Flux).
+- The cluster has logging to a centralized system.
+
+## Procedure
+
+### Step 1 — Detect
+
+1. Argo CD: `argocd app list -o yaml | jq '.[] | select(.status.health.status=="Degraded")'`.
+2. Flux CD: `flux get kustomizations --all-namespaces | grep False`.
+3. Confirm drift: `argocd app diff <app>` or `flux diff kustomization <name>`.
+
+### Step 2 — Diagnose
+
+4. Identify the sync status reason (`OutOfSync`, `Degraded`, `Unknown`).
+5. Inspect controller logs.
+6. Determine root cause:
+   - Failed health check.
+   - Failed pre-sync hook.
+   - Invalid manifest (CRD mismatch).
+   - Quota or rbac.
+   - Network partition from Git server.
+
+### Step 3 — Recover automatically
+
+7. If the controller can auto-sync, force a sync:
+   - Argo CD: `argocd app sync <app> --prune --force`.
+   - Flux: `flux reconcile kustomization <name> --with-source`.
+8. If the auto-sync succeeds, verify health.
+
+### Step 4 — Recover manually
+
+9. If auto-sync fails, manually reapply the desired state from Git:
+   - `kubectl apply -k <path-to-kustomize-output>` (Flux).
+   - `argocd app sync <app> --replace` (Argo).
+10. Document every manual apply as a `SyncOverride` annotation on the Application / Kustomization.
+
+### Step 5 — Address root cause
+
+11. If the failure was caused by an invalid manifest, revert the Git commit (revert PR, do not force-push).
+12. If the failure was caused by a CRD mismatch, update the CRD first, then the manifests.
+13. If the failure was caused by a network partition, restore network access and retry sync.
+
+### Step 6 — Verify
+
+14. Confirm `OutOfSync` status is `Synced`.
+15. Confirm `health.status` is `Healthy`.
+16. Confirm all replicas are Ready.
+
+### Step 7 — Postmortem
+
+17. File a postmortem per `INCIDENT_POSTMORTEM_REVIEW_PLAYBOOK.md`.
+18. Identify the drift origin (manifest change, controller bug, network).
+19. Add a regression test or guard rail.
+
+## Rollback
+
+If recovery introduces regressions:
+
+1. Revert the Git commit.
+2. Force a sync.
+3. Verify health.
+
+## References
+
+- `GITOPS_VERSION_GOVERNANCE.md`
+- `INCIDENT_POSTMORTEM_REVIEW_PLAYBOOK.md`
+- Argo CD: `https://argo-cd.readthedocs.io/en/stable/operator-manual/sync-options/`
+- Flux: `https://fluxcd.io/flux/cmd/flux_reconcile/`
@@ -0,0 +1,78 @@
+# Istio mTLS Rollout Playbook
+
+## Purpose
+
+Define the operational procedure for rolling out Istio mTLS in STRICT mode across a service mesh. The procedure uses a phased PERMISSIVE → STRICT transition to ensure that legacy plaintext clients do not lose connectivity.
+
+## Audience
+
+Platform engineers, SREs, and security engineers.
+
+## Pre-conditions
+
+- Istio 1.19+ installed (per `ISTIO_VERSION_GOVERNANCE.md`).
+- The mesh namespace exists.
+- AuthorizationPolicy CRDs are installed.
+- The team can roll back via GitOps revert.
+
+## Procedure
+
+### Phase 1 — Inventory
+
+1. List all workloads in the target namespace: `kubectl get deploy -n <ns> -o yaml`.
+2. Identify workloads without a sidecar (legacy plaintext).
+3. Identify workloads that originate traffic outside the mesh (ingress gateways, external clients).
+4. Document every legacy plaintext client.
+
+### Phase 2 — Audit policy baseline
+
+5. Confirm the mesh-wide `PeerAuthentication` is `PERMISSIVE` (no enforcement).
+6. Confirm `DestinationRule` `mesh-wide` TLS mode is `ISTIO_MUTUAL` (or not set).
+7. Confirm AuthorizationPolicy defaults are not blocking plaintext.
+
+### Phase 3 — Opt-in per workload
+
+8. For each workload, apply a `PeerAuthentication` in PERMISSIVE mode first.
+9. Observe mesh telemetry (Kiali / Prometheus) for failed TLS connections.
+10. Validate that the workload accepts both plaintext and mTLS traffic.
+
+### Phase 4 — Selective STRICT
+
+11. For each workload, change its `PeerAuthentication` to STRICT.
+12. Monitor the workload for 24 hours:
+    - `istio_requests_total{response_code=~"5.."}`.
+    - `istio_tcp_connection_closed_local{...}`.
+13. If errors spike, revert to PERMISSIVE and investigate.
+
+### Phase 5 — Mesh-wide STRICT
+
+14. Apply mesh-wide `PeerAuthentication` with `mtls.mode: STRICT`.
+15. Confirm every workload has its own `PeerAuthentication` aligned (STRICT or PERMISSIVE explicitly).
+16. Monitor mesh-wide for 24 hours.
+
+### Phase 6 — Lock down
+
+17. Set AuthorizationPolicy default-deny on every namespace.
+18. Add explicit ALLOW rules per workload pair.
+19. Validate with `kubectl authz-can-i` (if installed) or via synthetic test.
+
+### Phase 7 — Verify
+
+20. Confirm every connection in Kiali shows the lock icon (mTLS).
+21. Confirm access logs show `conn_security_policy: ISTIO_MUTUAL`.
+22. Confirm no plaintext traffic in `istio_tcp_connection_opened`.
+
+## Rollback
+
+If STRICT mode breaks a workload:
+
+1. Revert the `PeerAuthentication` to PERMISSIVE (or remove it).
+2. Verify traffic recovers.
+3. File a postmortem per `INCIDENT_POSTMORTEM_REVIEW_PLAYBOOK.md`.
+
+## References
+
+- `ISTIO_VERSION_GOVERNANCE.md`
+- `INCIDENT_POSTMORTEM_REVIEW_PLAYBOOK.md`
+- Istio mTLS: `https://istio.io/latest/docs/concepts/security/mutual-tls/`
+- Istio PeerAuthentication: `https://istio.io/latest/docs/reference/config/security/peer_authentication/`
@@ -212,6 +212,12 @@ This family contains repeatable procedures for development, deployment, incident
 - [DNSSEC Key Rollover Playbook](DNSSEC_ROLLOVER_PLAYBOOK.md)
 - [Incident Postmortem Architectural Review Playbook](INCIDENT_POSTMORTEM_REVIEW_PLAYBOOK.md)
 
+## 2026-09-05 Terraform module promotion, GitOps sync recovery, and Istio mTLS rollout playbooks (Batch 88)
+
+- [Terraform Module Promotion Playbook](TERRAFORM_MODULE_PROMOTION_PLAYBOOK.md)
+- [GitOps Sync Failure Recovery Playbook](GITOPS_SYNC_FAILURE_RECOVERY_PLAYBOOK.md)
+- [Istio mTLS Rollout Playbook](ISTIO_MTLS_ROLLOUT_PLAYBOOK.md)
+
 ## 2026-09-05 Message-broker upgrade, failover, and privacy-incident playbooks (Batch 80)
 
 - [Kafka Cluster Version Bump Playbook](KAFKA_VERSION_BUMP_PLAYBOOK.md)

@@ -0,0 +1,70 @@
+# Terraform Module Promotion Playbook
+
+## Purpose
+
+Define the operational procedure for promoting a Terraform module from internal use to verified-or-trusted third-party status under `TERRAFORM_VERSION_GOVERNANCE.md`. The promotion gates ensure that any module ingested by the project has been reviewed for state hygiene, source provenance, and license compliance.
+
+## Audience
+
+Platform engineers, SREs, and security engineers who publish or consume Terraform modules.
+
+## Pre-conditions
+
+- The module has been in internal use for at least 30 days.
+- The module has at least 3 internal consumers.
+- The module source repository is in the orchords org.
+
+## Procedure
+
+### Stage 1 — Internal
+
+1. The module lives in `github.com/<orchords-org>/terraform-modules/<name>`.
+2. The module README declares inputs, outputs, and usage.
+3. The module has unit tests (`terraform test`).
+4. The module has an example under `examples/<scenario>/`.
+5. The module has been applied in production in at least 3 internal consumers without rollback.
+
+### Stage 2 — Verified (verified-mirror)
+
+6. The module is mirrored to the internal private registry (`registry.<orchords-org>.internal/<name>`).
+7. The mirror pins the source to the internal repository via `source = "git::https://...#vX.Y.Z"`.
+8. The consumer MUST commit-pin the tag (not `latest`).
+
+### Stage 3 — Trusted third-party
+
+9. The module is published to the public Terraform Registry (or OpenTofu Registry) with:
+   - `source` field pointing back to the internal repository.
+   - `version` pinned.
+   - `license` declared (MPL-2.0 for OpenTofu works; Apache-2.0 for HashiCorp works).
+10. The module is approved by an owner not on the original author team (four-eyes review).
+11. The module has a `SECURITY.md` with a coordinated disclosure policy.
+
+### Stage 4 — Untrusted (NOT recommended)
+
+12. Untrusted modules (anonymous GitHub, gists, third-party without review) are forbidden.
+13. If a third-party untrusted module is required as an exception, an explicit risk acceptance is filed and the module is vendored into the internal mirror with a pinned commit SHA before use.
+
+### Promotion gates
+
+| Gate | Internal | Verified | Trusted | Untrusted |
+|---|---|---|---|---|
+| Source | internal repo | internal mirror | public registry | forbidden |
+| Pinning | commit SHA | tag | tag + checksum | vendored |
+| Review | 1 author | 1 author + 1 reviewer | 2 reviewers | exception only |
+| License declared | yes | yes | yes | n/a |
+| Unit tests | yes | yes | yes | n/a |
+
+## Rollback
+
+If a promoted module causes an incident:
+
+1. Mark the latest version as ` yanked` in the registry.
+2. Pin every consumer to the prior version.
+3. File a postmortem per `INCIDENT_POSTMORTEM_REVIEW_PLAYBOOK.md`.
+
+## References
+
+- `TERRAFORM_VERSION_GOVERNANCE.md`
+- `INCIDENT_POSTMORTEM_REVIEW_PLAYBOOK.md`
+- HashiCorp Module Registry: `https://registry.terraform.io/`
+- OpenTofu Registry: `https://registry.opentofu.org/`
@@ -0,0 +1,140 @@
+---
+title: GitOps and Argo CD / Flux Version Governance
+owner: Knowledge Engineering
+status: approved
+classification: public
+last-reviewed: 2026-09-05
+review-cycle: 180 days
+next-review: 2027-03-04
+source: CNCF Argo Project (https://argoproj.github.io/); CNCF Flux Project (https://fluxcd.io/); OpenGitOps specification (https://opengitops.dev/)
+---
+
+# GitOps and Argo CD / Flux Version Governance
+
+## Scope
+
+This card governs how `orchords-docs` evaluates GitOps tooling — Argo CD and Flux — and the supporting OpenGitOps principles. It is the reference input for any KB card that cites declarative cluster management, application sync, or progressive delivery.
+
+## Why this card exists
+
+GitOps is the operational practice of declaring the desired cluster state in Git and reconciling it via an automated agent. Two ecosystems dominate: CNCF Argo (Argo CD, Argo Rollouts, Argo Workflows, Argo Events) and CNCF Flux (Flux CD, Flagger, Helm Operator, SOPS). Without an explicit card, the KB cites GitOps patterns that do not survive an audit.
+
+## OpenGitOps principles
+
+OpenGitOps is the CNCF / Finniture specification (1.0, 2024) that defines four principles:
+
+1. **Declarative** — desired state is described declaratively.
+2. **Versioned and Immutable** — desired state is stored in a versioned, immutable way (Git).
+3. **Pulled Automatically** — agents pull the desired state and reconcile.
+4. **Continuously Reconciled** — agents continuously observe and reconcile.
+
+References: `https://opengitops.dev/`.
+
+## Argo CD version matrix
+
+| Version | First release | Status |
+|---|---|---|
+| 2.4.x | 2022 | stable |
+| 2.6.x | 2022 | stable |
+| 2.8.x | 2023 | stable |
+| 2.9.x | 2023 | stable |
+| 2.10.x | 2024 | stable |
+| 2.11.x | 2024 | stable |
+| 2.12.x | 2024 | stable |
+| 2.13.x | 2025 | stable |
+| 2.14.x | 2025 | current |
+
+References: `https://github.com/argoproj/argo-cd/releases`.
+
+## Flux CD version matrix
+
+| Version | First release | Status |
+|---|---|---|
+| 2.0.x | 2022 | stable |
+| 2.1.x | 2023 | stable |
+| 2.2.x | 2023 | stable |
+| 2.3.x | 2024 | stable |
+| 2.4.x | 2024 | stable |
+| 2.5.x | 2025 | stable |
+| 2.6.x | 2025 | current |
+
+References: `https://github.com/fluxcd/flux/releases`.
+
+## Application definition
+
+| Tool | Format |
+|---|---|
+| Argo CD | Application / ApplicationSet (CRD) |
+| Flux | Kustomization, HelmRelease (CRD) |
+| Helm | Chart (Helm v3) |
+| Kustomize | kustomization.yaml |
+
+Policy:
+
+- Each application lives in its own Git repository (preferred) or subdirectory.
+- Source-of-truth branch is `main`.
+- Tags mark the production-recommended version.
+
+## Sync strategies
+
+| Strategy | Description |
+|---|---|
+| Apply | declarative apply (Argo / Flux) |
+| Replace | replace resource on conflict |
+| Sync wave | ordered rollout |
+| Progressive delivery | canary / blue-green (Argo Rollouts, Flagger) |
+
+## Mandatory pre-flight (before adopting a new GitOps tool)
+
+1. OpenGitOps principles are documented.
+2. The tool version is within the support matrix.
+3. The repository structure is documented.
+4. The sync policy is documented.
+5. Secrets management is wired (SOPS, Sealed Secrets, External Secrets Operator).
+6. RBAC is configured.
+
+## Secrets management in GitOps
+
+| Tool | Mechanism |
+|---|---|
+| SOPS | encrypted YAML/JSON files; key in KMS / KMS-AAD / KMS-GCP |
+| Sealed Secrets | public-key-encrypted Kubernetes secrets |
+| External Secrets Operator | pulls from external secret store (Vault, AWS Secrets Manager) |
+
+Policy:
+
+- Secrets are NEVER stored in plaintext.
+- SOPS or External Secrets Operator is preferred.
+- Sealed Secrets acceptable for edge cases.
+
+## Progressive delivery
+
+| Tool | Notes |
+|---|---|
+| Argo Rollouts | canary, blue-green, traffic shifting |
+| Flagger | canary, A/B testing |
+| Argo Workflows | DAG-based workflows |
+
+Policy:
+
+- Progressive delivery for any service exposed to customer traffic.
+- Canary baseline: 5% canary for 10 minutes, then 25%, 50%, 100%.
+
+## Observability
+
+- `gitops_sync_status` (gauge, per app).
+- `gitops_sync_duration_seconds` (histogram).
+- `gitops_drift_count` (counter).
+- `gitops_applications_healthy` (gauge).
+- `gitops_applications_out_of_sync` (gauge).
+
+## Sources
+
+- OpenGitOps: `https://opengitops.dev/`
+- Argo CD: `https://argoproj.github.io/argo-cd/`
+- Flux CD: `https://fluxcd.io/`
+- Argo Rollouts: `https://argoproj.github.io/argo-rollouts/`
+- Flagger: `https://flagger.app/`
+- SOPS: `https://github.com/getsops/sops`
+- Sealed Secrets: `https://github.com/bitnami-labs/sealed-secrets`
+- External Secrets Operator: `https://external-secrets.io/`