Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
# Argo Rollouts Progressive Delivery Playbook

## Purpose

Define the operational procedure for using Argo Rollouts to perform a canary deployment with traffic shifting, automated analysis, and rollback.

## Audience

Platform engineers, SREs, and release engineers.

## Pre-conditions

- Argo Rollouts 1.5+ (per `ARGO_VERSION_GOVERNANCE.md`).
- The cluster has Argo Rollouts controller installed.
- The service is a `Rollout` resource (not `Deployment`).

## Procedure

### Step 1 — Author the Rollout

1. Define a `Rollout` resource:
- `strategy.canary.steps`.
- `trafficRouting.istio` or `trafficRouting.nginx`.
- `analysis.templates`.
2. Pin the Rollout to the same image tag pattern as the Deployment it replaces.

### Step 2 — Validate

3. `kubectl argo rollouts get rollout <name> --watch`.
4. `kubectl argo rollouts lint <name>`.
5. Confirm the Rollout is valid.

### Step 3 — Deploy the canary

6. Update the image:
- `kubectl argo rollouts set image <name> <container>=<new-image>`.
7. Confirm the canary starts.
8. Confirm traffic shifts per the strategy.

### Step 4 — Analyze

9. Confirm analysis templates run.
10. Confirm success rate / latency / error rate are within thresholds.
11. Promote or pause based on analysis.

### Step 5 — Promote

12. `kubectl argo rollouts promote <name>`.
13. Confirm the canary progresses to 100%.
14. Confirm the stable ReplicaSet scales up and the canary scales down.

### Step 6 — Verify

15. Confirm the new version is healthy.
16. Confirm smoke tests pass.
17. Confirm metrics are at baseline.

### Step 7 — Rollback if needed

18. `kubectl argo rollouts abort <name>`.
19. Confirm the Rollout reverts to the stable version.
20. File a postmortem per `INCIDENT_POSTMORTEM_REVIEW_PLAYBOOK.md`.

## Rollback

If the canary fails analysis:

1. `kubectl argo rollouts undo <name>`.
2. Confirm traffic returns to the previous version.
3. Investigate the regression.

## References

- `ARGO_VERSION_GOVERNANCE.md`
- Argo Rollouts: `https://argoproj.github.io/argo-rollouts/`
- Traffic routing: `https://argoproj.github.io/argo-rollouts/features/traffic-routing/`
70 changes: 70 additions & 0 deletions docs/knowledge/playbooks/CONSUL_TOKEN_ROTATION_PLAYBOOK.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
# Consul ACL Token Rotation Playbook

## Purpose

Define the operational procedure for rotating Consul ACL tokens (agent, default, replication, service) without service disruption. The procedure covers master tokens, agent tokens, and service identity tokens.

## Audience

Platform engineers, SREs, and security engineers.

## Pre-conditions

- Consul 1.10+ (per `CONSUL_VERSION_GOVERNANCE.md`).
- The team uses Consul ACLs.
- The team has the master token stored in a secret manager.

## Procedure

### Step 1 — Inventory

1. List all ACL tokens: `consul acl token list`.
2. Identify tokens by type: master, agent, replication, service identity.
3. Identify token age and policy bindings.

### Step 2 — Author the new token

4. Create the new token:
- `consul acl token create -description "<purpose>" -policy-name "<policy>"`.
5. Capture the new token `SecretID`.

### Step 3 — Roll out the new token

6. For agent tokens:
- Update the agent config: `acl.tokens.agent = "<new>"`.
- Reload the agent: `systemctl reload consul`.
7. For service identity tokens:
- Update the service config: `consul.acl.tokens.service = "<new>"`.
- Reload the service or restart.
8. For replication tokens:
- Update the replication config.
- Reload the agent.

### Step 4 — Verify

9. Confirm `consul members` shows the new tokens accepted.
10. Confirm ACL policies resolve correctly.
11. Confirm services still register.

### Step 5 — Revoke the old token

12. `consul acl token delete <old-token-id>`.
13. Confirm no agent is using the old token.

### Step 6 — Audit

14. Confirm the new token is in the audit log.
15. Confirm the old token revocation is in the audit log.

## Rollback

If a new token breaks a service:

1. Re-introduce the old token (revocation can be undone within 30 days via replication).
2. Reload the service.
3. Investigate the regression.

## References

- `CONSUL_VERSION_GOVERNANCE.md`
- Consul ACL: `https://developer.hashicorp.com/consul/docs/security/acl`
74 changes: 74 additions & 0 deletions docs/knowledge/playbooks/FLUX_HELM_RELEASE_UPGRADE_PLAYBOOK.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
# Flux Helm Release Upgrade Playbook

## Purpose

Define the operational procedure for upgrading a HelmRelease managed by Flux CD. The procedure covers chart version bumps and values updates, with safe rollback if analysis fails.

## Audience

Platform engineers, SREs, and GitOps operators.

## Pre-conditions

- Flux CD 2.3+ (per `FLUX_VERSION_GOVERNANCE.md`).
- The cluster has Helm Controller installed.
- The HelmRelease is reconciled by Flux.

## Procedure

### Step 1 — Detect

1. Confirm the current chart version: `flux get hr <name> -n <namespace>`.
2. Confirm a new chart version is available.

### Step 2 — Author the change

3. Edit the `HelmRelease` manifest in Git:
- `spec.chart.spec.version`.
- `spec.values`.
4. Commit the change.
5. Open a PR for review.

### Step 3 — Reconcile

6. Merge the PR.
7. Trigger reconciliation:
- `flux reconcile helmrelease <name> -n <namespace>`.
- Or wait for the reconcile interval (default 1m).

### Step 4 — Validate

8. `flux get hr <name> -n <namespace>` to confirm the new revision.
9. `flux logs` to inspect controller logs.
10. Confirm Helm test passes if defined.

### Step 5 — Verify

11. Confirm pods are Ready.
12. Confirm services are reachable.
13. Confirm metrics are at baseline.

### Step 6 — Rollback

14. If the new chart breaks behavior, revert the PR.
15. Reconcile again: `flux reconcile helmrelease <name>`.
16. Confirm the previous revision is rolled back.

### Step 7 — Cleanup

17. Remove the failed chart version from the Helm repository cache (optional).

## Rollback

If Helm upgrade fails midway:

1. `flux suspend helmrelease <name>` to pause reconciliation.
2. Manually roll back via `helm rollback`.
3. `flux resume helmrelease <name>` to resume.
4. Reconcile.

## References

- `FLUX_VERSION_GOVERNANCE.md`
- Flux Helm Controller: `https://fluxcd.io/flux/components/helm/`
- HelmRelease spec: `https://fluxcd.io/flux/components/helm/helmreleases/`
6 changes: 6 additions & 0 deletions docs/knowledge/playbooks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -283,3 +283,9 @@ This family contains repeatable procedures for development, deployment, incident
- [Prometheus High-Availability Playbook](PROMETHEUS_HIGH_AVAILABILITY_PLAYBOOK.md)
- [Grafana Dashboard Provisioning Playbook](GRAFANA_DASHBOARD_PROVISIONING_PLAYBOOK.md)
- [Kong Plugin Upgrade Playbook](KONG_PLUGIN_UPGRADE_PLAYBOOK.md)

## 2026-09-05 Consul ACL rotation, Argo Rollouts canary, and Flux Helm release upgrade playbooks (Batch 92)

- [Consul ACL Token Rotation Playbook](CONSUL_TOKEN_ROTATION_PLAYBOOK.md)
- [Argo Rollouts Progressive Delivery Playbook](ARGO_ROLLOUTS_PROGRESSIVE_DELIVERY_PLAYBOOK.md)
- [Flux Helm Release Upgrade Playbook](FLUX_HELM_RELEASE_UPGRADE_PLAYBOOK.md)
95 changes: 95 additions & 0 deletions docs/knowledge/reference/ARGO_VERSION_GOVERNANCE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
---
title: Argo Workflows / Argo CD / Argo Events / Argo Rollouts Version Governance
owner: Knowledge Engineering
status: approved
classification: public
last-reviewed: 2026-09-05
review-cycle: 180 days
next-review: 2027-03-04
source: Argo Project; CNCF Argo Workflows, Argo CD, Argo Events, Argo Rollouts
---

# Argo Workflows / Argo CD / Argo Events / Argo Rollouts Version Governance

## Scope

This card governs how `orchords-docs` evaluates the Argo project family across versions, CRD APIs, and deployment patterns.

## Why this card exists

Argo is the canonical Kubernetes-native workflow and deployment toolkit. Without an explicit card, the KB cites Argo practices that ignore the four-project split (Workflows, CD, Events, Rollouts), version skew rules, and ArgoCD Application / ApplicationSet APIs.

## Argo Workflows

Workflows is the canonical Kubernetes-native workflow engine:

| Version | Status |
|---|---|
| 3.4–3.5 | legacy |
| 3.6 | current LTS |
| 3.7 | current |

References: `https://github.com/argoproj/argo-workflows/releases`.

CRD API: `argoproj.io/v1alpha1` (stable for years).

## Argo CD

Argo CD is the canonical GitOps controller (per `GITOPS_VERSION_GOVERNANCE.md`).

| Version | Status |
|---|---|
| 2.4–2.6 | legacy |
| 2.7–2.10 | current |
| 2.11+ | current |

CRD API: `argoproj.io/v1alpha1`.

References: `https://github.com/argoproj/argo-cd/releases`.

## Argo Events

Argo Events is the event-driven workflow trigger:

| Version | Status |
|---|---|
| 1.7–1.8 | legacy |
| 1.9+ | current |

References: `https://github.com/argoproj/argo-events/releases`.

## Argo Rollouts

Argo Rollouts is the progressive-delivery controller:

| Version | Status |
|---|---|
| 1.5–1.6 | legacy |
| 1.7+ | current |

Strategies: BlueGreen, Canary, canary with analysis.

References: `https://github.com/argoproj/argo-rollouts/releases`.

## Cross-project compatibility

Argo components version-skew rules:

- Argo Workflows and Argo Events: aligned minor versions.
- Argo CD: independent.
- Argo Rollouts: independent.

## Cross-reference

| Domain | Card |
|---|---|
| GitOps | `GITOPS_VERSION_GOVERNANCE.md` |
| Kubernetes | `KUBERNETES_VERSION_GOVERNANCE.md` |

## Sources

- Argo Workflows: `https://github.com/argoproj/argo-workflows`
- Argo CD: `https://github.com/argoproj/argo-cd`
- Argo Events: `https://github.com/argoproj/argo-events`
- Argo Rollouts: `https://github.com/argoproj/argo-rollouts`
- CNCF Argo: `https://www.cncf.io/projects/argo/`
Loading