Support planned local switchover within a single DocumentDB HA cluster

[WentingWu666666]

Summary

Expose a user-facing way to perform a planned (administrator-initiated) switchover within a single DocumentDB HA cluster, leveraging CloudNativePG's native switchover mechanism. Today the operator only supports automatic local failover (delegated to CNPG) and manual cross-cluster promotion (multi-region). There is no supported way to deliberately move the primary role to a specific healthy replica instance without waiting for a failure.

Motivation

Planned switchover is a routine HA operation, distinct from failover:

• Node maintenance / drain — move the primary off a node before draining it, avoiding an ungraceful failover.
• Planned AZ maintenance — proactively relocate the primary to another zone.
• Rebalancing — restore a preferred primary placement after recovery.

Switchover is graceful (old primary archives WAL, uses pg_rewind, rejoins as replica) with minimal downtime and no data loss, unlike an unplanned failover.

Background: how CNPG supports this

CNPG performs a switchover when .status.targetPrimary is set to a different, healthy instance (e.g. via kubectl cnpg promote <cluster> <instance>). The operator already sets primaryUpdateMethod: switchover (operator/src/internal/cnpg/cnpg_cluster.go:76), so CNPG uses switchover during rolling updates — but we never trigger it on demand for a specific instance.

Proposed scope

• Provide a mechanism to request a local switchover to a target instance within one DocumentDB cluster (e.g. a kubectl documentdb switchover subcommand and/or an operator-driven trigger), mapping to setting the underlying CNPG Cluster.status.targetPrimary.
• Validate the target instance is healthy before initiating; reject switchover for single-instance clusters (instancesPerNode < 2).
• Surface progress/status so callers can wait for completion (similar to the existing promote wait logic).
• Keep this distinct from cross-cluster promotion (spec.clusterReplication.primary) — this is intra-cluster only.

Out of scope

• Changes to automatic local failover behavior (already handled by CNPG).
• Cross-cluster / multi-region promotion (already supported).

Open questions

1. Interface: reuse/extend the kubectl documentdb promote plugin, add a new switchover subcommand, or expose it via a DocumentDB spec/annotation field?
2. Should target selection be an explicit instance name, or a policy (e.g., "any replica in zone X")?
3. Do we need an e2e test under the lifecycle/status labels to cover planned switchover?

Acceptance criteria

• A documented, supported way to trigger a planned local switchover to a chosen healthy instance.
• Guardrails for single-instance and unhealthy-target cases.
• Docs updated (docs/.../operations/failover.md or a new switchover.md).
• Test coverage for the switchover path.

[documentdb-triage-tool]

🤖 Auto-triaged by documentdb-triage-tool.

Applied: ecosystem, go
Project fields suggested: Component kubectl-plugin · Priority P0 · Effort L · Status Triage
Confidence: 0.75 (mixed)

Reasoning

P0 keyword "data loss" in body; LLM: New user-facing planned switchover capability touching kubectl-plugin, controllers (CNPG cluster integration), and validation logic across multiple files with open design questions still unresolved.

If a label is wrong, remove it manually and ping @patty-chow so the rules can be tuned. The bot will not re-label items that already have component labels.