feat(ledger): gate future Raft protocol capabilities

cmd/ledgerctl/store/bootstrap_retry_test.go

@@ -34,6 +34,7 @@ import (
 	"github.com/formancehq/ledger/v3/internal/infra/backup"
 	"github.com/formancehq/ledger/v3/internal/infra/cache"
 	"github.com/formancehq/ledger/v3/internal/infra/health"
+	"github.com/formancehq/ledger/v3/internal/infra/membership"
 	"github.com/formancehq/ledger/v3/internal/infra/node"
 	"github.com/formancehq/ledger/v3/internal/infra/plan"
 	"github.com/formancehq/ledger/v3/internal/infra/state"
@@ -61,7 +62,7 @@ func newBootstrapRetryAdmission(t *testing.T, store *dal.Store) *admission.Admis
 	notifier.EXPECT().NotifyConfigChanged().AnyTimes()
 	machine, err := state.NewMachine(logger, registry, state.NewCacheSnapshotter(logger, registry, nil), store,
 		dal.NewSentinelFactory(store, false), meters, keys, shared, notifier, nil,
-		"", 0, func(*raftpb.Entry, *dal.WriteSession) error { return nil })
+		"", 0, func(*raftpb.Entry, *dal.WriteSession) error { return nil }, state.NoopCapabilityPolicyCommitHandler)
 	require.NoError(t, err)
 	require.NoError(t, state.NewRecovery(machine, store).RecoverState())
 	next := machine.State.LastAppliedIndex + 1
@@ -90,7 +91,7 @@ func newBootstrapRetryAdmission(t *testing.T, store *dal.Store) *admission.Admis
 		return future, nil
 	}).AnyTimes()
 
-	return admission.NewAdmission(store, logger, proposer,
+	return admission.NewAdmission(store, logger, proposer, &membership.Coordinator{},
 		builder, meters, gate, keys, shared, attrs, numscript.NewNumscriptCache(0), func(context.Context) error { return nil })
 }
 

cmd/server/server.go

@@ -873,6 +873,7 @@ func discoverPeersFromCluster(raftAddr string, tlsCfg bootstrap.TLSConfig, clust
 			Address:        p.GetRaftAddress(),
 			ServiceAddress: p.GetServiceAddress(),
 			InstanceID:     p.GetInstanceId(),
+			Capabilities:   append([]uint32(nil), p.GetCapabilities()...),
 		})
 	}
 

docs/ops/cluster-operations.md

@@ -722,6 +722,6 @@ After the reset, the preloader falls back to Pebble reads (0xF1 zone) for all ca
 - [Data Flows](../technical/architecture/data-flows.md) - Spool mechanics, entry processing
 - [Spool](../technical/architecture/subsystems/storage/spool.md) - Spool technical implementation
 - [Deployment](./deployment.md) - Helm chart configuration, CLI flags
-- [CLI Reference](./cli.md) - cluster commands
+- [CLI Reference](./cli.md) - `cluster promote-learner` and other cluster commands
 - [Memory Management](./memory.md) - Cache rotation threshold memory impact
 - [Availability and Disaster Recovery](./availability-and-recovery.md) - Failure-domain placement and quorum-loss recovery

docs/ops/upgrades.md

@@ -70,6 +70,20 @@ through each adjacent RC in order; do not jump to the newest RC. If the source
 cannot be identified or the current Operator already rejects the pinning step,
 stop and plan a separately validated migration. Do not delete the StatefulSet
 or PVCs to bypass the source check.
+### Maintainer-declared CLI break
+
+The `ledgerctl cluster add-learner` command is intentionally removed from the
+capability-evidence release. Existing scripts or runbooks invoking that command
+must be updated; the command is not kept as a compatibility alias and no
+protocol-version bump is required because the service gRPC contract is
+unchanged.
+
+To add a new node, configure its peer seed and start the Ledger binary. The
+bootstrap sequence calls `JoinAsLearner` and advertises the capabilities
+compiled into that binary. A node restarting with an existing WAL uses
+`RefreshMembership` to publish updated endpoints and capability evidence.
+The administrative `AddLearner` RPC remains an internal recovery API, but it is
+not a supported replacement for the removed CLI workflow.
 
 Before tagging a new release, verify the upgrade with state created by the
 supported source releases. Cover affected Pebble records, Raft log and snapshot
@@ -328,6 +342,16 @@ validation passes. For RC.1 to RC.2, follow the
 [maintenance-barrier rolling procedure](#account-metadata-address-validation).
 For RC.3 to RC.4, use the [operator-required stopped restart](#operator-required-stopped-restart).
 
+Capability metadata is the prerequisite for any future incompatible Raft entry,
+execution-plan, replay, or snapshot format. This release does not support
+mixed-version activation: stop every Ledger node, deploy the barrier-aware
+binary everywhere, restart the complete cluster, and only then enable the new
+feature. The capability reconciler does not arm features from membership
+advertisements. See the [capability contract](../technical/architecture/subsystems/consensus/capabilities.md)
+for the persisted replay boundary and coordinated rollout.
+
+Use this for RC updates after the mixed-version validation above passes:
+
 1. Record the current leader and voter set with `ledgerctl cluster status`.
 2. Restart one follower on the target binary.
 3. Wait until it reports normal, has caught up to the leader's commit index,
@@ -527,25 +551,27 @@ installation, backup/restore, rollback, or the dormant stopped-upgrade path.
 The pinned binary-level rc.1 test and relevant model, restore and migration
 suites remain separate gates.
 
-The same harness has a separate, future `rc3-to-rc4` transition:
+The same harness has a separate `rc3-to-rc4` transition for the mandatory
+operator-required stopped restart:
 
 ```bash
 UPGRADE_TRANSITION=rc3-to-rc4 RELEASE_BRANCH=release/v3.0 just test-release-operator-upgrade
 ```
 
-Run it against a release branch whose operator lists `v3.0.0-rc.4` as a
-mandatory stop boundary. The target operator and Ledger are still built from
-one exact release-branch HEAD and tagged locally as rc.4; the harness publishes no tag
+Run it after `v3.0.0-rc.3` has been published, against a release branch whose
+operator lists `v3.0.0-rc.4` as a mandatory stop boundary. The target operator
+and Ledger are built from one exact release-branch HEAD and tagged locally as
+rc.4; the harness publishes no tag
 or image. This path requires the operator's drain barrier to advance past
 `Draining`, observes the StatefulSet and all old Pods absent before accepting
 any target Pod, and checks that the replacement StatefulSet has a new UID while
 PVC UIDs persist. It restarts a target voter to exercise replay, verifies the
 pre-upgrade account, commits another transaction, checks Raft progress and
 runs each voter's audit/derived-state checker after drain release. It fails
-against an operator without the rc.4 boundary, so it is a release gate for the
-format-change PR, not evidence that rc.3 itself needs a stop. The rc.4 release
-still needs format-specific migration, snapshot, backup/restore and rollback
-validation outside this scenario.
+against an operator without the rc.4 boundary or if the mandatory stop is
+skipped, so it is a release gate for the format-change PR, not evidence that
+rc.3 itself needs a stop. The rc.4 release still needs format-specific migration,
+snapshot, backup/restore and rollback validation outside this scenario.
 
 ## Coordinated Restart
 

docs/technical/architecture/subsystems/consensus/README.md

@@ -13,6 +13,7 @@ The replication and ordering layer (`internal/infra/node`, `internal/infra/trans
 | [hybrid-logical-clock.md](hybrid-logical-clock.md) | Monotonic HLC timestamps across leader changes and clock skew. |
 | [removed-member-registry.md](removed-member-registry.md) | Universal member-identity invariant, committed-only peer routing, and replicated `(nodeID, instanceID)` set that prevents a removed member from silently rejoining and being auto-promoted; replay-safe per-event admission barriers, lookup costs, removal write ordering and durable restart states. |
 | [controlled-shutdown.md](controlled-shutdown.md) | Raft-ordered admission cutoff, local persistence confirmation, restart behavior, rolling-upgrade capability gate, and operator completion contract. |
+| [capabilities.md](capabilities.md) | Stable capability declarations, persisted activation policy, membership evidence, and mixed-version rollout boundaries. |
 
 ## Related
 

docs/technical/architecture/subsystems/consensus/capabilities.md

@@ -0,0 +1,179 @@
+# Coordinated-upgrade capabilities
+
+Ledger's capability metadata is the protocol foundation for future incompatible
+features deployed with a coordinated stop-and-restart. It is separate from the
+client-facing gRPC protocol revision. This release does not implement a
+mixed-version rolling activation path.
+
+## Wire contract
+
+`common.ClusterPolicy` carries the replicated activation state.
+`raft.Proposal.required_capabilities` carries stable producer-owned capability
+IDs. The fields are optional and therefore backward-readable by binaries that
+predate this foundation. IDs are never reused; semantic dependencies must be
+declared even when they do not add bytes to a proposal.
+
+The producer must rebuild its plan first, derive the capabilities from that
+final plan, stamp `required_capabilities`, and pass the typed result alongside
+the serialized proposal. Admission reuses that result without decoding the
+proposal a second time. A payload that depends on a capability but omits its
+declaration is invalid; the shared builder and stamping helper are the
+enforcement boundary for producers.
+
+## Admission and membership
+
+The pure decision contract in `internal/infra/capabilities` evaluates every
+current voter and learner. A member is represented by `(node_id, instance_id)`;
+an identity refresh is a new incarnation and cannot silently inherit evidence
+from the old one. Capability activation uses committed support evidence, not a
+live health poll: a member that has already acknowledged the capability may be
+temporarily unreachable when the activation is committed. Reachability still
+blocks admission of capability-dependent writes until that member is reachable
+again. Once a capability is active, every new membership announcement must
+include that capability.
+
+A proposal that declares a capability is admitted only when that capability is
+active and every member advertises it. Proposals with no declared capability
+remain unaffected. FSM apply never consults node-local capability flags.
+Membership evidence is an authenticated cluster fact, not a proof of code
+inspection: a node can only cause activation for IDs in the leader's local
+registry, and malformed (zero, duplicate, or unsorted) declarations are
+rejected. Operators must therefore treat a compromised binary as outside the
+trust model of this barrier.
+
+## Capabilities that change FSM semantics
+
+A capability can describe a protocol or semantic behavior, not only a payload
+feature. For example, `MirrorIngestRejectInvalidAssetV2` can identify the
+behavior that rejects an asset which an older FSM accepted.
+
+The capability has two roles in this case:
+
+1. The replicated `ClusterPolicy` controls whether the cluster may accept the
+   behavior.
+2. The order declares the capability that selects the behavior during FSM
+   apply.
+
+The serialized proposal currently carries `required_capabilities` at proposal
+scope. That is sufficient only when every order in the proposal has the same
+semantic contract. A proposal can batch multiple orders, so a future change
+that mixes old and new semantics must either add a persisted marker to each
+order (and dispatch apply from that marker) or reject mixed-semantic batches.
+The conceptual marker for a homogeneous proposal is:
+
+```text
+old proposal:  required_capabilities = []
+new proposal:  required_capabilities = [MirrorIngestRejectInvalidAssetV2]
+```
+
+The FSM must use the persisted declaration to select the behavior. It must not
+infer the behavior from the current policy alone. An old order can be replayed
+after the capability becomes active, and it must keep its old semantics.
+
+If a producer cannot mark the new orders, the change must remain backward
+compatible or add an explicit semantic version to the order. A capability by
+itself is an admission barrier; it is not a historical replay version.
+
+## Adding a protocol-breaking capability
+
+Use this workflow when a change can make an order apply differently on two
+binary versions.
+
+1. Name the semantic contract. Describe the old behavior, the new behavior,
+   and the orders that use each behavior.
+2. Allocate a new capability ID. Never reuse an ID for a different contract.
+3. Add the capability to the binary registry and derive it from the final
+   proposal in `RequiredCapabilities`.
+4. Stamp the capability on every new order that needs the new behavior. Do not
+   infer it from the current policy during FSM apply.
+5. Keep the old behavior for orders that do not carry the capability. Add
+   replay tests that apply both order forms with the new binary.
+6. Add admission tests for an inactive capability, an active capability, and a
+   member that does not advertise it.
+7. Deploy the compatible binary to all current members. Let each restarted
+   member refresh its membership evidence before activating the capability.
+8. Activate the capability through the replicated `ClusterPolicy`. The FSM
+   rejects a new member or membership refresh that omits an active capability.
+9. Keep the capability active after activation. The policy processor rejects
+   removal of an active capability.
+10. Update this document in the same change. Record the capability ID, the
+    semantic contract, the replay rule, and the rollout order.
+
+Do not add a capability for a bug fix that restores the intended behavior when
+the old behavior was invalid. Add one when valid historical orders need to keep
+their old behavior while new orders use a different rule.
+
+## Rollout and recovery
+
+Deploy the barrier-aware binary to every member, stop the cluster, and restart
+the complete cluster before enabling an incompatible feature. An old leader
+cannot be made safe by upgrading only the other nodes. Restart, snapshot
+install, restore, and downgrade code must preserve the policy. Membership
+changes are not used as a rolling activation mechanism. `Node.Propose` evaluates the authoritative
+policy while holding the proposal and index-tracker guards, before reserving a
+predicted index or queueing work; this prevents a rejected proposal from
+leaving a hole for queued successors. Removing a member remains allowed;
+adding, promoting, or refreshing one without all active capabilities is
+rejected.
+
+`Node.Propose` is the single final policy boundary for every producer,
+including mirror, events, maintenance, and backup flows; the shared plan
+runner does not duplicate this check. Callers must still stamp their proposal
+before use. Membership `ConfChange` operations carry and validate their
+capability evidence separately.
+
+## Admission cost budget and evidence
+
+The proposal gate has a deliberately bounded cost shape. For a proposal that
+declares `r` capabilities and a current membership of `m` voters and learners,
+the pure decision is `O(r + m*r)` and allocates only its normalized input and
+blocker report. The membership snapshot is copied once per gate evaluation;
+the gate must not perform network calls, wait for Raft, or hold a storage
+transaction while evaluating the result. Policy decoding and the snapshot
+copy remain serialized by the proposal admission boundary, so they are part
+of the write-path budget rather than background work.
+
+The deterministic behavior matrix is covered by
+`internal/infra/capabilities/capabilities_test.go`: empty declarations,
+inactive policy, active policy with converged voters and learners, missing
+evidence, and unreachable members. The representative cost shape is measured
+by:
+
+```text
+go test ./internal/infra/capabilities -run '^$' -bench BenchmarkEvaluate -benchmem -count=5
+```
+
+The benchmark uses three and seven members with zero, one, and four required
+capabilities. Any future capability registry or policy-provider change must
+rerun it and include the result in the change review. A regression must be
+explained against the `m*r` bound; if the bound no longer fits the admission
+budget, move decoding/snapshot preparation outside the serialized section and
+pass an immutable admission snapshot instead of widening the lock.
+
+## Restart refresh
+
+A normal restart does not run `JoinAsLearner`: the node restores its existing
+WAL `ConfState` and keeps the same `(node_id, instance_id)` incarnation. After
+the node and its Raft gRPC server are started, a node whose WAL carries
+`CLUSTER_JOINED` sends an internal `RefreshMembership` request to a configured
+peer. Followers forward that request to the current leader over the Raft gRPC
+server. The node can therefore receive inbound Raft traffic before the refresh
+commits; the refresh is an evidence update, not a pre-traffic barrier. During
+the rc.2/rc.3 rolling upgrade, an older peer that returns `Unimplemented` is
+treated as transient and the node keeps retrying in the background until a
+compatible peer is available. The capability-dependent protocol cutover is
+reserved for the coordinated rc.3/rc.4 restart, after every node runs the
+barrier-aware binary.
+The leader validates the identity and capability evidence against the active
+policy, then proposes a
+`ConfChangeUpdateNode` containing the current endpoints and capabilities.
+
+Only the committed change is authoritative. The update is replicated through
+Raft and applied to every member's peer store; a local restart-time upsert is
+not sufficient to activate a capability for admission. A fresh WAL instead
+uses `JoinAsLearner` and `ConfChangeAddLearnerNode`.
+
+The cluster-policy reconciler preserves committed capability activations while
+reconciling ordinary policy configuration; it does not arm capabilities from
+membership advertisements. A future feature must define its activation policy
+explicitly and enable it only after the coordinated restart has completed.