[flemzord]

Summary

Ledger v3 now uses RocksDB for its primary store, read index, and usage projection. This also updates checkpoints, restore and offline bootstrap, storage configuration, metrics, and Linux packaging. The Pebble dependency is removed from the server and release artifacts.

The first version starts with fresh RocksDB data. Ledger v3 has not had a stable Pebble release, so this PR does not convert development data or backups. The service gRPC protocol revision is 15 because storage metrics and index progress changed.

The RocksDB operator is now installed as ledger-next-operator and owns all six resources under ledger-next.formance.com/v1alpha1. The existing Pebble operator retains ledger.formance.com/v1alpha1. Workload names, selectors, Secrets, leader election and volume-protection policies are isolated. This allows fresh-data tests of both versions on the same Kubernetes cluster.

Validation

• Current head dc9f911af adds operator coexistence and the ACME benchmark protocol/results, and integrates release/v3.0 through b01200137. Local operator unit tests with race detection, envtest with both API groups installed, Helm isolation/render checks, and Antithesis workload routing tests passed. Repository baseline validation passed; root and operator lint reported zero issues. FSM/plan and cluster-local-state merge tests passed with race detection, as did the workload routing/stats/model tests. The broader local gate was interrupted before completion. Pinned qualification passed every mandatory gate and published both immutable candidate images for this exact SHA, including Dirty, business/cluster e2e, model and operator tests. The earlier Dirty formatting failure is fixed. Older Antithesis results below qualify the earlier storage implementation.

• Five 20-minute Antithesis qualifications exercised the new storage under cluster faults. The first found a Raft removal panic, and the third found a transient learner-join startup exit. Both paths have fixes and regression coverage; the fourth and fifth runs did not reproduce either exit. The fifth run passed all 53 Always assertions. Its maintenance reset had 60 passing examples and no failures; the stats qualification and exact volume convergence each had 63 passing examples and no failures. Its new findings concern scenario success or reachability with no example for the properties inspected, so the PR remains draft while coverage and intermittent findings are assessed.

• Pebble versus RocksDB comparison uses fresh single-node data, paired 90-second HTTP runs, and storage microbenchmarks. At 30 writes/15 reads per second and 50/25, all valid measured runs had zero request errors or dropped iterations. Across three runs of unchanged RocksDB code, write p99 ranged from -20.0% to +3.4% at 30/15 and -7.5% to +10.0% at 50/25. The third HTTP run was near parity at 50/25. Sampled mean RSS was 9–18% lower. Batch microbenchmarks consistently favored Pebble. These campaigns do not establish latency equivalence or maximum throughput.

• ACME-dev Pebble beta.5 pilot: three replicas, 120-second windows at 100 and 1,000 transactions/s (atomic bulk 50), then account/transaction reads and transaction lists. Zero workload errors, dropped iterations or restarts. At 1,000 tx/s, bulk p95 was 15.65 ms. This small warm-cache pilot on mixed shared nodes is beta feedback, not a RocksDB comparison or capacity limit. Benchmark Grafana reporting needs its URL configured; raw k6/telemetry artifacts were retained locally.

• ACME-dev RocksDB pilot: the separate ledger-next-operator release is deployed with immutable images for dc9f911af. A fresh three-replica next Cluster and Ledger CR became ready alongside the unchanged Pebble operator and its three healthy clusters. Final 120-second write/read runs exited 0 with zero HTTP/application errors, zero dropped iterations and verified balances. Bulk-write p95 was 16.04 ms at 100 tx/s and 15.74 ms at 1,000 tx/s, versus beta.5 at 15.59/15.65 ms. Account/transaction/list p95 was 21.08/21.17/11.44 ms, versus 22.38/22.28/11.48 ms. A first write run hit a k6 boundary-count threshold; a fresh-ledger retry passed with nominal-to-one-extra-bulk tolerance. Existing Benchmark Grafana reporting is unconfigured. These single-repeat, small warm-cache pilots on mixed/shared nodes compare versions, and do not establish engine parity or capacity. Raw summaries and resource telemetry are retained locally.

Risks and follow-up

• Old Pebble directories and backups cannot be opened by this build. This affects development data; no stable v3 Pebble release exists.
• Earlier Antithesis/model runs reported an asset-index account-query mismatch and a prepared log-query page mismatch. The latter did not recur in subsequent runs, which is not proof of resolution. Investigate those intermittent findings and the remaining scenario coverage before making this PR ready to merge.
• Repeat performance and checkpoint/recovery measurements on representative three-node clusters before setting a production performance acceptance threshold.