feat: add Holt and FDB metadata runtimes

[feichai0017]

Outcome

This PR adds two explicit metadata runtimes:

• holt:///absolute/path runs one exclusive standalone Holt store without a control plane.
• fdb:///absolute/fdb.cluster?prefix=NAME uses FoundationDB for shared metadata, catalog, routing, sessions, and leases.
• Clients discover the current owner through NoKV seeds. Client packages do not connect to FDB.
• This series removes the retired etcd dependency and metadata/control path.

Object bytes remain in S3-compatible storage such as RustFS. The metadata runtime does not store artifact payloads.

The implementation has no provider fallback, dual write, automatic migration, compatibility shim, or filename-based shard placement.

Architecture

• nokv-meta-store defines the provider-neutral transaction contract.
• nokv-meta-holt implements the local adapter. Holt remains the default build and standalone runtime.
• nokv-fdb owns the process-global FDB network, database, transaction, prefix, and error boundaries.
• nokv-meta-fdb maps TxnStore operations to session-fenced FDB transactions.
• nokv-control-fdb stores manifests, catalogs, routes, owner sessions, and heartbeats.
• nokv-server composes Holt or FDB from the metadata URL and transfers the FDB lease from bootstrap to steady-state serving.
• nokv-client resolves owners through NoKV seed endpoints.

The series also fixes failures found during live FDB qualification. The fixes cover recoverable provisioning, durable secondary-index replay, Ready-shard preservation, settled transient reads, owner-loss propagation, and continuous lease renewal.

FDB qualification

All ten FDB serving gates pass their scoped acceptance checks. The recorded distributed serving profile is QUALIFIED for FDB 7.3.79, three storage processes, three coordinators, three logs, double redundancy, and the pinned RustFS service.

Gate	Status	Covered behavior
1. Conformance	PASS	Point reads, scans, versions, predicates, conflicts, bounds, namespace isolation, and reopen
2. Unknown outcomes	PASS	Deterministic lost-ACK injection, all 17 mutation families, bootstrap, and cross-owner replay
3. Session fencing	PASS	Stale renew, read, write, and release rejection
4. Takeover	PASS	Concurrent contenders, owner loss, monotonic expiry, and successor takeover
5. Provision crashes	PASS	Prepare and finalize crash cuts with admission recovery
6. Serve crashes	PASS	Activation cuts, lease failure, local fail-close, stale-owner rejection, and successor mutation
7. Seed discovery	PASS	Failed-first fallback, owner refresh, stale hints, drift rejection, and final mutation
8. Lifecycle	PASS	Publication, replacement, snapshots, restore, revision GC, delete quarantine, and commit retirement
9. Limits	PASS	Measured FDB transaction sizes through 2,800,000 logical bytes and a 3,200,000-byte object round-trip
10. Performance	PASS	Uncontended and contended distributions with latency percentiles, throughput, retries, conflicts, and failures

Final Gates 8 through 10 used code source 11994bdca9a235aac70c5a68dc2a41ec856493f9 and one nokv-fdb binary with SHA-256 983e096c663341ba891f73115490fc22c1505401baff245d24b5f2a5490c2e46.

The Gate 10 measurements qualify this exact topology and workload. They are not a cross-machine product SLO and do not claim equivalent Holt and FDB durability.

See docs/development/fdb-root-fix-qualification-2026-08-31.md for hashes, topology, evidence bundles, results, and the full gate matrix.

Validation

• cargo fmt --all -- --check: PASS
• cargo test --workspace --all-features --exclude nokv-python in the FDB builder: PASS
• cargo test -p nokv-python with default features: PASS, 23 passed and 2 live S3 tests ignored
• cargo clippy --workspace --all-targets --all-features -- -D warnings in the FDB builder: PASS
• Real FDB adapter, control, fencing, crash-recovery, lifecycle, limit, and performance qualification: PASS
• python3 scripts/workbench/workbench_contract_test.py: PASS, 8 passed
• git diff --check: PASS
• DCO trailers on every non-merge branch commit: PASS

Commit and review boundary

The branch retains 73 commits for review, including three merges from main. All 70 non-merge commits include a Signed-off-by trailer.

The current diff contains 201 files, 37,142 insertions, and 32,999 deletions relative to main. The protected large-change-review check requires one eligible core maintainer to approve the exact current head before merge.

[Copilot]

Pull request overview

This PR introduces an explicit dual metadata-runtime model (standalone Holt vs shared FoundationDB) plus seed-based client discovery, and removes the superseded etcd metadata/control path. It also updates Workbench qualification plumbing, protocol envelopes, and metadata/index durability semantics to support the new serving and qualification boundaries.

Changes:

• Add FoundationDB runtime + adapters (nokv-fdb, nokv-meta-fdb, nokv-control-fdb) and wire/protocol support for seed discovery (DiscoverRoute).
• Update server/client composition and public CLI/Python surfaces to route via seeds and support Holt/FDB metadata URLs (with FDB remaining feature-gated / NOT QUALIFIED).
• Refactor metadata secondary-index publication to use generation-fenced staged/published visibility and add bounded cleanup primitives.

Reviewed changes

Copilot reviewed 50 out of 201 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
scripts/workbench/typed_live_qualification_test.py	Updates typed live qualification test dependency naming (etcd → metadata-store).
scripts/workbench/qualification_receipt_test.py	Removes etcd dependency from receipt tests for live producers.
scripts/workbench/qualification_invocation_manifest.json	Removes legacy producers from typed manifest; records omitted producers and updates receipt counts.
scripts/workbench/qualification_invocation_check_test.py	Makes expected report counts and artifact mapping derived from manifest/ledger.
scripts/workbench/python_sdk_qualification.py	Switches Python SDK live-gap dependencies to nokv-seed + object-store.
scripts/workbench/pre423_contract_ledger.py	Updates qualification policy hash.
scripts/workbench/pre423_contract_ledger.json	Adjusts producer required dependencies (etcd removal; seed/metadata-store naming).
scripts/workbench/live_workbench_test.py	Aligns workbench harness tests with seed/metadata changes and updated identity handling.
scripts/workbench/lingtai_mcp_qualification.py	Switches LingTai MCP gap dependencies from etcd to nokv-seed.
scripts/release/test_python_sdk_release.py	Tightens assertions to ensure protobuf/protoc steps are absent from release workflow.
docs/rustfs.md	Updates namespace provisioning semantics; removes legacy adoption mode description.
docs/product-design.md	Documents staged secondary-index generation + atomic visibility flip.
docs/index.md	Updates feature blurb to reflect staged index generation.
docs/development/pr_review_checklist.md	Updates checklist to reflect staged index generation + generation fencing.
docs/development/path-native-metadata-comparison.md	Updates publish/replace description for staged index generation + final publish command.
docs/development/code_contract.md	Adds/updates package boundaries (nokv-fdb/meta-fdb/control-fdb) and index generation-fencing contract.
docs/ai-training.md	Updates training pipeline step to reflect staged index generations.
crates/nokv/tests/cli_help.rs	Updates help-surface test to assert seed discovery + dual metadata runtimes + NOT QUALIFIED messaging.
crates/nokv/src/object_store.rs	Removes unused namespace-load surface from CLI object store adapter.
crates/nokv/src/fdb_main.rs	Adds nokv-fdb bin wrapper that includes main CLI implementation behind fdb feature.
crates/nokv/src/backend.rs	Extends error mapping for discovery failures and updated wire responses.
crates/nokv/Cargo.toml	Removes etcd default feature; adds fdb feature and nokv-fdb binary.
crates/nokv-types/src/workspace.rs	Adds PathIndexGenerationId fixed-bytes type.
crates/nokv-types/src/lib.rs	Re-exports PathIndexGenerationId.
crates/nokv-server/tests/fdb_provision.rs	Adds ignored live test for FDB provisioning crash-cut recovery and idempotent finalize.
crates/nokv-server/src/registry.rs	Adjusts routing failures (route_hint removal) and adds typed control fail-close preservation test.
crates/nokv-server/src/lib.rs	Splits Holt vs FDB runtime exports; adds metadata URL module exports and new routing discovery surface.
crates/nokv-server/src/legacy_rejection.rs	Updates legacy rejection to wrap responses in RpcResponse::Workspace.
crates/nokv-server/src/error.rs	Removes legacy recovery installer/publisher error variants and prepared owner admission variant.
crates/nokv-server/Cargo.toml	Adds fdb feature wiring to control/meta/runtime FDB crates and URL parsing deps.
crates/nokv-python/tests/test_api_surface.py	Updates Python API surface assertions for RoutingConfig.seeds.
crates/nokv-python/src/lib.rs	Updates module test to require RoutingConfig.seeds.
crates/nokv-python/src/client.rs	Refactors transport options build path; handles discovery errors as RPC failures for error codes.
crates/nokv-python/Cargo.toml	Drops explicit nokv-client etcd feature dependency.
crates/nokv-protocol/src/types.rs	Adds route discovery types (DiscoveredRoute, OwnerEndpoint, RouteState).
crates/nokv-protocol/src/response.rs	Adds RpcResponse envelope and discovery response/outcome.
crates/nokv-protocol/src/request.rs	Adds RpcRequest envelope and discovery request.
crates/nokv-protocol/src/lib.rs	Updates exports/docs for seed discovery and new RPC envelopes.
crates/nokv-protocol/src/error.rs	Adds discovery error codes and changes route_hint to Box<DiscoveredRoute>.
crates/nokv-meta/src/workspace/test_support.rs	Extends commit capture support and adds transaction planning target wrapper store.
crates/nokv-meta/src/workspace/snapshot.rs	Updates tests to use recovery receipts’ lsn where required.
crates/nokv-meta/src/workspace/remove.rs	Removes eager secondary-index row deletions; documents async cleanup semantics; updates tests accordingly.
crates/nokv-meta/src/workspace/recovery.rs	Clarifies recovery module applies to LocalJournal stores only.
crates/nokv-meta/src/workspace/publication_records.rs	Bumps publication record version; adds path digest + index generation to PathEntry.
crates/nokv-meta/src/workspace/namespace.rs	Updates tests for extended PathEntry fields.
crates/nokv-meta/src/workspace/mod.rs	Exposes new index locator + cleanup APIs and workspace format version.
crates/nokv-meta/src/workspace/keyspace.rs	Adds PathIndexLocator family and keyspace catalog entry.
crates/nokv-meta/src/workspace/generic_index.rs	Updates tests/fixtures to populate new PathEntry fields.
crates/nokv-meta/src/workspace/gc.rs	Updates tests to use recovery receipts’ lsn where required.
crates/nokv-meta/src/workspace/commit.rs	Adds path index locator row creation alongside path publication.
crates/nokv-meta/src/workspace/codec.rs	Renames/bumps workspace format version constant (marker versioning).
crates/nokv-meta-store/src/types.rs	Adds recovery mode + transaction planning target to store profile.
crates/nokv-meta-store/src/tests.rs	Updates fake store profile to include new profile fields.
crates/nokv-meta-store/src/store.rs	Documents new StoreError::Fenced semantics.
crates/nokv-meta-store/src/lib.rs	Re-exports RecoveryMode.
crates/nokv-meta-store/src/errors.rs	Adds fenced-owner error variant and display formatting.
crates/nokv-meta-store/src/conformance.rs	Tightens profile conformance checks (target bytes + recovery mode + ack/authority consistency).
crates/nokv-meta-holt/src/store.rs	Centralizes Holt local store profile initialization with new profile fields.
crates/nokv-meta-holt/src/lib.rs	Publishes Holt physical encoding version constant.
crates/nokv-meta-fdb/src/profile.rs	Defines FDB store limits/profile and conservative physical transaction guard.
crates/nokv-meta-fdb/src/options.rs	Adds FDB metadata adapter options and stable session-fence predicate type.
crates/nokv-meta-fdb/src/lib.rs	Introduces new FDB metadata adapter crate with feature-gated binding.
crates/nokv-meta-fdb/src/diagnostics.rs	Adds diagnostic counters/snapshot for FDB adapter transactions.
crates/nokv-meta-fdb/src/codec.rs	Adds key encoding/scan bound logic for FDB metadata subspace.
crates/nokv-meta-fdb/src/affected_bytes.rs	Adds affected-bytes estimation and physical guard enforcement for FDB txns.
crates/nokv-meta-fdb/README.md	Documents adapter intent, feature-gating, and live conformance requirements.
crates/nokv-meta-fdb/Cargo.toml	Adds new crate manifest with feature-gated FDB binding dependency.
crates/nokv-fdb/src/tests.rs	Adds tests for options/prefix/error classification/runtime registry semantics.
crates/nokv-fdb/src/runtime.rs	Implements process-global non-restartable FDB network runtime with explicit API version.
crates/nokv-fdb/src/prefix.rs	Implements versioned physical prefix/subspace envelope and lexicographic successor helper.
crates/nokv-fdb/src/options.rs	Adds validated cluster file + transaction timeout options.
crates/nokv-fdb/src/lifecycle.rs	Adds internal runtime registry that prevents restart after stop/failure.
crates/nokv-fdb/src/lib.rs	Introduces new FDB boundary crate with feature-gated database/runtime.
crates/nokv-fdb/src/error.rs	Adds stable error disposition classification and binding error wrapper type.
crates/nokv-fdb/src/database.rs	Adds database + transaction wrappers with explicit non-retrying operations.
crates/nokv-fdb/README.md	Documents runtime boundary behavior and feature-gated compilation.
crates/nokv-fdb/Cargo.toml	Adds new crate manifest with feature-gated foundationdb/futures deps.
crates/nokv-control/src/node_id.rs	Introduces NodeId type with canonical validation.
crates/nokv-control/src/lib.rs	Refactors control crate to provider-neutral catalog/ownership contracts (removes etcd module).
crates/nokv-control/src/distributed_store.rs	Adds DistributedControlStore trait for shared metadata modes.
crates/nokv-control/Cargo.toml	Removes etcd feature/deps; updates crate description.
crates/nokv-control-fdb/tests/fdb_control.rs	Adds ignored live test for concurrent contenders + takeover fencing.
crates/nokv-control-fdb/src/session_fence.rs	Adds stable session fence encoder for metadata adapter use.
crates/nokv-control-fdb/src/physical_keys.rs	Adds physical key layout under FDB store prefix subspaces.
crates/nokv-control-fdb/src/options.rs	Adds validated control store options and manifest binding checks.
crates/nokv-control-fdb/src/observer.rs	Adds local monotonic ownership observation helper for takeover TTL.
crates/nokv-control-fdb/src/lib.rs	Introduces control-fdb crate with feature-gated live store impl.
crates/nokv-control-fdb/README.md	Documents control store behavior and feature-gated compilation/testing.
crates/nokv-control-fdb/Cargo.toml	Adds new crate manifest.
crates/nokv-client/src/transport.rs	Updates protocol hello test name for v10 framing.
crates/nokv-client/src/lib.rs	Replaces etcd/control route resolver exports with seed-based resolver exports.
crates/nokv-client/src/generic_index.rs	Updates test transport helpers for RpcRequest/RpcResponse envelopes.
crates/nokv-client/src/error.rs	Adds ClientError::Discovery and ensures retryability/error rendering.
crates/nokv-client/src/artifact.rs	Updates test transport helpers and retry classification to include discovery errors.
crates/nokv-client/Cargo.toml	Removes control/etcd features and dependency.
Cargo.toml	Adds new workspace members/deps for FDB and URL/percent-encoding; removes etcd-client.
bench/src/serve_qualification/scenario.rs	Adds serve qualification scenario evidence helpers around ownership transitions.
bench/src/seed_qualification/scenario.rs	Adds seed discovery qualification scenario helpers for discovered routes.
bench/src/report.rs	Adds classified sample reporting and refactors measurement reporting to reuse it.
bench/src/metadata/fixture.rs	Updates metadata fixture seeding to populate new path index fields.
bench/src/lib.rs	Adds FDB live qualification modules and seed/serve qualification modules.
bench/src/fdb_performance_qualification/controls.rs	Adds system control capture for performance evidence.
bench/src/fdb_limits_qualification/inspection.rs	Adds live FDB prefix inspection for Gate 9 limit evidence.
bench/src/bin/nokv-workspace-bench.rs	Wraps bench request in RpcRequest::Workspace to match protocol changes.
bench/src/bin/nokv-fdb-unknown-outcome-qualification.rs	Adds live unknown-outcome qualification binary with explicit env gate.
bench/src/bin/nokv-fdb-serve-qualification.rs	Adds live serve qualification binary with explicit env gate.
bench/src/bin/nokv-fdb-seed-qualification.rs	Adds live seed qualification binary with explicit env gate.
bench/src/bin/nokv-fdb-performance-qualification.rs	Adds live performance qualification binary with explicit env gate.
bench/src/bin/nokv-fdb-limits-qualification.rs	Adds live limits qualification binary with explicit env gate.
bench/src/bin/nokv-fdb-lifecycle-qualification.rs	Adds live lifecycle qualification binary with explicit env gate.
bench/fault_injection/fdb_commit_unknown_fixture_api.h	Adds minimal fake FDB C API for commit-unknown fault injection.
bench/fault_injection/fake_fdb_c.c	Implements fake FDB C functions for fault injection harness.
bench/Cargo.toml	Reworks bench features and adds FDB qualification binaries/deps.
.github/workflows/release-python-sdk.yml	Removes protobuf/protoc setup steps now that etcd/protobuf build-time dependency is gone.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.