diff --git a/Cargo.lock b/Cargo.lock index e18c0da9..a3a8bfd9 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -813,6 +813,7 @@ version = "0.15.0" dependencies = [ "alloy", "ant-protocol", + "bao", "blake3", "bytes", "chrono", @@ -1322,6 +1323,16 @@ dependencies = [ "windows-link", ] +[[package]] +name = "bao" +version = "0.13.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3f9125f93587b894c196d58b0a752ce2213552d0d483c98ee41530e38202a511" +dependencies = [ + "arrayvec", + "blake3", +] + [[package]] name = "base16ct" version = "0.2.0" diff --git a/Cargo.toml b/Cargo.toml index b3217a87..0c245bd7 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -111,6 +111,7 @@ page_size = "0.6" # Protocol serialization postcard = { version = "1.1.3", features = ["use-std"] } +bao = "0.13.1" [target.'cfg(unix)'.dependencies] libc = "0.2" diff --git a/docs/REPLICATION_DESIGN.md b/docs/REPLICATION_DESIGN.md index 8d83094e..e14f4692 100644 --- a/docs/REPLICATION_DESIGN.md +++ b/docs/REPLICATION_DESIGN.md @@ -461,6 +461,8 @@ Rules: ## 15. Storage Audit Protocol (Anti-Outsourcing) +Protocol families: the audit message families do not ride the core replication protocol id. The digest-based lanes described in this section (responsible-chunk audit, post-replication possession probe, prune confirmation) share one possession-audit id, and the gossip-triggered subtree audit has its own. Core replication keeps its id unchanged, so an audit-protocol change never partitions replication across a mixed-version fleet. A receive guard MUST drop any message whose family disagrees with the id it arrived on, and responses MUST be sent on the same id the guard would accept them on. See ADR-0009. + Challenge-response for claimed holders: 1. Challenger creates unique challenge id + nonce. @@ -470,7 +472,7 @@ Challenge-response for claimed holders: 5. If `PeerKeySet` is empty, the audit tick is idle. 6. Challenger sends `challenged_peer_id` an ordered challenge key set equal to `PeerKeySet(challenged_peer_id)`. 7. Target responds with either per-key `AuditKeyDigest` values or a bootstrapping claim: - a. Per-key digests: for each challenged key `K_i` (in challenge order), target computes `AuditKeyDigest(K_i) = H(nonce || challenged_peer_id || K_i || record_bytes_i)`, where `record_bytes_i` is the full raw bytes of the record for `K_i`. Target returns the ordered list of per-key digests. If the target does not hold a challenged key, it MUST signal absence for that position (e.g., a sentinel/empty digest); it MUST NOT omit the position silently. + a. Per-key digests: for each challenged key `K_i` (in challenge order), target computes `AuditKeyDigest(K_i) = keyed_H(key = derive_key(context, nonce || challenged_peer_id || K_i), record_bytes_i)`, where `record_bytes_i` is the full raw bytes of the record for `K_i`, `derive_key` is BLAKE3 key-derivation mode and `context` is a versioned domain-separation string. Target returns the ordered list of per-key digests. If the target does not hold a challenged key, it MUST signal absence for that position (e.g., a sentinel/empty digest); it MUST NOT omit the position silently. b. Bootstrapping claim: target asserts it is still bootstrapping. Challenger applies the bootstrap-claim grace logic (Section 6.2 rule 4b): record `BootstrapClaimFirstSeen` if first observation, accept without penalty within the one-time `BOOTSTRAP_CLAIM_GRACE_PERIOD`, emit `BootstrapClaimAbuse` evidence if past grace period or if this is a repeated claim after the peer previously stopped claiming bootstrap. Audit tick ends (no digest verification). 8. On per-key digest response, challenger recomputes the expected `AuditKeyDigest(K_i)` for each challenged key from local copies and verifies equality per key before deadline. Each key is independently classified as passed (digest matches) or failed (mismatch, absent, or malformed). 9. On any per-key audit failures (timeout, malformed response, or one or more `AuditKeyDigest` mismatches/absences), challenger MUST perform a responsibility confirmation for each failed key before emitting penalty evidence: @@ -483,7 +485,7 @@ Audit-proof requirements: 1. Challenger MUST hold a local copy of each challenged record to recompute per-key digests. Audit selection is therefore limited to records the challenger stores. 2. Records are opaque bytes for replication; digest construction MUST operate over raw record bytes (no schema dependency) and be deterministic. -3. Each `AuditKeyDigest(K_i)` input MUST be exactly: `H(nonce || challenged_peer_id || K_i || record_bytes_i)`. Including `K_i` binds each digest to its specific key and prevents digest reordering attacks. +3. Each `AuditKeyDigest(K_i)` MUST be computed as `keyed_H(key = derive_key(context, nonce || challenged_peer_id || K_i), record_bytes_i)`. Including `K_i` in the derived key binds each digest to its specific key and prevents digest reordering attacks. The challenge material MUST enter as the hash key, not as a prefix to the hashed stream: a prefix binds only the leading bytes of the record, leaving the remainder independent of the nonce, whereas a derived key makes every compression of the record depend on it. 4. Each `AuditKeyDigest` MUST include full record bytes; key-only digests are invalid. 5. Nodes that advertise audit support MUST produce valid responses within `AUDIT_RESPONSE_TIMEOUT`. 6. Responses MUST include exactly one digest entry per challenged key in challenge order. A response is invalid if it has fewer or more entries than challenged keys. @@ -495,7 +497,8 @@ Audit challenge bound: Failure conditions: -- Timeout, malformed response, or per-key `AuditKeyDigest` mismatch/absence — subject to responsibility confirmation (step 9) before penalty. +- Malformed response or per-key `AuditKeyDigest` mismatch/absence — subject to responsibility confirmation (step 9) before penalty. +- Timeout — also subject to responsibility confirmation, but see the rollout gate below: for as long as a protocol-family change is rolling out across the fleet, the digest-based lanes MUST NOT penalise a timeout, because a peer on the older family cannot answer at all and its silence is not its fault. A response that arrives is judged on what it says throughout: mismatch, absence, malformed reply and explicit rejection are all penalised while the gate is set. The gate MUST be removed once the fleet has upgraded (see ADR-0009 for the criterion and owner), restoring the timeout penalty on these lanes. - Bootstrapping claim past `BOOTSTRAP_CLAIM_GRACE_PERIOD`, or repeated after the peer previously stopped claiming bootstrap (emits `BootstrapClaimAbuse`, not `AuditFailure`). Audit trigger and target selection: diff --git a/docs/adr/ADR-0009-audit-proof-shape-and-protocol-families.md b/docs/adr/ADR-0009-audit-proof-shape-and-protocol-families.md new file mode 100644 index 00000000..84095fd3 --- /dev/null +++ b/docs/adr/ADR-0009-audit-proof-shape-and-protocol-families.md @@ -0,0 +1,223 @@ +# ADR-0009: Audit proof shape and protocol families + +- **Status:** Proposed +- **Date:** 2026-07-29 +- **Decision owners:** Anselme (@grumbach) +- **Reviewers:** +- **Supersedes:** none +- **Superseded by:** none +- **Related:** ADR-0002 (gossip-triggered contiguous-subtree audit), ADR-0003 (possession verification), ant-node #181, V2-685 + +## Context + +ADR-0002 recorded the gossip-triggered subtree audit with a round 2 that returns +the audited chunks' original bytes. Fleet measurement showed that round-2 +response to be one of the largest steady-state replication traffic sources: on a +same-network control cohort the mean response was 6.19 MB, and production +sampling agreed at 5.79 to 6.15 MB across four separate days. The cost is paid +per audit event regardless of how often audits fire, so frequency caps alone +cannot bound it. + +Separately, the node has four audit lanes that each ask a peer to prove it holds +specific bytes: + +1. the subtree storage-commitment audit (ADR-0002), +2. the periodic responsible-chunk audit, +3. the post-replication possession probe (ADR-0003), +4. the prune-confirmation audit. + +Lanes 2, 3 and 4 are digest-based and share one wire message pair. Lane 1 built +its own per-leaf commitment. The lanes did not derive their per-audit freshness +the same way, which meant a change to one left the others on a different +footing. Keeping several different answers to the same question is a maintenance +and review hazard independent of any single lane's strength. + +ADR-0002 is not amended by this decision. It records what was decided then; this +ADR records what replaces its round-2 proof shape and how the freshness +derivation is unified. + +## Decision Drivers + +- Egress reduction is the primary goal; the audit must stop moving whole chunks + to prove they exist. +- One freshness construction across all audit lanes, not one per lane. +- A mixed-version fleet must keep replicating during the auto-upgrade window. + Partitioning core replication to ship an audit change is too blunt. +- A version skew must never be scored as a peer's fault. + +## Considered Options + +1. Keep full-chunk round-2 responses and rely on frequency caps alone. +2. Return a verified slice for round 2, and leave the other lanes as they were. +3. Return a verified slice for round 2, unify the freshness derivation across all + four lanes, and give each changed message family its own protocol id. +4. As option 3, but advance the single shared replication protocol id instead of + introducing per-family ids. + +## Decision + +We will take option 3. + +**Round-2 proof shape.** A chunk's address is its BLAKE3 root, and BLAKE3 is +internally a Merkle tree over 1 KiB blocks, so the responder returns a verified +slice for an opened block rather than the chunk. Because the address is public, +authenticity alone would not show possession, so round 1 additionally commits a +per-leaf root over the same blocks derived from the fresh per-audit nonce, and +round 2 checks both chains over the same block bytes. The block to open is drawn +with fresh randomness after the round-1 commitment arrives. The auditor also +anchors the responder's claimed content length against the address rather than +trusting it. + +**Freshness derivation.** All four lanes derive their per-audit freshness the +same way: the challenge material (nonce, challenged peer, key) is used to derive +a BLAKE3 key under a versioned domain-separation context, and the content is +hashed under that key. Deriving a key rather than prefixing the challenge means +the whole proof depends on the fresh nonce rather than only its leading bytes. + +**Protocol families.** Core replication keeps its existing id. The subtree audit +rides its own id, and the three digest-based lanes (one shared message pair) +ride a second one. A symmetric guard drops any message whose family disagrees +with the id it arrived on. + +**Unanswered round 2.** Because round 2 names the blocks to open only after the +round-1 roots are committed, a responder learns the draw before it decides +whether to reply. An unanswered round 2 that follows a valid round-1 proof +therefore revokes the holder credit carried by the commitment under audit, +scoped to that commitment. It stays in the graced timeout lane and takes no +trust penalty: honest peers drop replies, and version-skewed peers never reach +this state because they cannot complete the new round 1. What it removes is the +option of holding credit without ever completing a possession check. + +**Round-1 work bounds.** A single round-1 challenge selects a fixed-depth block +of the commitment tree sized to about the square root of the key count, so the +nonce cannot steer a responder into reading its whole store. Round 1 has its own +small admission pool, a per-peer responder cooldown, and its hashing runs off the +async executor. Round 2 is bound to a single-use session opened by a matching +round 1 from the same peer. + +Those bounds are all keyed by peer identity or by concurrency, and neither +bounds sustained work: a concurrency cap can be refilled the moment a slot frees, +and a per-peer cooldown can be refilled by presenting a different peer id. Round +1 is therefore also charged against a responder-wide budget over the chunk bytes +it reads and hashes, refilled at a fixed rate and keyed by nothing at all. The +budget carries debt, so an expensive proof is admitted proportionally less often +than a cheap one, and the sustained rate settles at the refill rate whatever the +caller's identity. It is sized above honest demand at the largest commitment the +system allows, so it costs honest auditing nothing. + +**Pre-admission bounds.** Family, session and admission checks all read fields of +a decoded message, so none of them can run before decoding. The audit families +therefore take their own wire-size ceiling, checked against the encoded length +before any parsing, sized against the largest legitimate audit body — the round-1 +proof at the commitment key-count cap. This keeps the work an unknown peer can +demand before admission proportional to what an audit can legitimately carry +rather than to the core replication ceiling, which is sized for hint batches that +no audit body contains. + +## Consequences + +### Positive + +- Round-2 responses fall from megabytes to kilobytes. A 990-node run measured + 14.49 KB over 69,903 responses against 6.19 MB on a simultaneous same-network + control cohort — a reduction of about 427x in decimal units. +- The cost per audit event is bounded regardless of audit rate, which frequency + caps cannot achieve on their own. +- One freshness construction across all four lanes instead of one per lane. +- Core replication is not partitioned by an audit change. A mixed-version run + measured 736/736 and 135/135 transfers with no failures, and the older cohort + continued to accept and store paid chunks throughout. +- A version skew on an audit lane produces no answer rather than a wrong answer, + so it lands in the timeout lane instead of the confirmed-failure lane. The + subtree lane already graces timeouts. The three digest lanes did not — they + reported a confirmed-failure trust event on a timeout — so a temporary rollout + gate (`GRACE_POSSESSION_AUDIT_TIMEOUTS`) graces them for the upgrade window. + See the trade-offs below: that gate must be removed in a follow-up. + +### Negative / Trade-offs + +- Round 2 now proves a sampled block rather than a whole chunk, so per-response + coverage is narrower and the guarantee rests on repeated sampling over time. + Stated precisely, because it is the load-bearing trade-off: the auditor holds + none of the audited bytes, so it cannot distinguish a correct nonced root from + a responder-chosen one. A peer retaining a fraction `p` of a chunk's blocks can + commit a root with genuine leaves for what it kept and arbitrary leaves for + what it dropped, and passes whenever every draw lands on a kept block — + roughly `p^leaves` per audit over the 3..=5 sampled leaves. The mandatory + final-block opening anchors the claimed length but adds no detection, since a + partial holder keeps the final block. The full-byte round 2 this replaces + caught any missing byte of a sampled chunk with certainty. + + The exchange is deliberate, and the comparison should be stated in both + directions. Against a node under-storing **in bulk** — the realistic case, a + node dropping data to save disk — detection is close to what the full-byte + audit gave, at roughly 1/430 of the egress; the fleet run caught a + 256 MB in-place corruption on the first audit that reached the node, by three + independent auditors, against zero false positives across ~43,000 audits in + the preceding 5.5 hours. Against a **fine-grained** partial deleter, one + shaving a little off every chunk, it is strictly weaker per audit than serving + every byte. The compensating lever is audit *frequency*, which is what the cost + reduction buys: the old shape made frequent auditing unaffordable. If a future + threat model needs sharper per-audit detection, the knob is openings per leaf, + at linear egress cost. +- Three protocol ids to reason about instead of one. +- A temporary rollout gate (`GRACE_POSSESSION_AUDIT_TIMEOUTS`) suppresses the + timeout penalty on the three digest lanes so a version skew cannot punish an + honest peer at confirmed-failure weight for the upgrade window. While it is + set, a peer that silently drops audit challenges is under-penalised: it still + takes the upstream unit transport decrement, but not the audit-severity one. + The guarded branches stay compiled rather than commented out so they cannot rot + while disabled. + + **Removal is owed, on an objective criterion.** Owner: Anselme (@grumbach), + the decision owner for this ADR. The gate exists only to cover peers that have + not yet upgraded, so the criterion is a fleet-version one: once the released + version carrying this change accounts for at least 99% of nodes seen in the + routing table over a seven-day window, and no supported release still on the + old audit protocol remains in the field, set the constant to `false`, delete + it, and inline the guarded branches. Until that is done, the timeout penalty on + those three lanes is suppressed for everyone, not only for old peers, so this + is a live weakening and not merely a compatibility shim. It should be tracked + as an open follow-up rather than closed with this change. +- Cross-version audits pause during the auto-upgrade window. Unanswered requests + still register a unit trust decrement upstream, which decays back to neutral; + the possession lanes probe more often than the subtree lane, so their dip is + larger. This is milder than either a stream of confirmed failures or a fleet + wide replication partition, but it is not zero. +- The responsible-chunk audit now returns no verdict when it cannot check a + challenged key against its own copy, so some ticks that previously recorded a + pass now record nothing. + +### Neutral / Operational + +- The wire counter for the round-2 response keeps its original name so the + measurement is directly comparable across versions. +- Each protocol family can be versioned independently from here on. + +## Validation + +- Focused unit tests pin the proof construction, the freshness derivation, the + canonical proof geometry, and the length anchoring. +- Attack proof-of-concept suites cover fabrication, substitution, replay, and + data-deletion behaviour against the responder. +- A real-QUIC multi-node end-to-end test covers an honest node passing, a + data-deleting node being caught, and repeated audits producing no false + positives. +- Routing tests assert that each message family is accepted only on its own + protocol id, in both directions. +- Unit tests pin the round-2 credit boundary (which outcomes revoke the pinned + commitment's credit, and that the revocation is scoped to that commitment), + that the round-1 work budget is not refilled by a fresh peer id and recovers at + its configured rate, and that the worst legitimate round-1 proof fits the + audit-family wire ceiling. +- Fleet evidence: response size against a same-network control cohort, + detection parity, false-positive rate, mixed-version transfer success, + round-1 CPU and memory against baseline, and node stability. +- Re-validation trigger: any change to the freshness derivation, the sampling + rule, the round-1 selection bound, or the set of protocol families. + +## Notes for AI-assisted work + +AI tools may help draft this ADR, but **must not mark it Accepted without human +review**. Accepted ADRs are immutable: create a new superseding ADR rather than +editing an Accepted ADR. diff --git a/src/replication/audit.rs b/src/replication/audit.rs index 64b192de..e19e0be9 100644 --- a/src/replication/audit.rs +++ b/src/replication/audit.rs @@ -11,7 +11,7 @@ use rand::seq::SliceRandom; use rand::Rng; use crate::ant_protocol::XorName; -use crate::replication::config::{ReplicationConfig, REPLICATION_PROTOCOL_ID}; +use crate::replication::config::{ReplicationConfig, POSSESSION_AUDIT_PROTOCOL_ID}; use crate::replication::protocol::{ compute_audit_digest, AuditChallenge, AuditResponse, ReplicationMessage, ReplicationMessageBody, ABSENT_KEY_DIGEST, @@ -48,7 +48,11 @@ pub enum AuditTickResult { /// The peer claiming bootstrap status. peer: PeerId, }, - /// No eligible peers for audit this tick. + /// No eligible peers for audit this tick, OR the auditor could not run a + /// meaningful check (the responsible-chunk audit found a challenged key's + /// own local copy unreadable). Nothing was proven either way: no holder + /// credit, no trust event, no penalty. The audit simply did not happen this + /// tick, exactly like having no peer to audit. Idle, /// Audit skipped (not enough local keys). InsufficientKeys, @@ -244,7 +248,7 @@ pub async fn audit_tick_with_repair_proofs( let response = match p2p_node .send_request( &challenged_peer, - REPLICATION_PROTOCOL_ID, + POSSESSION_AUDIT_PROTOCOL_ID, encoded, audit_timeout, ) @@ -535,62 +539,154 @@ async fn verify_digests( } let challenged_peer_bytes = challenged_peer.as_bytes(); - let mut failed_keys = Vec::new(); + let mut checks = Vec::with_capacity(keys.len()); for (i, key) in keys.iter().enumerate() { - let received_digest = &digests[i]; - - // Check for absent sentinel. - if *received_digest == ABSENT_KEY_DIGEST { - failed_keys.push(AuditKeyFailure::absent(*key)); - continue; - } - - // Recompute expected digest from local copy. + // Read the auditor's own copy (logging the gone-vs-error distinction), + // then classify the peer's digest against it. let local_bytes = match storage.get_raw(key).await { - Ok(Some(bytes)) => bytes, + Ok(Some(bytes)) => Some(bytes), Ok(None) => { // We should hold this key (we sampled it), but it's gone. warn!( "Audit: local key {} disappeared during audit", hex::encode(key) ); - continue; + None } Err(e) => { warn!("Audit: failed to read local key {}: {e}", hex::encode(key)); - continue; + None } }; + checks.push(classify_local_digest( + &digests[i], + key, + nonce, + challenged_peer_bytes, + local_bytes.as_deref(), + )); + } - let expected = compute_audit_digest(nonce, challenged_peer_bytes, key, &local_bytes); - if *received_digest != expected { - failed_keys.push(AuditKeyFailure::digest_mismatch(*key)); + match aggregate_digest_checks(challenged_peer, checks, keys.len()) { + // Nothing provably failed: Passed (everything verified) or Idle (a skip). + Ok(verdict) => verdict, + // Step 9: Responsibility confirmation for failed keys. + Err(failed_keys) => { + handle_classified_audit_failure( + challenged_peer, + challenge_id, + &failed_keys, + AuditFailureReason::DigestMismatch, + keys.len(), + p2p_node, + config, + ) + .await } } +} + +/// Outcome of checking one challenged key's digest against the auditor's own copy. +#[cfg_attr(test, derive(Debug))] +enum DigestCheck { + /// The peer's digest matches the digest recomputed from local bytes. + Verified, + /// The auditor's own copy was unreadable, so the peer's digest for this key + /// could not be checked. Not the peer's fault, and not a pass either. + Skipped, + /// A provable failure: the absent sentinel, or a digest that disagrees with + /// the local copy. + Failed(AuditKeyFailure), +} + +/// Classify one challenged key's response against the auditor's local copy. +/// +/// `local_bytes` is `None` when the auditor's own copy is unreadable (gone or a +/// read error). The peer's digest is then unverifiable, so the key is +/// [`DigestCheck::Skipped`] and never silently treated as a pass. The absent +/// sentinel is the peer's own admission and stays a provable failure even then. +fn classify_local_digest( + received_digest: &[u8; 32], + key: &XorName, + nonce: &[u8; 32], + challenged_peer_bytes: &[u8; 32], + local_bytes: Option<&[u8]>, +) -> DigestCheck { + if *received_digest == ABSENT_KEY_DIGEST { + return DigestCheck::Failed(AuditKeyFailure::absent(*key)); + } + let Some(bytes) = local_bytes else { + return DigestCheck::Skipped; + }; + if *received_digest == compute_audit_digest(nonce, challenged_peer_bytes, key, bytes) { + DigestCheck::Verified + } else { + DigestCheck::Failed(AuditKeyFailure::digest_mismatch(*key)) + } +} +/// Fold per-key [`DigestCheck`] outcomes into the responsible-audit result. +/// +/// `Ok` when no key provably failed: `Passed` iff every key verified, else +/// `Idle` (see [`no_failure_verdict`]). `Err(failed_keys)` when at least one key +/// failed, for the caller's responsibility-confirmation path. +/// +/// Pure, so the security-relevant fold — a skipped key must count as `skipped`, +/// never as `verified` — is unit-testable without a live `P2PNode`. +fn aggregate_digest_checks( + challenged_peer: &PeerId, + checks: Vec, + challenged: usize, +) -> Result> { + let mut verified = 0usize; + let mut skipped = 0usize; + let mut failed_keys = Vec::new(); + for check in checks { + match check { + DigestCheck::Verified => verified += 1, + DigestCheck::Skipped => skipped += 1, + DigestCheck::Failed(failure) => failed_keys.push(failure), + } + } if failed_keys.is_empty() { - info!( - "Audit: peer {challenged_peer} passed (all {} keys verified)", - keys.len() - ); - return AuditTickResult::Passed { - challenged_peer: *challenged_peer, - keys_checked: keys.len(), - }; + Ok(no_failure_verdict( + challenged_peer, + challenged, + verified, + skipped, + )) + } else { + Err(failed_keys) } +} - // Step 9: Responsibility confirmation for failed keys. - handle_classified_audit_failure( - challenged_peer, - challenge_id, - &failed_keys, - AuditFailureReason::DigestMismatch, - keys.len(), - p2p_node, - config, - ) - .await +/// Decide the verdict when no key provably failed. +/// +/// A `Passed` here means "the whole response was verified against local bytes". +/// If any key was skipped because the auditor's own copy was unreadable, the +/// response was not fully verified, so the tick yields no verdict (`Idle`: no +/// success trust event, no penalty) rather than a pass that reflects only the +/// keys that happened to be readable. `verified == 0` is the same no-verdict +/// case. +fn no_failure_verdict( + challenged_peer: &PeerId, + challenged: usize, + verified: usize, + skipped: usize, +) -> AuditTickResult { + if skipped > 0 || verified == 0 { + warn!( + "Audit: {skipped} of {challenged} challenged keys for {challenged_peer} were \ + locally unverifiable; audit did not conclude this tick (Idle, no verdict)" + ); + return AuditTickResult::Idle; + } + info!("Audit: peer {challenged_peer} passed (all {verified} keys verified)"); + AuditTickResult::Passed { + challenged_peer: *challenged_peer, + keys_checked: verified, + } } // --------------------------------------------------------------------------- @@ -876,6 +972,97 @@ mod tests { PeerId::from_bytes(bytes) } + // -- classify_local_digest / no-verdict fold ------------------------------ + + #[test] + fn classify_local_digest_verified_on_matching_local_bytes() { + let (key, nonce, peer, bytes) = ([1u8; 32], [2u8; 32], [3u8; 32], b"chunk-bytes"); + let good = compute_audit_digest(&nonce, &peer, &key, bytes); + assert!(matches!( + classify_local_digest(&good, &key, &nonce, &peer, Some(bytes)), + DigestCheck::Verified + )); + } + + #[test] + fn classify_local_digest_skipped_when_local_copy_unreadable() { + // An unreadable own copy means the peer's digest cannot be checked, so + // the outcome is Skipped (never Verified) whatever the digest value is. + let (key, nonce, peer) = ([1u8; 32], [2u8; 32], [3u8; 32]); + assert!(matches!( + classify_local_digest(&[0xAB; 32], &key, &nonce, &peer, None), + DigestCheck::Skipped + )); + } + + #[test] + fn classify_local_digest_failed_on_digest_mismatch() { + let (key, nonce, peer, bytes) = ([1u8; 32], [2u8; 32], [3u8; 32], b"chunk-bytes"); + assert!(matches!( + classify_local_digest(&[0xCD; 32], &key, &nonce, &peer, Some(bytes)), + DigestCheck::Failed(_) + )); + } + + #[test] + fn classify_local_digest_absent_sentinel_fails_even_when_local_unreadable() { + // Precedence: the peer's own admission of absence stays a provable + // failure and must not be downgraded to Skipped just because the + // auditor's copy is also unreadable. + let (key, nonce, peer) = ([1u8; 32], [2u8; 32], [3u8; 32]); + match classify_local_digest(&ABSENT_KEY_DIGEST, &key, &nonce, &peer, None) { + DigestCheck::Failed(f) => assert_eq!(f.kind, AuditKeyFailureKind::Absent), + other => panic!("absent sentinel must fail, got {other:?}"), + } + } + + #[test] + fn all_verified_yields_passed_with_the_verified_count() { + let peer = PeerId::from_bytes([9u8; 32]); + let checks = vec![DigestCheck::Verified, DigestCheck::Verified]; + match aggregate_digest_checks(&peer, checks, 2) { + Ok(AuditTickResult::Passed { keys_checked, .. }) => assert_eq!(keys_checked, 2), + other => panic!("expected Passed, got {other:?}"), + } + } + + #[test] + fn any_skipped_key_yields_idle_not_a_partial_pass() { + // The security-relevant fold: a response that was only partly checkable + // must not mint a pass reflecting just the readable keys. + let peer = PeerId::from_bytes([9u8; 32]); + let checks = vec![DigestCheck::Verified, DigestCheck::Skipped]; + match aggregate_digest_checks(&peer, checks, 2) { + Ok(AuditTickResult::Idle) => {} + other => panic!("expected Idle, got {other:?}"), + } + } + + #[test] + fn all_skipped_yields_idle_not_a_vacuous_pass() { + let peer = PeerId::from_bytes([9u8; 32]); + let checks = vec![DigestCheck::Skipped, DigestCheck::Skipped]; + match aggregate_digest_checks(&peer, checks, 2) { + Ok(AuditTickResult::Idle) => {} + other => panic!("expected Idle, got {other:?}"), + } + } + + #[test] + fn a_failed_key_still_routes_to_the_failure_path_despite_skips() { + // A provable failure outranks a skip: the caller must still run + // responsibility confirmation rather than fall into the Idle lane. + let peer = PeerId::from_bytes([9u8; 32]); + let checks = vec![ + DigestCheck::Skipped, + DigestCheck::Failed(AuditKeyFailure::digest_mismatch([7u8; 32])), + ]; + match aggregate_digest_checks(&peer, checks, 2) { + Err(failed) => assert_eq!(failed.len(), 1), + Ok(other) => panic!("expected the failure path, got {other:?}"), + } + } + // -- handle_audit_challenge: present keys --------------------------------- #[tokio::test] diff --git a/src/replication/commitment_state.rs b/src/replication/commitment_state.rs index 027edc9a..3d8e4c29 100644 --- a/src/replication/commitment_state.rs +++ b/src/replication/commitment_state.rs @@ -3,8 +3,9 @@ //! Phase 2b of the v12 storage-bound audit design. Builds, signs, and //! caches a [`StorageCommitment`] over the responder's currently-stored //! key set; serves audit lookups by `expected_commitment_hash`; retains -//! the previous commitment across one rotation so an audit pinned to it -//! does not false-fail at the rotation boundary (v5/v12 §4 retention). +//! recently-gossiped commitments for a bounded TTL window (with a slot +//! backstop) so an audit pinned to a still-answerable commitment does not +//! false-fail across rotations. //! //! Rotation strategy: //! @@ -341,7 +342,7 @@ pub(crate) const GOSSIP_ANSWERABILITY_TTL: Duration = Duration::from_secs(3 * 36 /// be slightly early. Adding this margin on reload guarantees an honest node /// never *under*-retains across a restart (it may over-retain by the margin, /// which is harmless — it only makes the responder answer a little longer, and a -/// data-deleter still fails the round-2 byte challenge). Sized well above the +/// data-deleter still fails the round-2 slice challenge). Sized well above the /// persist interval + gossip cadence, far below the TTL. const RESTART_STAMP_GRACE: Duration = Duration::from_secs(5 * 60); @@ -588,11 +589,11 @@ impl ResponderCommitmentState { /// Whether `key` is committed under any retained slot (the current /// commitment plus any still-in-window gossiped ones) — i.e. whether a peer - /// could still pin a recently gossiped root and demand this key's bytes in a - /// round-2 byte challenge. + /// could still pin a recently gossiped root and open this key's blocks in a + /// round-2 slice challenge. /// /// This is the SAME predicate the round-2 responder uses to decide a key is - /// "committed" (`handle_subtree_byte_challenge` calls `built.proof_for(key)` + /// "committed" (`handle_subtree_slice_challenge` calls `built.proof_for(key)` /// on the pinned slot, which is committed iff `contains_key`), folded over /// every retained slot. The pruner consults it before deleting an /// out-of-range key, so "the pruner will not delete it" and "the responder diff --git a/src/replication/config.rs b/src/replication/config.rs index bd5225e9..db044c39 100644 --- a/src/replication/config.rs +++ b/src/replication/config.rs @@ -14,7 +14,7 @@ use std::time::Duration; use rand::Rng; -use crate::ant_protocol::{CLOSE_GROUP_SIZE, MAX_CHUNK_SIZE}; +use crate::ant_protocol::CLOSE_GROUP_SIZE; // --------------------------------------------------------------------------- // Static constants (compile-time reference profile) @@ -131,19 +131,21 @@ pub const MAX_CONCURRENT_REPLICATION_SENDS: usize = 3; /// Maximum number of concurrent in-flight audit-responder tasks. /// -/// The responsible-chunk (audit #2), subtree (round 1), and byte (round 2) -/// challenge handlers are all spawned off the serial replication message loop so -/// their disk reads don't stall replication. This caps how many run at once +/// The LIGHT audit-responder handlers — responsible-chunk audits and subtree +/// slice (round 2) — are spawned off the serial replication message loop so their +/// disk reads don't stall replication. (The HEAVY subtree round 1 has its own +/// tighter pool, [`MAX_CONCURRENT_SUBTREE_ROUND1`].) This caps how many run at once /// across the engine, restoring backpressure: a peer flooding audit challenges -/// cannot fan out unbounded `get_raw` reads or multi-MiB byte serves. When the -/// cap is hit, the challenge is dropped and the caller's audit-specific timeout -/// policy applies. The cap must therefore stay high enough for honest audit -/// traffic while still throttling flooders. +/// cannot fan out unbounded `get_raw` reads. When the cap is hit, the challenge +/// is dropped and the caller's audit-specific timeout policy applies. The cap +/// must therefore stay high enough for honest audit traffic while still +/// throttling flooders. /// Sized to cover a handful of concurrent honest auditors (the per-peer /// gossip-audit cooldown is 30 min, so genuine concurrent audits are few) while -/// bounding the byte round's worst-case resident bytes -/// (`N × MAX_BYTE_CHALLENGE_KEYS × MAX_CHUNK_SIZE`). -pub const MAX_CONCURRENT_AUDIT_RESPONSES: usize = 32; +/// bounding the round-2 worst-case disk reads (each request reads at most +/// `BYTE_SPOTCHECK_MAX` distinct chunks — the openings are coalesced and the +/// distinct-key count is capped — to build its slice proofs). +pub const MAX_CONCURRENT_AUDIT_RESPONSES: usize = 16; /// Maximum concurrent in-flight audit-responder tasks from any SINGLE peer. /// @@ -157,6 +159,75 @@ pub const MAX_CONCURRENT_AUDIT_RESPONSES: usize = 32; /// headroom beyond the legitimate round-1 + round-2 overlap. pub const MAX_AUDIT_RESPONSES_PER_PEER: u32 = 4; +/// Dedicated global concurrency cap for the HEAVY subtree-audit round 1. +/// +/// Round 1 hashes every leaf of the selected `sqrt(key_count)` subtree (up to +/// ~1000 chunks × `MAX_CHUNK_SIZE` for a maximal commitment), far heavier than a +/// responsible-chunk or slice (round-2) response. Giving it its own tiny pool — +/// rather than sharing [`MAX_CONCURRENT_AUDIT_RESPONSES`] — keeps a burst of +/// round-1 proofs from starving the light audits, and bounds concurrent +/// multi-gigabyte hashing to this many at once. Two allows overlap without +/// admitting many simultaneous full-subtree hashes; there is little benefit in +/// more concurrent large LMDB scans against one disk. +pub const MAX_CONCURRENT_SUBTREE_ROUND1: usize = 2; + +/// Per-peer concurrency cap for the heavy subtree-audit round 1. One in-flight +/// round-1 proof per source at a time (an honest auditor never needs more). +pub const MAX_SUBTREE_ROUND1_PER_PEER: u32 = 1; + +/// Per-peer responder-side cooldown between heavy subtree round-1 proofs. +/// +/// An honest auditor already self-limits to one gossip-triggered subtree audit +/// per peer per 30 min, so matching that as a responder-side floor costs honest +/// traffic nothing while bounding the sustained round-1 work a single identity +/// can extract (a concurrency cap alone lets a peer refill its slot forever). +pub const SUBTREE_ROUND1_RESPONDER_COOLDOWN: Duration = Duration::from_secs(30 * 60); + +/// Lifetime of a single-use round-1 → round-2 session. +/// +/// A round-2 slice challenge is only served if the same peer completed a matching +/// round 1 within this window. Long enough for the auditor to verify round 1 and +/// send round 2, far shorter than commitment retention; ephemeral, so a loss +/// across a restart just +/// drops that round to the (graced) timeout lane. +pub const SUBTREE_SESSION_TTL: Duration = Duration::from_secs(2 * 60); + +/// Capacity backstop on the live round-1 session map (bounds memory if many +/// peers open sessions; oldest are evicted past this). +pub const MAX_SUBTREE_SESSIONS: usize = 4 * MAX_CONCURRENT_SUBTREE_ROUND1 * 256; + +/// Sustained rate at which the responder-wide round-1 work budget refills, in +/// bytes of chunk content per second. +/// +/// [`MAX_CONCURRENT_SUBTREE_ROUND1`] bounds how many round-1 proofs run at once +/// and [`SUBTREE_ROUND1_RESPONDER_COOLDOWN`] bounds how often one peer id may +/// ask, but neither bounds sustained work: the cooldown is keyed by identity, so +/// a party holding several identities refills its allowance by rotating between +/// them and keeps the small pool permanently busy. This budget is keyed by +/// nothing at all — it is charged for the bytes read and hashed no matter who +/// asked — so identity count cannot buy more of it. +/// +/// Sized to clear honest demand even at the worst commitment size, since +/// starving honest audits would cost more than it saves. The 990-node run +/// served about 24 audits per node per hour. At the `MAX_COMMITMENT_KEY_COUNT` +/// cap one proof reads close to 4 GiB, so honest demand there is ~96 GiB/h, +/// against the 225 GiB/h this allows — and a flooder is held to about twice +/// honest load rather than to whatever the disk will bear. For the far more +/// common mid-sized commitment, where a proof reads a few hundred MiB, the +/// headroom is more than an order of magnitude. +/// +/// Signed because the budget it refills carries debt when a proof costs more +/// than is left. +pub const SUBTREE_ROUND1_WORK_REFILL_BYTES_PER_SEC: i64 = 64 * 1024 * 1024; + +/// Burst capacity of the round-1 work budget, in bytes of chunk content. +/// +/// Also its starting fill, so a freshly started node can serve audits at once. +/// Two maximal proofs' worth, matching [`MAX_CONCURRENT_SUBTREE_ROUND1`], so a +/// legitimate burst is never refused for arriving together — the budget limits +/// the sustained rate, which is what a concurrency cap cannot do. +pub const SUBTREE_ROUND1_WORK_BURST_BYTES: i64 = 8 * 1024 * 1024 * 1024; + /// Concurrent fetches cap, derived from hardware thread count. /// /// Uses `std::thread::available_parallelism()` so the node scales to the @@ -191,18 +262,15 @@ pub const AUDIT_TICK_INTERVAL_MAX: Duration = Duration::from_secs(AUDIT_TICK_INT /// for the round-1 proof, whose payload is hashes (KB-scale). const AUDIT_RESPONSE_FLOOR_SECS: u64 = 4; -/// Floor on the round-2 BYTE-challenge deadline. -/// -/// Unlike round 1 (KB of hashes), the byte challenge ships up to -/// `MAX_BYTE_CHALLENGE_KEYS` full chunks (2 × 4 MiB = 8 MiB) back over the -/// wire, so the envelope must also cover a cold QUIC handshake, the -/// multi-MiB upload back to the auditor, and a busy honest peer's disk read. -/// The round-1 4 s floor is still sized for a hashes-only reply; round 2 needs -/// a larger base for the §4 byte-serving envelope. 5 s matches the -/// cross-continent-RTT + handshake + 8 MiB transfer budget while keeping a relay -/// that must fetch the bytes over a residential link outside it (the scaled -/// term adds the per-byte estimate on top). Mirrors main's more generous -/// byte-round base. +/// Floor on the round-2 SLICE-challenge deadline. +/// +/// The round-2 reply is only a few KB per opening (a 1 KiB block plus two short +/// hash chains), but an honest responder still reads each opened chunk's full +/// bytes from disk to build its Bao slice and nonced opening, so the floor must +/// cover a cold QUIC handshake plus a busy honest peer's full-chunk disk read. +/// The round-1 4 s floor is sized for a hashes-only reply; round 2 keeps a +/// slightly larger 5 s base for the disk-read envelope, with the per-byte scaled +/// term (one full chunk per opening) added on top. const BYTE_AUDIT_RESPONSE_FLOOR_SECS: u64 = 5; /// Conservative honest-responder read throughput, in bytes per second. @@ -245,53 +313,121 @@ const PRUNE_HYSTERESIS_DURATION_SECS: u64 = 3 * 24 * 60 * 60; // 3 days /// Minimum continuous out-of-range duration before pruning a key. pub const PRUNE_HYSTERESIS_DURATION: Duration = Duration::from_secs(PRUNE_HYSTERESIS_DURATION_SECS); -/// Protocol identifier for replication operations. -/// -/// Bumped to `v2` for the v12 storage-bound audit. That change extends the -/// wire types (`NeighborSyncRequest`/`Response` carry an optional trailing -/// `StorageCommitment`, and the gossip-triggered storage-commitment audit adds -/// the `SubtreeAuditChallenge`/`SubtreeAuditResponse` and `SubtreeByteChallenge`/ -/// `SubtreeByteResponse` messages). The bump is for SEMANTIC interop, not -/// decode failure: postcard tolerates the appended optional field (an old -/// decoder reads the fields it knows and ignores the trailer — pinned by the -/// `old_decoder_tolerates_new_neighbor_sync_*` tests in `protocol.rs`), but -/// tolerating bytes is not interoperating. A v1 node cannot decode the NEW -/// message variants at all (unknown enum discriminant) and never acts on a -/// piggybacked commitment, so mixed-version replication would half-function — -/// audit challenges unanswered, commitments silently dropped — and a v2 node -/// could read that silence as misbehaviour. Rather than reason about each -/// such case, we route v12 replication on a distinct protocol id: a node only -/// delivers messages whose topic matches its own id (see the topic check in -/// `mod.rs`), so v1 and v2 nodes simply do not exchange replication traffic -/// during a mixed-version window. This is the rollout-safe behaviour: no -/// half-interpreted exchange, no spurious eviction. Replication between -/// matched-version peers is unaffected. (DHT routing/lookups are a separate -/// protocol and continue to span both versions.) +/// Protocol identifier for core replication operations (fresh replication, +/// neighbour sync, verification, fetch, repair, commitment fetch). +/// +/// Kept at `v2`: none of these messages changed on the wire in the V2-685 slice +/// audit, so v2 and v3 nodes interoperate on all of them. The two message +/// families whose *semantics* changed each ride their own id +/// ([`SUBTREE_AUDIT_PROTOCOL_ID`], [`POSSESSION_AUDIT_PROTOCOL_ID`]) so a version +/// bump in either cannot partition core replication. A node filters inbound +/// messages by exact topic match (see the dispatch in `mod.rs`). pub const REPLICATION_PROTOCOL_ID: &str = "autonomi.ant.replication.v2"; +/// Protocol identifier for the digest-based possession audits. +/// +/// Carries the `AuditChallenge`/`AuditResponse` pair shared by the +/// responsible-chunk audit, the post-replication possession probe, and the +/// prune-confirmation audit. +/// +/// These messages keep their wire *shape*, but the digest they carry is now the +/// domain-separated keyed construction +/// ([`crate::replication::protocol::compute_audit_digest`]) rather than the +/// earlier flat prefix hash, so a peer on the old construction and a peer on the +/// new one compute different digests for the same bytes. Left on the shared core +/// id, that difference would read as a *confirmed* `DigestMismatch` on every +/// cross-version exchange, which carries [`AUDIT_FAILURE_TRUST_WEIGHT`] and runs +/// the responsibility-confirmation path — a false-positive penalty on honest +/// peers for the whole upgrade window. +/// +/// Routing them on their own id converts that into a benign pause: a +/// cross-version challenge is simply not answered, so it lands in the graced +/// timeout lane instead of the confirmed-failure lane. The rollout effect is the +/// same bounded one described on [`SUBTREE_AUDIT_PROTOCOL_ID`]: `saorsa-core`'s +/// `send_request` still records a unit trust failure per unanswered request, and +/// these lanes probe more often than the 30-minute subtree cooldown, so the dip +/// is larger than the subtree lane's while still decaying back to neutral. That +/// is strictly milder than a stream of weight-5 confirmed failures, and milder +/// than bumping the shared id, which would also break sync, quorum, fetch, +/// repair and commitment fetch. +/// +/// `v2` here is the digest generation, not the message shape. +pub const POSSESSION_AUDIT_PROTOCOL_ID: &str = "autonomi.ant.replication.possession-audit.v2"; + +/// Protocol identifier for the subtree storage-commitment audit (ADR-0002 / +/// V2-685), both rounds: `SubtreeAuditChallenge`/`Response` (round 1) and +/// `SubtreeSliceChallenge`/`Response` (round 2). +/// +/// These are the only replication messages whose wire format changed for the +/// slice audit (round 1's `SubtreeLeaf` now carries `content_len` + `nonced_root` +/// instead of a flat `nonced_hash`; round 2 replaced full-byte responses with Bao +/// verified slices). Routing them on their own id — instead of bumping the whole +/// [`REPLICATION_PROTOCOL_ID`] — means a mixed-version fleet keeps doing fresh +/// replication, neighbour sync, fetch and repair across versions; only +/// cross-version subtree *audits* pause during the ~24 h auto-upgrade window. +/// +/// Rollout effect is bounded, not zero: `saorsa-core`'s `send_request` records a +/// unit trust failure on any unanswered request (before ant-node's graced-timeout +/// policy), so a v3 auditor's subtree challenge to a still-v2 peer (and the +/// reverse, where a v2 subtree audit is dropped by the id/body guard) each ding +/// that peer's EMA trust once per 30-minute audit cooldown. Trust decays back to +/// neutral (a worst-case dip recovers above the 0.35 routing-swap threshold in +/// ~1 online day, and a successful audit after upgrade adds a unit success), and +/// crossing 0.35 only makes a peer replaceable in a full bucket — it does not +/// delete data or ban the peer. This is strictly milder than bumping the shared +/// id, which would fail every cross-version request path (sync/quorum/prune/ +/// possession/repair/commitment-fetch) with no per-peer limiter. A truly +/// zero-penalty rollout needs an upstream `send_request` that does not +/// auto-report trust; tracked as a saorsa-core follow-up. +pub const SUBTREE_AUDIT_PROTOCOL_ID: &str = "autonomi.ant.replication.subtree-audit.v1"; + /// 10 MiB — maximum replication wire message size (accommodates hint batches). const REPLICATION_MESSAGE_SIZE_MIB: usize = 10; /// Maximum replication wire message size. pub const MAX_REPLICATION_MESSAGE_SIZE: usize = REPLICATION_MESSAGE_SIZE_MIB * 1024 * 1024; -/// Headroom reserved for the envelope (enum tags, ids, length prefixes) when -/// sizing a round-2 byte-challenge batch against the wire cap. -const BYTE_CHALLENGE_RESPONSE_HEADROOM: usize = 64 * 1024; +/// Maximum wire size for a message on either audit protocol family. +/// +/// The 10 MiB core ceiling is sized for hint batches, which no audit body +/// carries. Applying it to the audit families would let a peer make this node +/// allocate and decode megabytes of attacker-shaped collection before any +/// family, session or admission check has run — those checks all read fields of +/// the decoded body, so they cannot come first. Checking the encoded length +/// against a family-appropriate ceiling can, because it needs nothing but the +/// byte count. +/// +/// Sized against the largest legitimate audit body, the round-1 +/// `SubtreeAuditResponse::Proof`: at the `MAX_COMMITMENT_KEY_COUNT` cap a +/// subtree is at most 1,024 leaves of ~100 bytes each, plus the sibling cut +/// hashes and the signed commitment, so ~110 KiB. Every other audit body — +/// both challenges, the round-2 response — is far smaller. This leaves roughly +/// 4x headroom over the worst legitimate case while cutting the pre-admission +/// allocation ceiling by a factor of 20. +pub const MAX_AUDIT_MESSAGE_SIZE: usize = 512 * 1024; +const _: () = assert!( + MAX_AUDIT_MESSAGE_SIZE < MAX_REPLICATION_MESSAGE_SIZE, + "the audit-family ceiling must be tighter than the core one, or it is pointless" +); -/// Maximum keys per round-2 [`SubtreeByteChallenge`] (per-batch cap). +/// Maximum block openings per round-2 [`SubtreeSliceChallenge`]. /// -/// Sized so the WORST-CASE response (every requested chunk at -/// `MAX_CHUNK_SIZE`) still encodes under [`MAX_REPLICATION_MESSAGE_SIZE`]. -/// The auditor splits its spot-check sample into batches of this size (one -/// challenge per batch, same nonce/pin); the responder rejects any single -/// challenge requesting more. +/// Each opening is a Bao verified slice (a 1 KiB block plus O(log n) BLAKE3 +/// parent hashes) plus a nonced block-tree sibling chain — a few KB, so even +/// this many openings encode far under [`MAX_REPLICATION_MESSAGE_SIZE`] with no +/// batching. The auditor draws up to *two* openings per sampled leaf — one +/// fresh-random block (possession) and one at the claimed final block (a length +/// pin: opening the final block forces Bao's EOF validation, which authenticates +/// the true content length and defeats a forged-short `content_len` that would +/// otherwise shrink the challenge space). With at most `BYTE_SPOTCHECK_MAX` +/// leaves that is `2 × BYTE_SPOTCHECK_MAX` openings; this cap sits just above +/// that, and the responder rejects any challenge requesting more (a forged- +/// auditor guard: each opening forces one full chunk read to build its proof). /// -/// [`SubtreeByteChallenge`]: crate::replication::protocol::SubtreeByteChallenge -pub const MAX_BYTE_CHALLENGE_KEYS: usize = - (MAX_REPLICATION_MESSAGE_SIZE - BYTE_CHALLENGE_RESPONSE_HEADROOM) / MAX_CHUNK_SIZE; +/// [`SubtreeSliceChallenge`]: crate::replication::protocol::SubtreeSliceChallenge +pub const MAX_SLICE_OPENINGS: usize = 10; const _: () = assert!( - MAX_BYTE_CHALLENGE_KEYS >= 1, - "wire cap must fit at least one max-size chunk per byte-challenge response" + MAX_SLICE_OPENINGS >= 1, + "at least one block opening must be allowed per slice challenge" ); /// Rollout gate for ADR-0004 quote-arithmetic enforcement. @@ -421,6 +557,52 @@ pub const PENDING_VERIFY_MAX_AGE: Duration = Duration::from_secs(PENDING_VERIFY_ /// Trust event weight for confirmed audit failures. pub const AUDIT_FAILURE_TRUST_WEIGHT: f64 = 5.0; +/// ROLLOUT GATE (V2-685) — **temporary. Must be turned off in a follow-up.** +/// +/// # Why this exists +/// +/// The three digest-based audit lanes — the responsible-chunk audit, the +/// post-replication possession probe, and prune confirmation — moved onto +/// [`POSSESSION_AUDIT_PROTOCOL_ID`] because their digest construction changed. +/// During the auto-upgrade window a v3 auditor's challenge to a still-v2 peer is +/// therefore never answered, and an unanswered challenge times out. +/// +/// Those three lanes report an `ApplicationFailure` carrying +/// [`AUDIT_FAILURE_TRUST_WEIGHT`] +/// on a timeout — weight 5, the confirmed-failure weight. Left alone, every +/// honest peer on the other side of the version boundary would be penalised at +/// confirmed-failure severity, repeatedly, for the entire window, for a skew +/// that is not its fault. (The subtree lane already graces its timeouts and is +/// unaffected; see `handle_subtree_failed_audit`.) +/// +/// # What it does while `true` +/// +/// The three lanes grace a timeout exactly as the subtree lane does: no +/// application trust event and no holder-credit revocation. This is not a free +/// pass — `saorsa-core`'s `send_request` still records its own unit transport +/// failure for an unanswered request, so a peer that never answers still drifts +/// down, just at weight 1 instead of 5. +/// +/// The grace covers *silence only*. A peer that answers is being judged on what +/// it said, so every confirmed outcome — digest mismatch, absent key, malformed +/// reply, explicit rejection — is penalised throughout the window exactly as +/// before. This is deliberately not an amnesty for cheating; it is only a +/// refusal to read "did not reply" as "lost the data". +/// +/// # FOLLOW-UP — required, do not ship this permanently +/// +/// Once the fleet has upgraded past the possession-audit protocol move, set this +/// to `false`, then delete the constant and inline the guarded branches so the +/// timeout penalty is restored unconditionally. Until that lands, a peer that +/// silently drops audit challenges is under-penalised. Tracked on V2-685. +/// +/// The guarded code is gated by this constant rather than commented out on +/// purpose: it stays compiled, type-checked, linted and covered by tests while +/// disabled, so it cannot rot before it is switched back on. +/// +/// [`POSSESSION_AUDIT_PROTOCOL_ID`]: crate::replication::config::POSSESSION_AUDIT_PROTOCOL_ID +pub const GRACE_POSSESSION_AUDIT_TIMEOUTS: bool = true; + /// Probability of launching a subtree audit when a peer's *changed* commitment /// is ingested via gossip (ADR-0002). Keeps audits occasional surprise exams. pub const AUDIT_ON_GOSSIP_PROBABILITY: f64 = 0.2; @@ -618,6 +800,10 @@ pub struct ReplicationConfig { /// Upper bound of the possession-check delay window (ADR-0003). Defaults /// to [`POSSESSION_CHECK_DELAY_MAX`]. pub possession_check_delay_max: Duration, + /// Per-peer responder-side cooldown between heavy subtree round-1 proofs. + /// Defaults to [`SUBTREE_ROUND1_RESPONDER_COOLDOWN`]; tests set + /// it low so rapid back-to-back audits of one holder are not rate-dropped. + pub subtree_round1_responder_cooldown: Duration, } impl Default for ReplicationConfig { @@ -645,6 +831,7 @@ impl Default for ReplicationConfig { bootstrap_complete_timeout_secs: BOOTSTRAP_COMPLETE_TIMEOUT_SECS, possession_check_delay_min: POSSESSION_CHECK_DELAY_MIN, possession_check_delay_max: POSSESSION_CHECK_DELAY_MAX, + subtree_round1_responder_cooldown: SUBTREE_ROUND1_RESPONDER_COOLDOWN, } } } @@ -824,12 +1011,11 @@ impl ReplicationConfig { // an honest HDD-backed peer at sqrt(N)=10 stored chunks could // miss the budget under load. let multiplied = total_bytes.saturating_mul(self.audit_response_honest_multiplier); - // Resolve the scaled term in MILLISECONDS, not seconds: at the - // byte-round sizes (MAX_BYTE_CHALLENGE_KEYS = 2 → 8 MiB) the per-second - // quotient `multiplied / bps` integer-truncates to 0, leaving only the - // floor. The §4 finding was that byte-serving challenges need the - // sub-second honest-read estimate (e.g. 8 MiB × 5 / 50 MB/s ≈ 840 ms) - // instead of dropping it. + // Resolve the scaled term in MILLISECONDS, not seconds: at small + // sample sizes (e.g. a 2-key challenge → 8 MiB) the per-second quotient + // `multiplied / bps` integer-truncates to 0, leaving only the floor. + // Small challenges still need the sub-second honest-read estimate + // (e.g. 8 MiB × 5 / 50 MB/s ≈ 840 ms) instead of dropping it. let scaled_ms = multiplied.saturating_mul(1000) / bps; // saturating_add avoids a panic if the floor plus the scaled term would // overflow `Duration::MAX`. @@ -837,20 +1023,26 @@ impl ReplicationConfig { .saturating_add(Duration::from_millis(scaled_ms)) } - /// Deadline for the round-2 BYTE challenge serving `challenged_key_count` - /// full chunks back to the auditor. + /// Deadline for the round-2 SLICE challenge opening `openings` blocks. /// - /// Same per-byte scaling as [`Self::audit_response_timeout`] (so a relay - /// that must fetch the bytes over a residential link still blows it), but on - /// a higher floor (`BYTE_AUDIT_RESPONSE_FLOOR_SECS`) because the reply - /// carries up to - /// `MAX_BYTE_CHALLENGE_KEYS × MAX_CHUNK_SIZE` of chunk data — handshake + - /// multi-MiB upload + a busy honest disk read do not fit the hashes-only - /// round-1 floor (the §4 finding). + /// The reply itself is only a few KB per opening (a 1 KiB block plus two + /// short hash chains), but an honest responder still reads each opened + /// chunk's full bytes from disk to build its Bao slice and nonced opening. So + /// the deadline is sized to that honest full-chunk disk read (the same + /// per-byte scaling as [`Self::audit_response_timeout`], one full chunk per + /// opening), on the `BYTE_AUDIT_RESPONSE_FLOOR_SECS` floor to absorb the + /// handshake and a busy disk. + /// + /// Unlike the old full-byte round 2, security here does NOT rest on this + /// deadline being too tight for a relay to fetch bytes — the possession + /// guarantee is the round-1 `nonced_root` commitment (uncomputable without + /// all the bytes, under a fresh nonce), so the deadline can be generous + /// without weakening the audit. It exists only to bound how long the auditor + /// waits for an honest reply. #[must_use] - pub fn byte_audit_response_timeout(&self, challenged_key_count: usize) -> Duration { + pub fn slice_audit_response_timeout(&self, openings: usize) -> Duration { let scaled = self - .audit_response_timeout(challenged_key_count) + .audit_response_timeout(openings) .saturating_sub(self.audit_response_floor); Duration::from_secs(BYTE_AUDIT_RESPONSE_FLOOR_SECS).saturating_add(scaled) } @@ -944,13 +1136,31 @@ mod tests { } #[test] - fn replication_protocol_id_is_v2() { - // The v12 storage-bound audit changes replication SEMANTICS. The - // protocol id MUST advance past v1 so v1 and v2 nodes never exchange - // replication traffic they can only half-interpret (rollout safety — - // see the const's doc). If this regresses to v1, mixed-version nodes - // would talk past each other and risk spurious penalties. + fn core_replication_id_stays_v2_changed_families_ride_their_own_ids() { + // Core replication stays on v2 (mixed-version fleets keep replicating). + // The two families whose semantics changed each ride their own id, so + // either can be bumped without partitioning core replication, and a + // message of one family never lands on another family's handler — see + // the id/body guard in `mod.rs`. assert_eq!(REPLICATION_PROTOCOL_ID, "autonomi.ant.replication.v2"); + assert_eq!( + SUBTREE_AUDIT_PROTOCOL_ID, + "autonomi.ant.replication.subtree-audit.v1" + ); + assert_eq!( + POSSESSION_AUDIT_PROTOCOL_ID, + "autonomi.ant.replication.possession-audit.v2" + ); + let ids = [ + REPLICATION_PROTOCOL_ID, + SUBTREE_AUDIT_PROTOCOL_ID, + POSSESSION_AUDIT_PROTOCOL_ID, + ]; + for (i, a) in ids.iter().enumerate() { + for b in ids.iter().skip(i + 1) { + assert_ne!(a, b, "protocol ids must be pairwise distinct"); + } + } } #[test] diff --git a/src/replication/mod.rs b/src/replication/mod.rs index 7b3d21fa..e7f152c4 100644 --- a/src/replication/mod.rs +++ b/src/replication/mod.rs @@ -29,6 +29,7 @@ pub mod pruning; pub mod quorum; pub mod recent_provers; pub mod scheduling; +pub mod slice; pub mod storage_commitment_audit; pub mod subtree; pub mod types; @@ -62,8 +63,11 @@ use crate::replication::commitment_state::{ PeerCommitmentRecord, PersistedRetention, ResponderCommitmentState, GOSSIP_ANSWERABILITY_TTL, }; use crate::replication::config::{ - max_parallel_fetch, storage_admission_width, ReplicationConfig, MAX_AUDIT_RESPONSES_PER_PEER, - MAX_CONCURRENT_AUDIT_RESPONSES, MAX_CONCURRENT_REPLICATION_SENDS, REPLICATION_PROTOCOL_ID, + max_parallel_fetch, storage_admission_width, ReplicationConfig, MAX_AUDIT_MESSAGE_SIZE, + MAX_AUDIT_RESPONSES_PER_PEER, MAX_CONCURRENT_AUDIT_RESPONSES, MAX_CONCURRENT_REPLICATION_SENDS, + MAX_CONCURRENT_SUBTREE_ROUND1, MAX_SUBTREE_ROUND1_PER_PEER, MAX_SUBTREE_SESSIONS, + POSSESSION_AUDIT_PROTOCOL_ID, REPLICATION_PROTOCOL_ID, SUBTREE_AUDIT_PROTOCOL_ID, + SUBTREE_ROUND1_WORK_BURST_BYTES, SUBTREE_ROUND1_WORK_REFILL_BYTES_PER_SEC, SUBTREE_SESSION_TTL, }; use crate::replication::paid_list::PaidList; use crate::replication::protocol::{ @@ -948,6 +952,64 @@ impl FirstAuditScheduler { /// Prefix used by saorsa-core's request-response mechanism. const RR_PREFIX: &str = "/rr/"; +/// Match an inbound topic against the replication protocol ids, in both the bare +/// gossip form and the `/rr/` request-response form. +/// +/// Returns the matched id (core [`REPLICATION_PROTOCOL_ID`], +/// [`SUBTREE_AUDIT_PROTOCOL_ID`] or [`POSSESSION_AUDIT_PROTOCOL_ID`]) and +/// whether it was the RR form. The matched id is carried into the handler so it +/// can enforce that each message family only arrives on its own id. +fn match_replication_protocol(topic: &str) -> Option<(&'static str, bool)> { + for id in [ + REPLICATION_PROTOCOL_ID, + SUBTREE_AUDIT_PROTOCOL_ID, + POSSESSION_AUDIT_PROTOCOL_ID, + ] { + if topic == id { + return Some((id, false)); + } + if let Some(rest) = topic.strip_prefix(RR_PREFIX) { + if rest == id { + return Some((id, true)); + } + } + } + None +} + +/// Whether a decoded body belongs on the protocol id it arrived on: +/// subtree-audit bodies on [`SUBTREE_AUDIT_PROTOCOL_ID`], possession-audit +/// bodies on [`POSSESSION_AUDIT_PROTOCOL_ID`], every other body on +/// [`REPLICATION_PROTOCOL_ID`]. +/// +/// The receive guard drops any mismatch (a cross-version or misrouted message); +/// sharing this one predicate between the guard and its regression test means a +/// change to the rule cannot pass the test unnoticed. +fn body_matches_protocol(body: &ReplicationMessageBody, protocol: &str) -> bool { + protocol == response_protocol_for(body) +} + +/// The protocol id a body belongs on — the single source of truth for BOTH +/// directions: the receive guard ([`body_matches_protocol`]) and the outbound +/// response selector in `send_replication_response_checked`. +/// +/// Sharing one function is what makes the family isolation symmetric. It also +/// removes a dependency on transport behaviour: saorsa-core correlates an RR +/// response by `(peer, msg_id)` rather than by protocol name, so a possession +/// `AuditResponse` sent on the core id would still reach an auditor waiting on +/// the possession id — but a bare (non-RR) response sent that way is dropped by +/// the peer's own guard, and correlation is a detail this layer should not rely +/// on. +fn response_protocol_for(body: &ReplicationMessageBody) -> &'static str { + if body.is_subtree_audit() { + SUBTREE_AUDIT_PROTOCOL_ID + } else if body.is_possession_audit() { + POSSESSION_AUDIT_PROTOCOL_ID + } else { + REPLICATION_PROTOCOL_ID + } +} + fn fresh_offer_payment_context() -> VerificationContext { VerificationContext::FreshReplication } @@ -1250,8 +1312,10 @@ pub struct ReplicationEngine { /// Limits concurrent outbound replication sends to prevent bandwidth /// saturation on home broadband connections. send_semaphore: Arc, - /// Bounds concurrent IN-FLIGHT audit-responder tasks (subtree round 1 + - /// byte round 2). Those are spawned off the serial message loop so disk + /// Bounds concurrent IN-FLIGHT LIGHT audit-responder tasks (responsible-chunk + /// audits + subtree slice round 2). The heavy subtree round 1 has its own + /// tighter pool ([`SubtreeRound1Limiter`]). Those are spawned off the serial + /// message loop so disk /// reads don't block replication; the semaphore restores a global /// backpressure ceiling so the node can't fan out unbounded `get_raw` reads /// / multi-MiB byte serves. @@ -1264,6 +1328,11 @@ pub struct ReplicationEngine { /// per-peer cap guarantees no single source can hold more than its share, /// so a flood self-throttles without denying service to everyone else. audit_responder_inflight: Arc>>, + /// Resource controls for the HEAVY subtree-audit round 1: its own + /// tight admission pool (so a burst of full-subtree hashing can't starve the + /// light audits), a per-peer rate cooldown, and single-use round-1 → round-2 + /// sessions binding a slice challenge to a matching round 1. + subtree_round1: SubtreeRound1Limiter, /// Receiver for fresh-write events from the chunk PUT handler. /// /// When present, `start()` spawns a drainer task that calls @@ -1360,6 +1429,7 @@ impl ReplicationEngine { send_semaphore: Arc::new(Semaphore::new(MAX_CONCURRENT_REPLICATION_SENDS)), audit_responder_semaphore: Arc::new(Semaphore::new(MAX_CONCURRENT_AUDIT_RESPONSES)), audit_responder_inflight: Arc::new(RwLock::new(HashMap::new())), + subtree_round1: SubtreeRound1Limiter::new(config.subtree_round1_responder_cooldown), fresh_write_rx: Some(fresh_write_rx), possession_check_tx, possession_check_rx: Some(possession_check_rx), @@ -1697,8 +1767,9 @@ impl ReplicationEngine { /// Drains scheduled possession-check events and, for each, waits a /// randomised 5-15 minute settle delay before probing every responsible /// peer for actual possession. A peer that cryptographically fails to prove - /// possession, including by timeout, is penalised at `AuditChallenge` - /// severity. + /// possession is penalised at `AuditChallenge` severity. A peer that simply + /// does not answer normally is too, but that is suspended for the + /// possession-audit protocol rollout — see `GRACE_POSSESSION_AUDIT_TIMEOUTS`. fn start_possession_check_scheduler(&mut self) { let Some(mut rx) = self.possession_check_rx.take() else { return; @@ -2023,6 +2094,7 @@ impl ReplicationEngine { let sync_state = Arc::clone(&self.sync_state); let audit_responder_semaphore = Arc::clone(&self.audit_responder_semaphore); let audit_responder_inflight = Arc::clone(&self.audit_responder_inflight); + let subtree_round1 = self.subtree_round1.clone(); // ADR-0002 gossip-audit trigger: bundled state so an ingested *changed* // commitment can spawn a probabilistic, cooldown-gated subtree audit. @@ -2047,24 +2119,28 @@ impl ReplicationEngine { data, .. } = event { - // Determine if this is a replication message - // and whether it arrived via the /rr/ request-response - // path (which wraps payloads in RequestResponseEnvelope). - let rr_info = if topic == REPLICATION_PROTOCOL_ID { - Some((data.clone(), None)) - } else if topic.starts_with(RR_PREFIX) - && &topic[RR_PREFIX.len()..] == REPLICATION_PROTOCOL_ID - { - P2PNode::parse_request_envelope(&data) - .filter(|(_, is_resp, _)| !is_resp) - .map(|(msg_id, _, payload)| (payload, Some(msg_id))) - } else { - None - }; - if let Some((payload, rr_message_id)) = rr_info { + // Determine which replication protocol this message + // rode (core or subtree-audit) and whether it arrived + // via the /rr/ request-response path (which wraps + // payloads in a RequestResponseEnvelope). + let rr_info = match_replication_protocol(&topic).and_then( + |(matched_id, is_rr)| { + if is_rr { + P2PNode::parse_request_envelope(&data) + .filter(|(_, is_resp, _)| !is_resp) + .map(|(msg_id, _, payload)| { + (matched_id, payload, Some(msg_id)) + }) + } else { + Some((matched_id, data.clone(), None)) + } + }, + ); + if let Some((matched_id, payload, rr_message_id)) = rr_info { match handle_replication_message( &source, &payload, + matched_id, &p2p, &storage, &paid_list, @@ -2083,6 +2159,7 @@ impl ReplicationEngine { &gossip_audit, &audit_responder_semaphore, &audit_responder_inflight, + &subtree_round1, rr_message_id.as_deref(), ).await { Ok(()) => {} @@ -3013,6 +3090,278 @@ impl Drop for AuditResponderGuard { } } +/// A live round-1 → round-2 subtree-audit session: proof of a matching round 1. +struct SubtreeSession { + commitment_hash: [u8; 32], + nonce: [u8; 32], + inserted: Instant, +} + +/// Responder-wide token bucket over the chunk bytes round-1 proof building may +/// read and hash, refilled continuously at +/// [`SUBTREE_ROUND1_WORK_REFILL_BYTES_PER_SEC`] up to +/// [`SUBTREE_ROUND1_WORK_BURST_BYTES`]. +/// +/// Deliberately keyed by nothing. The per-peer cooldown limits how often one +/// identity may ask, so it is refilled by acquiring more identities; this is +/// charged for work done regardless of who asked, so it is not. +/// +/// Charged after the fact, with the bytes the proof actually covered: the cost +/// of a request is not known until the pinned commitment has been resolved and +/// its subtree selected, both of which happen inside the handler. Admission +/// therefore asks only whether the balance is positive, and a proof that costs +/// more than is left drives the balance NEGATIVE rather than stopping at zero. +/// Carrying the debt is what makes the bound real: without it a maximal proof +/// would cost the same as a trivial one, since either way the next request only +/// has to wait for the balance to climb back above zero. With it, sustained +/// throughput settles at refill ÷ cost-per-proof, so expensive proofs are +/// admitted proportionally less often. +struct Round1WorkBudget { + /// Signed, so an over-large proof leaves debt to work off. + balance: i64, + last_refill: Instant, +} + +impl Round1WorkBudget { + /// Deepest debt carried, so one huge proof cannot lock out honest audits + /// for longer than the burst takes to refill. + const MAX_DEBT: i64 = -SUBTREE_ROUND1_WORK_BURST_BYTES; + /// Nanoseconds per second, for the sub-second part of a refill. + const NANOS_PER_SEC: i64 = 1_000_000_000; + + fn new() -> Self { + Self { + balance: SUBTREE_ROUND1_WORK_BURST_BYTES, + last_refill: Instant::now(), + } + } + + /// Add the tokens accrued since the last touch, capped at the burst size. + fn refill(&mut self, now: Instant) { + let elapsed = now.saturating_duration_since(self.last_refill); + self.last_refill = now; + let whole_secs = i64::try_from(elapsed.as_secs()) + .unwrap_or(i64::MAX) + .saturating_mul(SUBTREE_ROUND1_WORK_REFILL_BYTES_PER_SEC); + let sub_sec = i64::from(elapsed.subsec_nanos()) + .saturating_mul(SUBTREE_ROUND1_WORK_REFILL_BYTES_PER_SEC) + / Self::NANOS_PER_SEC; + self.balance = self + .balance + .saturating_add(whole_secs.saturating_add(sub_sec)) + .min(SUBTREE_ROUND1_WORK_BURST_BYTES); + } + + /// Whether budget remains for a new round-1 proof. + fn has_budget(&mut self, now: Instant) -> bool { + self.refill(now); + self.balance > 0 + } + + /// Charge `bytes` of completed proof work, carrying debt down to + /// [`Self::MAX_DEBT`]. + fn charge(&mut self, bytes: i64, now: Instant) { + self.refill(now); + self.balance = self.balance.saturating_sub(bytes).max(Self::MAX_DEBT); + } +} + +/// Resource controls for the HEAVY subtree-audit round 1: a tight +/// admission pool separate from the light responsible/slice audits, a per-peer +/// rate cooldown, and single-use round-1 → round-2 sessions so a round-2 slice +/// challenge is only served after a matching round 1. +/// +/// It also holds the responder-wide [`Round1WorkBudget`], the only one of those +/// bounds not keyed by peer identity, and so the only one that bounds sustained +/// work rather than concurrency or per-identity frequency. +#[derive(Clone)] +struct SubtreeRound1Limiter { + semaphore: Arc, + inflight: Arc>>, + cooldown: Arc>>, + /// Per-peer minimum spacing between served round-1 proofs (config-driven; + /// [`SUBTREE_ROUND1_RESPONDER_COOLDOWN`] in production, near-zero in tests). + cooldown_interval: Duration, + sessions: Arc>>, + /// Identity-independent ceiling on sustained round-1 work. + work: Arc>, +} + +impl SubtreeRound1Limiter { + fn new(cooldown_interval: Duration) -> Self { + Self { + semaphore: Arc::new(Semaphore::new(MAX_CONCURRENT_SUBTREE_ROUND1)), + inflight: Arc::new(RwLock::new(HashMap::new())), + cooldown: Arc::new(RwLock::new(HashMap::new())), + cooldown_interval, + sessions: Arc::new(RwLock::new(HashMap::new())), + work: Arc::new(RwLock::new(Round1WorkBudget::new())), + } + } + + /// Charge completed round-1 proof work against the responder-wide budget. + async fn charge_work(&self, content_bytes: i64) { + self.work + .write() + .await + .charge(content_bytes, Instant::now()); + } + + /// Admit one heavy round-1 proof for `source`: take a concurrency permit + /// FIRST (so a full pool never wastes the peer's cooldown allowance), then + /// the responder-wide work budget, then the per-peer rate cooldown. `None` + /// drops the challenge (the remote auditor applies its own graced-timeout + /// policy). + async fn admit(&self, source: &PeerId) -> Option { + let guard = admit_audit_responder_with_limits( + &self.semaphore, + &self.inflight, + source, + MAX_CONCURRENT_SUBTREE_ROUND1, + MAX_SUBTREE_ROUND1_PER_PEER, + ) + .await + .ok()?; + // Checked before the per-peer cooldown is stamped, so a peer refused for + // want of budget is not also charged its next allowance. + if !self.work.write().await.has_budget(Instant::now()) { + return None; // guard drops here, releasing the permit + slot + } + let now = Instant::now(); + let mut cooldown = self.cooldown.write().await; + if let Some(&last) = cooldown.get(source) { + if now.duration_since(last) < self.cooldown_interval { + return None; // guard drops here, releasing the permit + slot + } + } + // Evict lapsed entries (their cooldown has expired, so they no longer + // limit) and cap capacity, so peer-id churn can't grow this map unbounded. + cooldown.retain(|_, &mut last| now.duration_since(last) < self.cooldown_interval); + if cooldown.len() >= MAX_SUBTREE_SESSIONS { + if let Some(oldest) = cooldown.iter().min_by_key(|(_, &t)| t).map(|(k, _)| *k) { + cooldown.remove(&oldest); + } + } + cooldown.insert(*source, now); + Some(guard) + } + + /// Record a single-use session once a round-1 proof is built and about to be + /// sent, so the matching round 2 is admitted exactly once. + async fn open_session( + &self, + source: PeerId, + challenge_id: u64, + commitment_hash: [u8; 32], + nonce: [u8; 32], + ) { + let now = Instant::now(); + let mut sessions = self.sessions.write().await; + sessions.retain(|_, e| now.duration_since(e.inserted) < SUBTREE_SESSION_TTL); + if sessions.len() >= MAX_SUBTREE_SESSIONS { + if let Some(oldest) = sessions + .iter() + .min_by_key(|(_, e)| e.inserted) + .map(|(k, _)| *k) + { + sessions.remove(&oldest); + } + } + sessions.insert( + (source, challenge_id), + SubtreeSession { + commitment_hash, + nonce, + inserted: now, + }, + ); + } + + /// Atomically consume the round-2 session for this exchange. `true` iff a + /// live session matching `(source, challenge_id, commitment_hash, nonce)` + /// existed (and is now removed); a miss silently drops round 2 to the graced + /// timeout lane (sessions are ephemeral and can be lost across a restart). + async fn consume_session( + &self, + source: &PeerId, + challenge_id: u64, + commitment_hash: &[u8; 32], + nonce: &[u8; 32], + ) -> bool { + let mut sessions = self.sessions.write().await; + let matches = sessions.get(&(*source, challenge_id)).is_some_and(|e| { + Instant::now().duration_since(e.inserted) < SUBTREE_SESSION_TTL + && &e.commitment_hash == commitment_hash + && &e.nonce == nonce + }); + if matches { + sessions.remove(&(*source, challenge_id)); + } + matches + } +} + +/// Outcome of admitting a round-2 slice challenge. +enum SliceAdmission { + /// Admitted: the guard holds the global permit and the per-peer slot, and + /// the single-use round-1 session has been consumed. + Admitted(AuditResponderGuard), + /// Refused at a responder ceiling. The round-1 session is left INTACT. + Capacity(AuditResponderAdmissionFailure), + /// No live round-1 session matched this challenge. + NoSession, +} + +/// Admit a round-2 slice challenge: take the responder permit BEFORE consuming +/// the single-use round-1 session. +/// +/// The order is the point: a single-use token must not be spent on work that is +/// then refused. Consuming first and testing admission second leaves the session +/// destroyed by a purely local capacity drop, so the refusal is not recoverable +/// even in principle. +/// +/// Scope of the benefit today, stated honestly: `request_slice_proof` issues one +/// `send_request` and maps any failure straight to `SliceRound::Timeout`, so the +/// auditor does not currently re-send round 2 within an audit. The preserved +/// session is therefore not yet *recovering* an audit — it keeps a refusal +/// truthful (temporary means temporary) and keeps the invariant available for a +/// retry, rather than baking "capacity drop is permanent" into the state +/// machine. If an application-level retry is ruled out for good, this ordering +/// still costs nothing over the alternative. +/// +/// Cost of the ordering: the permit and per-peer slot are held across the +/// session probe, which is one in-memory map lookup and no chunk work. The +/// per-peer cap still bounds a peer sending unsessioned challenges to the same +/// share it could already occupy with well-formed ones, so the admission +/// surface is unchanged. +async fn admit_slice_challenge( + semaphore: &Arc, + inflight: &Arc>>, + round1: &SubtreeRound1Limiter, + source: &PeerId, + challenge: &protocol::SubtreeSliceChallenge, +) -> SliceAdmission { + let guard = match admit_audit_responder(semaphore, inflight, source).await { + Ok(guard) => guard, + Err(failure) => return SliceAdmission::Capacity(failure), + }; + if !round1 + .consume_session( + source, + challenge.challenge_id, + &challenge.expected_commitment_hash, + &challenge.nonce, + ) + .await + { + // Release the permit and per-peer slot before the caller replies: no + // chunk work follows, so holding them would shrink the pool for nothing. + drop(guard); + return SliceAdmission::NoSession; + } + SliceAdmission::Admitted(guard) +} + /// Try to admit one audit-responder task for `source`: take a global permit AND /// a per-peer slot (both bounded). Returns `Err` with the binding ceiling and /// its decision-time counters (caller drops the challenge, leaving the remote @@ -3026,8 +3375,26 @@ async fn admit_audit_responder( inflight: &Arc>>, source: &PeerId, ) -> std::result::Result { - let global_limit = MAX_CONCURRENT_AUDIT_RESPONSES; - let peer_limit = MAX_AUDIT_RESPONSES_PER_PEER; + admit_audit_responder_with_limits( + semaphore, + inflight, + source, + MAX_CONCURRENT_AUDIT_RESPONSES, + MAX_AUDIT_RESPONSES_PER_PEER, + ) + .await +} + +/// Admission core shared by the light audit pool ([`admit_audit_responder`]) and +/// the tight heavy subtree round-1 pool: take a global permit AND a per-peer slot +/// under the given limits. +async fn admit_audit_responder_with_limits( + semaphore: &Arc, + inflight: &Arc>>, + source: &PeerId, + global_limit: usize, + peer_limit: u32, +) -> std::result::Result { // `available_permits()` is a cheap atomic load; `global_limit - available` // is the best-effort in-flight count at decision time. Not synchronized with // the per-peer lock, so it is a snapshot, not a single atomic view. @@ -3092,6 +3459,7 @@ async fn admit_audit_responder( async fn handle_replication_message( source: &PeerId, data: &[u8], + inbound_protocol: &str, p2p_node: &Arc, storage: &Arc, paid_list: &Arc, @@ -3110,11 +3478,62 @@ async fn handle_replication_message( gossip_audit: &GossipAuditTrigger, audit_responder_semaphore: &Arc, audit_responder_inflight: &Arc>>, + subtree_round1: &SubtreeRound1Limiter, rr_message_id: Option<&str>, ) -> Result<()> { + // Size guard BEFORE decoding, keyed on the BODY's family — not on the id the + // message arrived on. + // + // Every later check — protocol family, live round-1 session, responder + // admission — reads fields of the decoded body, so none of them can run + // first. Decoding is therefore the first thing an unknown peer can make this + // node do, and the audit bodies carry variable-length collections, so under + // the 10 MiB core ceiling a sessionless peer could force multi-megabyte + // allocation and decode work per message with nothing spent on its side. + // Audit bodies are ~110 KiB at worst (see `MAX_AUDIT_MESSAGE_SIZE`), so a + // tighter family ceiling costs honest traffic nothing. + // + // Selecting that ceiling from `inbound_protocol` did NOT close the path it + // claimed to: an audit body addressed to the CORE id skipped the audit + // ceiling entirely, was decoded under the 10 MiB allowance, and only then + // dropped by `body_matches_protocol` — after its collections had been + // allocated. A `SubtreeSliceChallenge` carrying 200,000 openings encodes to + // ~6.6 MB and sailed through. The ceiling has to follow the body, and the + // body's discriminant is readable from the first few bytes without decoding + // anything attacker-sized, so it can be read first. + // + // A prefix too malformed to classify gets the strict ceiling: a message we + // cannot classify is not one to decode generously. + let peeked_family = protocol::peek_variant_index(data).map(protocol::family_of_variant); + let is_audit_body = peeked_family.map_or(true, protocol::BodyFamily::is_audit); + if is_audit_body && data.len() > MAX_AUDIT_MESSAGE_SIZE { + debug!( + "Dropping oversized audit-family message from {source} on {inbound_protocol}: \ + {} bytes > {MAX_AUDIT_MESSAGE_SIZE}", + data.len() + ); + return Ok(()); + } + let msg = ReplicationMessage::decode(data) .map_err(|e| Error::Protocol(format!("Failed to decode replication message: {e}")))?; + // Symmetric id/body guard: subtree-audit bodies are valid ONLY on the audit + // id, and core bodies ONLY on the core id. postcard::from_bytes ignores + // trailing bytes, so a mixed-version peer's message could otherwise decode + // into a valid-looking but wrong body (e.g. an old round-1 `Proof` on the + // core id misreading its bytes as the new `content_len`/`nonced_root`). The + // outer enum discriminants are unchanged across versions, so this drop by + // (id, is_subtree_audit) is exact. + if !body_matches_protocol(&msg.body, inbound_protocol) { + debug!( + "Dropping replication body (variant {}) on protocol {inbound_protocol}: \ + wrong id for its family (cross-version or misrouted)", + msg.body.variant_index() + ); + return Ok(()); + } + match msg.body { ReplicationMessageBody::FreshReplicationOffer(ref offer) => { handle_fresh_offer( @@ -3217,7 +3636,8 @@ async fn handle_replication_message( // block all other replication traffic until its digests complete // (head-of-line blocking). The same flood-fair admission applies: a // global ceiling AND a per-peer cap, dropping the challenge if either - // is hit. Responsible/prune audit timeouts are penalised by the + // is hit. A dropped challenge reads as a timeout to the auditor, and + // once the rollout gate is removed that is penalised again by the // caller, so the caps must remain high enough for honest audit load; // the per-peer share still prevents one flooder from starving others. let guard = match admit_audit_responder( @@ -3278,33 +3698,32 @@ async fn handle_replication_message( "Audit challenge received: kind=subtree source={source} request_response={}", rr_message_id.is_some(), ); - let guard = match admit_audit_responder( - audit_responder_semaphore, - audit_responder_inflight, - source, - ) - .await - { - Ok(guard) => guard, - Err(failure) => { - protocol::record_audit_drop(protocol::AuditDropKind::Subtree); - warn!( - "Audit challenge reply not sent: kind=subtree response=dropped \ - source={source} {failure}" - ); - return Ok(()); - } + // Round 1 is the HEAVY path (rebuilds + hashes the whole sqrt-subtree), + // so it uses its own tight admission pool + per-peer rate cooldown, + // separate from the light responsible/slice audits, and a miss silently + // drops (subtree auditors grace timeouts). + let Some(guard) = subtree_round1.admit(source).await else { + protocol::record_audit_drop(protocol::AuditDropKind::Subtree); + warn!( + "Audit challenge reply not sent: kind=subtree response=dropped \ + source={source} (heavy round-1 pool full or per-peer cooldown)" + ); + return Ok(()); }; let bootstrapping = *is_bootstrapping.read().await; let storage = Arc::clone(storage); let p2p_node = Arc::clone(p2p_node); let my_commitment_state = Arc::clone(my_commitment_state); + let subtree_round1 = subtree_round1.clone(); let source = *source; let request_id = msg.request_id; let rr_message_id = rr_message_id.map(ToOwned::to_owned); tokio::spawn(async move { - let _guard = guard; // global permit + per-peer slot, held until done - let response = storage_commitment_audit::handle_subtree_challenge( + let _guard = guard; // heavy permit + per-peer slot, held until done + let storage_commitment_audit::Round1Work { + response, + content_bytes, + } = storage_commitment_audit::handle_subtree_challenge_measured( &challenge, &storage, p2p_node.peer_id(), @@ -3312,6 +3731,35 @@ async fn handle_replication_message( Some(&my_commitment_state), ) .await; + // Charge the work actually done, on EVERY outcome. + // + // This used to charge only the `Proof` arm, reasoning that the + // rejecting paths either read nothing or reflected this node's + // own broken storage. The second half of that was wrong: a + // retained commitment containing one unreadable key still costs + // a full run of reads and keyed-BLAKE3 passes over every leaf + // before it, and then rejects. An attacker who finds such a + // commitment could replay subtrees over it indefinitely for + // free. The per-peer cooldown does not catch that either, since + // it is escapable by rotating identity — the responder-wide + // budget is the only bound that applies, so it has to see the + // work. A zero charge is a no-op, so the untouched paths are + // unaffected. + subtree_round1.charge_work(content_bytes).await; + // A round-1 proof authorizes exactly one matching round 2: open a + // single-use session so a slice challenge cannot be served without + // a live round-1 exchange. + if let crate::replication::protocol::SubtreeAuditResponse::Proof { .. } = &response + { + subtree_round1 + .open_session( + source, + challenge.challenge_id, + challenge.expected_commitment_hash, + challenge.nonce, + ) + .await; + } let response_kind = subtree_audit_response_kind(&response); let sent = send_replication_response_checked( &source, @@ -3337,32 +3785,71 @@ async fn handle_replication_message( }); Ok(()) } - ReplicationMessageBody::SubtreeByteChallenge(challenge) => { - // Round 2 of the storage audit (ADR-0002): serve the original bytes - // for the auditor's spot-check keys, or signal `Absent` for a - // committed key we can no longer produce. Reads chunk bytes from - // disk, so likewise spawned off the serial loop (§5) under the same - // flood-fair admission (codex#1 + codex-r2 A). + ReplicationMessageBody::SubtreeSliceChallenge(challenge) => { + // Round 2 of the storage audit (ADR-0002 / V2-685): open one 1 KiB + // block of each of the auditor's spot-check keys with a Bao verified + // slice + nonced block-tree opening, or signal `Absent` for a + // committed key we can no longer produce. Reads chunk bytes from disk + // to build the proofs, so likewise spawned off the serial loop + // under the same flood-fair admission (a global ceiling plus a + // per-peer cap). info!( - "Audit challenge received: kind=byte source={source} request_response={}", + "Audit challenge received: kind=slice source={source} request_response={}", rr_message_id.is_some(), ); - let guard = match admit_audit_responder( + let guard = match admit_slice_challenge( audit_responder_semaphore, audit_responder_inflight, + subtree_round1, source, + &challenge, ) .await { - Ok(guard) => guard, - Err(failure) => { - protocol::record_audit_drop(protocol::AuditDropKind::Byte); + SliceAdmission::Admitted(guard) => guard, + // Capacity drop: same silent-shed contract as the other two audit + // handlers. The session is still live, so the auditor's retry once + // load clears can still be served. + SliceAdmission::Capacity(failure) => { + protocol::record_audit_drop(protocol::AuditDropKind::Slice); warn!( - "Audit challenge reply not sent: kind=byte response=dropped \ + "Audit challenge reply not sent: kind=slice response=dropped \ source={source} {failure}" ); return Ok(()); } + // No live round-1 session: reply with a cheap `Transient` rejection + // rather than dropping silently. Sessions are ephemeral (an honest + // responder that restarts between rounds loses its session), and an + // unanswered `send_request` would make saorsa-core record a + // transport trust failure against that honest responder — an + // ongoing effect, not just a rollout-window one. A `Transient` + // reply routes the auditor to the graced timeout lane (no trust + // penalty; the responder re-earns pinned credit on the next audit) + // and does no chunk work, so it is not a DoS lever. + SliceAdmission::NoSession => { + protocol::record_audit_drop(protocol::AuditDropKind::Slice); + debug!( + "Slice challenge without a live round-1 session source={source} \ + challenge_id={} → Transient reject", + challenge.challenge_id + ); + send_replication_response_checked( + source, + p2p_node, + msg.request_id, + ReplicationMessageBody::SubtreeSliceResponse( + protocol::SubtreeSliceResponse::Rejected { + challenge_id: challenge.challenge_id, + kind: protocol::RejectKind::Transient, + reason: "no live round-1 session".to_string(), + }, + ), + rr_message_id, + ) + .await; + return Ok(()); + } }; let bootstrapping = *is_bootstrapping.read().await; let storage = Arc::clone(storage); @@ -3373,7 +3860,7 @@ async fn handle_replication_message( let rr_message_id = rr_message_id.map(ToOwned::to_owned); tokio::spawn(async move { let _guard = guard; // global permit + per-peer slot, held until done - let response = storage_commitment_audit::handle_subtree_byte_challenge( + let response = storage_commitment_audit::handle_subtree_slice_challenge( &challenge, &storage, p2p_node.peer_id(), @@ -3381,24 +3868,24 @@ async fn handle_replication_message( Some(&my_commitment_state), ) .await; - let response_kind = subtree_byte_response_kind(&response); + let response_kind = subtree_slice_response_kind(&response); let sent = send_replication_response_checked( &source, &p2p_node, request_id, - ReplicationMessageBody::SubtreeByteResponse(response), + ReplicationMessageBody::SubtreeSliceResponse(response), rr_message_id.as_deref(), ) .await; if sent { info!( - "Audit challenge reply sent: kind=byte response={response_kind} \ + "Audit challenge reply sent: kind=slice response={response_kind} \ source={source} request_response={}", rr_message_id.is_some(), ); } else { warn!( - "Audit challenge reply not sent: kind=byte response={response_kind} \ + "Audit challenge reply not sent: kind=slice response={response_kind} \ source={source} request_response={}", rr_message_id.is_some(), ); @@ -3454,7 +3941,7 @@ async fn handle_replication_message( | ReplicationMessageBody::FetchResponse(_) | ReplicationMessageBody::AuditResponse(_) | ReplicationMessageBody::SubtreeAuditResponse(_) - | ReplicationMessageBody::SubtreeByteResponse(_) + | ReplicationMessageBody::SubtreeSliceResponse(_) | ReplicationMessageBody::GetCommitmentByPinResponse(_) => Ok(()), } } @@ -4053,11 +4540,11 @@ fn subtree_audit_response_kind(response: &protocol::SubtreeAuditResponse) -> &'s } } -fn subtree_byte_response_kind(response: &protocol::SubtreeByteResponse) -> &'static str { +fn subtree_slice_response_kind(response: &protocol::SubtreeSliceResponse) -> &'static str { match response { - protocol::SubtreeByteResponse::Items { .. } => "items", - protocol::SubtreeByteResponse::Bootstrapping { .. } => "bootstrapping", - protocol::SubtreeByteResponse::Rejected { .. } => "rejected", + protocol::SubtreeSliceResponse::Items { .. } => "items", + protocol::SubtreeSliceResponse::Bootstrapping { .. } => "bootstrapping", + protocol::SubtreeSliceResponse::Rejected { .. } => "rejected", } } @@ -4106,25 +4593,23 @@ async fn send_replication_response_checked( } }; // V2-684: per-peer served-bytes attribution for the heavy serve paths. - // `FetchResponse` + `SubtreeByteResponse` carry ~99% of served bytes; - // `NeighborSyncResponse` is included for completeness. Other response - // variants (verification/audit/commitment) are intentionally excluded. + // `FetchResponse` carries ~99% of served bytes; `NeighborSyncResponse` is + // included for completeness. Other response variants (verification/audit/ + // commitment) are intentionally excluded — the round-2 audit reply is now a + // few-KB verified slice (V2-685), not a full-chunk transfer, so it is light. if matches!( msg.body, - ReplicationMessageBody::FetchResponse(_) - | ReplicationMessageBody::SubtreeByteResponse(_) - | ReplicationMessageBody::NeighborSyncResponse(_) + ReplicationMessageBody::FetchResponse(_) | ReplicationMessageBody::NeighborSyncResponse(_) ) { protocol::record_served(peer, encoded.len()); } + let protocol = response_protocol_for(&msg.body); let result = if let Some(msg_id) = rr_message_id { p2p_node - .send_response(peer, REPLICATION_PROTOCOL_ID, msg_id, encoded) + .send_response(peer, protocol, msg_id, encoded) .await } else { - p2p_node - .send_message(peer, REPLICATION_PROTOCOL_ID, encoded, &[]) - .await + p2p_node.send_message(peer, protocol, encoded, &[]).await }; if let Err(e) = result { debug!("Failed to send replication response to {peer}: {e}"); @@ -5433,12 +5918,32 @@ async fn handle_subtree_audit_result( /// bootstrapping); every confirmed storage-integrity reason does. /// /// Responsible-chunk `AuditChallenge` failures use this directly: timeouts keep -/// the bootstrap claim but are still reported as audit failures, matching the -/// pre-ADR-0002 behaviour. +/// the bootstrap claim, matching the pre-ADR-0002 behaviour. Whether a timeout +/// is also *penalised* is a separate question — see +/// [`audit_failure_reports_trust_penalty`]. fn audit_failure_clears_bootstrap_claim(reason: &AuditFailureReason) -> bool { !matches!(reason, AuditFailureReason::Timeout) } +/// Whether an audit failure with this reason reports an application trust event +/// at [`AUDIT_FAILURE_TRUST_WEIGHT`](config::AUDIT_FAILURE_TRUST_WEIGHT). +/// +/// ROLLOUT GATE — see [`GRACE_POSSESSION_AUDIT_TIMEOUTS`]. While that gate is +/// set, a `Timeout` on the digest lanes is graced, because a peer on the other +/// side of the possession-audit protocol move never answers and its silence is +/// not evidence about its storage. Every other reason is a confirmed failure and +/// is always penalised. When the gate is removed this becomes `true` for every +/// reason, restoring the unconditional penalty. +/// +/// [`GRACE_POSSESSION_AUDIT_TIMEOUTS`]: config::GRACE_POSSESSION_AUDIT_TIMEOUTS +fn audit_failure_reports_trust_penalty(reason: &AuditFailureReason) -> bool { + if config::GRACE_POSSESSION_AUDIT_TIMEOUTS { + !matches!(reason, AuditFailureReason::Timeout) + } else { + true + } +} + /// Handle the result of a responsible-chunk audit tick (audit #2): emit trust /// events and manage bootstrap-claim state. /// @@ -5494,12 +5999,19 @@ async fn handle_audit_result( } else { debug!("Audit timeout for {challenged_peer}; retaining active bootstrap claim"); } - p2p_node - .report_trust_event( - challenged_peer, - TrustEvent::ApplicationFailure(config::AUDIT_FAILURE_TRUST_WEIGHT), - ) - .await; + if audit_failure_reports_trust_penalty(reason) { + p2p_node + .report_trust_event( + challenged_peer, + TrustEvent::ApplicationFailure(config::AUDIT_FAILURE_TRUST_WEIGHT), + ) + .await; + } else { + debug!( + "Audit timeout for {challenged_peer} graced during the possession-audit \ + protocol rollout (no confirmed-failure penalty)" + ); + } } } AuditTickResult::BootstrapClaim { peer } => { @@ -6227,14 +6739,16 @@ async fn rebuild_and_rotate_commitment( // the Merkle root commits to the SET OF KEYS, not to the bytes. The // commitment therefore binds "which keys I claim to hold"; it does NOT // by itself prove byte possession. Byte possession is enforced by the - // audit-verify path, which recomputes `bytes_hash == BLAKE3(local_bytes)` - // and the per-key digest against the AUDITOR'S OWN local copy of the - // bytes — so a responder that holds the key list but dropped the bytes - // still fails (`missing bytes for committed key` / digest mismatch). - // This is sound ONLY while keys are content addresses. If this module - // is ever reused for non-content-addressed records (`bytes_hash != key`), - // the `(k, k)` shortcut would let a byte-less node forge a valid root and - // MUST be replaced with `(key, BLAKE3(bytes))` computed from real bytes. + // round-2 slice audit: a Bao verified slice decoded against the chunk + // ADDRESS plus a keyed nonced block-tree opening under a fresh per-audit + // nonce, so a responder that holds the key list but dropped the bytes + // cannot answer. This is sound ONLY while keys are content addresses; + // the round-1 verifier enforces `bytes_hash == key` on every audited leaf + // (`evaluate_subtree_structure`), so a non-content-addressed + // `(key, bytes_hash)` leaf is rejected rather than letting a byte-less node + // earn credit for `key`. If this module is ever reused for + // non-content-addressed records, that `(k, k)` shortcut AND the verifier + // gate must be replaced with `(key, BLAKE3(bytes))` computed from real bytes. let entries: Vec<_> = keys.into_iter().take(cap).map(|k| (k, k)).collect(); // No-op-rotation guard: compute just the Merkle root from `entries` @@ -6321,18 +6835,393 @@ mod tests { use std::time::Instant; use std::time::SystemTime; + #[test] + fn match_replication_protocol_accepts_both_ids_bare_and_rr() { + // Core id, bare gossip form and /rr/ request-response form. + assert_eq!( + match_replication_protocol(REPLICATION_PROTOCOL_ID), + Some((REPLICATION_PROTOCOL_ID, false)) + ); + assert_eq!( + match_replication_protocol(&format!("{RR_PREFIX}{REPLICATION_PROTOCOL_ID}")), + Some((REPLICATION_PROTOCOL_ID, true)) + ); + // Subtree-audit id, both forms. + assert_eq!( + match_replication_protocol(SUBTREE_AUDIT_PROTOCOL_ID), + Some((SUBTREE_AUDIT_PROTOCOL_ID, false)) + ); + assert_eq!( + match_replication_protocol(&format!("{RR_PREFIX}{SUBTREE_AUDIT_PROTOCOL_ID}")), + Some((SUBTREE_AUDIT_PROTOCOL_ID, true)) + ); + // Possession-audit id, both forms. + assert_eq!( + match_replication_protocol(POSSESSION_AUDIT_PROTOCOL_ID), + Some((POSSESSION_AUDIT_PROTOCOL_ID, false)) + ); + assert_eq!( + match_replication_protocol(&format!("{RR_PREFIX}{POSSESSION_AUDIT_PROTOCOL_ID}")), + Some((POSSESSION_AUDIT_PROTOCOL_ID, true)) + ); + // Foreign topics (incl. a bare /rr/ and an unrelated protocol) don't match. + assert_eq!(match_replication_protocol("autonomi.ant.dht.v1"), None); + assert_eq!(match_replication_protocol(RR_PREFIX), None); + assert_eq!( + match_replication_protocol("autonomi.ant.replication.v3"), + None + ); + } + + // The receive guard drops a body whose family disagrees with the id it rode: + // subtree-audit bodies only on the subtree id, possession-audit bodies only on + // the possession id, core bodies only on the core id. This is what stops a + // mixed-version peer's message from being honoured on the wrong handler after + // a postcard misdecode. The test drives the SAME `body_matches_protocol` the + // production guard uses, over real bodies, so a regression in the rule fails + // here. + #[test] + fn body_matches_protocol_is_symmetric_over_real_bodies() { + use crate::replication::protocol::{ + AuditChallenge, FreshReplicationOffer, ReplicationMessageBody, SubtreeSliceChallenge, + }; + // A subtree-audit body (models a v2 SubtreeByteChallenge, which decodes to + // this variant 13 under the new enum), a possession-audit body, and a + // core body. + let audit = ReplicationMessageBody::SubtreeSliceChallenge(SubtreeSliceChallenge { + challenge_id: 1, + nonce: [0u8; 32], + challenged_peer_id: [0u8; 32], + expected_commitment_hash: [0u8; 32], + openings: vec![], + }); + let possession = ReplicationMessageBody::AuditChallenge(AuditChallenge { + challenge_id: 1, + nonce: [0u8; 32], + challenged_peer_id: [0u8; 32], + keys: vec![[0u8; 32]], + }); + let core = ReplicationMessageBody::FreshReplicationOffer(FreshReplicationOffer { + key: [0u8; 32], + data: vec![], + proof_of_payment: vec![], + }); + // Correct routing is kept. + assert!(body_matches_protocol(&audit, SUBTREE_AUDIT_PROTOCOL_ID)); + assert!(body_matches_protocol( + &possession, + POSSESSION_AUDIT_PROTOCOL_ID + )); + assert!(body_matches_protocol(&core, REPLICATION_PROTOCOL_ID)); + // Every cross-routing is dropped. In particular an older peer's + // possession audit arriving on the shared core id is dropped rather than + // answered with a digest from a different generation, which would score + // as a confirmed mismatch against an honest peer. + for (body, wrong) in [ + (&audit, REPLICATION_PROTOCOL_ID), + (&audit, POSSESSION_AUDIT_PROTOCOL_ID), + (&possession, REPLICATION_PROTOCOL_ID), + (&possession, SUBTREE_AUDIT_PROTOCOL_ID), + (&core, SUBTREE_AUDIT_PROTOCOL_ID), + (&core, POSSESSION_AUDIT_PROTOCOL_ID), + ] { + assert!( + !body_matches_protocol(body, wrong), + "body must not be accepted on {wrong}" + ); + } + + // Send and receive must agree. A RESPONSE is routed by the same rule, so + // the id a body is sent on is always an id the peer's guard accepts. + // Before this was shared, possession responses went out on the core id + // and only worked because saorsa-core correlates RR replies by + // (peer, msg_id) rather than by protocol name — a bare possession + // response was dropped by the receiving guard. + for (body, expected) in [ + (&audit, SUBTREE_AUDIT_PROTOCOL_ID), + (&possession, POSSESSION_AUDIT_PROTOCOL_ID), + (&core, REPLICATION_PROTOCOL_ID), + ] { + assert_eq!( + response_protocol_for(body), + expected, + "a response must be sent on the family's own id" + ); + assert!( + body_matches_protocol(body, response_protocol_for(body)), + "the id we send on must be one the receive guard accepts" + ); + } + + // The possession RESPONSE body (not just the challenge) routes to the + // possession id too — that is the direction that was actually wrong. + let possession_response = ReplicationMessageBody::AuditResponse( + crate::replication::protocol::AuditResponse::Digests { + challenge_id: 1, + digests: vec![[0u8; 32]], + }, + ); + assert_eq!( + response_protocol_for(&possession_response), + POSSESSION_AUDIT_PROTOCOL_ID + ); + } + fn test_peer(b: u8) -> PeerId { let mut bytes = [0u8; 32]; bytes[0] = b; PeerId::from_bytes(bytes) } + // The heavy round-1 limiter enforces the per-peer rate cooldown + // and single-use round-1 → round-2 sessions. + #[tokio::test] + async fn subtree_round1_limiter_cooldown_and_single_use_session() { + let limiter = SubtreeRound1Limiter::new(Duration::from_secs(3600)); + let peer = test_peer(1); + + // First round-1 is admitted; drop the guard so concurrency is free again. + let guard = limiter.admit(&peer).await; + assert!(guard.is_some(), "first round-1 admitted"); + drop(guard); + // A second round-1 within the cooldown is dropped even though the heavy + // pool now has a free slot — the rate cooldown, not concurrency, blocks it. + assert!( + limiter.admit(&peer).await.is_none(), + "second round-1 within cooldown is rate-dropped" + ); + // A different peer has its own cooldown. + assert!(limiter.admit(&test_peer(2)).await.is_some()); + + // Session: opened by round 1, consumed exactly once by the matching round 2. + let hash = [7u8; 32]; + let nonce = [9u8; 32]; + limiter.open_session(peer, 42, hash, nonce).await; + // Wrong nonce / commitment does not match. + assert!(!limiter.consume_session(&peer, 42, &hash, &[0u8; 32]).await); + assert!(!limiter.consume_session(&peer, 42, &[0u8; 32], &nonce).await); + // A round 2 with no prior round 1 (wrong challenge_id) misses. + assert!(!limiter.consume_session(&peer, 99, &hash, &nonce).await); + // The matching round 2 consumes it — and only once (single-use). + assert!(limiter.consume_session(&peer, 42, &hash, &nonce).await); + assert!(!limiter.consume_session(&peer, 42, &hash, &nonce).await); + } + + // The concurrency pool and the per-peer cooldown are both keyed by peer id, + // so a party holding several identities refills its allowance by rotating + // between them and can keep the heavy pool busy indefinitely. The work + // budget is keyed by nothing: it is charged for bytes proved, whoever asked, + // so a fresh identity is refused exactly like a repeat caller once it is + // spent. That is the difference between bounding concurrency and bounding + // sustained work. + #[tokio::test] + async fn round1_work_budget_is_not_refilled_by_a_fresh_identity() { + // Cooldown disabled so only the work budget can refuse anything. + let limiter = SubtreeRound1Limiter::new(Duration::ZERO); + assert!( + limiter.admit(&test_peer(1)).await.is_some(), + "a node starts with budget in hand so it can serve audits at once" + ); + + // Serve enough proof work to run the balance into debt. Charging exactly + // the burst would leave it at zero, which the next nanosecond of refill + // lifts back above the line — the debt is the point. + limiter + .charge_work(2 * SUBTREE_ROUND1_WORK_BURST_BYTES) + .await; + + for id in 2..8u8 { + assert!( + limiter.admit(&test_peer(id)).await.is_none(), + "a never-seen peer must still be refused while the budget is spent" + ); + } + } + + // The budget is a rate, not a quota: it comes back on its own, so an honest + // auditor blocked by a flood is only delayed. Carrying the debt is what + // prices an expensive proof above a cheap one — without it, a proof reading + // a maximal subtree would cost no more of the next caller's wait than a + // one-leaf proof. + #[test] + fn round1_work_budget_carries_debt_and_refills_over_time() { + let now = Instant::now(); + let at = |secs: u64| { + now.checked_add(Duration::from_secs(secs)) + .unwrap_or_else(Instant::now) + }; + let mut budget = Round1WorkBudget { + balance: 0, + last_refill: now, + }; + assert!(!budget.has_budget(now), "empty means empty"); + + // A proof costing four seconds' worth of refill leaves four seconds of + // debt, so the wait scales with what was actually served. + budget.charge(4 * SUBTREE_ROUND1_WORK_REFILL_BYTES_PER_SEC, now); + assert!( + !budget.has_budget(at(3)), + "still in debt three seconds after an over-large proof" + ); + assert!( + budget.has_budget(at(5)), + "the debt is worked off at the refill rate" + ); + + // Idle time does not bank unbounded credit for a later flood. + budget.refill(at(24 * 60 * 60)); + assert_eq!( + budget.balance, SUBTREE_ROUND1_WORK_BURST_BYTES, + "refill is capped at the burst size" + ); + } + fn test_key(b: u8) -> crate::ant_protocol::XorName { let mut k = [0u8; 32]; k[0] = b; k } + /// Build a round-2 slice challenge matching an open session. `openings` is + /// empty: these tests exercise admission and session handling only, which + /// run before any block is opened. + fn slice_challenge( + challenge_id: u64, + commitment_hash: [u8; 32], + nonce: [u8; 32], + ) -> protocol::SubtreeSliceChallenge { + protocol::SubtreeSliceChallenge { + challenge_id, + nonce, + challenged_peer_id: [0u8; 32], + expected_commitment_hash: commitment_hash, + openings: Vec::new(), + } + } + + // Regression (Copilot, PR #181): a round-2 slice challenge refused at the + // responder caps MUST NOT burn the single-use round-1 session. + // + // The handler used to consume the session first and admit second, so a + // transient local capacity drop permanently destroyed that audit exchange — + // every retry hit the `no live round-1 session` path and got `Transient`, + // even after load cleared. Turning momentary local load into a deterministic + // round-2 miss costs the responder its whole-slice credit for that round. + #[tokio::test] + async fn capacity_refused_slice_challenge_preserves_round1_session() { + let semaphore = Arc::new(Semaphore::new(MAX_CONCURRENT_AUDIT_RESPONSES)); + let inflight = Arc::new(RwLock::new(HashMap::new())); + let round1 = SubtreeRound1Limiter::new(Duration::ZERO); + let peer = test_peer(0xB1); + let (id, hash, nonce) = (77u64, [3u8; 32], [4u8; 32]); + let challenge = slice_challenge(id, hash, nonce); + + round1.open_session(peer, id, hash, nonce).await; + + // Saturate this peer's share so the next admission must be refused. + let mut hold = Vec::new(); + for _ in 0..MAX_AUDIT_RESPONSES_PER_PEER { + match admit_audit_responder(&semaphore, &inflight, &peer).await { + Ok(guard) => hold.push(guard), + Err(err) => panic!("unexpected admission failure below the cap: {err:?}"), + } + } + + let refused = + admit_slice_challenge(&semaphore, &inflight, &round1, &peer, &challenge).await; + assert!( + matches!(refused, SliceAdmission::Capacity(_)), + "a saturated peer share must refuse on capacity, not on session" + ); + + // Load clears. The session must have survived the refusal. + drop(hold); + let retried = + admit_slice_challenge(&semaphore, &inflight, &round1, &peer, &challenge).await; + assert!( + matches!(retried, SliceAdmission::Admitted(_)), + "the round-1 session must survive a capacity refusal so the retry succeeds" + ); + + // Still single-use: the successful admission consumed it exactly once. + drop(retried); + let replayed = + admit_slice_challenge(&semaphore, &inflight, &round1, &peer, &challenge).await; + assert!( + matches!(replayed, SliceAdmission::NoSession), + "a consumed session must not be replayable" + ); + } + + // The permit taken for the session probe is released when the probe misses, + // so an unsessioned flood cannot pin the responder pool shut. + #[tokio::test] + async fn unsessioned_slice_challenge_releases_its_admission_slot() { + let semaphore = Arc::new(Semaphore::new(MAX_CONCURRENT_AUDIT_RESPONSES)); + let inflight = Arc::new(RwLock::new(HashMap::new())); + let round1 = SubtreeRound1Limiter::new(Duration::ZERO); + let peer = test_peer(0xB2); + let challenge = slice_challenge(1, [0u8; 32], [0u8; 32]); + + // Far more unsessioned challenges than the per-peer cap would allow if + // the slot leaked on the miss path. + for _ in 0..(MAX_AUDIT_RESPONSES_PER_PEER * 4) { + let outcome = + admit_slice_challenge(&semaphore, &inflight, &round1, &peer, &challenge).await; + assert!( + matches!(outcome, SliceAdmission::NoSession), + "no session was ever opened, so every attempt must miss" + ); + } + + assert_eq!( + semaphore.available_permits(), + MAX_CONCURRENT_AUDIT_RESPONSES, + "every global permit must be returned" + ); + assert!( + inflight.read().await.get(&peer).is_none_or(|n| *n == 0), + "no per-peer slot may be left occupied" + ); + } + + // The rollout gate must grace exactly ONE thing — a timeout — and nothing + // else. A confirmed storage-integrity failure is still penalised while the + // gate is set, otherwise moving the possession lanes onto their own protocol + // id would have handed cheating peers an amnesty for the upgrade window. + // + // FOLLOW-UP: when `GRACE_POSSESSION_AUDIT_TIMEOUTS` is set to false and the + // gate deleted, the timeout expectation below flips to `true`. That is the + // intended end state, and this test is where the flip is reflected. + #[test] + fn rollout_gate_graces_only_timeouts() { + for reason in [ + AuditFailureReason::DigestMismatch, + AuditFailureReason::KeyAbsent, + AuditFailureReason::MalformedResponse, + AuditFailureReason::Rejected, + ] { + assert!( + audit_failure_reports_trust_penalty(&reason), + "{reason:?} is a confirmed failure and must be penalised even during rollout" + ); + } + assert_eq!( + audit_failure_reports_trust_penalty(&AuditFailureReason::Timeout), + !config::GRACE_POSSESSION_AUDIT_TIMEOUTS, + "a timeout is penalised exactly when the rollout gate is off" + ); + + // Holder credit is a separate axis that was already timeout-safe; the + // gate must not have disturbed it. + assert!(!audit_failure_revokes_holder_credit( + &AuditFailureReason::Timeout + )); + assert!(audit_failure_revokes_holder_credit( + &AuditFailureReason::DigestMismatch + )); + } + #[tokio::test] async fn audit_responder_admission_reports_per_peer_cap_full() { let semaphore = Arc::new(Semaphore::new(MAX_CONCURRENT_AUDIT_RESPONSES)); diff --git a/src/replication/possession.rs b/src/replication/possession.rs index 71869218..c44b5731 100644 --- a/src/replication/possession.rs +++ b/src/replication/possession.rs @@ -3,8 +3,8 @@ //! After a node fresh-replicates a chunk, every close-group peer responsible //! for it is checked 5-15 minutes later for actual possession. The check is a //! single-key cryptographic -//! [`AuditChallenge`]: the probed -//! peer must return `BLAKE3(nonce ‖ peer_id ‖ key ‖ bytes)` computed over the +//! [`AuditChallenge`]: the probed peer must return +//! `compute_audit_digest(nonce, peer_id, key, bytes)` computed over the //! chunk it claims to hold. It cannot produce that digest without the bytes, so //! — unlike a self-reported presence flag — a peer cannot escape the check by //! falsely asserting possession. A peer that holds the chunk earns nothing — @@ -15,10 +15,17 @@ //! push is irrelevant: a peer the push never reached is still checked and //! penalised if it lacks the chunk. //! -//! A peer unreachable at check time is penalised immediately at audit severity, -//! matching the responsible-chunk `AuditChallenge` path. A matching bootstrap -//! claim uses the shared bootstrap-claim grace/abuse tracker; peer-side -//! malformed, rejected, or mismatched responses are audit failures. +//! Peer-side malformed, rejected, or mismatched responses are audit failures. A +//! matching bootstrap claim uses the shared bootstrap-claim grace/abuse tracker. +//! +//! A peer unreachable at check time would normally be penalised at audit +//! severity too, matching the responsible-chunk `AuditChallenge` path, but that +//! is currently suspended: this lane moved to its own protocol id, so a peer on +//! the far side of the upgrade never answers, and +//! [`GRACE_POSSESSION_AUDIT_TIMEOUTS`] graces silence until the fleet has moved +//! over. See `probe_outcome_is_penalised`. +//! +//! [`GRACE_POSSESSION_AUDIT_TIMEOUTS`]: crate::replication::config::GRACE_POSSESSION_AUDIT_TIMEOUTS use std::sync::Arc; use std::time::{Duration, Instant}; @@ -32,7 +39,8 @@ use tokio_util::sync::CancellationToken; use crate::ant_protocol::XorName; use crate::logging::{debug, warn}; use crate::replication::config::{ - ReplicationConfig, AUDIT_FAILURE_TRUST_WEIGHT, REPLICATION_PROTOCOL_ID, + ReplicationConfig, AUDIT_FAILURE_TRUST_WEIGHT, GRACE_POSSESSION_AUDIT_TIMEOUTS, + POSSESSION_AUDIT_PROTOCOL_ID, }; use crate::replication::protocol::{ compute_audit_digest, AuditChallenge, AuditResponse, ReplicationMessage, @@ -56,6 +64,7 @@ pub struct PossessionCheckEvent { } /// Verdict of cryptographically probing a single peer for possession of a chunk. +#[derive(Clone, Copy)] #[cfg_attr(test, derive(Debug, PartialEq, Eq))] enum ProbeOutcome { /// Peer returned a digest proving it holds the chunk's bytes. @@ -63,8 +72,9 @@ enum ProbeOutcome { /// Peer failed the audit challenge: absent sentinel, digest mismatch, /// rejection, mismatched challenge ID, wrong digest count, or malformed reply. Failed, - /// No response (transport error / deadline). Penalised immediately at - /// audit-failure severity. + /// No response (transport error / deadline). Penalised at audit-failure + /// severity — except while the rollout gate graces silence; see + /// [`probe_outcome_is_penalised`]. Timeout, /// Peer returned a matching bootstrap claim. Graced only through the shared /// bootstrap-claim tracker. @@ -73,6 +83,31 @@ enum ProbeOutcome { Inconclusive, } +/// Whether a probe outcome is reported as an audit-severity failure. +/// +/// `Failed` is the peer answering and being wrong — the absent sentinel, a +/// digest that does not match, a rejection. That is evidence, and it is +/// penalised throughout. +/// +/// `Timeout` is silence, and while the [`GRACE_POSSESSION_AUDIT_TIMEOUTS`] +/// rollout gate is set it is graced: a peer on the far side of the +/// possession-audit protocol move never answers this lane, so its silence says +/// nothing about whether it holds the chunk. Penalising it would punish honest +/// peers for a version skew for the whole upgrade window. +/// +/// Exhaustive rather than a catch-all, so a new outcome cannot silently inherit +/// "not penalised", and extracted as a predicate so the rollout decision can be +/// asserted without driving a live probe. +#[must_use] +fn probe_outcome_is_penalised(outcome: ProbeOutcome) -> bool { + match outcome { + ProbeOutcome::Failed => true, + ProbeOutcome::Timeout => !GRACE_POSSESSION_AUDIT_TIMEOUTS, + // Proven, claimed-and-tracked, or never actually asked. + ProbeOutcome::Present | ProbeOutcome::BootstrapClaim | ProbeOutcome::Inconclusive => false, + } +} + /// Pick a randomised delay in `[min, max]` to wait before a possession check /// runs. The bounds come from `ReplicationConfig` (defaulting to /// `POSSESSION_CHECK_DELAY_MIN`/`MAX`) so tests can shorten them. @@ -149,7 +184,14 @@ pub(crate) async fn run_possession_check( .await; } ProbeOutcome::Timeout => { - report_possession_audit_failure(&peer, &key_hex, "timed out", p2p_node).await; + if probe_outcome_is_penalised(ProbeOutcome::Timeout) { + report_possession_audit_failure(&peer, &key_hex, "timed out", p2p_node).await; + } else { + debug!( + "Possession check: {peer} timed out for {key_hex}; graced during the \ + possession-audit protocol rollout (not penalised)" + ); + } } ProbeOutcome::BootstrapClaim => { handle_possession_bootstrap_claim(&peer, &key_hex, p2p_node, config, sync_state) @@ -281,7 +323,7 @@ async fn probe_once( }; let response = match p2p_node - .send_request(peer, REPLICATION_PROTOCOL_ID, encoded, probe_timeout) + .send_request(peer, POSSESSION_AUDIT_PROTOCOL_ID, encoded, probe_timeout) .await { Ok(response) => response, @@ -364,6 +406,35 @@ mod tests { use super::*; use crate::replication::config::{POSSESSION_CHECK_DELAY_MAX, POSSESSION_CHECK_DELAY_MIN}; + // The possession probe's half of the rollout-gate contract (dirvine, PR + // #181). Silence is graced while the gate is set; every outcome where the + // peer actually answered and was wrong stays penalised. + #[test] + fn possession_rollout_gate_graces_silence_but_never_a_bad_answer() { + assert!( + probe_outcome_is_penalised(ProbeOutcome::Failed), + "an absent sentinel or mismatched digest is evidence — always penalised" + ); + assert_eq!( + probe_outcome_is_penalised(ProbeOutcome::Timeout), + !GRACE_POSSESSION_AUDIT_TIMEOUTS, + "silence is penalised exactly when the rollout gate is off" + ); + // A proven holder, a tracked bootstrap claim, and a probe that never + // left this node are all not failures at all — the gate must not have + // made any of them one. + for outcome in [ + ProbeOutcome::Present, + ProbeOutcome::BootstrapClaim, + ProbeOutcome::Inconclusive, + ] { + assert!( + !probe_outcome_is_penalised(outcome), + "{outcome:?} is not an audit failure" + ); + } + } + const PEER_ID: [u8; 32] = [0x42; 32]; const NONCE: [u8; 32] = [0x7a; 32]; const CHALLENGE_ID: u64 = 0xDEAD_BEEF; diff --git a/src/replication/protocol.rs b/src/replication/protocol.rs index efc2366f..54592304 100644 --- a/src/replication/protocol.rs +++ b/src/replication/protocol.rs @@ -137,10 +137,10 @@ pub enum ReplicationMessageBody { SubtreeAuditChallenge(SubtreeAuditChallenge), /// Response to a contiguous-subtree storage audit challenge (round 1). SubtreeAuditResponse(SubtreeAuditResponse), - /// Surprise byte challenge for the spot-checked leaves (round 2). - SubtreeByteChallenge(SubtreeByteChallenge), - /// Response carrying the requested chunks' original bytes (round 2). - SubtreeByteResponse(SubtreeByteResponse), + /// Surprise slice challenge for the spot-checked leaves (round 2). + SubtreeSliceChallenge(SubtreeSliceChallenge), + /// Response carrying verified slices for the opened blocks (round 2). + SubtreeSliceResponse(SubtreeSliceResponse), // === Commitment fetch by pin (ADR-0004) === // APPENDED at the end so postcard variant discriminants of all the @@ -203,12 +203,133 @@ impl ReplicationMessageBody { Self::AuditResponse(_) => 10, Self::SubtreeAuditChallenge(_) => 11, Self::SubtreeAuditResponse(_) => 12, - Self::SubtreeByteChallenge(_) => 13, - Self::SubtreeByteResponse(_) => 14, + Self::SubtreeSliceChallenge(_) => 13, + Self::SubtreeSliceResponse(_) => 14, Self::GetCommitmentByPin(_) => 15, Self::GetCommitmentByPinResponse(_) => 16, } } + + /// Whether this body is a subtree storage-commitment audit message (both + /// rounds). These ride [`SUBTREE_AUDIT_PROTOCOL_ID`], not the core + /// [`REPLICATION_PROTOCOL_ID`]; the receive dispatch uses this to enforce + /// that audit bodies only arrive on the audit id and core bodies only on the + /// core id, so a mixed-version peer's message can never be honoured on the + /// wrong handler even if postcard misdecodes it into a valid-looking value. + /// + /// [`SUBTREE_AUDIT_PROTOCOL_ID`]: crate::replication::config::SUBTREE_AUDIT_PROTOCOL_ID + /// [`REPLICATION_PROTOCOL_ID`]: crate::replication::config::REPLICATION_PROTOCOL_ID + #[must_use] + pub fn is_subtree_audit(&self) -> bool { + family_of_variant(self.variant_index()) == BodyFamily::SubtreeAudit + } + + /// Whether this body is a digest-based possession audit message. + /// + /// One wire pair (`AuditChallenge`/`AuditResponse`) serves three callers: + /// the responsible-chunk audit, the post-replication possession probe, and + /// the prune-confirmation audit. They ride [`POSSESSION_AUDIT_PROTOCOL_ID`] + /// because the digest they carry is keyed by the challenge material, so a + /// peer on an older digest generation must not be answered on this lane at + /// all rather than be scored as a confirmed mismatch. + /// + /// [`POSSESSION_AUDIT_PROTOCOL_ID`]: crate::replication::config::POSSESSION_AUDIT_PROTOCOL_ID + #[must_use] + pub fn is_possession_audit(&self) -> bool { + family_of_variant(self.variant_index()) == BodyFamily::PossessionAudit + } +} + +/// Which protocol family a body belongs to. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum BodyFamily { + /// Core replication: everything that is not an audit. + Core, + /// The digest-based possession pair (`AuditChallenge`/`AuditResponse`). + PossessionAudit, + /// The four subtree storage-commitment audit bodies, both rounds. + SubtreeAudit, +} + +impl BodyFamily { + /// Whether this family takes the tighter audit wire ceiling. + #[must_use] + pub(crate) fn is_audit(self) -> bool { + matches!(self, Self::PossessionAudit | Self::SubtreeAudit) + } +} + +/// The family of a body, keyed by its postcard discriminant. +/// +/// Keyed by discriminant rather than by `&self` on purpose: the receive path has +/// to classify a message BEFORE decoding it, because the pre-decode size ceiling +/// is the only check that can run before an attacker-sized collection is +/// allocated. The discriminant is the one field reachable that early (see +/// [`peek_variant_index`]). +/// +/// Everything that classifies a body reads this one table — +/// [`ReplicationMessageBody::is_subtree_audit`], +/// [`ReplicationMessageBody::is_possession_audit`], the outbound response +/// routing, the post-decode family guard, and the pre-decode ceiling — so the +/// pre- and post-decode views of "which family is this" cannot drift apart. A +/// drift is exactly what let an audit body sent on the core id skip the audit +/// ceiling and decode under the 10 MiB core allowance. +/// +/// Indices match [`ReplicationMessageBody::variant_index`], which is declaration +/// order and postcard-stable. +#[must_use] +pub(crate) fn family_of_variant(index: usize) -> BodyFamily { + match index { + 9 | 10 => BodyFamily::PossessionAudit, + 11..=14 => BodyFamily::SubtreeAudit, + _ => BodyFamily::Core, + } +} + +/// Maximum bytes a postcard varint can occupy for a `u64`. +const MAX_VARINT_LEN: usize = 10; + +/// Read one postcard varint (LEB128, little-endian base-128) from `bytes`. +/// +/// Returns the value and how many bytes it consumed. Bounded by `max_bytes` and +/// by the 64-bit shift, so a hostile prefix cannot spin here. +fn read_varint(bytes: &[u8], max_bytes: usize) -> Option<(u64, usize)> { + let mut value: u64 = 0; + let mut shift: u32 = 0; + for (i, byte) in bytes.iter().take(max_bytes).enumerate() { + value |= u64::from(byte & 0x7F).checked_shl(shift)?; + if byte & 0x80 == 0 { + return Some((value, i + 1)); + } + shift = shift.checked_add(7)?; + if shift >= u64::BITS { + return None; + } + } + None +} + +/// Read a message's body discriminant without decoding the body. +/// +/// `ReplicationMessage` is `{ request_id: u64, body: ReplicationMessageBody }`, +/// and postcard writes struct fields in order with no length prefix: a varint +/// `request_id`, then the enum's varint discriminant. So the discriminant sits +/// behind at most [`MAX_VARINT_LEN`] bytes and costs two bounded varint reads to +/// reach — no allocation, and nothing attacker-sized is touched. +/// +/// `None` means the prefix is not even well-formed enough to classify; the +/// caller treats that as "apply the strictest ceiling", since a message we +/// cannot classify is one we should not decode generously. +/// +/// A test round-trips every variant through `encode` to prove this agrees with +/// [`ReplicationMessageBody::variant_index`] for real messages, rather than +/// trusting this reading of postcard's format. +#[must_use] +pub(crate) fn peek_variant_index(data: &[u8]) -> Option { + let (_request_id, consumed) = read_varint(data, MAX_VARINT_LEN)?; + let rest = data.get(consumed..)?; + let (variant, _) = read_varint(rest, MAX_VARINT_LEN)?; + usize::try_from(variant).ok() } // --------------------------------------------------------------------------- @@ -238,8 +359,14 @@ pub(crate) enum AuditDropKind { Responsible = 0, /// Subtree audit round-1 challenge. Subtree = 1, - /// Subtree audit round-2 byte challenge. - Byte = 2, + /// Subtree audit round-2 slice challenge. + /// + /// The emitted counter is still named `audit_dropped_byte` (see + /// `log_audit_outcome_summary`): round 2 no longer serves full chunk bytes, + /// but keeping the wire-visible counter name fixed is what lets a v2 fleet + /// and a v3 fleet be compared with no field mapping. The name is a metric + /// identifier, not a description of the payload. + Slice = 2, } /// One slot per [`AuditFailureReason`] variant, per [`AuditOutcomeKind`]. @@ -384,6 +511,14 @@ pub(crate) fn log_traffic_summary() { group = 3, subtree_audit_response_tx_bytes = tb(12), subtree_audit_response_tx_count = tc(12), subtree_audit_response_rx_bytes = rb(12), subtree_audit_response_rx_count = rc(12), + // Indices 13/14 are the round-2 pair, renamed on the wire to + // `SubtreeSliceChallenge`/`SubtreeSliceResponse`. The COUNTER names stay + // `subtree_byte_*` on purpose: they are the field names an operator + // queries, and holding them fixed is what let the V2-685 testnet compare + // a 0.15.0 cohort against a slice-audit cohort with no field mapping + // (6.19 MB vs 14.49 KB per response, same query). Renaming them would + // silently zero every existing dashboard and alert. The variant index, + // not the label, is the source of truth for what is being counted. subtree_byte_challenge_tx_bytes = tb(13), subtree_byte_challenge_tx_count = tc(13), subtree_byte_challenge_rx_bytes = rb(13), subtree_byte_challenge_rx_count = rc(13), subtree_byte_response_tx_bytes = tb(14), subtree_byte_response_tx_count = tc(14), @@ -404,9 +539,9 @@ pub(crate) fn log_traffic_summary() { // // Follow-up to V2-623: the per-variant table above says how many bytes each // message type served, but not WHICH peers pulled them. This adds per-peer -// attribution for the heavy serve paths (`FetchResponse`, `SubtreeByteResponse`, -// `NeighborSyncResponse` — ~99% of served bytes), emitted as a top-10-by-bytes -// INFO line on the same cadence and target as `log_traffic_summary`. +// attribution for the heavy serve paths (`FetchResponse`, `NeighborSyncResponse` +// — ~99% of served bytes), emitted as a top-10-by-bytes INFO line on the same +// cadence and target as `log_traffic_summary`. // // Design (mirrors V2-623's process-global-static choice for the same reason — // the serve choke point is a free function with no engine handle): @@ -832,7 +967,8 @@ pub enum AuditResponse { /// [`SubtreeAuditResponse::Proof`] for that selected subtree against the pinned /// commitment, or a [`SubtreeAuditResponse::Rejected`] if it genuinely cannot /// (for a recently gossiped pinned commitment a rejection is a confirmed -/// failure, since the responder retains its last two gossiped commitments). +/// failure, since the responder retains its recently gossiped commitments for a +/// bounded TTL window). #[derive(Debug, Clone, Serialize, Deserialize)] pub struct SubtreeAuditChallenge { /// Unique challenge identifier. @@ -864,11 +1000,12 @@ pub enum SubtreeAuditResponse { /// the root from the proof, and requires it to equal the commitment /// root (structure). /// - /// The leaves carry only hashes (`bytes_hash`, `nonced_hash`), so this round + /// The leaves carry only hashes (`bytes_hash`, `nonced_root`), so this round /// proves the tree SHAPE is committed — not that the bytes are still held. /// Real possession is proven in **round 2**: the auditor picks a few of the - /// just-verified leaves and sends a [`SubtreeByteChallenge`] requesting their - /// original chunk bytes FROM the responder (see that type). + /// just-verified leaves and sends a [`SubtreeSliceChallenge`] opening one + /// random 1 KiB block of each against both the chunk address and the round-1 + /// `nonced_root` (see that type). Proof { /// The challenge this response answers. challenge_id: u64, @@ -923,56 +1060,79 @@ pub enum RejectKind { /// timeout lane (holder credit revoked, no trust penalty). Transient, /// Any other rejection (wrong target peer, no commitment state, malformed - /// proof plan, oversized byte challenge, …). CONFIRMED failure. + /// proof plan, oversized slice challenge, …). CONFIRMED failure. Protocol, } -/// Round 2 of the storage audit (ADR-0002): the **surprise byte challenge**. +/// A single block the round-2 slice challenge opens: which committed key, and +/// which 1 KiB block within it. +/// +/// The `block_index` is drawn with FRESH auditor randomness after round 1 (never +/// nonce-derived), so the responder cannot have prepared only the opened block. +#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)] +pub struct SubtreeSliceOpening { + /// The committed key whose block is opened. + pub key: XorName, + /// The 1 KiB block index within the chunk, in `0..block_count(content_len)`. + pub block_index: u32, +} + +/// Round 2 of the storage audit (ADR-0002 / V2-685): the **surprise slice +/// challenge**. /// -/// After the auditor has structurally verified a [`SubtreeAuditResponse::Proof`] -/// it picks a small sample of that subtree's just-proven leaves with FRESH -/// randomness (chosen now, after the proof is committed — NOT derived from the -/// round-1 nonce, so the responder could not have predicted it at proof-build -/// time) and asks the responder to return the ORIGINAL chunk bytes for exactly -/// those keys. The auditor then checks each returned chunk against the committed -/// leaf: -/// - `BLAKE3(bytes) == leaf.bytes_hash` (the chunk's content address), AND -/// - `compute_audit_digest(nonce, peer, key, bytes) == leaf.nonced_hash`. +/// After structurally verifying a [`SubtreeAuditResponse::Proof`] the auditor +/// picks a small sample of the just-proven leaves with FRESH randomness (chosen +/// now, after the proof is committed — NOT derived from the round-1 nonce) and, +/// for each, a fresh-random 1 KiB block index. It asks the responder to open +/// exactly those blocks. For each opened block the responder returns a Bao +/// verified slice plus a nonced block-tree opening; the auditor checks, over the +/// same block bytes: +/// - the Bao slice against `leaf.bytes_hash` (the chunk's content address), AND +/// - the nonced opening against `leaf.nonced_root` (round-1 possession commit). /// -/// This makes possession non-delegable to the auditor: the auditor needs to -/// hold NONE of the responder's chunks. A responder that committed to a chunk it -/// no longer holds cannot fabricate bytes that hash to the committed address (a -/// preimage break), so it is caught regardless of who audits it. +/// This makes possession non-delegable *and* cheap to prove: the response is a +/// few KB, not up to two 4 MiB chunks. A responder that did not hold the bytes at +/// round-1 commit time cannot have committed a correct `nonced_root`, and cannot +/// fold an after-the-fact-fetched block to a foreign root without a preimage +/// break — so it is caught regardless of who audits it. #[derive(Debug, Clone, Serialize, Deserialize)] -pub struct SubtreeByteChallenge { +pub struct SubtreeSliceChallenge { /// The same `challenge_id` as the round-1 [`SubtreeAuditChallenge`], so the /// responder/auditor correlate the two rounds. pub challenge_id: u64, - /// The same nonce as round 1 — needed for the freshness (`nonced_hash`) - /// check and to bind these bytes to this audit. + /// The same nonce as round 1 — binds each nonced block opening to this audit. pub nonce: [u8; 32], - /// The challenged peer ID (bound into each leaf's possession hash). + /// The challenged peer ID (bound into every nonced block leaf). pub challenged_peer_id: [u8; 32], /// The pinned commitment hash from round 1, so the responder resolves the - /// SAME tree it just proved and serves bytes only for keys it committed to. + /// SAME tree it just proved and opens blocks only for keys it committed to. pub expected_commitment_hash: [u8; 32], - /// The exact keys whose original bytes the responder must return. These are - /// the auditor's freshly-randomised spot-check sample of the round-1 subtree - /// (chosen after the proof was received; not nonce-derived). - pub keys: Vec, + /// The exact blocks to open: the auditor's freshly-randomised spot-check + /// sample of the round-1 subtree (chosen after the proof was received; not + /// nonce-derived), up to two blocks per sampled leaf (a fresh-random block + /// plus the final block, for the content-length pin). + pub openings: Vec, } -/// One requested chunk in a [`SubtreeByteResponse`]. +/// One opened block in a [`SubtreeSliceResponse`]. #[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)] -pub enum SubtreeByteItem { - /// The responder holds this committed key and returns its original bytes. +pub enum SubtreeSliceItem { + /// The responder holds this committed key and opens the requested block. Present { /// The requested key. key: XorName, - /// The original chunk bytes (the auditor re-hashes to verify). - bytes: Vec, + /// The 1 KiB block index that was opened. + block_index: u32, + /// Bao verified slice: the block bytes plus the BLAKE3 parent hashes that + /// authenticate them against the chunk address. The auditor decodes this + /// to recover the verified block bytes. + bao_slice: Vec, + /// Sibling hashes on the path from this block up to the committed + /// `nonced_root`, bottom-up. The auditor folds the block leaf with these + /// to prove the block was committed under round 1's fresh nonce. + nonced_siblings: Vec<[u8; 32]>, }, - /// The responder committed to this key but cannot serve its bytes. This is a + /// The responder committed to this key but cannot serve its block. This is a /// PROVABLE cheat (it published a commitment over a chunk it does not hold), /// so the auditor counts it as a confirmed failure — NOT a graced timeout. /// Distinguishing this explicit signal from silence is what separates a @@ -983,29 +1143,43 @@ pub enum SubtreeByteItem { }, } -/// Response to a [`SubtreeByteChallenge`] (round 2). One item per requested key, -/// in the requested order. +/// Response to a [`SubtreeSliceChallenge`] (round 2). +/// +/// The contract is **coalesced and order-independent**: the responder groups the +/// requested openings by key (reading and hashing each chunk once), so +/// - a `Present` item is unique per distinct `(key, block_index)`; +/// - an `Absent` item is at most one per key and covers ALL of that key's +/// requested openings (it has no `block_index`); +/// - duplicate requested openings do not multiply work or response items; +/// - item order is unspecified — the auditor matches by identity, not position. +/// +/// The auditor rejects a response whose identities collide (a duplicate +/// `(key, block_index)`, or a key that is both `Present` and `Absent`) as +/// malformed, so first-match ambiguity cannot decide a verdict. /// -/// Sizing rule: a challenge carries at most -/// [`MAX_BYTE_CHALLENGE_KEYS`](super::config::MAX_BYTE_CHALLENGE_KEYS) keys — -/// the auditor batches its sample, the responder rejects larger requests — so -/// the WORST-CASE `Items` response (every chunk at `MAX_CHUNK_SIZE`) always -/// encodes under [`MAX_REPLICATION_MESSAGE_SIZE`]. +/// Each item is a few KB (a 1 KiB block plus O(log n) hashes on two short +/// chains), so even the worst-case sample fits far under +/// [`MAX_REPLICATION_MESSAGE_SIZE`] with no batching — the auditor bounds the +/// sample to [`MAX_SLICE_OPENINGS`](super::config::MAX_SLICE_OPENINGS) and the +/// responder rejects larger requests. #[derive(Debug, Clone, Serialize, Deserialize)] -pub enum SubtreeByteResponse { - /// The responder's per-key answers (bytes or an explicit absent signal). +pub enum SubtreeSliceResponse { + /// The responder's per-opening answers (a verified slice or an absent signal). Items { /// The challenge this response answers. challenge_id: u64, - /// One entry per requested key. - items: Vec, + /// One item per DISTINCT requested opening, coalesced and + /// order-independent (see the type-level contract above): duplicate + /// openings do not multiply items, and one `Absent` covers all of a + /// key's openings. + items: Vec, }, /// Peer is still bootstrapping (should not happen mid-audit, but handled). Bootstrapping { /// The challenge this response answers. challenge_id: u64, }, - /// The responder rejects the byte challenge outright. `kind` drives the + /// The responder rejects the slice challenge outright. `kind` drives the /// auditor's accounting (ADR-0004 A1: grace removed): [`RejectKind::Transient`] /// routes to the timeout lane (no trust penalty, holder credit revoked); every /// other kind is a confirmed failure, like round 1. @@ -1023,11 +1197,27 @@ pub enum SubtreeByteResponse { // Audit digest helper // --------------------------------------------------------------------------- -/// Compute `AuditKeyDigest(K_i) = BLAKE3(nonce || challenged_peer_id || K_i || record_bytes_i)`. +/// Domain-separation context for the audit digest key derivation. /// -/// Returns the 32-byte BLAKE3 digest binding the nonce, peer identity, key, -/// and record content together so a peer cannot forge proofs without holding -/// the actual data. +/// Versioned alongside the possession-audit protocol id so no other BLAKE3 use +/// in this codebase, and no future revision of this digest, can derive a +/// colliding key from the same challenge material. +pub const AUDIT_DIGEST_KEY_CONTEXT: &str = + "autonomi.ant.replication.possession-audit.v2.digest-key"; + +/// Compute the possession digest for one challenged key. +/// +/// `BLAKE3_keyed(BLAKE3_derive_key(ctx, nonce || challenged_peer_id || K_i), record_bytes_i)`, +/// with `ctx` the versioned [`AUDIT_DIGEST_KEY_CONTEXT`]. Returns the 32-byte +/// digest binding the nonce, peer identity, key, and record content together so +/// a peer cannot produce it without holding the actual data. +/// +/// The challenge material enters as the BLAKE3 *key*, not as an input prefix. +/// In keyed mode the key replaces the IV of every block's compression, so the +/// whole digest depends on the fresh per-audit nonce rather than only its +/// leading bytes. This matches the construction the subtree audit already uses +/// for its per-leaf commitments, so every audit lane binds the nonce the same +/// way. #[must_use] pub fn compute_audit_digest( nonce: &[u8; 32], @@ -1035,12 +1225,12 @@ pub fn compute_audit_digest( key: &XorName, record_bytes: &[u8], ) -> [u8; 32] { - let mut hasher = blake3::Hasher::new(); - hasher.update(nonce); - hasher.update(challenged_peer_id); - hasher.update(key); - hasher.update(record_bytes); - *hasher.finalize().as_bytes() + let mut key_hasher = blake3::Hasher::new_derive_key(AUDIT_DIGEST_KEY_CONTEXT); + key_hasher.update(nonce); + key_hasher.update(challenged_peer_id); + key_hasher.update(key); + let digest_key = *key_hasher.finalize().as_bytes(); + *blake3::keyed_hash(&digest_key, record_bytes).as_bytes() } // --------------------------------------------------------------------------- @@ -1130,7 +1320,7 @@ mod tests { for (kind, kind_index) in [ (AuditDropKind::Responsible, 0usize), (AuditDropKind::Subtree, 1usize), - (AuditDropKind::Byte, 2usize), + (AuditDropKind::Slice, 2usize), ] { let before = AUDIT_DROPPED[kind_index].load(Ordering::Relaxed); record_audit_drop(kind); @@ -1141,31 +1331,324 @@ mod tests { } } - // === Round-2 byte response sizing === + // === Round-2 slice response sizing === #[test] - fn max_batch_worst_case_byte_response_fits_wire_cap() { - // The auditor batches its round-2 sample to MAX_BYTE_CHALLENGE_KEYS per - // challenge precisely so this worst case — every requested chunk at - // MAX_CHUNK_SIZE — still encodes. If this fails, honest responders - // would hit encode errors and fail otherwise valid byte challenges. - let items: Vec = (0..crate::replication::config::MAX_BYTE_CHALLENGE_KEYS) - .map(|i| SubtreeByteItem::Present { + fn max_slice_response_is_tiny_relative_to_wire_cap() { + // A worst-case round-2 slice response is MAX_SLICE_OPENINGS openings, each + // a 1 KiB block plus a Bao proof and a nonced sibling chain. For a 4 MiB + // chunk that is ~4096 BLAKE3 chunks → ~12 parent hashes per chain. We + // overestimate generously (16 KiB slice + 24 sibling hashes per opening) + // and assert it encodes far under the wire cap — the whole point of the + // change is that round 2 is now KB-scale, not up to 8 MiB. + let items: Vec = (0..crate::replication::config::MAX_SLICE_OPENINGS) + .map(|i| SubtreeSliceItem::Present { key: [u8::try_from(i).unwrap_or(u8::MAX); 32], - bytes: vec![0xAB; crate::ant_protocol::MAX_CHUNK_SIZE], + block_index: u32::try_from(i).unwrap_or(u32::MAX), + bao_slice: vec![0xAB; 16 * 1024], + nonced_siblings: vec![[0x5A; 32]; 24], }) .collect(); let msg = ReplicationMessage { request_id: 7, - body: ReplicationMessageBody::SubtreeByteResponse(SubtreeByteResponse::Items { + body: ReplicationMessageBody::SubtreeSliceResponse(SubtreeSliceResponse::Items { challenge_id: 7, items, }), }; let encoded = msg .encode() - .expect("worst-case max-batch byte response must fit the wire cap"); + .expect("worst-case slice response must fit the wire cap"); + // Comfortably under 1 MiB, itself a fraction of the 10 MiB wire cap. + assert!(encoded.len() <= 1024 * 1024); assert!(encoded.len() <= MAX_REPLICATION_MESSAGE_SIZE); + assert!( + encoded.len() <= crate::replication::config::MAX_AUDIT_MESSAGE_SIZE, + "round 2 must fit the tighter audit-family ceiling" + ); + } + + // The audit families take a much tighter wire ceiling than the 10 MiB core + // one, checked before decoding so an unknown peer cannot make this node + // allocate megabytes before any admission check has run. That ceiling is + // only safe if it clears the largest body an honest audit can produce, which + // is the round-1 proof at the commitment key-count cap: 1,024 leaves plus + // the sibling cut hashes and the signed commitment. Build that worst case + // and pin the headroom, so a later change to leaf size or commitment + // encoding cannot quietly start dropping honest proofs. + #[test] + fn max_round1_proof_fits_the_audit_family_ceiling() { + use crate::replication::commitment::{StorageCommitment, MAX_COMMITMENT_KEY_COUNT}; + use crate::replication::config::MAX_AUDIT_MESSAGE_SIZE; + use crate::replication::subtree::{max_subtree_leaves, SubtreeLeaf, SubtreeProof}; + + let leaf_count = max_subtree_leaves(MAX_COMMITMENT_KEY_COUNT) as usize; + let leaves: Vec = (0..leaf_count) + .map(|_| SubtreeLeaf { + key: [0xAB; 32], + bytes_hash: [0xCD; 32], + content_len: u32::MAX, + nonced_root: [0xEF; 32], + }) + .collect(); + let msg = ReplicationMessage { + request_id: 11, + body: ReplicationMessageBody::SubtreeAuditResponse(SubtreeAuditResponse::Proof { + challenge_id: 11, + commitment: StorageCommitment { + root: [0x01; 32], + key_count: MAX_COMMITMENT_KEY_COUNT, + sender_peer_id: [0x02; 32], + // Over-sized stand-ins for the ML-DSA-65 key and signature. + sender_public_key: vec![0x03; 4096], + signature: vec![0x04; 8192], + }, + proof: SubtreeProof { + leaves, + // One per level down to the subtree root; 32 is far past the + // real depth at this key count. + sibling_cut_hashes: vec![[0x05; 32]; 32], + }, + }), + }; + let encoded = msg + .encode() + .expect("worst-case round-1 proof must fit the wire cap"); + assert!( + encoded.len() <= MAX_AUDIT_MESSAGE_SIZE, + "worst legitimate round-1 proof is {} bytes, over the audit ceiling of \ + {MAX_AUDIT_MESSAGE_SIZE}", + encoded.len() + ); + } + + /// Every body variant, in declaration order, so a test can walk the whole + /// enum rather than a hand-picked sample. + fn all_bodies() -> Vec { + let z = [0u8; 32]; + vec![ + ReplicationMessageBody::FreshReplicationOffer(FreshReplicationOffer { + key: z, + data: vec![], + proof_of_payment: vec![], + }), + ReplicationMessageBody::FreshReplicationResponse(FreshReplicationResponse::Accepted { + key: z, + }), + ReplicationMessageBody::PaidNotify(PaidNotify { + key: z, + proof_of_payment: vec![], + }), + ReplicationMessageBody::NeighborSyncRequest(NeighborSyncRequest { + replica_hints: vec![], + paid_hints: vec![], + bootstrapping: false, + commitment: None, + }), + ReplicationMessageBody::NeighborSyncResponse(NeighborSyncResponse { + replica_hints: vec![], + paid_hints: vec![], + bootstrapping: false, + rejected_keys: vec![], + commitment: None, + }), + ReplicationMessageBody::VerificationRequest(VerificationRequest { + keys: vec![], + paid_list_check_indices: vec![], + }), + ReplicationMessageBody::VerificationResponse(VerificationResponse { results: vec![] }), + ReplicationMessageBody::FetchRequest(FetchRequest { key: z }), + ReplicationMessageBody::FetchResponse(FetchResponse::NotFound { key: z }), + ReplicationMessageBody::AuditChallenge(AuditChallenge { + challenge_id: 1, + nonce: z, + challenged_peer_id: z, + keys: vec![], + }), + ReplicationMessageBody::AuditResponse(AuditResponse::Digests { + challenge_id: 1, + digests: vec![], + }), + ReplicationMessageBody::SubtreeAuditChallenge(SubtreeAuditChallenge { + challenge_id: 1, + nonce: z, + challenged_peer_id: z, + expected_commitment_hash: z, + }), + ReplicationMessageBody::SubtreeAuditResponse(SubtreeAuditResponse::Bootstrapping { + challenge_id: 1, + }), + ReplicationMessageBody::SubtreeSliceChallenge(SubtreeSliceChallenge { + challenge_id: 1, + nonce: z, + challenged_peer_id: z, + expected_commitment_hash: z, + openings: vec![], + }), + ReplicationMessageBody::SubtreeSliceResponse(SubtreeSliceResponse::Bootstrapping { + challenge_id: 1, + }), + ReplicationMessageBody::GetCommitmentByPin(GetCommitmentByPin { pin: z }), + ReplicationMessageBody::GetCommitmentByPinResponse( + GetCommitmentByPinResponse::NotRetained { pin: z }, + ), + ] + } + + // The pre-decode ceiling reads the body's discriminant straight off the + // wire, so `peek_variant_index` MUST agree with `variant_index()` for every + // variant, at request ids that straddle postcard's varint width boundaries. + // If it ever disagreed, a body would be sized against the wrong family's + // ceiling — which is the bug this whole path exists to prevent. + #[test] + fn peeked_discriminant_matches_the_decoded_one_for_every_variant() { + // 1-byte, 2-byte, 5-byte and 10-byte `request_id` varints. + for request_id in [0u64, 1, 127, 128, 300, u64::from(u32::MAX), u64::MAX] { + for body in all_bodies() { + let expected = body.variant_index(); + let encoded = ReplicationMessage { request_id, body } + .encode() + .expect("encode"); + assert_eq!( + peek_variant_index(&encoded), + Some(expected), + "peek disagreed with variant_index at request_id {request_id}" + ); + } + } + } + + // A truncated or nonsense prefix must not classify as `Core`, because Core + // is the LENIENT ceiling. Failing to classify has to mean "apply the strict + // one", never "let it through". + #[test] + fn unclassifiable_prefix_yields_no_family() { + assert_eq!(peek_variant_index(&[]), None); + // A varint that never terminates: all continuation bits set. + assert_eq!(peek_variant_index(&[0xFF; 32]), None); + // A well-formed request_id with nothing after it. + assert_eq!(peek_variant_index(&[0x01]), None); + } + + // Regression (dirvine, PR #181): the pre-decode ceiling used to be chosen + // from the protocol id the message arrived on, so an audit body addressed to + // the CORE id skipped the audit ceiling and was decoded under the 10 MiB core + // allowance — its collections allocated — before the family guard dropped it. + // + // The reproduction from that review: a valid `SubtreeSliceChallenge` with + // 200,000 openings encodes to ~6.6 MB, far past the 512 KiB audit ceiling. + // Classifying by discriminant catches it wherever it is addressed. + #[test] + fn oversized_audit_body_is_classified_as_audit_wherever_it_is_addressed() { + let z = [0u8; 32]; + let challenge = SubtreeSliceChallenge { + challenge_id: 1, + nonce: z, + challenged_peer_id: z, + expected_commitment_hash: z, + openings: (0..200_000u32) + .map(|i| SubtreeSliceOpening { + key: z, + block_index: i, + }) + .collect(), + }; + let encoded = ReplicationMessage { + request_id: 1, + body: ReplicationMessageBody::SubtreeSliceChallenge(challenge), + } + .encode() + .expect("encode"); + + assert!( + encoded.len() > crate::replication::config::MAX_AUDIT_MESSAGE_SIZE, + "the reproduction must exceed the audit ceiling to be meaningful; got {} bytes", + encoded.len() + ); + // The classification the receive path performs, before any decode. + let family = peek_variant_index(&encoded).map(family_of_variant); + assert_eq!(family, Some(BodyFamily::SubtreeAudit)); + assert!( + family.is_none_or(BodyFamily::is_audit), + "an audit body must take the audit ceiling regardless of the id it rode" + ); + } + + // The family table is the single source of truth: the `&self` predicates, + // the response routing and the pre-decode ceiling all read it, so they + // cannot drift. Walk every variant and check both views agree. + #[test] + fn family_table_agrees_with_the_body_predicates() { + for body in all_bodies() { + let family = family_of_variant(body.variant_index()); + assert_eq!(body.is_subtree_audit(), family == BodyFamily::SubtreeAudit); + assert_eq!( + body.is_possession_audit(), + family == BodyFamily::PossessionAudit + ); + assert_eq!( + family.is_audit(), + body.is_subtree_audit() || body.is_possession_audit() + ); + } + } + + // `is_subtree_audit()` classifies exactly the four subtree-audit variants + // (both rounds), and NOTHING else — crucially not the digest-based + // `AuditChallenge`/`AuditResponse` pair, which rides its own + // `POSSESSION_AUDIT_PROTOCOL_ID` via `is_possession_audit()`. The receive + // guard and the response-routing both key off this one predicate, so it + // must be exact: a body that answers `true` here is required to arrive on + // the subtree id, and is dropped on any other. + #[test] + fn is_subtree_audit_covers_both_rounds_only() { + let z = [0u8; 32]; + let audit = [ + ReplicationMessageBody::SubtreeAuditChallenge(SubtreeAuditChallenge { + challenge_id: 1, + nonce: z, + challenged_peer_id: z, + expected_commitment_hash: z, + }), + ReplicationMessageBody::SubtreeAuditResponse(SubtreeAuditResponse::Bootstrapping { + challenge_id: 1, + }), + ReplicationMessageBody::SubtreeSliceChallenge(SubtreeSliceChallenge { + challenge_id: 1, + nonce: z, + challenged_peer_id: z, + expected_commitment_hash: z, + openings: vec![], + }), + ReplicationMessageBody::SubtreeSliceResponse(SubtreeSliceResponse::Bootstrapping { + challenge_id: 1, + }), + ]; + for body in &audit { + assert!( + body.is_subtree_audit(), + "variant {} must be subtree-audit", + body.variant_index() + ); + } + let core = [ + ReplicationMessageBody::FreshReplicationOffer(FreshReplicationOffer { + key: z, + data: vec![], + proof_of_payment: vec![], + }), + // Periodic possession audit — NOT the subtree audit. It rides the + // possession id, not the core one; what matters here is only that + // `is_subtree_audit()` does not claim it. + ReplicationMessageBody::AuditResponse(AuditResponse::Bootstrapping { challenge_id: 1 }), + ]; + for body in &core { + assert!( + !body.is_subtree_audit(), + "variant {} must be core, not subtree-audit", + body.variant_index() + ); + } } // === Fresh Replication roundtrip === @@ -1693,6 +2176,46 @@ mod tests { // === Audit digest computation === + #[test] + fn audit_digest_is_the_domain_separated_keyed_construction() { + // Pin the exact KEYED construction: the challenge material enters as the + // BLAKE3 key (via `derive_key`, mixed into every block's compression), + // never as an input prefix. A regression to a flat + // `BLAKE3(nonce || peer || key || bytes)` produces a different digest and + // fails this exact-equality check. The context comes from the single + // source of truth so the test cannot drift from the implementation. + // A multi-block record exercises the key mixing past the first block. + let nonce = [0x01; 32]; + let peer_id = [0x02; 32]; + let key: XorName = [0x03; 32]; + let record_bytes = vec![0x5A; 8 * 1024]; + + let mut key_hasher = blake3::Hasher::new_derive_key(AUDIT_DIGEST_KEY_CONTEXT); + key_hasher.update(&nonce); + key_hasher.update(&peer_id); + key_hasher.update(&key); + let digest_key = *key_hasher.finalize().as_bytes(); + let expected = *blake3::keyed_hash(&digest_key, &record_bytes).as_bytes(); + + assert_eq!( + compute_audit_digest(&nonce, &peer_id, &key, &record_bytes), + expected, + "digest must be BLAKE3_keyed(derive_key(ctx, nonce||peer||key), bytes)" + ); + + // And it must NOT be the superseded flat prefix hash. + let mut flat = blake3::Hasher::new(); + flat.update(&nonce); + flat.update(&peer_id); + flat.update(&key); + flat.update(&record_bytes); + assert_ne!( + compute_audit_digest(&nonce, &peer_id, &key, &record_bytes), + *flat.finalize().as_bytes(), + "keyed digest must differ from the flat prefix construction" + ); + } + #[test] fn audit_digest_is_deterministic() { let nonce = [0x01; 32]; diff --git a/src/replication/pruning.rs b/src/replication/pruning.rs index 87bf1f26..f58d3727 100644 --- a/src/replication/pruning.rs +++ b/src/replication/pruning.rs @@ -55,7 +55,8 @@ use crate::ant_protocol::XorName; use crate::replication::commitment_state::ResponderCommitmentState; use crate::replication::config::{ storage_admission_width, ReplicationConfig, AUDIT_FAILURE_TRUST_WEIGHT, - MAX_PRUNE_AUDIT_CHALLENGES_PER_PASS, REPLICATION_PROTOCOL_ID, + GRACE_POSSESSION_AUDIT_TIMEOUTS, MAX_PRUNE_AUDIT_CHALLENGES_PER_PASS, + POSSESSION_AUDIT_PROTOCOL_ID, }; use crate::replication::paid_list::PaidList; use crate::replication::protocol::{ @@ -1377,25 +1378,43 @@ async fn peer_proves_records( else { return Vec::new(); }; - let Some(decoded) = - send_prune_audit_challenge(&peer, encoded, key_count, p2p_node, config).await - else { - // No decoded response means a timeout or malformed reply. Prune - // confirmation reuses `AuditChallenge` semantics, so this is an immediate - // audit failure just like a decoded bad proof below. Keep the historical - // one-report-per-peer-per-pass guard by attempting each key against the - // shared `report_state`. - let mut audit_failure_reported = false; - for key in &challenge_keys { - if report_prune_audit_failure_once(&peer, key, p2p_node, config, report_state).await { - audit_failure_reported = true; - break; + let decoded = match send_prune_audit_challenge(&peer, encoded, key_count, p2p_node, config) + .await + { + Ok(decoded) => decoded, + // No usable response. Prune confirmation reuses `AuditChallenge` + // semantics, so this is an immediate audit failure just like a decoded + // bad proof below. Keep the historical one-report-per-peer-per-pass + // guard by attempting each key against the shared `report_state`. + // + // ROLLOUT GATE — see `GRACE_POSSESSION_AUDIT_TIMEOUTS`. A peer on the + // far side of the possession-audit protocol move never answers, so an + // unanswered challenge is not evidence about it. Only that case is + // graced: a peer that DID answer with undecodable bytes is a real + // protocol failure, not a version skew, and is still penalised. + Err(failure) => { + if !prune_send_failure_is_penalised(failure) { + debug!( + "Prune audit: {peer} did not answer; graced during the \ + possession-audit protocol rollout (not penalised)" + ); + return Vec::new(); } + let mut audit_failure_reported = false; + for key in &challenge_keys { + if report_prune_audit_failure_once(&peer, key, p2p_node, config, report_state).await + { + audit_failure_reported = true; + break; + } + } + if audit_failure_reported { + debug!( + "Prune audit: reported one failure for unanswered/malformed batch from {peer}" + ); + } + return Vec::new(); } - if audit_failure_reported { - debug!("Prune audit: reported one failure for timed-out/malformed batch from {peer}"); - } - return Vec::new(); }; let statuses = prune_audit_response_statuses(decoded, challenge_id, &peer, &challenge_material); @@ -1480,34 +1499,70 @@ fn encode_prune_audit_challenge( Some((encoded, key_count)) } +/// Why a prune-audit challenge yielded no usable response. +/// +/// The two cases must stay distinguishable: only `Unanswered` is graced by the +/// [`GRACE_POSSESSION_AUDIT_TIMEOUTS`] rollout gate. Collapsing both into one +/// "no response" case would let the gate also excuse a matched-version peer that +/// answered with garbage, which is a real protocol failure, not a version skew. +#[derive(Clone, Copy)] +#[cfg_attr(test, derive(Debug, PartialEq, Eq))] +enum PruneAuditSendFailure { + /// No answer within the deadline, or a transport error. Not evidence about + /// the peer's storage — and the expected outcome for a peer on the far side + /// of the possession-audit protocol move. + Unanswered, + /// The peer answered, but the bytes did not decode as a replication message. + /// A matched-version peer should never do this. + Malformed, +} + +/// Whether a prune-audit send failure is penalised at audit severity, or graced +/// by the [`GRACE_POSSESSION_AUDIT_TIMEOUTS`] rollout gate. +/// +/// Silence is graced while the gate is set, because a peer on the far side of +/// the possession-audit protocol move never answers and its silence says nothing +/// about its storage. An undecodable *answer* is not silence: the peer replied, +/// so it is on this protocol and its reply is simply wrong. That stays a +/// confirmed failure throughout, which is what keeps the gate from being an +/// amnesty rather than a grace. +/// +/// Extracted as a predicate so the rollout decision is assertable directly, +/// rather than only reachable through a live prune pass. +#[must_use] +fn prune_send_failure_is_penalised(failure: PruneAuditSendFailure) -> bool { + match failure { + PruneAuditSendFailure::Unanswered => !GRACE_POSSESSION_AUDIT_TIMEOUTS, + PruneAuditSendFailure::Malformed => true, + } +} + async fn send_prune_audit_challenge( peer: &PeerId, encoded: Vec, key_count: usize, p2p_node: &Arc, config: &ReplicationConfig, -) -> Option { +) -> std::result::Result { let timeout = config.audit_response_timeout(key_count); let response = match p2p_node - .send_request(peer, REPLICATION_PROTOCOL_ID, encoded, timeout) + .send_request(peer, POSSESSION_AUDIT_PROTOCOL_ID, encoded, timeout) .await { Ok(response) => response, Err(e) => { debug!("Prune audit challenge with {key_count} keys against {peer} failed: {e}"); - return None; + return Err(PruneAuditSendFailure::Unanswered); } }; - let decoded = match ReplicationMessage::decode(&response.data) { - Ok(msg) => msg, + match ReplicationMessage::decode(&response.data) { + Ok(msg) => Ok(msg), Err(e) => { warn!("Failed to decode prune audit response from {peer}: {e}"); - return None; + Err(PruneAuditSendFailure::Malformed) } - }; - - Some(decoded) + } } fn prune_audit_response_statuses( @@ -1773,6 +1828,38 @@ async fn peer_is_currently_responsible( mod tests { use super::*; + // Regression (dirvine, PR #181): the prune lane collapsed "no answer" and + // "answered with undecodable bytes" into a single `None`, so the rollout + // gate excused both. The implementation now distinguishes them, and THIS is + // the test that holds the distinction in place — the generic + // `rollout_gate_graces_only_timeouts` test lives on the responsible-audit + // predicate and passed happily while the prune collapse was still present, + // so it could never have caught this. + // + // The gate grants grace to silence, not to a wrong answer. A peer that + // replies at all is on this protocol, so a reply that does not decode is a + // real protocol failure and stays penalised for the whole upgrade window. + #[test] + fn prune_rollout_gate_graces_silence_but_never_a_bad_answer() { + assert!( + !prune_send_failure_is_penalised(PruneAuditSendFailure::Unanswered), + "an unanswered prune challenge must be graced while the rollout gate is set" + ); + assert!( + prune_send_failure_is_penalised(PruneAuditSendFailure::Malformed), + "an undecodable REPLY is a protocol failure, not a version skew — always penalised" + ); + + // Tie the first assertion to the gate rather than to a constant, so the + // follow-up that removes the gate flips this test instead of leaving it + // silently asserting the wrong thing. + assert_eq!( + prune_send_failure_is_penalised(PruneAuditSendFailure::Unanswered), + !GRACE_POSSESSION_AUDIT_TIMEOUTS, + "silence is penalised exactly when the rollout gate is off" + ); + } + fn peer_id_from_byte(b: u8) -> PeerId { let mut bytes = [0u8; 32]; bytes[0] = b; diff --git a/src/replication/recent_provers.rs b/src/replication/recent_provers.rs index b793c228..e10be92c 100644 --- a/src/replication/recent_provers.rs +++ b/src/replication/recent_provers.rs @@ -44,8 +44,9 @@ use crate::ant_protocol::XorName; /// Maximum number of cached provers per key. /// -/// Sized at 2× `CLOSE_GROUP_SIZE = 8`, giving 8 slack slots for churn -/// without unbounded growth. LRU-evicted within the cap. +/// Comfortably above `CLOSE_GROUP_SIZE = 7` (a key has at most 7 responsible +/// holders), leaving slack slots for churn without unbounded growth. +/// LRU-evicted within the cap. pub const MAX_PROVERS_PER_KEY: usize = 16; /// Maximum age of a cached prover entry before it is considered stale. @@ -312,6 +313,36 @@ mod tests { assert!(cache.is_credited_holder(&key(1), &peer(2), &hash(0xAB))); } + #[test] + fn forget_peer_drops_credit_across_all_commitments() { + // A confirmed audit failure (e.g. a `ResponsiveBootstrap` on a stale pin + // H1) revokes ALL of the peer's holder credit, INCLUDING credit earned + // under a newer commitment H2: the contradiction is identity-level, not + // scoped to one key set. Another peer's credit is untouched. + let mut cache = RecentProvers::new(); + let now = Instant::now(); + let h1 = hash(0x11); + let h2 = hash(0x22); + cache.record_proof(key(1), peer(1), h1, now); + cache.record_proof(key(2), peer(1), h2, now); + cache.record_proof(key(1), peer(2), h1, now); + + cache.forget_peer(&peer(1)); + + assert!( + !cache.is_credited_holder(&key(1), &peer(1), &h1), + "H1 credit dropped" + ); + assert!( + !cache.is_credited_holder(&key(2), &peer(1), &h2), + "H2 credit dropped too (peer-wide, not pin-scoped)" + ); + assert!( + cache.is_credited_holder(&key(1), &peer(2), &h1), + "another peer's credit is preserved" + ); + } + #[test] fn forget_commitment_drops_only_matching_entries() { let mut cache = RecentProvers::new(); diff --git a/src/replication/slice.rs b/src/replication/slice.rs new file mode 100644 index 00000000..a8ea28d6 --- /dev/null +++ b/src/replication/slice.rs @@ -0,0 +1,1076 @@ +//! BLAKE3 verified-slice possession proofs for the storage-commitment audit +//! (ADR-0002 round 2, V2-685). +//! +//! The audit's round 2 used to make a challenged peer return the **complete +//! original bytes** of the sampled chunks (up to two 4 MiB chunks per response), +//! at 5.8 to 6.2 MB per response measured in production. What that cost the +//! fleet per day moved with the audit *rate*, not the response size, so +//! frequency caps could move the daily total but never the cost of an +//! individual audit. This module replaces the response shape instead: one +//! opened 1 KiB block is a few KB rather than a full chunk, about 427x smaller, +//! and that holds whatever the rate does. +//! +//! Two axes move in opposite directions, and it is worth separating them. The +//! *freshness* construction gets strictly stronger: the nonce now enters every +//! block leaf, closing a preprocessing gap in the old flat `nonced_hash`. The +//! *coverage* gets weaker: round 2 samples blocks instead of checking every +//! byte. See "What a pass does and does not prove" below — that trade is the +//! whole design, not a footnote to it. +//! +//! ## Why two chains +//! +//! A chunk's address is `BLAKE3(content)` (its content hash), and BLAKE3 is +//! internally a Merkle tree over 1 KiB blocks, so a **Bao verified slice** proves +//! that a given block is the real content at a given offset *against the address +//! the auditor already knows* — content authenticity, for free, no full chunk. +//! But authenticity alone is checkable from purely public data (the address is +//! public), so a node that stores nothing could pass it by fetching the block on +//! demand from an honest holder. To bind **possession at commitment time** the +//! responder also commits, per leaf in round 1, a fresh **nonced block tree**: +//! a Merkle root over the same 1 KiB blocks whose leaves are **keyed** BLAKE3 +//! hashes, `keyed_BLAKE3(key=f(nonce ‖ peer ‖ key), block_index ‖ len ‖ block)`. +//! +//! The nonce is fed as the BLAKE3 **key**, not a message prefix, so it mixes +//! into every chunk compression of every block leaf: there is no +//! nonce-independent chaining value a responder can precompute across audits. +//! (A plain `BLAKE3(nonce ‖ … ‖ block)` prefix would leave the block's tail in a +//! second, nonce-free BLAKE3 chunk whose CV is precomputable, letting a node +//! store ~10.7% less than each block and still reconstruct the leaf for any +//! fresh nonce — a smaller version of the old flat-`nonced_hash` gap.) Building +//! the *correct* `nonced_root` therefore requires all of a chunk's bytes at +//! round-1 commit time, and the auditor picks which block to open with fresh +//! randomness *after* the roots are committed (cut-and-choose): a responder +//! cannot connect a real, after-the-fact-fetched block to a garbage committed +//! root without a preimage break. +//! +//! In round 2 the auditor verifies both chains over the **same** block bytes: +//! the Bao chain against the address (authenticity) and the nonced chain against +//! the round-1 `nonced_root` (possession). Either failing is a confirmed cheat. +//! +//! ## What a pass does and does not prove +//! +//! This is a **sampling** check, and the strength claim should be read as such. +//! The auditor holds none of the audited bytes, so it cannot tell a correct +//! `nonced_root` from a responder-chosen one. A peer holding a fraction `p` of a +//! chunk's blocks can commit a root built from real leaves for the blocks it +//! kept and arbitrary leaves for the rest, and it passes whenever every fresh +//! draw lands on a kept block. Per audit that is roughly `p^leaves` over the +//! 3..=5 sampled leaves; the mandatory final-block opening pins the claimed +//! length but adds no detection, since a partial holder simply keeps the final +//! block. +//! +//! So a pass means "held the sampled blocks at commit time", and repeated audits +//! drive the probability of sustained under-storage down geometrically rather +//! than catching it outright on the first try. The full-byte round 2 this +//! replaces caught any missing byte of a sampled chunk with certainty — that is +//! the difference, and it is inherent: verifying every byte means moving every +//! byte. +//! +//! For `p^leaves` to describe *detection* rather than merely a pass, declining +//! to answer must not be cheaper than answering. The responder sees the drawn +//! blocks before it replies, so an unanswered round 2 following a valid round-1 +//! proof revokes the holder credit for the commitment under audit (see +//! `storage_commitment_audit::round2_revokes_pinned_credit`). It stays in the +//! graced timeout lane and costs no trust, so an honest dropped reply is +//! harmless; what it removes is the option of holding credit while never +//! completing a possession check. +//! +//! Be precise about the comparison, in both directions: +//! +//! - Against a node under-storing **in bulk** — the realistic case, a node +//! dropping data to save disk — detection is close to what the full-byte +//! audit gave, at roughly 1/430 of the egress (measured about 427x on a +//! same-network control cohort: 14.49 KB against 6.19 MB). A 990-node run +//! caught a 256 MB in-place corruption on the first audit that reached the +//! node. +//! - Against a **fine-grained** partial deleter, one shaving a little off every +//! chunk, this is strictly weaker per audit than serving every byte. The +//! compensating lever is audit *frequency*, which is exactly what the cost +//! reduction buys: the old shape made frequent auditing unaffordable. +//! +//! If a threat model ever needs sharper per-audit detection, the knob is +//! openings per leaf, at linear egress cost. + +use std::io::{Cursor, Read}; + +use crate::ant_protocol::XorName; + +/// Block size for slice audits: one BLAKE3 chunk (1 KiB). A block is the unit +/// both the Bao authenticity proof and the nonced possession tree open on. +/// +/// Matching BLAKE3's internal 1 KiB chunk means a single opened block maps to a +/// single BLAKE3 leaf, so the Bao proof for one block is minimal. +pub const AUDIT_BLOCK_SIZE: u64 = 1024; + +/// Domain tag for a nonced block-tree leaf. Distinct from every other hash in +/// the protocol so a leaf can never be reinterpreted as a node or a commitment. +const DOMAIN_BLOCK_LEAF: &[u8] = b"autonomi.ant.audit.slice.block-leaf.v1"; + +/// Domain tag for a nonced block-tree internal node. +const DOMAIN_BLOCK_NODE: &[u8] = b"autonomi.ant.audit.slice.block-node.v1"; + +/// Domain-separation context for deriving the per-audit BLAKE3 key of the +/// nonced block tree. +/// +/// Versioned alongside the subtree-audit protocol id, and distinct from the +/// possession lane's [`AUDIT_DIGEST_KEY_CONTEXT`], so the two lanes cannot +/// derive a colliding key from the same challenge material. +/// +/// [`AUDIT_DIGEST_KEY_CONTEXT`]: crate::replication::protocol::AUDIT_DIGEST_KEY_CONTEXT +const BLOCK_KEY_CONTEXT: &str = "autonomi.ant.audit.slice.block-key.v1"; + +/// Per-audit keying material for the nonced block tree, derived from the fresh +/// nonce, the challenged peer and the chunk key. Constant across a chunk's blocks. +/// +/// Uses BLAKE3's `derive_key` mode, the same construction as the possession +/// lane's [`compute_audit_digest`] — that is what ADR-0009's "one freshness +/// derivation across all audit lanes" means concretely. `derive_key` separates +/// domains at the mode level (its own flag bits), so this key cannot collide +/// with a plain or keyed hash of the same bytes; a plain hash over a domain +/// prefix would separate only by convention. +/// +/// The result is then used as a BLAKE3 **key**, not a message prefix. That is +/// what forces every BLAKE3 chunk of a block leaf to depend on the nonce: keyed +/// BLAKE3 mixes the key into the initial state of every chunk compression, so +/// there is no nonce-independent chaining value. Prefixing the nonce instead +/// leaves the block's tail in a second, nonce-free BLAKE3 chunk whose chaining +/// value is precomputable (leaf input is `domain ‖ nonce ‖ … ‖ block`, ~1166 +/// bytes, so the last ~142 block bytes fall in chunk 1 with no nonce), letting a +/// node store ~10.7% less than each block and still reconstruct the leaf for any +/// fresh nonce. +/// +/// [`compute_audit_digest`]: crate::replication::protocol::compute_audit_digest +#[must_use] +fn nonced_block_key(nonce: &[u8; 32], peer: &[u8; 32], key: &XorName) -> [u8; 32] { + let mut h = blake3::Hasher::new_derive_key(BLOCK_KEY_CONTEXT); + h.update(nonce); + h.update(peer); + h.update(key); + *h.finalize().as_bytes() +} + +/// Number of 1 KiB blocks covering `content_len` bytes. +/// +/// Always at least 1 (an empty chunk is one empty block) so every committed key +/// opens at least one block, and the block index the auditor draws is always in +/// range. +#[must_use] +pub fn block_count(content_len: u64) -> u32 { + if content_len == 0 { + return 1; + } + u32::try_from(content_len.div_ceil(AUDIT_BLOCK_SIZE)).unwrap_or(u32::MAX) +} + +/// Byte range `[start, end)` of block `index` within `content_len` bytes. +/// +/// The final block may be short; an out-of-range index clamps to an empty range +/// at `content_len` (callers never pass one — the auditor draws indices in +/// `0..block_count`). +#[must_use] +pub fn block_range(content_len: u64, index: u32) -> (u64, u64) { + let start = u64::from(index) + .saturating_mul(AUDIT_BLOCK_SIZE) + .min(content_len); + let end = start.saturating_add(AUDIT_BLOCK_SIZE).min(content_len); + (start, end) +} + +/// Slice a block's bytes out of a full chunk. Returns an empty slice for an +/// out-of-range index (never happens for auditor-drawn indices). +#[must_use] +fn block_bytes(content: &[u8], index: u32) -> &[u8] { + let (start, end) = block_range(content.len() as u64, index); + let start = usize::try_from(start).unwrap_or(usize::MAX); + let end = usize::try_from(end).unwrap_or(usize::MAX); + content.get(start..end).unwrap_or(&[]) +} + +/// Nonced block-leaf hash: keyed by the per-audit key (nonce/peer/key) so every +/// BLAKE3 chunk of the leaf depends on the nonce, binding the block index, block +/// length and block bytes with no nonce-independent state to precompute. +/// +/// See [`nonced_block_key`] for why the nonce is a key, not a prefix. +#[must_use] +fn nonced_block_leaf( + nonce: &[u8; 32], + peer: &[u8; 32], + key: &XorName, + index: u32, + block: &[u8], +) -> [u8; 32] { + let audit_key = nonced_block_key(nonce, peer, key); + let mut h = blake3::Hasher::new_keyed(&audit_key); + h.update(DOMAIN_BLOCK_LEAF); + h.update(&index.to_le_bytes()); + let block_len = u32::try_from(block.len()).unwrap_or(u32::MAX); + h.update(&block_len.to_le_bytes()); + h.update(block); + *h.finalize().as_bytes() +} + +/// Combine two child hashes into a nonced block-tree internal node. +#[must_use] +fn nonced_block_node(left: &[u8; 32], right: &[u8; 32]) -> [u8; 32] { + let mut h = blake3::Hasher::new(); + h.update(DOMAIN_BLOCK_NODE); + h.update(left); + h.update(right); + *h.finalize().as_bytes() +} + +/// Fold one level of a left-packed Merkle tree, self-pairing an unpaired last +/// node (`node(x, x)`) exactly like the commitment tree. +#[must_use] +fn fold_level(level: &[[u8; 32]]) -> Vec<[u8; 32]> { + let mut next = Vec::with_capacity(level.len().div_ceil(2)); + let mut i = 0; + while i < level.len() { + let left = level.get(i).copied().unwrap_or([0u8; 32]); + // Self-pair the last node when the level has an odd length. + let right = level.get(i + 1).copied().unwrap_or(left); + next.push(nonced_block_node(&left, &right)); + i += 2; + } + next +} + +/// The nonced block-tree leaves for a chunk's `content`, in block order. +#[must_use] +fn nonced_leaves( + nonce: &[u8; 32], + peer: &[u8; 32], + key: &XorName, + content: &[u8], +) -> Vec<[u8; 32]> { + let count = block_count(content.len() as u64); + (0..count) + .map(|i| nonced_block_leaf(nonce, peer, key, i, block_bytes(content, i))) + .collect() +} + +/// Compute the nonced block-tree root over a chunk's `content` (responder, round +/// 1). Requires every byte of the chunk, under the fresh nonce. +#[must_use] +pub fn nonced_block_root( + nonce: &[u8; 32], + peer: &[u8; 32], + key: &XorName, + content: &[u8], +) -> [u8; 32] { + let mut level = nonced_leaves(nonce, peer, key, content); + // A single leaf is its own root (matches the commitment tree's convention). + while level.len() > 1 { + level = fold_level(&level); + } + level.first().copied().unwrap_or([0u8; 32]) +} + +/// Sibling hashes on the path from block `index` up to the nonced root, +/// bottom-up (leaf level first). `None` if `index` is out of range for the tree. +/// +/// The verifier folds the recomputed leaf with these siblings using node-index +/// parity, so the sibling ordering is positional, not left/right-tagged. +#[must_use] +pub fn nonced_block_siblings( + nonce: &[u8; 32], + peer: &[u8; 32], + key: &XorName, + content: &[u8], + index: u32, +) -> Option> { + siblings_from_leaves(&nonced_leaves(nonce, peer, key, content), index) +} + +/// Sibling chain for `index` from prebuilt nonced leaves (no per-leaf rehash). +/// +/// Folding 32-byte CVs is cheap; the cost is computing the leaves, so callers +/// that open several blocks of one chunk build the leaves once and fold here per +/// block. `None` if `index` is out of range. +#[must_use] +fn siblings_from_leaves(leaves: &[[u8; 32]], index: u32) -> Option> { + if usize::try_from(index).ok()? >= leaves.len() { + return None; + } + let mut level = leaves.to_vec(); + let mut node_index = index as usize; + let mut siblings = Vec::new(); + while level.len() > 1 { + // Sibling is the other child of this node's parent; the last node of an + // odd level self-pairs, so its sibling is itself. + let sibling_index = node_index ^ 1; + let sibling = level + .get(sibling_index) + .or_else(|| level.get(node_index)) + .copied() + .unwrap_or([0u8; 32]); + siblings.push(sibling); + node_index /= 2; + level = fold_level(&level); + } + Some(siblings) +} + +/// Verify a nonced block opening (auditor, round 2): recompute the block leaf +/// from the served bytes and fold it with `siblings` to the committed +/// `nonced_root`. +/// +/// `block` must be the Bao-verified block bytes for `index`, so this proves the +/// responder committed a nonced root over the *real* content at round-1 time. +#[must_use] +#[allow(clippy::too_many_arguments)] +pub fn verify_nonced_block( + nonce: &[u8; 32], + peer: &[u8; 32], + key: &XorName, + index: u32, + block: &[u8], + siblings: &[[u8; 32]], + nonced_root: &[u8; 32], + block_count: u32, +) -> bool { + // Enforce the canonical tree geometry: the sibling chain MUST be exactly the + // depth of a `block_count`-leaf tree. `nonced_root` is responder-chosen in + // round 1, so without this a partial holder could set `nonced_root` to a + // single block's leaf and pass with zero siblings whenever the fresh draw + // happens to land on that block. Pinning the depth binds the claimed + // left-packed geometry and rejects such degraded proofs. + if siblings.len() != nonced_tree_depth(block_count) { + return false; + } + // An index past the last block names no leaf of the committed tree, so there + // is nothing it could legitimately open. Production requests are always in + // range; rejecting here makes a malformed or forged one fail deterministically + // and cheaply, rather than folding a hash chain to reach the same answer. + if index >= block_count { + return false; + } + let mut node_index = index as usize; + let mut cur = nonced_block_leaf(nonce, peer, key, index, block); + for sibling in siblings { + cur = if node_index % 2 == 0 { + nonced_block_node(&cur, sibling) + } else { + nonced_block_node(sibling, &cur) + }; + node_index /= 2; + } + &cur == nonced_root +} + +/// Canonical sibling-chain length for a `block_count`-leaf nonced tree. +/// +/// The number of `div_ceil(2)` folds from the leaf level down to a single root +/// (0 for a single-block chunk). Matches [`nonced_block_siblings`]'s output. +#[must_use] +pub fn nonced_tree_depth(block_count: u32) -> usize { + let mut n = block_count.max(1) as usize; + let mut depth = 0; + while n > 1 { + n = n.div_ceil(2); + depth += 1; + } + depth +} + +/// Extract a Bao verified slice for block `index` of `content` (responder, round +/// 2). The slice carries the block bytes plus the O(log n) BLAKE3 parent hashes +/// that verify it against the chunk address. +/// +/// # Errors +/// +/// Returns the underlying IO error only if the in-memory Bao extraction fails, +/// which cannot happen for a well-formed in-memory chunk; surfaced as a +/// `Result` rather than a panic so the responder degrades to a rejection. +pub fn extract_block_slice(content: &[u8], index: u32) -> std::io::Result> { + // The outboard carries the BLAKE3 tree hashes separately from the content, so + // the extractor reads the real chunk bytes plus just the parent hashes on the + // block's path — no need to materialise a full Bao encoding of the chunk. + let (outboard, _hash) = bao::encode::outboard(content); + extract_block_slice_with_outboard(content, &outboard, index) +} + +/// Extract a Bao verified slice for block `index` reusing a prebuilt `outboard`. +/// +/// Building the outboard hashes the whole chunk, so when several blocks of the +/// same chunk are opened in one challenge (round + final, plus any dedup), the +/// caller builds the outboard once (see [`ChunkOpener`]) and calls this per +/// block instead of re-hashing. Borrows `content` directly (no copy). +/// +/// # Errors +/// +/// Surfaces an in-memory Bao extraction error as a `Result` rather than a panic; +/// it cannot happen for a well-formed in-memory chunk and outboard. +pub fn extract_block_slice_with_outboard( + content: &[u8], + outboard: &[u8], + index: u32, +) -> std::io::Result> { + let (start, end) = block_range(content.len() as u64, index); + let len = end - start; + let mut extractor = bao::encode::SliceExtractor::new_outboard( + Cursor::new(content), + Cursor::new(outboard), + start, + len, + ); + let mut slice = Vec::new(); + extractor.read_to_end(&mut slice)?; + Ok(slice) +} + +/// Reusable per-chunk responder state for building round-2 openings. +/// +/// Building the Bao outboard and the nonced block-tree leaves each hash the full +/// chunk. When one challenge opens several blocks of the same chunk (the normal +/// random + final pair, plus any duplicates), doing that work once and serving +/// every opening from it keeps a multi-opening challenge to a single hashing pass +/// over the bytes instead of one per opening (V2-685 round-2 amplification fix). +pub struct ChunkOpener<'a> { + content: &'a [u8], + outboard: Vec, + leaves: Vec<[u8; 32]>, +} + +impl<'a> ChunkOpener<'a> { + /// Build the chunk's reusable opening state once: its Bao outboard and its + /// nonced block-leaves (two hash passes over the resident chunk). Repeated + /// openings of the same chunk reuse this state instead of rehashing per opening. + #[must_use] + pub fn new(nonce: &[u8; 32], peer: &[u8; 32], key: &XorName, content: &'a [u8]) -> Self { + let (outboard, _hash) = bao::encode::outboard(content); + let leaves = nonced_leaves(nonce, peer, key, content); + Self { + content, + outboard, + leaves, + } + } + + /// Number of 1 KiB blocks in the chunk (always ≥ 1). + #[must_use] + pub fn block_count(&self) -> u32 { + u32::try_from(self.leaves.len()).unwrap_or(u32::MAX) + } + + /// Bao verified slice for `index`, reusing the prebuilt outboard. + /// + /// # Errors + /// Surfaces an in-memory Bao extraction error as a `Result` rather than a panic. + pub fn bao_slice(&self, index: u32) -> std::io::Result> { + extract_block_slice_with_outboard(self.content, &self.outboard, index) + } + + /// Nonced-tree sibling chain for `index`, reusing the prebuilt leaves. + #[must_use] + pub fn nonced_siblings(&self, index: u32) -> Option> { + siblings_from_leaves(&self.leaves, index) + } +} + +/// Size of the Bao slice's leading length header (a little-endian `u64` giving +/// the encoded content's total length, in bytes). This is the first field of +/// every Bao slice; the decoder authenticates it against the tree root, so it +/// cannot be forged to disagree with the `address` the slice verifies against. +const BAO_LENGTH_HEADER_SIZE: usize = 8; + +/// Verify a Bao slice for block `index` against the chunk `address` +/// (`BLAKE3(content)`), returning the verified block bytes (auditor, round 2). +/// +/// `content_len` is the responder's round-1 claim. It is NOT bound by the signed +/// commitment (the Merkle leaf hashes only `key ‖ bytes_hash`), so a malicious +/// responder can claim any length. Left unchecked, a **deflation** lie forges +/// possession: claiming `content_len = 1024` for a 4 MiB chunk collapses +/// `block_count` to 1, so the auditor can only ever open block 0, and a Bao +/// slice for the prefix range `[0, 1024)` verifies fine against the true +/// full-content address — a node storing ~1 KiB passes an audit for the whole +/// chunk. To close this, the slice's own authenticated length header (which the +/// decoder validates against `address`, so it equals the *true* content length) +/// must match the claimed `content_len`. A responder that lies about the length +/// then either fails the header check (claim ≠ true length) or is forced to +/// report the true length — in which case the block index is drawn from the full +/// range and possession of all blocks is required. +#[must_use] +pub fn verify_block_slice( + slice: &[u8], + address: &[u8; 32], + content_len: u64, + index: u32, +) -> Option> { + // Authenticate the claimed length against the slice's own header before + // trusting `content_len` for the block range. The header is the first 8 + // bytes of every Bao slice and is validated against `address` by the decode + // below (a forged header changes the tree shape and fails the root check), + // so requiring it to equal `content_len` forces the claim to be the true + // content length — defeating the deflation forgery described above. + let header = slice.get(..BAO_LENGTH_HEADER_SIZE)?; + let declared_len = u64::from_le_bytes(header.try_into().ok()?); + if declared_len != content_len { + return None; + } + + // Reject an out-of-range index outright. `block_range` clamps one to the + // empty range `[content_len, content_len)`, which would decode to an empty + // block and return `Some(vec![])` — a malformed request reading as a + // partially valid answer. Nothing legitimate opens a block that does not + // exist. + if index >= block_count(content_len) { + return None; + } + + let (start, end) = block_range(content_len, index); + let len = end - start; + let hash = blake3::Hash::from_bytes(*address); + let mut decoder = bao::decode::SliceDecoder::new(Cursor::new(slice), &hash, start, len); + let mut verified = Vec::new(); + decoder.read_to_end(&mut verified).ok()?; + if verified.len() as u64 != len { + return None; + } + Some(verified) +} + +#[cfg(test)] +#[allow( + clippy::unwrap_used, + clippy::expect_used, + clippy::panic, + clippy::cast_possible_truncation +)] +mod tests { + use super::*; + + const NONCE: [u8; 32] = [0x11; 32]; + const PEER: [u8; 32] = [0x22; 32]; + const KEY: XorName = [0x33; 32]; + + /// Deterministic pseudo-content of a given length (avoids RNG in tests). + fn content_of(len: usize) -> Vec { + (0..len).map(|i| (i * 31 + 7) as u8).collect() + } + + // -- block geometry ----------------------------------------------------- + + #[test] + fn block_count_is_ceil_with_empty_floor() { + assert_eq!(block_count(0), 1); + assert_eq!(block_count(1), 1); + assert_eq!(block_count(1024), 1); + assert_eq!(block_count(1025), 2); + assert_eq!(block_count(4096), 4); + assert_eq!(block_count(4 * 1024 * 1024), 4096); + assert_eq!(block_count(4 * 1024 * 1024 - 1), 4096); + } + + #[test] + fn block_range_covers_content_without_gaps_or_overlap() { + let len = 1024 * 3 + 500; + let count = block_count(len); + let mut expected_start = 0u64; + for i in 0..count { + let (s, e) = block_range(len, i); + assert_eq!(s, expected_start); + assert!(e <= len); + assert!(e > s || len == 0); + expected_start = e; + } + assert_eq!(expected_start, len); + } + + // -- bao slice == blake3 address --------------------------------------- + + #[test] + fn bao_root_equals_blake3_address_across_lengths() { + // The whole design rests on Bao's root being the chunk's BLAKE3 address. + for len in [0usize, 1, 1023, 1024, 1025, 2048, 4096, 10_000, 1 << 20] { + let content = content_of(len); + let (_outboard, bao_hash) = bao::encode::outboard(&content); + let blake = blake3::hash(&content); + assert_eq!( + bao_hash.as_bytes(), + blake.as_bytes(), + "bao root must equal blake3(content) at len {len}" + ); + } + } + + // Both verifiers reject a block index past the end of the chunk. Nothing the + // auditor generates is ever out of range, so this is about failing cleanly: + // `block_range` clamps an over-large index to an empty range, which without + // the check reads back as a successfully verified empty block rather than as + // the malformed request it is. + #[test] + fn verifiers_reject_a_block_index_past_the_end() { + let len = 4096usize; + let content = content_of(len); + let address = *blake3::hash(&content).as_bytes(); + let count = block_count(len as u64); + + // A real slice for the last block does not become a valid opening for a + // block that does not exist. + let slice = extract_block_slice(&content, count - 1).expect("extract"); + assert!( + verify_block_slice(&slice, &address, len as u64, count).is_none(), + "the first index past the end must be rejected, not read as empty" + ); + assert!(verify_block_slice(&slice, &address, len as u64, u32::MAX).is_none()); + + // Same on the possession chain, with a genuine sibling path. + let siblings = + nonced_block_siblings(&NONCE, &PEER, &KEY, &content, count - 1).expect("siblings"); + let root = nonced_block_root(&NONCE, &PEER, &KEY, &content); + let (s, e) = block_range(len as u64, count - 1); + let block = &content[s as usize..e as usize]; + assert!( + !verify_nonced_block(&NONCE, &PEER, &KEY, count, block, &siblings, &root, count), + "an index past the last leaf names nothing in the committed tree" + ); + } + + #[test] + fn slice_roundtrip_verifies_every_block() { + for len in [1usize, 1024, 1025, 4096, 5000, 1 << 16] { + let content = content_of(len); + let address = *blake3::hash(&content).as_bytes(); + let count = block_count(len as u64); + for i in 0..count { + let slice = extract_block_slice(&content, i).expect("extract"); + let verified = + verify_block_slice(&slice, &address, len as u64, i).expect("verify slice"); + let (s, e) = block_range(len as u64, i); + assert_eq!(verified.as_slice(), &content[s as usize..e as usize]); + } + } + } + + // Regression: content_len deflation must NOT forge possession. + // + // `content_len` is not bound by the signed round-1 commitment (the Merkle + // leaf hashes only key+bytes_hash), so a malicious responder can claim + // content_len=1024 for a 1 MiB chunk. That would collapse block_count to 1, + // let the auditor only ever open block 0, and — before the fix — pass a Bao + // slice for the prefix [0,1024) against the true full-content address while + // storing ~1 KiB. verify_block_slice now authenticates the slice's own length + // header against the claim, so the deflation is rejected at chain 1. + #[test] + fn content_len_deflation_is_rejected() { + let full = content_of(1 << 20); // 1 MiB chunk, truly 1024 blocks. + let address = *blake3::hash(&full).as_bytes(); + + // Attacker keeps only the first block's Bao slice (extracted honestly for + // range [0,1024) of the real content), discarding the rest of the 1 MiB. + let stored_slice = extract_block_slice(&full, 0).expect("extract block 0"); + + // Round 1: attacker claims content_len = 1024 (one block). + let claimed_len = 1024u64; + assert_eq!(block_count(claimed_len), 1, "deflated claim = one block"); + + // Round 2: only block 0 could be drawn. Chain 1 must reject because the + // slice's authenticated length header (1 MiB) disagrees with the claim. + assert!( + verify_block_slice(&stored_slice, &address, claimed_len, 0).is_none(), + "deflated content_len must fail the slice length-header check" + ); + + // An honest responder reporting the true length still verifies block 0. + let honest = verify_block_slice(&stored_slice, &address, (1 << 20) as u64, 0); + assert_eq!( + honest.expect("honest full-length slice must verify").len(), + 1024 + ); + } + + /// Rewrite a Bao slice's 8-byte little-endian length header in place. + fn rewrite_slice_len(slice: &mut [u8], forged_len: u64) { + slice[..BAO_LENGTH_HEADER_SIZE].copy_from_slice(&forged_len.to_le_bytes()); + } + + // Regression for the header-rewrite length forgery (the residual attack the + // plain header==content_len check does NOT catch on its own). + // + // Bao does not cryptographically bind the length header for a PREFIX slice + // that never touches EOF: the unopened right subtree is an opaque hash. So an + // attacker can claim `content_len` just past a subtree boundary (e.g. 2 KiB+1 + // for a 4 KiB / 4-block chunk), REWRITE each slice header to that forged + // length, supply the true right-subtree CV as the opaque root sibling, and + // pass any opening in the retained left half — storing only that half. The + // header check passes because header == forged content_len. + // + // The defence is to ALSO open the CLAIMED FINAL block: that decode reaches + // EOF, where Bao authenticates the encoded length against the address, so a + // forged-short length fails. This test shows both halves: a left-half opening + // slips past the header check, but the final-block opening rejects the forgery. + #[test] + fn header_rewrite_length_forgery_is_caught_at_the_final_block() { + // True chunk: 4 blocks. Root = parent(parent(b0,b1), parent(b2,b3)). + let full = content_of(4096); + let address = *blake3::hash(&full).as_bytes(); + + // Forged length: 2049 bytes → block_count 3 (b0, b1 full, b2' one byte). + // The claimed final block is index 2; blocks 0..2 are the retained half. + let forged_len = 2049u64; + assert_eq!(block_count(forged_len), 3); + let forged_final = block_count(forged_len) - 1; + + // Left-half opening (block 0): the attacker rewrites an honest block-0 + // slice's header to the forged length. The retained slice already carries + // the true right-subtree CV as the opaque root sibling, so it verifies + // against the real address AND passes the header==content_len check. + let mut left = extract_block_slice(&full, 0).expect("extract block 0"); + rewrite_slice_len(&mut left, forged_len); + assert!( + verify_block_slice(&left, &address, forged_len, 0).is_some(), + "header rewrite lets a left-half opening slip past the header check — \ + this is exactly why the final block must also be opened" + ); + + // Final-block opening (block 2 under the forged length): no slice the + // attacker can offer verifies, because reaching the forged EOF forces Bao + // to authenticate the length against the true address. The best attempt — + // an honest block-2 slice of the true content with a rewritten header — + // is rejected. + let mut fake_final = extract_block_slice(&full, 2).expect("extract block 2"); + rewrite_slice_len(&mut fake_final, forged_len); + assert!( + verify_block_slice(&fake_final, &address, forged_len, forged_final).is_none(), + "final-block opening under a forged short length must fail (EOF length auth)" + ); + } + + #[test] + fn slice_against_wrong_address_fails() { + let content = content_of(4096); + let mut wrong = *blake3::hash(&content).as_bytes(); + wrong[0] ^= 0x01; + let slice = extract_block_slice(&content, 2).expect("extract"); + assert!(verify_block_slice(&slice, &wrong, 4096, 2).is_none()); + } + + #[test] + fn tampered_slice_bytes_fail_verification() { + let content = content_of(4096); + let address = *blake3::hash(&content).as_bytes(); + let mut slice = extract_block_slice(&content, 1).expect("extract"); + // Corrupt the payload bytes near the end of the slice (a data byte). + if let Some(b) = slice.last_mut() { + *b ^= 0xFF; + } + assert!(verify_block_slice(&slice, &address, 4096, 1).is_none()); + } + + #[test] + fn slice_for_one_block_cannot_serve_a_different_block() { + let content = content_of(4096); + let address = *blake3::hash(&content).as_bytes(); + let slice = extract_block_slice(&content, 0).expect("extract"); + // A slice built for block 0 must not verify as block 2. + assert!(verify_block_slice(&slice, &address, 4096, 2).is_none()); + } + + // -- nonced block tree -------------------------------------------------- + + #[test] + fn nonced_openings_roundtrip_every_block() { + for len in [1usize, 1024, 1025, 3000, 4096, 9001] { + let content = content_of(len); + let root = nonced_block_root(&NONCE, &PEER, &KEY, &content); + let count = block_count(len as u64); + for i in 0..count { + let siblings = + nonced_block_siblings(&NONCE, &PEER, &KEY, &content, i).expect("siblings"); + let block = block_bytes(&content, i); + assert!( + verify_nonced_block(&NONCE, &PEER, &KEY, i, block, &siblings, &root, count), + "len {len} block {i} must verify" + ); + } + } + } + + #[test] + fn nonced_opening_rejects_wrong_block_bytes() { + let content = content_of(4096); + let root = nonced_block_root(&NONCE, &PEER, &KEY, &content); + let siblings = nonced_block_siblings(&NONCE, &PEER, &KEY, &content, 1).expect("siblings"); + let mut wrong = block_bytes(&content, 1).to_vec(); + wrong[0] ^= 0x01; + let bc = block_count(content.len() as u64); + assert!(!verify_nonced_block( + &NONCE, &PEER, &KEY, 1, &wrong, &siblings, &root, bc + )); + } + + #[test] + fn nonced_opening_binds_nonce_peer_and_key() { + let content = content_of(4096); + let root = nonced_block_root(&NONCE, &PEER, &KEY, &content); + let siblings = nonced_block_siblings(&NONCE, &PEER, &KEY, &content, 2).expect("siblings"); + let block = block_bytes(&content, 2); + let bc = block_count(content.len() as u64); + // Correct binding verifies. + assert!(verify_nonced_block( + &NONCE, &PEER, &KEY, 2, block, &siblings, &root, bc + )); + // A different nonce, peer, or key must not verify against the same root. + let other = [0xAB; 32]; + assert!(!verify_nonced_block( + &other, &PEER, &KEY, 2, block, &siblings, &root, bc + )); + assert!(!verify_nonced_block( + &NONCE, &other, &KEY, 2, block, &siblings, &root, bc + )); + assert!(!verify_nonced_block( + &NONCE, &PEER, &other, 2, block, &siblings, &root, bc + )); + } + + // Regression: the nonce must KEY every BLAKE3 chunk of a block + // leaf, not merely prefix the message. A prefixed leaf (`BLAKE3(nonce ‖ … ‖ + // block)`, ~1166 bytes) leaves the block's tail in a second, nonce-free + // BLAKE3 chunk whose chaining value is precomputable, letting a node store + // ~10.7% less than each block and rebuild the leaf for any fresh nonce. Lock + // in the keyed construction and prove it differs from the vulnerable one. + #[test] + fn leaf_is_keyed_by_the_nonce_not_prefixed() { + let block = content_of(1024); + let index = 3u32; + let block_len = block.len() as u32; + let got = nonced_block_leaf(&NONCE, &PEER, &KEY, index, &block); + + // Required construction: keyed BLAKE3, nonce in the key. + let audit_key = nonced_block_key(&NONCE, &PEER, &KEY); + let mut keyed = blake3::Hasher::new_keyed(&audit_key); + keyed.update(DOMAIN_BLOCK_LEAF); + keyed.update(&index.to_le_bytes()); + keyed.update(&block_len.to_le_bytes()); + keyed.update(&block); + assert_eq!( + got, + *keyed.finalize().as_bytes(), + "leaf must be keyed BLAKE3 with the nonce-derived key" + ); + + // The old, vulnerable prefixed construction must NOT match. + let mut prefixed = blake3::Hasher::new(); + prefixed.update(DOMAIN_BLOCK_LEAF); + prefixed.update(&NONCE); + prefixed.update(&PEER); + prefixed.update(&KEY); + prefixed.update(&index.to_le_bytes()); + prefixed.update(&block_len.to_le_bytes()); + prefixed.update(&block); + assert_ne!( + got, + *prefixed.finalize().as_bytes(), + "leaf must not use the precomputable prefixed-nonce construction" + ); + } + + // The documented strength claim, made executable: a partial holder CAN pass, + // and what stops it is sampling, not the hash construction. + // + // A peer keeping blocks 0..k of a chunk commits a root whose kept-block + // leaves are genuine and whose dropped-block leaves are garbage. Every + // opening on a kept block verifies against that root; every opening on a + // dropped block fails. This is the whole basis for describing round 2 as a + // sampling check rather than a proof of complete possession. + #[test] + fn partial_holder_passes_on_kept_blocks_and_fails_on_dropped_ones() { + let content = content_of(8 * 1024); // 8 blocks + let count = block_count(content.len() as u64); + assert_eq!(count, 8); + let kept = 5u32; // blocks 0..5 retained, 5..8 dropped + + // Build the tree a partial holder would commit: real leaves for kept + // blocks, arbitrary (wrong) leaves for the dropped ones. + let leaves: Vec<[u8; 32]> = (0..count) + .map(|i| { + if i < kept { + nonced_block_leaf(&NONCE, &PEER, &KEY, i, block_bytes(&content, i)) + } else { + // No bytes for this block — anything at all goes here. + nonced_block_leaf(&NONCE, &PEER, &KEY, i, b"dropped") + } + }) + .collect(); + let mut level = leaves.clone(); + while level.len() > 1 { + level = fold_level(&level); + } + let forged_root = level.first().copied().expect("non-empty tree"); + + // An opening on a KEPT block verifies against the forged root: the peer + // passes this draw despite not holding the whole chunk. + for i in 0..kept { + let siblings = siblings_from_leaves(&leaves, i).expect("in range"); + assert!( + verify_nonced_block( + &NONCE, + &PEER, + &KEY, + i, + block_bytes(&content, i), + &siblings, + &forged_root, + count, + ), + "a kept block must verify — this is the false-pass path the docs describe" + ); + } + + // An opening on a DROPPED block cannot be answered: the peer has no + // bytes whose leaf folds to the committed root at that index. + for i in kept..count { + let siblings = siblings_from_leaves(&leaves, i).expect("in range"); + assert!( + !verify_nonced_block( + &NONCE, + &PEER, + &KEY, + i, + block_bytes(&content, i), + &siblings, + &forged_root, + count, + ), + "a dropped block must fail — sampling is what catches the partial holder" + ); + } + } + + // ADR-0009 claims ONE freshness derivation across every audit lane. That is + // only true if this lane and the possession lane derive their key the same + // way: BLAKE3 `derive_key` mode over `nonce ‖ peer ‖ key`, under a + // per-lane context string. The slice lane previously used a plain hash with + // a domain prefix, which separates domains only by convention, so the two + // constructions did not actually match and the claim was overstated. + #[test] + fn block_key_uses_the_same_derivation_as_the_possession_lane() { + let mut expected = blake3::Hasher::new_derive_key(BLOCK_KEY_CONTEXT); + expected.update(&NONCE); + expected.update(&PEER); + expected.update(&KEY); + assert_eq!( + nonced_block_key(&NONCE, &PEER, &KEY), + *expected.finalize().as_bytes(), + "block key must use derive_key mode, like compute_audit_digest" + ); + + // Mode separation is the point: the same material under a plain hash, + // or under keyed mode, must land somewhere else entirely. + let mut plain = blake3::Hasher::new(); + plain.update(BLOCK_KEY_CONTEXT.as_bytes()); + plain.update(&NONCE); + plain.update(&PEER); + plain.update(&KEY); + assert_ne!( + nonced_block_key(&NONCE, &PEER, &KEY), + *plain.finalize().as_bytes(), + "derive_key mode must not coincide with a plain domain-prefixed hash" + ); + + // The two lanes must never derive the SAME key from the same challenge + // material — that is what the distinct context strings buy. + assert_ne!( + nonced_block_key(&NONCE, &PEER, &KEY), + { + let mut other = blake3::Hasher::new_derive_key( + crate::replication::protocol::AUDIT_DIGEST_KEY_CONTEXT, + ); + other.update(&NONCE); + other.update(&PEER); + other.update(&KEY); + *other.finalize().as_bytes() + }, + "the slice and possession lanes must derive different keys" + ); + } + + #[test] + fn nonced_root_changes_with_any_block_edit() { + let content = content_of(5000); + let root = nonced_block_root(&NONCE, &PEER, &KEY, &content); + let mut edited = content; + // Flip a byte in the LAST block; the root must change (all blocks covered). + if let Some(b) = edited.last_mut() { + *b ^= 0x01; + } + let root2 = nonced_block_root(&NONCE, &PEER, &KEY, &edited); + assert_ne!(root, root2); + } + + #[test] + fn nonced_siblings_out_of_range_is_none() { + let content = content_of(2048); + let count = block_count(content.len() as u64); + assert!(nonced_block_siblings(&NONCE, &PEER, &KEY, &content, count).is_none()); + } + + // -- the combined possession property ---------------------------------- + + #[test] + fn relay_cannot_open_against_a_foreign_committed_root() { + // A responder that did NOT hold the bytes at round 1 commits a root over + // garbage (or a guess). Even if it later fetches the real block, folding + // the real leaf to that committed root would be a preimage break: model + // this by taking a root from DIFFERENT content and checking that the real + // block + honest siblings never fold to it. + let real = content_of(4096); + let garbage = content_of(4097); // different content => different tree + let foreign_root = nonced_block_root(&NONCE, &PEER, &KEY, &garbage); + let siblings = nonced_block_siblings(&NONCE, &PEER, &KEY, &real, 0).expect("siblings"); + let block = block_bytes(&real, 0); + assert!(!verify_nonced_block( + &NONCE, + &PEER, + &KEY, + 0, + block, + &siblings, + &foreign_root, + block_count(real.len() as u64) + )); + } + + // Canonical-depth regression: `nonced_root` is responder-chosen in round 1, + // so a partial holder could set it to a single block's leaf and pass with + // zero siblings whenever the fresh draw lands on that block. Pinning the + // sibling-chain length to the tree depth rejects that (and any wrong-depth + // chain) even when it would fold to the supplied root. + #[test] + fn canonical_depth_rejects_wrong_sibling_count() { + let content = content_of(4096); // 4 blocks → canonical depth 2 + let bc = block_count(content.len() as u64); + assert_eq!(nonced_tree_depth(bc), 2); + let block0 = block_bytes(&content, 0); + + // Degraded escape: claim root = block 0's own leaf, supply zero siblings. + // Folds to that root, but depth 0 != 2, so rejected. + let leaf0 = nonced_block_leaf(&NONCE, &PEER, &KEY, 0, block0); + assert!(!verify_nonced_block( + &NONCE, + &PEER, + &KEY, + 0, + block0, + &[], + &leaf0, + bc + )); + + // The honest, correct-depth opening still verifies. + let root = nonced_block_root(&NONCE, &PEER, &KEY, &content); + let siblings = nonced_block_siblings(&NONCE, &PEER, &KEY, &content, 0).expect("siblings"); + assert_eq!(siblings.len(), 2); + assert!(verify_nonced_block( + &NONCE, &PEER, &KEY, 0, block0, &siblings, &root, bc + )); + + // A too-long chain (extra sibling) is rejected on depth alone. + let mut too_long = siblings; + too_long.push([0u8; 32]); + assert!(!verify_nonced_block( + &NONCE, &PEER, &KEY, 0, block0, &too_long, &root, bc + )); + } +} diff --git a/src/replication/storage_commitment_audit.rs b/src/replication/storage_commitment_audit.rs index 14e5104f..458d7e93 100644 --- a/src/replication/storage_commitment_audit.rs +++ b/src/replication/storage_commitment_audit.rs @@ -9,6 +9,7 @@ //! [`handle_subtree_challenge`]; the pure proof maths live in //! [`crate::replication::subtree`]. +use std::collections::{HashMap, HashSet}; use std::sync::Arc; use std::time::Duration; @@ -19,11 +20,12 @@ use crate::ant_protocol::XorName; use crate::replication::commitment::{commitment_hash, StorageCommitment}; use crate::replication::commitment_state::ResponderCommitmentState; use crate::replication::config::{ - ReplicationConfig, MAX_BYTE_CHALLENGE_KEYS, REPLICATION_PROTOCOL_ID, + ReplicationConfig, MAX_SLICE_OPENINGS, SUBTREE_AUDIT_PROTOCOL_ID, }; use crate::replication::protocol::{ RejectKind, ReplicationMessage, ReplicationMessageBody, SubtreeAuditChallenge, - SubtreeAuditResponse, SubtreeByteChallenge, SubtreeByteItem, SubtreeByteResponse, + SubtreeAuditResponse, SubtreeSliceChallenge, SubtreeSliceItem, SubtreeSliceOpening, + SubtreeSliceResponse, }; use crate::replication::recent_provers::RecentProvers; use crate::replication::subtree::{ @@ -44,14 +46,22 @@ use crate::replication::audit::AuditTickResult; // Auditor side // --------------------------------------------------------------------------- -/// ADR-0002 round-2 byte challenge samples a SMALL surprise set of the proven +/// ADR-0002 round-2 slice challenge samples a SMALL surprise set of the proven /// leaves (3..=5). Small enough that the responder's honest local-disk read of -/// the original chunks stays well inside the possession-in-time deadline, while +/// the original chunks stays well inside the response deadline (a liveness bound), while /// a relay forced to fetch them over the network blows it; large enough that /// faking a fraction `x` of leaves survives only `(1 - x)^k`. const BYTE_SPOTCHECK_MIN: u32 = 3; const BYTE_SPOTCHECK_MAX: u32 = 5; +// Each sampled leaf produces up to two openings (a fresh-random block plus the +// claimed final block, deduplicated when they coincide), so the opening cap must +// cover twice the leaf sample. +const _: () = assert!( + 2 * BYTE_SPOTCHECK_MAX as usize <= MAX_SLICE_OPENINGS, + "MAX_SLICE_OPENINGS must cover two openings per sampled leaf" +); + /// ADR-0004 A1: with grace removed, the responder retries a TRANSIENT chunk-read /// error a few times before rejecting `Transient` (which routes to the timeout /// lane). A momentary disk blip usually clears within these attempts; only a @@ -139,14 +149,16 @@ struct AuditCtx<'a> { /// /// 1. **Structure** (round 1) — the returned subtree rebuilds to the pinned /// root, within a size-scaled deadline. -/// 2. **Real bytes** (round 2) — the auditor demands the ORIGINAL chunk content -/// for a 3..=5 FRESHLY-RANDOM sample of the proven leaves (chosen after the -/// proof arrives, not nonce-derived — see `random_spotcheck_leaves`) FROM the -/// responder, and recomputes both the content-address hash and the nonce -/// freshness hash from that served content. The auditor holds none of the -/// peer's chunks. -/// 3. **Timing** — each round's deadline is sized to an honest local-disk read, -/// so a relay forced to fetch over the network blows it. +/// 2. **Verified slice** (round 2) — for a 3..=5 FRESHLY-RANDOM sample of the +/// proven leaves (chosen after the proof arrives, not nonce-derived — see +/// `random_spotcheck_leaves`), the auditor opens one random 1 KiB block per +/// leaf (plus its final block) and verifies a Bao slice against the chunk +/// address and a nonced-tree opening against the round-1 `nonced_root`, over +/// the bytes the responder serves. The auditor holds none of the peer's +/// chunks, and possession is the round-1 commitment, not a full-chunk transfer. +/// 3. **Timing** — a response deadline still bounds liveness, but (unlike the old +/// full-byte round 2) it is no longer the possession proof; that is the +/// round-1 `nonced_root` commitment. /// /// A timeout (either round) is reported as [`AuditFailureReason::Timeout`] (the /// caller applies the strike/grace policy). Any structural failure, served @@ -197,7 +209,7 @@ pub async fn run_subtree_audit( let timeout = config.audit_response_timeout(subtree_leaves); let response = match p2p_node - .send_request(challenged_peer, REPLICATION_PROTOCOL_ID, encoded, timeout) + .send_request(challenged_peer, SUBTREE_AUDIT_PROTOCOL_ID, encoded, timeout) .await { Ok(resp) => resp, @@ -231,63 +243,92 @@ pub async fn run_subtree_audit( dispatch_subtree_response(resp_msg.body, &ctx).await } -/// Outcome of the round-2 byte challenge round-trip (auditor side). -enum ByteRound { - /// The responder returned per-key items (verified by the caller). - Served(Vec), - /// The responder rejected the byte challenge (confirmed failure for a +/// Outcome of the round-2 slice challenge round-trip (auditor side). +enum SliceRound { + /// The responder returned per-opening items (verified by the caller). + Served(Vec), + /// The responder rejected the slice challenge (confirmed failure for a /// recently pinned commitment). Rejected, /// The responder rejected with `Transient` (a local read error): routed to - /// the non-response/timeout lane — no trust penalty, but holder credit is - /// revoked, because the peer answered and could not prove possession, so it - /// must not keep stale credit. Distinct from a silent network `Timeout`, - /// which keeps credit (a dropped packet is not evidence of loss). + /// the non-response/timeout lane — no trust penalty, but the pinned + /// commitment's holder credit is revoked, because the peer answered and + /// could not prove possession, so it must not keep stale credit. + /// + /// This lands in the same place as [`SliceRound::Timeout`] below, which also + /// revokes commitment-scoped credit without a trust penalty. The two are + /// kept separate because they say different things about the peer — one + /// could not read its disk, the other did not reply at all — and only the + /// reason is logged differently. TransientReject, - /// No response within the byte deadline, or a transport error (graced - /// timeout). Keeps holder credit. + /// No response within the slice deadline, or a transport error. Routed to + /// the non-response/timeout lane: no trust penalty, but the pinned + /// commitment's holder credit is revoked. + /// + /// Silence here is not the same as a silent peer in general. The responder + /// answered round 1 in this same exchange, so it was live and reachable, and + /// it saw which blocks were drawn before deciding whether to reply. Leaving + /// credit in place would make "answer only when the draw is favourable" a + /// free strategy: an unfavourable sample would cost nothing and the credit + /// earned from an earlier favourable one would survive. Revoking scopes the + /// cost to the pinned commitment, so the peer must actually answer a round 2 + /// to hold credit, while an honest node that drops one reply keeps its trust + /// intact and re-earns credit on its next audit. Timeout, + /// The responder claimed `Bootstrapping` in round 2 after answering a valid + /// round-1 proof. A node that just produced a signed subtree proof is + /// provably not bootstrapping, so this responsive contradiction is a + /// confirmed failure (not a graced timeout): it revokes the peer's holder + /// credit and takes the trust penalty. + /// + /// This classification relies on `is_bootstrapping` being ONE-WAY (true → + /// false; see `mod.rs`): a round-1 proof implies the snapshot was already + /// `false`, and a restart drops the single-use round-1 session so round 2 is + /// answered `Transient` before this branch is ever reached. An honest running + /// node therefore cannot produce this response; only a malicious, incompatible, + /// or future broken-state peer can. If a "re-bootstrap" (false → true) + /// transition is ever added, it MUST clear live sessions or this policy must + /// be revisited. + ResponsiveBootstrap, /// Malformed / unexpected round-2 response body. Malformed, } -/// Round 2: ask the responder for the ORIGINAL chunk content of one BATCH of -/// auditor-selected spot-check `keys` (at most [`MAX_BYTE_CHALLENGE_KEYS`], so -/// the worst-case response of max-size chunks fits the wire cap), sized to a -/// possession-in-time deadline (honest local-disk read of `keys.len()` chunks). -/// The responder cannot have predicted which keys are sampled. -async fn request_byte_proof(ctx: &AuditCtx<'_>, keys: &[XorName]) -> ByteRound { - let challenge = SubtreeByteChallenge { +/// Round 2: ask the responder to open `openings` blocks (one 1 KiB block per +/// sampled leaf, at most [`MAX_SLICE_OPENINGS`]) with a Bao verified slice plus a +/// nonced block-tree opening each. The reply is a few KB total, so there is no +/// batching. The responder cannot have predicted which leaves — or which block +/// within each — are opened (fresh post-proof randomness). +async fn request_slice_proof(ctx: &AuditCtx<'_>, openings: &[SubtreeSliceOpening]) -> SliceRound { + let challenge = SubtreeSliceChallenge { challenge_id: ctx.challenge_id, nonce: ctx.nonce, challenged_peer_id: *ctx.challenged_peer.as_bytes(), expected_commitment_hash: ctx.expected_commitment_hash, - keys: keys.to_vec(), + openings: openings.to_vec(), }; let msg = ReplicationMessage { request_id: ctx.challenge_id, - body: ReplicationMessageBody::SubtreeByteChallenge(challenge), + body: ReplicationMessageBody::SubtreeSliceChallenge(challenge), }; let encoded = match msg.encode() { Ok(data) => data, Err(e) => { - warn!("Audit: failed to encode byte challenge: {e}"); - return ByteRound::Malformed; + warn!("Audit: failed to encode slice challenge: {e}"); + return SliceRound::Malformed; } }; - // Deadline sized to "honest responder reads `keys.len()` local chunks AND - // ships them back": a relay forced to fetch them over the network blows it - // (graced timeout, never a confirmed failure — same possession-in-time - // principle as round 1). Uses the byte-round floor, which is high enough for - // the multi-MiB reply (handshake + upload + busy disk) — the round-1 - // hashes-only floor would be too tight for 2 × 4 MiB (§4). - let timeout = ctx.config.byte_audit_response_timeout(keys.len()); + // Deadline sized to "honest responder reads `openings.len()` full local + // chunks to build their proofs": generous, because possession is now + // guaranteed by the round-1 nonced commitment, not by this deadline being + // too tight for a relay to fetch bytes. + let timeout = ctx.config.slice_audit_response_timeout(openings.len()); let response = match ctx .p2p_node .send_request( ctx.challenged_peer, - REPLICATION_PROTOCOL_ID, + SUBTREE_AUDIT_PROTOCOL_ID, encoded, timeout, ) @@ -296,31 +337,48 @@ async fn request_byte_proof(ctx: &AuditCtx<'_>, keys: &[XorName]) -> ByteRound { Ok(resp) => resp, Err(e) => { debug!( - "Audit: byte challenge to {} timed out / failed: {e}", + "Audit: slice challenge to {} timed out / failed: {e}", ctx.challenged_peer ); - return ByteRound::Timeout; + return SliceRound::Timeout; } }; let resp_msg = match ReplicationMessage::decode(&response.data) { Ok(m) => m, Err(e) => { - warn!("Audit: failed to decode byte response: {e}"); - return ByteRound::Malformed; + warn!("Audit: failed to decode slice response: {e}"); + return SliceRound::Malformed; } }; - match resp_msg.body { - ReplicationMessageBody::SubtreeByteResponse(SubtreeByteResponse::Items { - challenge_id, + classify_slice_response(resp_msg.body, ctx.challenge_id, ctx.challenged_peer) +} + +/// Classify a decoded round-2 body into a [`SliceRound`]. +/// +/// Pure over the wire body so the round-2 accounting boundary is testable +/// without a live `P2PNode`; [`request_slice_proof`] calls exactly this on the +/// bytes it decoded. +/// +/// Every arm is guarded on `challenge_id`: a body carrying any other id is +/// `Malformed`, so a responder cannot answer a hard challenge with a softer +/// verdict minted for a different one. +fn classify_slice_response( + body: ReplicationMessageBody, + challenge_id: u64, + challenged_peer: &PeerId, +) -> SliceRound { + match body { + ReplicationMessageBody::SubtreeSliceResponse(SubtreeSliceResponse::Items { + challenge_id: id, items, - }) if challenge_id == ctx.challenge_id => ByteRound::Served(items), - ReplicationMessageBody::SubtreeByteResponse(SubtreeByteResponse::Rejected { - challenge_id, + }) if id == challenge_id => SliceRound::Served(items), + ReplicationMessageBody::SubtreeSliceResponse(SubtreeSliceResponse::Rejected { + challenge_id: id, kind, reason, - }) if challenge_id == ctx.challenge_id => { + }) if id == challenge_id => { // ADR-0004 A1: grace removed. UnknownCommitment/Protocol repudiation // of a pinned root is a confirmed failure; a Transient read error // routes to the timeout lane (credit revoked, no trust penalty) — the @@ -329,31 +387,57 @@ async fn request_byte_proof(ctx: &AuditCtx<'_>, keys: &[XorName]) -> ByteRound { match grade_reject(kind) { RejectGrade::Confirmed => { warn!( - "Audit: {} rejected byte challenge ({kind:?}; confirmed): {reason}", - ctx.challenged_peer + "Audit: {challenged_peer} rejected slice challenge \ + ({kind:?}; confirmed): {reason}" ); - ByteRound::Rejected + SliceRound::Rejected } RejectGrade::TimeoutLane => { debug!( - "Audit: {} returned Transient for byte challenge (timeout lane): {reason}", - ctx.challenged_peer + "Audit: {challenged_peer} returned Transient for slice challenge \ + (timeout lane): {reason}" ); - ByteRound::TransientReject + SliceRound::TransientReject } } } - // A node claiming bootstrap MID-AUDIT (it answered round 1) is treated - // as a timeout: it didn't prove possession but the round-1 proof shows - // it isn't bootstrapping, so the bootstrap-claim-abuse detector (round 1) - // owns that lane; here we just don't credit it. - ReplicationMessageBody::SubtreeByteResponse(SubtreeByteResponse::Bootstrapping { - challenge_id, - }) if challenge_id == ctx.challenge_id => ByteRound::Timeout, - _ => ByteRound::Malformed, + // A node claiming bootstrap MID-AUDIT (it just answered round 1 with a + // valid signed proof) is contradicting itself: a bootstrapping node has + // no committed data to prove. A graced timeout here would let it keep the + // holder credit it earned earlier while dodging every round-2 possession + // check, so this is a confirmed failure with credit revocation. + ReplicationMessageBody::SubtreeSliceResponse(SubtreeSliceResponse::Bootstrapping { + challenge_id: id, + }) if id == challenge_id => SliceRound::ResponsiveBootstrap, + _ => SliceRound::Malformed, } } +/// Whether a round-2 outcome must revoke the holder credit carried by the +/// commitment the auditor pinned. +/// +/// True for every outcome that ends round 2 without a completed possession +/// proof but stays in the graced timeout lane — an explicit `Transient` reject +/// and silence alike. Both mean the peer did not prove it still holds the bytes +/// it committed to, so it must not keep standing earned by an earlier audit. +/// +/// Silence matters most here. By round 2 the responder has already answered +/// round 1, so it is demonstrably live, and it learns which blocks were drawn +/// before it decides whether to reply. If silence kept credit, replying only to +/// favourable draws would cost nothing: unfavourable samples would be free and +/// credit from an earlier favourable one would persist, so the sampling +/// probability would describe answered passes rather than detection. Revoking +/// makes every unanswered round 2 cost the peer its standing for that +/// commitment, while an honest peer that drops one reply takes no trust penalty +/// and re-earns credit on its next audit. +/// +/// The confirmed lanes (`Rejected`, `ResponsiveBootstrap`, `Malformed`) are +/// excluded because they revoke more broadly downstream, via +/// `apply_audit_failure_credit_revocation` → `forget_peer`. +const fn round2_revokes_pinned_credit(round: &SliceRound) -> bool { + matches!(round, SliceRound::TransientReject | SliceRound::Timeout) +} + /// Map a decoded response body to an audit outcome (auditor side). A response /// whose `challenge_id` doesn't match, or any non-subtree body, is malformed. async fn dispatch_subtree_response( @@ -465,11 +549,12 @@ pub(crate) enum AuditVerdict { /// Runs the cheap gates in fail-fast order: pin / identity / signature → /// structure (the returned subtree rebuilds to the pinned root). It does **not** /// prove byte possession — the leaves carry only the public `bytes_hash` (the -/// chunk address) and a `nonced_hash` the responder computed itself. Possession -/// is proven in round 2 ([`verify_byte_response`]), where the auditor demands -/// the original chunk bytes for a freshly-random (post-proof) sample and -/// recomputes both hashes from the SERVED content. This removes any dependency -/// on the auditor holding the peer's chunks. +/// chunk address), a `content_len`, and a `nonced_root` the responder computed +/// itself. Possession is proven in round 2 ([`verify_slice_response`]), where the +/// auditor opens a freshly-random (post-proof) block per sampled leaf and +/// verifies a Bao slice against the address plus a nonced-tree opening against +/// the round-1 `nonced_root` — over the SERVED bytes, so the auditor never holds +/// the peer's chunks. /// /// Returns [`StructureVerdict::Valid`] (proceed to round 2) or a confirmed /// [`AuditFailureReason`] mapped from the failing gate. @@ -500,6 +585,22 @@ pub(crate) fn evaluate_subtree_structure( if let StructureVerdict::Invalid(_) = verify_subtree_proof(proof, nonce, commitment) { return Err(AuditFailureReason::DigestMismatch); } + + // -- Content-address binding (possession-forgery guard) -- + // The commitment is only sound for CONTENT-ADDRESSED chunks, where the key IS + // the content hash (`key == BLAKE3(content)`), so an honest leaf is always + // `(key, key)` (see the commitment builder's `(k, k)` shortcut). The signed + // Merkle leaf hashes `key ‖ bytes_hash` WITHOUT forcing them equal, yet round 2 + // authenticates the served slice against `bytes_hash` while holder credit is + // recorded for `key`. A leaf with `bytes_hash != key` would therefore let a peer + // earn credit for an expensive address `key` by proving possession of an + // unrelated (e.g. one-byte) chunk hashing to `bytes_hash`. Reject any such leaf: + // for honest content-addressed data the two are identical, so this never fails + // an honest holder, and it re-binds Chain 1's `bytes_hash` check to the credited + // `key`. + if proof.leaves.iter().any(|l| l.bytes_hash != l.key) { + return Err(AuditFailureReason::DigestMismatch); + } Ok(()) } @@ -510,17 +611,19 @@ pub(crate) fn evaluate_subtree_structure( /// CRITICAL (ADR-0002 soundness): the sample MUST NOT be derivable from /// anything the responder knew when it built the round-1 proof. The structural /// root check binds only `(key, bytes_hash)` (both public — `bytes_hash` is the -/// chunk's network address), NOT `nonced_hash`. So a relay holding only public -/// addresses can fabricate a structurally-valid proof with bogus `nonced_hash` +/// chunk's network address), NOT `nonced_root`. So a relay holding only public +/// addresses can fabricate a structurally-valid proof with a bogus `nonced_root` /// on every leaf and, if it could predict which leaves round 2 opens, fetch /// only those and pass — earning holder credit for leaves it never held. /// /// Picking the sample with fresh CSPRNG randomness AFTER the proof is received /// turns round 1 into a commitment and round 2 into an unpredictable challenge /// (cut-and-choose): to pass with probability above `(1 - faked_fraction)^count` -/// the responder must have produced a correct `nonced_hash` — which requires the -/// real bytes — for essentially every leaf at round-1 commit time. The auditor -/// still holds none of the peer's chunks. +/// the responder must have produced a correct `nonced_root` — which requires the +/// real bytes under the fresh nonce — for essentially every leaf at round-1 +/// commit time. The auditor still holds none of the peer's chunks. The block +/// index opened within each sampled leaf is likewise drawn fresh +/// ([`random_block_index`]), so no single block can be prepared in advance. fn random_spotcheck_leaves( proof: &SubtreeProof, count: u32, @@ -553,54 +656,150 @@ fn random_spotcheck_leaves( .collect() } -/// Round-2 verdict (ADR-0002): the responder served the original chunk content -/// for the auditor's spot-check sample; verify possession from THAT content. +/// Draw a fresh-random 1 KiB block index in `0..block_count(content_len)`. /// -/// `served(key)` returns what the responder returned for a requested key: -/// `Some(Some(bytes))` for [`SubtreeByteItem::Present`], `Some(None)` for an -/// explicit [`SubtreeByteItem::Absent`], and `None` if the responder omitted the -/// key entirely (treated like `Absent` — a committed key it would not serve). +/// Called AFTER the round-1 proof is in hand, per sampled leaf, so the responder +/// could not have prepared only the opened block (the cut-and-choose property, +/// now at block granularity). +fn random_block_index(content_len: u32) -> u32 { + let count = crate::replication::slice::block_count(u64::from(content_len)); + if count <= 1 { + return 0; + } + rand::thread_rng().gen_range(0..count) +} + +/// The two block indices opened for one sampled leaf: a fresh-random block (the +/// cut-and-choose possession check) and the claimed final block. /// -/// For each sampled leaf the auditor recomputes, from the SERVED content: -/// - `BLAKE3(content) == leaf.bytes_hash` (the chunk's content address), AND -/// - `BLAKE3(nonce ‖ peer ‖ key ‖ content) == leaf.nonced_hash` (freshness), -/// i.e. `compute_audit_digest(nonce, peer, key, content)`. +/// The final block is opened so the auditor's Bao decode reaches EOF, which is +/// where Bao authenticates the encoded length against the address. Without it a +/// responder can forge a *shorter* `content_len` (which is not signed by the +/// round-1 commitment), collapse the challenge space, and pass while storing only +/// the prefix — the length header alone is not bound for a prefix slice that +/// never touches EOF. Opening the final block pins the true length: a forged +/// short length fails the final-block decode against the real address. /// -/// The freshness inputs are byte-identical to what the responder used to BUILD -/// the leaf in round 1 (`subtree_leaf` → `nonced_leaf_hash`): the SAME four -/// inputs, so an honest holder's served content reproduces `nonced_hash` -/// exactly. Round 1 commits over the data (the `nonced_hash` is uncomputable -/// without the bytes); round 2 reveals a random subset to prove the commitment -/// was not fabricated. +/// Returns one index when the random draw already lands on the final block (or +/// the chunk is a single block), two otherwise, never more than two. +fn block_indices_for_leaf(content_len: u32) -> Vec { + let count = crate::replication::slice::block_count(u64::from(content_len)); + let final_index = count.saturating_sub(1); + let random = random_block_index(content_len); + if random == final_index { + vec![final_index] + } else { + vec![random, final_index] + } +} + +/// Round-2 verdict (ADR-0002 / V2-685): the responder opened one 1 KiB block per +/// sampled leaf with a Bao verified slice plus a nonced block-tree opening; +/// verify possession from those. /// -/// Both checks are over the content the responder sent, so the auditor needs to -/// hold none of the peer's chunks. Any `Absent`/omitted committed key, or any -/// served content that fails a hash, is a provable lie → confirmed -/// [`AuditFailureReason::DigestMismatch`]. All sampled leaves verifying → -/// `Pass { checked }`. -pub(crate) fn verify_byte_response( - leaves: &[&crate::replication::subtree::SubtreeLeaf], +/// `openings` pairs each sampled leaf with the block index the auditor drew for +/// it. `items` is what the responder returned. For each opening the auditor: +/// 1. finds the responder's `Present` item for exactly this `(key, block_index)` +/// (a missing / `Absent` / wrong-block item is a provable lie); +/// 2. **Chain 1** — decodes the Bao slice against `leaf.bytes_hash` (the chunk +/// address), recovering the verified block bytes. This proves the block is +/// the real content at that offset, without the auditor holding the chunk. +/// 3. **Chain 2** — folds the nonced block leaf (recomputed from those verified +/// bytes under the audit nonce/peer/key/index) with the returned siblings to +/// `leaf.nonced_root`. This proves the responder committed a nonced tree +/// over the real content at round-1 time, so it held the bytes then. +/// +/// Both chains are over the SAME block bytes and the auditor holds none of the +/// peer's chunks. Any missing/absent opening, a slice that fails to decode +/// against the address, or a nonced opening that does not fold to the committed +/// root, is a provable lie → confirmed [`AuditFailureReason::DigestMismatch`]. +/// All openings verifying → `Pass { checked }`. +pub(crate) fn verify_slice_response( + openings: &[(crate::replication::subtree::SubtreeLeaf, u32)], nonce: &[u8; 32], challenged_peer_bytes: &[u8; 32], - served: impl Fn(&XorName) -> Option>>, + items: &[SubtreeSliceItem], ) -> AuditVerdict { + // Validate the response shape before matching. A well-formed response has at + // most one item per SOLICITED identity; the per-opening `find_map` below + // returns the FIRST match, so a duplicate `(key, block_index)` `Present`, a + // key that is both `Present` and `Absent`, a repeated `Absent`, or an + // UNSOLICITED item would let the responder's item order (or padding) decide + // the verdict. Bounding `items.len()` to the request count also keeps this + // check (and the matching below) at O(openings) — a peer cannot pad the + // response with thousands of unique items to force quadratic validation work. + if items.len() > openings.len() { + return AuditVerdict::Fail(AuditFailureReason::MalformedResponse); + } + let requested_blocks: HashSet<(XorName, u32)> = + openings.iter().map(|(leaf, i)| (leaf.key, *i)).collect(); + let requested_keys: HashSet = openings.iter().map(|(leaf, _)| leaf.key).collect(); + let mut present: HashSet<(XorName, u32)> = HashSet::new(); + let mut absent: HashSet = HashSet::new(); + for it in items { + let ok = match it { + SubtreeSliceItem::Present { + key, block_index, .. + } => { + requested_blocks.contains(&(*key, *block_index)) + && present.insert((*key, *block_index)) + && !absent.contains(key) + } + SubtreeSliceItem::Absent { key } => { + requested_keys.contains(key) + && absent.insert(*key) + && !present.iter().any(|(k, _)| k == key) + } + }; + if !ok { + return AuditVerdict::Fail(AuditFailureReason::MalformedResponse); + } + } + let mut checked = 0usize; - for leaf in leaves { - // Present{bytes} -> Some(Some(bytes)); Absent -> Some(None); omitted -> None. - // A committed key the responder cannot / will not serve is a provable lie. - let Some(Some(content)) = served(&leaf.key) else { + for (leaf, block_index) in openings { + let block_index = *block_index; + // Match the responder's item for exactly this (key, block_index). A + // missing item, an explicit Absent, or a different block is a provable lie. + let served = items.iter().find_map(|it| match it { + SubtreeSliceItem::Present { + key, + block_index: served_index, + bao_slice, + nonced_siblings, + } if key == &leaf.key && *served_index == block_index => { + Some(Some((bao_slice.as_slice(), nonced_siblings.as_slice()))) + } + SubtreeSliceItem::Absent { key } if key == &leaf.key => Some(None), + _ => None, + }); + let Some(Some((bao_slice, nonced_siblings))) = served else { + return AuditVerdict::Fail(AuditFailureReason::DigestMismatch); + }; + + // Chain 1: authenticate the block against the chunk address. + let Some(block) = crate::replication::slice::verify_block_slice( + bao_slice, + &leaf.bytes_hash, + u64::from(leaf.content_len), + block_index, + ) else { return AuditVerdict::Fail(AuditFailureReason::DigestMismatch); }; - let plain = *blake3::hash(&content).as_bytes(); - let nonced = crate::replication::subtree::nonced_leaf_hash( + + // Chain 2: prove the block was committed under round 1's fresh nonce, + // with the sibling chain pinned to the canonical depth for the chunk's + // block count (bound the claimed tree geometry). + if !crate::replication::slice::verify_nonced_block( nonce, challenged_peer_bytes, &leaf.key, - &content, - ); - if leaf.bytes_hash != plain || leaf.nonced_hash != nonced { - // Served content does not hash to the committed address / freshness - // hash: cannot be the chunk it committed to. + block_index, + &block, + nonced_siblings, + &leaf.nonced_root, + crate::replication::slice::block_count(u64::from(leaf.content_len)), + ) { return AuditVerdict::Fail(AuditFailureReason::DigestMismatch); } checked += 1; @@ -613,13 +812,14 @@ pub(crate) fn verify_byte_response( /// **Round 1** (this proof): pin + identity + signature + structure. If the /// proof structurally rebuilds to the pinned root, the tree SHAPE is committed — /// but not yet that the bytes are held. **Round 2**: the auditor picks a small -/// freshly-random (post-proof) sample of the just-proven leaves and sends a -/// [`SubtreeByteChallenge`] demanding their original chunk content FROM the -/// responder, then verifies that content against the committed `bytes_hash` -/// (content address) and `nonced_hash` (freshness). A responder that committed -/// to a chunk it no longer holds cannot serve content that hashes to the -/// committed address, so it fails — regardless of what the auditor holds. On a -/// full pass, credits the peer as a proven holder. +/// freshly-random (post-proof) sample of the just-proven leaves, draws a fresh +/// block index for each, and sends a [`SubtreeSliceChallenge`] opening those +/// blocks. It verifies each opened block against the committed `bytes_hash` +/// (Bao slice → content address) and `nonced_root` (nonced block-tree opening → +/// round-1 possession commit). A responder that committed to a chunk it no +/// longer held cannot have committed a correct `nonced_root`, so it fails — +/// regardless of what the auditor holds. On a full pass, credits the peer as a +/// proven holder. async fn verify_subtree_response( ctx: &AuditCtx<'_>, commitment: &StorageCommitment, @@ -640,13 +840,14 @@ async fn verify_subtree_response( return failed(challenged_peer, challenge_id, reason); } - // -- Round 2: surprise byte challenge for a 3..=5 FRESHLY-RANDOM sample. -- + // -- Round 2: surprise slice challenge for a 3..=5 FRESHLY-RANDOM sample. -- // The sample is chosen now, with CSPRNG randomness, AFTER the round-1 proof // is in hand — NOT derived from the round-1 nonce. The responder committed - // every leaf's `nonced_hash` in round 1 without knowing which leaves we will - // open, so it cannot have fabricated the un-opened ones (cut-and-choose). - // We cap the sample at the ADR's 3..=5 band (clamped to the subtree size) so - // the round-2 message and the responder's disk read stay cheap. + // every leaf's `nonced_root` in round 1 without knowing which leaves — or + // which block within each — we will open, so it could not have prepared only + // the opened blocks (cut-and-choose). The reply is a few KB per opening (a + // 1 KiB block plus two short hash chains), so one round-2 message serves the + // whole sample; no batching. let sample_n = ctx .config .audit_spotcheck_count() @@ -662,85 +863,74 @@ async fn verify_subtree_response( AuditFailureReason::DigestMismatch, ); } - // The sample is challenged in batches of MAX_BYTE_CHALLENGE_KEYS so each - // response — worst case, every requested chunk at MAX_CHUNK_SIZE — still - // encodes under MAX_REPLICATION_MESSAGE_SIZE. Each batch carries its own - // possession-in-time deadline (sized to its own length), so splitting does - // not widen the per-chunk window a relay would need to fetch over the - // network. - // - // CRITICAL: verify each batch's served bytes AS IT ARRIVES, against that - // batch's own sampled leaves, and return a CONFIRMED failure immediately. - // Deferring all verification until every batch is collected would let a - // later batch's timeout-lane Timeout (`round_failure`) mask a deterministic - // failure already proven by an earlier batch (an absent committed key or a - // hash mismatch) — a confirmed cheat would be downgraded to a timeout. A - // Timeout/Rejected/Malformed only becomes the verdict if NO earlier batch - // already produced confirmed bad bytes. - let verdict = 'rounds: { - for batch in sampled.chunks(MAX_BYTE_CHALLENGE_KEYS) { - let batch_keys: Vec = batch.iter().map(|l| l.key).collect(); - match request_byte_proof(ctx, &batch_keys).await { - ByteRound::Served(items) => { - // Verify THIS batch now. A confirmed failure here is final — - // a later batch's timeout must not be able to overwrite it. - let v = verify_byte_response( - batch, - &ctx.nonce, - challenged_peer.as_bytes(), - |key| { - items.iter().find_map(|it| match it { - SubtreeByteItem::Present { key: k, bytes } if k == key => { - Some(Some(bytes.clone())) - } - SubtreeByteItem::Absent { key: k } if k == key => Some(None), - _ => None, - }) - }, - ); - if let AuditVerdict::Fail(reason) = v { - break 'rounds AuditVerdict::Fail(reason); - } - } - // The responder rejected the byte challenge for a recently - // pinned commitment → confirmed failure, same as round 1. - ByteRound::Rejected => { - break 'rounds AuditVerdict::Fail(AuditFailureReason::Rejected) - } - // Transient reject (a local read error): ADR-0004 A1 routes it to - // the timeout lane — no trust penalty, but revoke the holder - // credit for THIS pinned commitment (the peer answered and could - // not prove possession) before taking the Timeout verdict. Scoped - // to the commitment hash, not the whole peer, so it never erases - // credit the peer re-earned for a newer commitment. - ByteRound::TransientReject => { - if let Some(credit) = ctx.credit { - credit - .recent_provers - .write() - .await - .forget_commitment(&ctx.expected_commitment_hash); - } - break 'rounds AuditVerdict::Fail(AuditFailureReason::Timeout); - } - // No response within the byte deadline (or transport error) → - // timeout (graced by the caller's strike policy — could be - // honest slowness). Keeps credit (a dropped packet is not - // evidence of loss). Only reached when no earlier batch already - // confirmed bad bytes. - ByteRound::Timeout => { - break 'rounds AuditVerdict::Fail(AuditFailureReason::Timeout) - } - // Malformed/unexpected round-2 body. - ByteRound::Malformed => { - break 'rounds AuditVerdict::Fail(AuditFailureReason::MalformedResponse) - } - } + // Pair each sampled leaf with up to two block indices: a fresh-random block + // (possession) and the claimed final block (a length pin — see + // `block_indices_for_leaf`). Own the leaves so the borrow on `proof` ends + // before the await. The sample is <= BYTE_SPOTCHECK_MAX and each leaf yields + // <= 2 openings, so the total is <= 2 * BYTE_SPOTCHECK_MAX <= MAX_SLICE_OPENINGS + // (statically asserted); `take` is a defensive backstop. + let openings_with_leaves: Vec<(crate::replication::subtree::SubtreeLeaf, u32)> = sampled + .iter() + .flat_map(|leaf| { + let leaf = (*leaf).clone(); + block_indices_for_leaf(leaf.content_len) + .into_iter() + .map(move |block_index| (leaf.clone(), block_index)) + }) + .take(MAX_SLICE_OPENINGS) + .collect(); + let openings: Vec = openings_with_leaves + .iter() + .map(|(leaf, block_index)| SubtreeSliceOpening { + key: leaf.key, + block_index: *block_index, + }) + .collect(); + + let round = request_slice_proof(ctx, &openings).await; + + // Round 2 ended without a completed possession proof: drop the holder credit + // this pinned commitment carries, before mapping the outcome to a verdict. + // Scoped to the commitment hash, so credit the peer re-earned for a newer + // commitment survives. + if round2_revokes_pinned_credit(&round) { + if let Some(credit) = ctx.credit { + credit + .recent_provers + .write() + .await + .forget_commitment(&ctx.expected_commitment_hash); + } + } + + let verdict = match round { + // The responder served openings: verify both chains for every one. Any + // failing chain is a confirmed cheat. + SliceRound::Served(items) => verify_slice_response( + &openings_with_leaves, + &ctx.nonce, + challenged_peer.as_bytes(), + &items, + ), + // Confirmed round-2 failures. `Rejected`: the responder repudiated the + // slice challenge for a recently pinned commitment. `ResponsiveBootstrap`: + // it claimed bootstrap AFTER producing a valid round-1 proof, which is a + // self-contradiction (a bootstrapping node has no committed data to + // prove). Both are classified as any other confirmed `Rejected`, which + // revokes the peer's holder credit and takes the trust penalty downstream + // (`apply_audit_failure_credit_revocation` → `forget_peer`). + SliceRound::Rejected | SliceRound::ResponsiveBootstrap => { + AuditVerdict::Fail(AuditFailureReason::Rejected) } - // Every batch served bytes that verified. - AuditVerdict::Pass { - checked: sampled.len(), + // Round 2 produced no proof, either as an explicit `Transient` (a local + // read error) or as silence. Both route to the timeout lane: no trust + // penalty, the strike policy still graces them, and the credit for the + // pinned commitment was already revoked above. + SliceRound::TransientReject | SliceRound::Timeout => { + AuditVerdict::Fail(AuditFailureReason::Timeout) } + // Malformed/unexpected round-2 body. + SliceRound::Malformed => AuditVerdict::Fail(AuditFailureReason::MalformedResponse), }; match verdict { @@ -761,7 +951,7 @@ async fn verify_subtree_response( } info!( "Audit: peer {challenged_peer} passed subtree audit ({} leaves, {checked} \ - byte-checked)", + block openings verified)", proof.leaves.len() ); AuditTickResult::Passed { @@ -896,6 +1086,74 @@ pub async fn handle_subtree_challenge( self_peer_id: &PeerId, is_bootstrapping: bool, commitment_state: Option<&Arc>, +) -> SubtreeAuditResponse { + handle_subtree_challenge_measured( + challenge, + storage, + self_peer_id, + is_bootstrapping, + commitment_state, + ) + .await + .response +} + +/// A round-1 response together with the chunk bytes spent producing it. +pub struct Round1Work { + /// What to send back. + pub response: SubtreeAuditResponse, + /// Chunk content read from LMDB and hashed BEFORE this response was + /// produced. + /// + /// Counted on the rejecting paths too, which is the point. A subtree is read + /// leaf by leaf, so a commitment holding one unreadable key still costs a + /// full run of reads and keyed-BLAKE3 passes over every leaf before it. If + /// only the `Proof` arm were charged, an attacker who found such a + /// commitment could replay subtrees over it indefinitely for free — and + /// since the per-peer cooldown is escapable by rotating identity, the + /// responder-wide work budget is the only bound that would have caught it. + pub content_bytes: i64, +} + +/// [`handle_subtree_challenge`], additionally reporting the read-and-hash work +/// it performed so the caller can charge it on every exit path. +pub async fn handle_subtree_challenge_measured( + challenge: &SubtreeAuditChallenge, + storage: &LmdbStorage, + self_peer_id: &PeerId, + is_bootstrapping: bool, + commitment_state: Option<&Arc>, +) -> Round1Work { + // The accumulator is threaded in rather than returned per-arm so that every + // exit reports its work by construction: a new early return cannot forget to + // account for the reads that already happened. + let mut content_bytes = 0i64; + let response = subtree_challenge_response( + challenge, + storage, + self_peer_id, + is_bootstrapping, + commitment_state, + &mut content_bytes, + ) + .await; + Round1Work { + response, + content_bytes, + } +} + +/// The round-1 responder proper. `content_bytes` accrues the chunk content read +/// and hashed so far, and is meaningful on every return path, not just the +/// successful one. +#[allow(clippy::too_many_lines)] +async fn subtree_challenge_response( + challenge: &SubtreeAuditChallenge, + storage: &LmdbStorage, + self_peer_id: &PeerId, + is_bootstrapping: bool, + commitment_state: Option<&Arc>, + content_bytes: &mut i64, ) -> SubtreeAuditResponse { if is_bootstrapping { return SubtreeAuditResponse::Bootstrapping { @@ -989,13 +1247,41 @@ pub async fn handle_subtree_challenge( }; } }; - leaves.push(crate::replication::subtree::subtree_leaf( - &challenge.nonce, - &challenge.challenged_peer_id, - key, - &bytes, - )); - // bytes drops here. + // Charge at the READ, not at the end. The disk read and the keyed-BLAKE3 + // pass below are both already owed at this point, and every remaining + // exit from this loop — a later missing key, a persistent read error, a + // failed hashing task — discards the leaves but not the work. Accruing + // here is what makes a commitment with one bad key cost the attacker + // something per replay instead of nothing. + *content_bytes = + content_bytes.saturating_add(i64::try_from(bytes.len()).unwrap_or(i64::MAX)); + // Hash the leaf (a full keyed-BLAKE3 pass over the chunk) on a blocking + // thread, not the async worker: a maximal subtree is ~sqrt(N) leaves of up + // to MAX_CHUNK_SIZE each, so doing this inline would tie up a Tokio worker + // for the whole proof and let a few round-1 requests starve the runtime. + // One chunk is resident at a time, so peak memory is bounded. + let nonce = challenge.nonce; + let peer = challenge.challenged_peer_id; + let leaf_key = *key; + let leaf = match tokio::task::spawn_blocking(move || { + crate::replication::subtree::subtree_leaf(&nonce, &peer, &leaf_key, &bytes) + }) + .await + { + Ok(leaf) => leaf, + Err(e) => { + warn!( + "Subtree audit: leaf hashing task failed for key {}: {e}", + hex::encode(key) + ); + return SubtreeAuditResponse::Rejected { + challenge_id: challenge.challenge_id, + kind: RejectKind::Transient, + reason: format!("leaf hashing task error: {e}"), + }; + } + }; + leaves.push(leaf); } SubtreeAuditResponse::Proof { @@ -1008,57 +1294,130 @@ pub async fn handle_subtree_challenge( } } -/// Handle a round-2 byte challenge (responder side), ADR-0002. +/// Build every requested opening for one committed key from a single hashing +/// pass over its bytes: a Bao verified slice (authenticity against the chunk +/// address) plus a nonced block-tree opening (possession against round 1's +/// `nonced_root`) per block index. +/// +/// The Bao outboard and nonced tree each hash the full chunk, so this builds +/// them once (via [`ChunkOpener`]) and serves all `indices` from them rather +/// than re-hashing per opening (V2-685 round-2 amplification fix). `indices` is +/// deduplicated by the caller. CPU-heavy, so callers run it on a blocking thread. +/// +/// Returns `Err((kind, reason))` for the terminal cases that abort the whole +/// response: a block index out of range (only a forged/buggy auditor sends one → +/// `Protocol`), or a surprise in-memory Bao extraction error (`Transient`, so an +/// honest holder is not branded a deleter). +fn build_slice_items_for_key( + nonce: [u8; 32], + peer: [u8; 32], + key: XorName, + bytes: &[u8], + indices: &[u32], +) -> Result, (RejectKind, String)> { + let opener = crate::replication::slice::ChunkOpener::new(&nonce, &peer, &key, bytes); + let count = opener.block_count(); + let mut items = Vec::with_capacity(indices.len()); + for &block_index in indices { + if block_index >= count { + return Err(( + RejectKind::Protocol, + format!( + "block index {block_index} out of range for key {}", + hex::encode(key) + ), + )); + } + let bao_slice = match opener.bao_slice(block_index) { + Ok(slice) => slice, + Err(e) => { + warn!( + "Subtree slice audit: bao extraction failed for key {}: {e}", + hex::encode(key) + ); + return Err((RejectKind::Transient, format!("bao extraction error: {e}"))); + } + }; + // `block_index < count` was just checked, so siblings must exist. A + // `None` here is an internal proof-building inconsistency, not an honest + // condition: abort the whole response as `Transient` (routes to the + // timeout lane) rather than emit an empty-siblings `Present` that would + // make an honest holder fail its own audit. + let Some(nonced_siblings) = opener.nonced_siblings(block_index) else { + warn!( + "Subtree slice audit: no nonced siblings for in-range block {block_index} of key {}", + hex::encode(key) + ); + return Err(( + RejectKind::Transient, + format!("nonced sibling build inconsistency for block {block_index}"), + )); + }; + items.push(SubtreeSliceItem::Present { + key, + block_index, + bao_slice, + nonced_siblings, + }); + } + Ok(items) +} + +/// Handle a round-2 slice challenge (responder side), ADR-0002 / V2-685. /// /// The auditor has already structurally verified this node's round-1 subtree -/// proof and now demands the ORIGINAL chunk bytes for a small freshly-random -/// sample of those leaves. For each requested key the responder either returns -/// the bytes ([`SubtreeByteItem::Present`]) or — if it committed to the key but -/// can no longer produce it — an explicit [`SubtreeByteItem::Absent`], which the -/// auditor counts as a provable failure (committing to bytes you don't hold). +/// proof and now opens up to two 1 KiB blocks (a fresh-random block plus the +/// final block) of a small freshly-random sample of those leaves. For each +/// opening the responder reads the committed chunk and builds a two-chain +/// opening (a Bao verified slice for authenticity against the chunk address, and +/// a nonced block-tree opening for possession against round 1's `nonced_root`), +/// returning [`SubtreeSliceItem::Present`]. If it committed to the key but can no +/// longer +/// produce the bytes it returns [`SubtreeSliceItem::Absent`], which the auditor +/// counts as a provable failure. /// /// A key the responder never committed to (not in the pinned tree) is also /// returned `Absent`: the auditor only ever samples keys it saw in round 1, so -/// in practice this guards against a malformed/forged byte challenge rather than -/// an honest mismatch. -pub async fn handle_subtree_byte_challenge( - challenge: &SubtreeByteChallenge, +/// in practice this guards against a malformed/forged challenge rather than an +/// honest mismatch. +pub async fn handle_subtree_slice_challenge( + challenge: &SubtreeSliceChallenge, storage: &LmdbStorage, self_peer_id: &PeerId, is_bootstrapping: bool, commitment_state: Option<&Arc>, -) -> SubtreeByteResponse { +) -> SubtreeSliceResponse { if is_bootstrapping { - return SubtreeByteResponse::Bootstrapping { + return SubtreeSliceResponse::Bootstrapping { challenge_id: challenge.challenge_id, }; } if challenge.challenged_peer_id != *self_peer_id.as_bytes() { - return SubtreeByteResponse::Rejected { + return SubtreeSliceResponse::Rejected { challenge_id: challenge.challenge_id, kind: RejectKind::Protocol, reason: "challenged_peer_id does not match this node".to_string(), }; } - // An honest auditor batches its sample to MAX_BYTE_CHALLENGE_KEYS per - // challenge so the worst-case response fits the wire cap. Reject larger - // requests up front: serving them could only produce an unencodable - // response (and invites disk-read amplification from a forged auditor). - if challenge.keys.len() > MAX_BYTE_CHALLENGE_KEYS { - let requested = challenge.keys.len(); - return SubtreeByteResponse::Rejected { + // An honest auditor opens at most MAX_SLICE_OPENINGS blocks per challenge. + // Reject larger requests up front: each opening forces a full chunk read to + // build its proof, so an oversized request is a disk-read amplification lever + // for a forged auditor. + if challenge.openings.len() > MAX_SLICE_OPENINGS { + let requested = challenge.openings.len(); + return SubtreeSliceResponse::Rejected { challenge_id: challenge.challenge_id, kind: RejectKind::Protocol, reason: format!( - "byte challenge requests {requested} keys; max {MAX_BYTE_CHALLENGE_KEYS} per challenge" + "slice challenge requests {requested} openings; max {MAX_SLICE_OPENINGS} per challenge" ), }; } let Some(state) = commitment_state else { - return SubtreeByteResponse::Rejected { + return SubtreeSliceResponse::Rejected { challenge_id: challenge.challenge_id, kind: RejectKind::Protocol, reason: "no commitment state".to_string(), @@ -1068,76 +1427,164 @@ pub async fn handle_subtree_byte_challenge( // retain it (rotated past it), reject as `UnknownCommitment`. With audit // grace removed (ADR-0004 A1) the auditor treats a responsive miss on an // in-window pin as a confirmed failure — answerability is restart-durable and - // pins are challenged only in-window. We serve bytes only for keys committed + // pins are challenged only in-window. We open blocks only for keys committed // under this pin. let Some(built) = state.lookup_by_hash(&challenge.expected_commitment_hash) else { - return SubtreeByteResponse::Rejected { + return SubtreeSliceResponse::Rejected { challenge_id: challenge.challenge_id, kind: RejectKind::UnknownCommitment, reason: "unknown commitment hash".to_string(), }; }; - let mut items = Vec::with_capacity(challenge.keys.len()); - for key in &challenge.keys { - // Serve ONLY keys committed under this pin. A key the auditor asks for - // that is not in the pinned tree is `Absent` — never served from local - // storage just because we happen to hold it (§15: serving an - // uncommitted-but-held key would let a forged challenge harvest bytes - // and muddy the possession proof, which must be about THIS commitment). - if built.proof_for(key).is_none() { - items.push(SubtreeByteItem::Absent { key: *key }); + // Coalesce openings by key, preserving first-seen order and deduplicating + // block indices per key, so each committed chunk is read from LMDB and hashed + // at most once even when the auditor opens several of its blocks (the normal + // random + final pair, or a forged duplicate). Without this a ten-opening + // request could re-read and re-hash the same chunk ten times. + let mut key_order: Vec = Vec::new(); + let mut indices_by_key: HashMap> = HashMap::new(); + for opening in &challenge.openings { + let entry = indices_by_key.entry(opening.key).or_default(); + if entry.is_empty() { + key_order.push(opening.key); + } + if !entry.contains(&opening.block_index) { + entry.push(opening.block_index); + } + } + + // Coalescing only saves work when keys REPEAT. A forged auditor could still + // spread its MAX_SLICE_OPENINGS across that many DISTINCT keys to force a + // full-chunk read each (up to ~40 MiB). An honest auditor samples at most + // BYTE_SPOTCHECK_MAX leaves, so cap distinct keys to that. + if key_order.len() > BYTE_SPOTCHECK_MAX as usize { + let distinct = key_order.len(); + return SubtreeSliceResponse::Rejected { + challenge_id: challenge.challenge_id, + kind: RejectKind::Protocol, + reason: format!( + "slice challenge spans {distinct} distinct keys; max {BYTE_SPOTCHECK_MAX}" + ), + }; + } + + let mut items = Vec::with_capacity(challenge.openings.len()); + for key in key_order { + let indices = indices_by_key.remove(&key).unwrap_or_default(); + // Open ONLY keys committed under this pin. A key not in the pinned tree + // is `Absent` — never served from local storage just because we happen to + // hold it (§15: serving an uncommitted-but-held key would let a forged + // challenge harvest data and muddy the possession proof for THIS commit). + if built.proof_for(&key).is_none() { + items.push(SubtreeSliceItem::Absent { key }); continue; } - match get_raw_retrying(storage, key).await { - // Committed key, bytes present → serve them. - Ok(Some(bytes)) => items.push(SubtreeByteItem::Present { key: *key, bytes }), - // Committed key, definitively absent → provable failure (§7: this is - // a real "I don't hold it" answer, distinct from a read error). - Ok(None) => { - warn!( - "Subtree byte audit: committed key {} requested but bytes absent", - hex::encode(key) - ); - items.push(SubtreeByteItem::Absent { key: *key }); - } - // Persistent transient read error after retries → do NOT brand the - // peer a deleter. Reject `Transient`; the auditor routes it to the - // timeout lane so a flaky LMDB read never manufactures a confirmed - // possession failure on an honest holder (which also gains no credit). - Err(e) => { - warn!( - "Subtree byte audit: storage read error for committed key {}: {e} \ - (rejecting as transient, not a confirmed failure)", - hex::encode(key) - ); - return SubtreeByteResponse::Rejected { - challenge_id: challenge.challenge_id, - kind: RejectKind::Transient, - reason: format!("transient storage read error: {e}"), - }; - } + match serve_committed_key_openings(challenge, storage, key, indices).await { + KeyServe::Items(mut built_items) => items.append(&mut built_items), + KeyServe::Absent => items.push(SubtreeSliceItem::Absent { key }), + KeyServe::Reject(reject) => return reject, } } - SubtreeByteResponse::Items { + SubtreeSliceResponse::Items { challenge_id: challenge.challenge_id, items, } } +/// Outcome of serving all requested openings for one committed key. +enum KeyServe { + /// Openings built for this key; append to the response. + Items(Vec), + /// Committed but the bytes are gone → provable `Absent`. + Absent, + /// A terminal condition (out-of-range index, read/build error) that aborts + /// the whole response. + Reject(SubtreeSliceResponse), +} + +/// Read a committed key's chunk once and build every requested opening from it. +/// +/// The Bao outboard + nonced tree hash the whole chunk, so the CPU-heavy build +/// runs on a blocking thread to keep an audit-responder flood off the Tokio pool. +/// `indices` is already deduplicated by the caller. +async fn serve_committed_key_openings( + challenge: &SubtreeSliceChallenge, + storage: &LmdbStorage, + key: XorName, + indices: Vec, +) -> KeyServe { + let reject = |kind, reason| { + KeyServe::Reject(SubtreeSliceResponse::Rejected { + challenge_id: challenge.challenge_id, + kind, + reason, + }) + }; + match get_raw_retrying(storage, &key).await { + Ok(Some(bytes)) => { + let nonce = challenge.nonce; + let peer = challenge.challenged_peer_id; + match tokio::task::spawn_blocking(move || { + build_slice_items_for_key(nonce, peer, key, &bytes, &indices) + }) + .await + { + Ok(Ok(built_items)) => KeyServe::Items(built_items), + Ok(Err((kind, reason))) => reject(kind, reason), + Err(e) => { + warn!( + "Subtree slice audit: proof build task failed for key {}: {e}", + hex::encode(key) + ); + reject( + RejectKind::Transient, + format!("proof build task error: {e}"), + ) + } + } + } + // Committed key, definitively absent → provable failure (§7: a real "I + // don't hold it" answer, distinct from a read error). + Ok(None) => { + warn!( + "Subtree slice audit: committed key {} requested but bytes absent", + hex::encode(key) + ); + KeyServe::Absent + } + // Persistent transient read error after retries → do NOT brand the peer a + // deleter. Reject `Transient`; the auditor routes it to the timeout lane + // so a flaky LMDB read never manufactures a confirmed possession failure + // on an honest holder (which also gains no credit). + Err(e) => { + warn!( + "Subtree slice audit: storage read error for committed key {}: {e} \ + (rejecting as transient, not a confirmed failure)", + hex::encode(key) + ); + reject( + RejectKind::Transient, + format!("transient storage read error: {e}"), + ) + } + } +} + #[cfg(test)] #[allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)] mod tests { use super::*; use crate::replication::commitment_state::BuiltCommitment; - use crate::replication::subtree::{build_subtree_proof, nonced_leaf_hash, SubtreeLeaf}; + use crate::replication::subtree::{build_subtree_proof, SubtreeLeaf}; use saorsa_pqc::api::sig::ml_dsa_65; + use std::time::Instant; /// ADR-0004 A1 grade flip (grace removed): a responsive `UnknownCommitment` /// or `Protocol` rejection is a CONFIRMED failure; only `Transient` routes to /// the timeout lane. This pure decision backs both audit rounds - /// (`Confirmed → AuditFailureReason::Rejected` / `ByteRound::Rejected`; + /// (`Confirmed → AuditFailureReason::Rejected` / `SliceRound::Rejected`; /// `TimeoutLane → AuditFailureReason::Timeout` + pinned-credit revocation). #[test] fn grade_reject_removes_grace_for_unknown_commitment() { @@ -1154,44 +1601,275 @@ mod tests { ); } + /// A peer id for classification tests (never dialled). + fn classify_peer() -> PeerId { + PeerId::from_bytes([0x5Au8; 32]) + } + + // A node that answers round 1 with a valid signed proof and then claims + // `Bootstrapping` in round 2 is contradicting itself: a bootstrapping node + // has no committed data to prove. Grading that as a graced timeout would be + // the cheapest possible way to dodge every possession check while KEEPING + // the holder credit earned in round 1 — so it must be a confirmed failure. + #[test] + fn responsive_bootstrap_is_a_confirmed_failure_not_a_graced_timeout() { + let id = 4242u64; + let round = classify_slice_response( + ReplicationMessageBody::SubtreeSliceResponse(SubtreeSliceResponse::Bootstrapping { + challenge_id: id, + }), + id, + &classify_peer(), + ); + assert!( + matches!(round, SliceRound::ResponsiveBootstrap), + "mid-audit Bootstrapping must classify as ResponsiveBootstrap" + ); + + // The production arm maps `ResponsiveBootstrap` alongside `Rejected` to + // `AuditFailureReason::Rejected`, and THAT reason is what revokes the + // peer's holder credit wholesale. Pin the policy both ways, since the + // whole point of the classification is landing on the revoking side. + assert!( + super::super::audit_failure_revokes_holder_credit(&AuditFailureReason::Rejected), + "a confirmed round-2 failure must revoke holder credit" + ); + assert!( + !super::super::audit_failure_revokes_holder_credit(&AuditFailureReason::Timeout), + "the timeout lane must not revoke credit WHOLESALE — the contrast that \ + makes classifying ResponsiveBootstrap as confirmed load-bearing. The \ + timeout lane still drops the pinned commitment's own credit; see \ + round2_revokes_pinned_credit" + ); + } + + /// A responder that answered round 1 and then goes quiet must not keep the + /// holder credit for the commitment under audit. + /// + /// Round 2 names the blocks after the roots are committed, so a responder + /// chooses whether to reply already knowing the draw. If silence kept + /// credit, answering only favourable draws would be free and standing earned + /// once would survive every dodged check afterwards. Both no-proof outcomes + /// therefore drop the pinned credit, while the confirmed lanes are excluded + /// here because they revoke the peer's credit wholesale downstream. + #[test] + fn round2_without_a_proof_drops_the_pinned_commitment_credit() { + assert!( + round2_revokes_pinned_credit(&SliceRound::Timeout), + "silence after a valid round-1 proof must cost the pinned credit" + ); + assert!( + round2_revokes_pinned_credit(&SliceRound::TransientReject), + "a transient reject also ends round 2 with no possession proof" + ); + assert!( + !round2_revokes_pinned_credit(&SliceRound::Served(vec![])), + "a served response is judged on its contents, not here" + ); + for confirmed in [ + SliceRound::Rejected, + SliceRound::ResponsiveBootstrap, + SliceRound::Malformed, + ] { + assert!( + !round2_revokes_pinned_credit(&confirmed), + "confirmed failures revoke wholesale downstream, not scoped here" + ); + } + } + + /// The scoped revocation must not erase credit the peer re-earned under a + /// different commitment: an unanswered round 2 costs the pin it was about, + /// not the peer's whole standing. + #[test] + fn pinned_credit_revocation_is_scoped_to_the_audited_commitment() { + let peer = classify_peer(); + let (audited, newer) = ([0x11u8; 32], [0x22u8; 32]); + let key: XorName = [0xC3u8; 32]; + let now = Instant::now(); + let mut provers = RecentProvers::new(); + provers.record_proof(key, peer, audited, now); + provers.record_proof(key, peer, newer, now); + + // What `verify_subtree_response` does when round 2 yields no proof. + provers.forget_commitment(&audited); + + assert!( + !provers.is_credited_holder(&key, &peer, &audited), + "credit for the audited pin must be gone after an unanswered round 2" + ); + assert!( + provers.is_credited_holder(&key, &peer, &newer), + "credit re-earned under a newer commitment must survive" + ); + } + + // Every classification arm is guarded on `challenge_id`. A responder must not + // be able to answer the challenge it was actually sent with a body minted for + // a different one — in particular it must not downgrade a live challenge by + // replaying a `Bootstrapping` or `Transient` body from another exchange. + #[test] + fn slice_response_for_another_challenge_is_malformed() { + let (sent, other) = (7u64, 8u64); + let peer = classify_peer(); + let bodies = [ + ReplicationMessageBody::SubtreeSliceResponse(SubtreeSliceResponse::Bootstrapping { + challenge_id: other, + }), + ReplicationMessageBody::SubtreeSliceResponse(SubtreeSliceResponse::Items { + challenge_id: other, + items: vec![], + }), + ReplicationMessageBody::SubtreeSliceResponse(SubtreeSliceResponse::Rejected { + challenge_id: other, + kind: RejectKind::Transient, + reason: String::new(), + }), + ]; + for body in bodies { + assert!( + matches!( + classify_slice_response(body, sent, &peer), + SliceRound::Malformed + ), + "a body carrying another challenge_id must be malformed" + ); + } + + // A non-round-2 body on the round-2 path is malformed too. + assert!(matches!( + classify_slice_response( + ReplicationMessageBody::SubtreeAuditResponse( + crate::replication::protocol::SubtreeAuditResponse::Bootstrapping { + challenge_id: sent, + } + ), + sent, + &peer, + ), + SliceRound::Malformed + )); + } + // The two-round audit splits into SHIPPED pure functions exercised directly // here (no reimplementation that could drift): // - round 1: `evaluate_subtree_structure` (pin/identity/signature + // structural root rebuild), // - sampling: `random_spotcheck_leaves` (3..=5 FRESHLY-RANDOM leaves chosen // after the proof is in hand — see its doc for the soundness argument), and - // - round 2: `verify_byte_response` (recompute content-address + freshness - // from the bytes the RESPONDER served — the auditor holds nothing). + // - round 2: `verify_slice_response` (decode the Bao slice against the chunk + // address + fold the nonced opening to the round-1 `nonced_root`, both from + // what the RESPONDER served — the auditor holds nothing). fn key(i: u32) -> XorName { let mut k = [0u8; 32]; k[..4].copy_from_slice(&i.to_be_bytes()); k } - /// The "chunk content" for a key in these fixtures. The committed tree's leaf - /// `bytes_hash` is `BLAKE3(chunk_bytes(key))`, mirroring the general - /// `(key, BLAKE3(content))` commitment; round 2 serves exactly this content. - fn chunk_bytes(k: &XorName) -> Vec { - let mut v = k.to_vec(); - v.extend_from_slice(b"chunk-body"); + + // The auditor's per-opening matcher returns the FIRST item with a matching + // identity, so an oversized, unsolicited, or colliding response must be + // rejected as malformed BEFORE matching — otherwise item order (or padding) + // could decide the verdict or force quadratic validation work. + #[test] + fn verify_slice_response_rejects_malformed_item_sets() { + let leaf = |k: XorName| SubtreeLeaf { + key: k, + bytes_hash: [0u8; 32], + content_len: 0, + nonced_root: [0u8; 32], + }; + let present = |k: XorName, i: u32| SubtreeSliceItem::Present { + key: k, + block_index: i, + bao_slice: vec![], + nonced_siblings: vec![], + }; + let malformed = |verdict| { + matches!( + verdict, + AuditVerdict::Fail(AuditFailureReason::MalformedResponse) + ) + }; + let nonce = [0u8; 32]; + let peer = [0u8; 32]; + // The auditor requested (key1, block 0), (key1, block 1), (key2, block 0). + let openings = vec![(leaf(key(1)), 0u32), (leaf(key(1)), 1), (leaf(key(2)), 0)]; + + // More items than requested openings. + let oversized = vec![ + present(key(1), 0), + present(key(1), 1), + present(key(2), 0), + present(key(2), 0), + ]; + assert!(malformed(verify_slice_response( + &openings, &nonce, &peer, &oversized + ))); + + // An item that was not requested. + let unsolicited = vec![present(key(9), 0)]; + assert!(malformed(verify_slice_response( + &openings, + &nonce, + &peer, + &unsolicited + ))); + + // Duplicate (key, block_index). + let dup = vec![present(key(1), 0), present(key(1), 0)]; + assert!(malformed(verify_slice_response( + &openings, &nonce, &peer, &dup + ))); + + // A key that is both Present and Absent. + let conflict = vec![present(key(1), 0), SubtreeSliceItem::Absent { key: key(1) }]; + assert!(malformed(verify_slice_response( + &openings, &nonce, &peer, &conflict + ))); + + // A well-formed, unique, solicited response passes the identity guard (the + // empty proofs then fail verification as DigestMismatch, NOT malformed). + let ok = vec![present(key(1), 0), present(key(1), 1), present(key(2), 0)]; + assert!(!malformed(verify_slice_response( + &openings, &nonce, &peer, &ok + ))); + } + /// Deterministic chunk content for fixture index `i`. Fixture keys are + /// CONTENT-ADDRESSED (`ckey(i) == BLAKE3(chunk_bytes(i))`), so a committed + /// leaf is `(key, key)` exactly as production, and round 2 serves this content. + fn chunk_bytes(i: u32) -> Vec { + let mut v = b"chunk-body".to_vec(); + v.extend_from_slice(&i.to_le_bytes()); v } + /// Content-addressed key for fixture index `i` (so `bytes_hash == key`). + fn ckey(i: u32) -> XorName { + *blake3::hash(&chunk_bytes(i)).as_bytes() + } + + /// The content behind a committed fixture key (reverse of `ckey`), so round-2 + /// fixtures can serve the real bytes for any sampled leaf. + fn content_for_key(k: &XorName) -> Vec { + (0..16_384u32) + .find(|&i| &ckey(i) == k) + .map(chunk_bytes) + .expect("fixture content for committed key") + } + /// Build an honest committed tree of `n` keys + a valid round-1 proof for /// `nonce`. Returns `(built, proof, peer_id)`. The auditor pins `built.hash()`. fn honest(n: u32, nonce: &[u8; 32]) -> (BuiltCommitment, SubtreeProof, [u8; 32]) { let (pk, sk) = ml_dsa_65().generate_keypair().unwrap(); let peer_id = *blake3::hash(&pk.to_bytes()).as_bytes(); let pk_b = pk.to_bytes(); - let entries: Vec<_> = (0..n) - .map(|i| { - let k = key(i); - (k, *blake3::hash(&chunk_bytes(&k)).as_bytes()) - }) - .collect(); + // Content-addressed: bytes_hash == key, exactly as production commits. + let entries: Vec<_> = (0..n).map(|i| (ckey(i), ckey(i))).collect(); let built = BuiltCommitment::build(entries, &peer_id, &sk, &pk_b).unwrap(); let proof = - build_subtree_proof(built.tree(), nonce, &peer_id, |k| Some(chunk_bytes(k))).unwrap(); + build_subtree_proof(built.tree(), nonce, &peer_id, |k| Some(content_for_key(k))) + .unwrap(); (built, proof, peer_id) } @@ -1205,8 +1883,8 @@ mod tests { evaluate_subtree_structure(built.commitment(), proof, nonce, &built.hash(), peer) } - /// The 3..=5 spot-check leaves the auditor would demand bytes for in round 2. - /// Now freshly-random (post-proof) rather than nonce-derived; the `_nonce`/ + /// The 3..=5 spot-check leaves the auditor would open in round 2. Now + /// freshly-random (post-proof) rather than nonce-derived; the `_nonce`/ /// `_key_count` params are kept so existing call sites read unchanged. fn sample<'a>( proof: &'a SubtreeProof, @@ -1216,12 +1894,43 @@ mod tests { random_spotcheck_leaves(proof, 8u32.clamp(BYTE_SPOTCHECK_MIN, BYTE_SPOTCHECK_MAX)) } - // A round-2 `served` closure that returns the HONEST content for every key. - // The nested-Option shape is the `verify_byte_response` callback contract: - // Present{bytes} -> Some(Some(bytes)); Absent -> Some(None); omitted -> None. - #[allow(clippy::option_option, clippy::unnecessary_wraps)] - fn served_honest(key: &XorName) -> Option>> { - Some(Some(chunk_bytes(key))) + /// Pair each sampled leaf with a block index for round 2. The fixtures use + /// short (single-block) chunks, so block 0 is the only block; the production + /// auditor draws a fresh random index via `random_block_index`. + fn openings_for(sample: &[&SubtreeLeaf]) -> Vec<(SubtreeLeaf, u32)> { + sample.iter().map(|l| ((*l).clone(), 0u32)).collect() + } + + /// Honest responder: for each opening build a real Bao slice + nonced opening + /// from the true chunk content, exactly as `handle_subtree_slice_challenge` + /// would. + fn served_honest_items( + openings: &[(SubtreeLeaf, u32)], + nonce: &[u8; 32], + peer: &[u8; 32], + ) -> Vec { + openings + .iter() + .map(|(leaf, block_index)| { + let content = content_for_key(&leaf.key); + let bao_slice = + crate::replication::slice::extract_block_slice(&content, *block_index).unwrap(); + let nonced_siblings = crate::replication::slice::nonced_block_siblings( + nonce, + peer, + &leaf.key, + &content, + *block_index, + ) + .unwrap(); + SubtreeSliceItem::Present { + key: leaf.key, + block_index: *block_index, + bao_slice, + nonced_siblings, + } + }) + .collect() } // ---- round 1: structure -------------------------------------------------- @@ -1232,15 +1941,44 @@ mod tests { let (built, proof, peer) = honest(400, &nonce); // Round 1. assert!(structure(&built, &proof, &nonce, &peer).is_ok()); - // Round 2: honest responder serves the real content for the sample. + // Round 2: honest responder opens real slices for the sample. let s = sample(&proof, &nonce, built.commitment().key_count); assert!(!s.is_empty()); - match verify_byte_response(&s, &nonce, &peer, served_honest) { + let openings = openings_for(&s); + let items = served_honest_items(&openings, &nonce, &peer); + match verify_slice_response(&openings, &nonce, &peer, &items) { AuditVerdict::Pass { checked } => assert!(checked >= 1, "must verify >=1 leaf"), other @ AuditVerdict::Fail(_) => panic!("expected Pass, got {other:?}"), } } + /// Possession-forgery guard: a peer that signs a commitment whose leaves + /// decouple the credited `key` from the authenticated content hash + /// (`bytes_hash != key`) is rejected at round 1 — even though the structural + /// root still rebuilds from `(key, bytes_hash)`. Without the guard such a peer + /// could earn holder credit for an expensive address `key` while only proving + /// possession of an unrelated (e.g. one-byte) chunk hashing to `bytes_hash`. + #[test] + fn leaf_with_bytes_hash_decoupled_from_key_is_rejected() { + let nonce = [5u8; 32]; + let (pk, sk) = ml_dsa_65().generate_keypair().unwrap(); + let peer_id = *blake3::hash(&pk.to_bytes()).as_bytes(); + let pk_b = pk.to_bytes(); + // Every leaf commits (key, bytes_hash) with bytes_hash != key. + let entries: Vec<_> = (0..64u32).map(|i| (ckey(i), ckey(i + 10_000))).collect(); + let built = BuiltCommitment::build(entries, &peer_id, &sk, &pk_b).unwrap(); + let proof = + build_subtree_proof(built.tree(), &nonce, &peer_id, |k| Some(content_for_key(k))) + .unwrap(); + // The structural root rebuilds (it is a genuinely signed tree), so the + // decoupled-address gate is what must reject it. + assert_eq!( + structure(&built, &proof, &nonce, &peer_id), + Err(AuditFailureReason::DigestMismatch), + "a leaf whose bytes_hash != key must be rejected at round 1" + ); + } + #[test] fn commitment_bound_to_another_peer_rejected() { let nonce = [3u8; 32]; @@ -1301,76 +2039,88 @@ mod tests { let (built, proof, peer) = honest(400, &nonce); assert!(structure(&built, &proof, &nonce, &peer).is_ok()); let s = sample(&proof, &nonce, built.commitment().key_count); - // Responder returns Absent for the FIRST sampled key, honest for the rest. - let victim = s.first().map(|l| l.key).unwrap(); - let v = verify_byte_response(&s, &nonce, &peer, |k| { - if *k == victim { - Some(None) // explicit Absent - } else { - Some(Some(chunk_bytes(k))) - } - }); + let openings = openings_for(&s); + // Responder returns Absent for the FIRST opening, honest for the rest. + let victim = openings.first().map(|(l, _)| l.key).unwrap(); + let mut items = served_honest_items(&openings, &nonce, &peer); + if let Some(slot) = items.first_mut() { + *slot = SubtreeSliceItem::Absent { key: victim }; + } + let v = verify_slice_response(&openings, &nonce, &peer, &items); assert_eq!(v, AuditVerdict::Fail(AuditFailureReason::DigestMismatch)); } #[test] fn omitted_committed_key_is_confirmed_failure() { - // A responder that simply omits a sampled committed key from its items + // A responder that simply omits a sampled committed opening from its items // (neither Present nor Absent) is treated identically to Absent: it // committed to the key and won't serve it → confirmed failure. let nonce = [9u8; 32]; let (built, proof, peer) = honest(400, &nonce); let s = sample(&proof, &nonce, built.commitment().key_count); - let victim = s.first().map(|l| l.key).unwrap(); - let v = verify_byte_response(&s, &nonce, &peer, |k| { - if *k == victim { - None // omitted entirely - } else { - Some(Some(chunk_bytes(k))) - } - }); + let openings = openings_for(&s); + let mut items = served_honest_items(&openings, &nonce, &peer); + items.remove(0); // omit the first opening entirely + let v = verify_slice_response(&openings, &nonce, &peer, &items); assert_eq!(v, AuditVerdict::Fail(AuditFailureReason::DigestMismatch)); } #[test] fn fake_storage_garbage_bytes_is_confirmed_failure() { - // A "fake-storage" responder claims possession but serves garbage. The - // garbage does not hash to the committed content address (`bytes_hash`), - // so the round-2 content-address check fails → confirmed failure. No - // auditor holdings involved. + // A "fake-storage" responder claims possession but opens a slice built + // from garbage content. The garbage slice does not decode against the + // committed content address (`bytes_hash`), so chain 1 fails → confirmed + // failure. No auditor holdings involved. let nonce = [9u8; 32]; let (built, proof, peer) = honest(400, &nonce); let s = sample(&proof, &nonce, built.commitment().key_count); - let v = verify_byte_response(&s, &nonce, &peer, |k| { - let mut garbage = blake3::hash(k).as_bytes().to_vec(); - garbage.extend_from_slice(b"adversary-fake-storage"); - Some(Some(garbage)) - }); + let openings = openings_for(&s); + let items: Vec = openings + .iter() + .map(|(leaf, bi)| { + let mut garbage = blake3::hash(&leaf.key).as_bytes().to_vec(); + garbage.extend_from_slice(b"adversary-fake-storage"); + let bao_slice = + crate::replication::slice::extract_block_slice(&garbage, *bi).unwrap(); + let nonced_siblings = crate::replication::slice::nonced_block_siblings( + &nonce, &peer, &leaf.key, &garbage, *bi, + ) + .unwrap(); + SubtreeSliceItem::Present { + key: leaf.key, + block_index: *bi, + bao_slice, + nonced_siblings, + } + }) + .collect(); + let v = verify_slice_response(&openings, &nonce, &peer, &items); assert_eq!(v, AuditVerdict::Fail(AuditFailureReason::DigestMismatch)); } #[test] - fn correct_content_address_but_stale_freshness_fails() { - // Suppose a responder could serve bytes that hash to the content address - // (it holds the chunk) — then BOTH checks pass; that is honest. But if - // it serves bytes whose freshness hash does not match (e.g. replaying a - // different nonce's digest is impossible since we recompute it here), the - // freshness check must catch any content that doesn't reproduce the - // committed `nonced_hash`. We model a leaf whose committed nonced_hash was - // built under a DIFFERENT nonce, so the audit nonce's recompute differs. + fn correct_content_address_but_stale_nonced_root_fails() { + // A responder can serve the real block (chain 1, the Bao slice against the + // address, passes), but if its committed `nonced_root` does not correspond + // to the audit's nonce over that content, the nonced opening (chain 2) + // cannot fold to it. We model a leaf whose committed `nonced_root` was + // built under a DIFFERENT nonce; the honest opening under the audit nonce + // then fails to match it. let nonce = [9u8; 32]; let (built, mut proof, peer) = honest(400, &nonce); - // Rewrite EVERY leaf's nonced_hash to one bound to a different nonce but - // keep its bytes_hash correct (so each leaf's content-address check is - // fine; only freshness is wrong). Tampering all leaves means the - // freshly-random sample is guaranteed to land on a stale-freshness leaf. let other_nonce = [0xEEu8; 32]; for leaf in &mut proof.leaves { - leaf.nonced_hash = - nonced_leaf_hash(&other_nonce, &peer, &leaf.key, &chunk_bytes(&leaf.key)); + leaf.nonced_root = crate::replication::slice::nonced_block_root( + &other_nonce, + &peer, + &leaf.key, + &content_for_key(&leaf.key), + ); } let s = sample(&proof, &nonce, built.commitment().key_count); - let v = verify_byte_response(&s, &nonce, &peer, served_honest); + let openings = openings_for(&s); + let items = served_honest_items(&openings, &nonce, &peer); + let v = verify_slice_response(&openings, &nonce, &peer, &items); assert_eq!(v, AuditVerdict::Fail(AuditFailureReason::DigestMismatch)); } @@ -1384,8 +2134,13 @@ mod tests { let (built, proof, peer) = honest(256, &nonce); assert!(structure(&built, &proof, &nonce, &peer).is_ok()); let s = sample(&proof, &nonce, built.commitment().key_count); - // Responder is a total deleter: Absent for everything. - let v = verify_byte_response(&s, &nonce, &peer, |_| Some(None)); + let openings = openings_for(&s); + // Responder is a total deleter: Absent for every opening. + let items: Vec = openings + .iter() + .map(|(l, _)| SubtreeSliceItem::Absent { key: l.key }) + .collect(); + let v = verify_slice_response(&openings, &nonce, &peer, &items); assert_eq!(v, AuditVerdict::Fail(AuditFailureReason::DigestMismatch)); } @@ -1410,8 +2165,10 @@ mod tests { let nonce = [11u8; 32]; let (built, proof, peer) = honest(400, &nonce); let s = sample(&proof, &nonce, built.commitment().key_count); - match verify_byte_response(&s, &nonce, &peer, served_honest) { - AuditVerdict::Pass { checked } => assert_eq!(checked, s.len()), + let openings = openings_for(&s); + let items = served_honest_items(&openings, &nonce, &peer); + match verify_slice_response(&openings, &nonce, &peer, &items) { + AuditVerdict::Pass { checked } => assert_eq!(checked, openings.len()), other @ AuditVerdict::Fail(_) => panic!("expected Pass, got {other:?}"), } } @@ -1420,9 +2177,9 @@ mod tests { #[test] fn structure_fail_short_circuits_before_round_2() { - // A structurally invalid proof is rejected in round 1; the byte challenge + // A structurally invalid proof is rejected in round 1; the slice challenge // is never issued. We assert the round-1 gate returns Err so the auditor - // (verify_subtree_response) never reaches request_byte_proof. + // (verify_subtree_response) never reaches request_slice_proof. let nonce = [5u8; 32]; let (built, mut proof, peer) = honest(300, &nonce); if let Some(first) = proof.leaves.first_mut() { @@ -1431,23 +2188,27 @@ mod tests { assert!(structure(&built, &proof, &nonce, &peer).is_err()); } - /// Build an honest committed tree whose keys are deliberately "FAR": their - /// addresses live at the high end of the XOR space (top bytes = 0xFF). On the - /// auditor side these are the leaves `observe_closeness` counts toward `far`. + /// Build an honest committed tree whose keys are content-addressed but biased + /// to the FAR half of the XOR space (top bit set), so `observe_closeness` + /// counts them toward `far`. Keys stay content-addressed (`bytes_hash == key`) + /// so round 2 serves real bytes. fn honest_far(n: u32, nonce: &[u8; 32]) -> (BuiltCommitment, SubtreeProof, [u8; 32]) { let (pk, sk) = ml_dsa_65().generate_keypair().unwrap(); let peer_id = *blake3::hash(&pk.to_bytes()).as_bytes(); let pk_b = pk.to_bytes(); - let entries: Vec<_> = (0..n) - .map(|i| { - let mut k = [0xFFu8; 32]; - k[28..].copy_from_slice(&i.to_be_bytes()); - (k, *blake3::hash(&chunk_bytes(&k)).as_bytes()) - }) - .collect(); + let mut entries: Vec<(XorName, [u8; 32])> = Vec::new(); + let mut i = 0u32; + while entries.len() < n as usize { + let k = ckey(i); + if k[0] >= 0x80 { + entries.push((k, k)); + } + i = i.saturating_add(1); + } let built = BuiltCommitment::build(entries, &peer_id, &sk, &pk_b).unwrap(); let proof = - build_subtree_proof(built.tree(), nonce, &peer_id, |k| Some(chunk_bytes(k))).unwrap(); + build_subtree_proof(built.tree(), nonce, &peer_id, |k| Some(content_for_key(k))) + .unwrap(); (built, proof, peer_id) } @@ -1461,12 +2222,24 @@ mod tests { let (built_far, proof_far, peer_far) = honest_far(400, &nonce); assert!(structure(&built_far, &proof_far, &nonce, &peer_far).is_ok()); let sf = sample(&proof_far, &nonce, built_far.commitment().key_count); - let v_far = verify_byte_response(&sf, &nonce, &peer_far, served_honest); + let of = openings_for(&sf); + let v_far = verify_slice_response( + &of, + &nonce, + &peer_far, + &served_honest_items(&of, &nonce, &peer_far), + ); let (built_near, proof_near, peer_near) = honest(400, &nonce); assert!(structure(&built_near, &proof_near, &nonce, &peer_near).is_ok()); let sn = sample(&proof_near, &nonce, built_near.commitment().key_count); - let v_near = verify_byte_response(&sn, &nonce, &peer_near, served_honest); + let on = openings_for(&sn); + let v_near = verify_slice_response( + &on, + &nonce, + &peer_near, + &served_honest_items(&on, &nonce, &peer_near), + ); match (&v_far, &v_near) { (AuditVerdict::Pass { checked: cf }, AuditVerdict::Pass { checked: cn }) => { @@ -1486,7 +2259,8 @@ mod tests { let _l = SubtreeLeaf { key: key(1), bytes_hash: [0u8; 32], - nonced_hash: [0u8; 32], + content_len: 0, + nonced_root: [0u8; 32], }; } } diff --git a/src/replication/subtree.rs b/src/replication/subtree.rs index 1aa73faf..2fc8e2ee 100644 --- a/src/replication/subtree.rs +++ b/src/replication/subtree.rs @@ -14,10 +14,11 @@ //! 1. **Structure** — [`verify_subtree_proof`] re-derives the selected branch //! from `(nonce, key_count)`, rebuilds the root from the returned leaves and //! cut-hashes, and requires it to equal the pinned root. -//! 2. **Real bytes** — [`select_spotcheck_indices`] picks a few leaves within -//! the subtree; the caller fetches their bytes and checks both the plain -//! content hash and the nonce freshness hash. Faking a fraction `x` of -//! leaves survives only `(1 - x)^k`. +//! 2. **Verified slice** — [`select_spotcheck_indices`] picks a few leaves +//! within the subtree; the caller opens one random block per leaf and checks +//! a Bao slice against the chunk address plus a nonced-tree opening against +//! the round-1 keyed `nonced_root`. Faking a fraction `x` of leaves survives +//! only `(1 - x)^k`. //! //! ## Tree geometry (must match [`super::commitment::MerkleTree`]) //! @@ -31,7 +32,6 @@ //! intersected with `0..N`. use super::commitment::{leaf_hash, node_hash, StorageCommitment, MAX_COMMITMENT_KEY_COUNT}; -use super::protocol::compute_audit_digest; use crate::ant_protocol::XorName; use serde::{Deserialize, Serialize}; @@ -44,14 +44,29 @@ pub const SMALL_TREE_FULL_AUDIT_FLOOR: u32 = 4; pub struct SubtreeLeaf { /// The committed key (chunk address) at this leaf position. pub key: XorName, - /// `BLAKE3(record_bytes)` — the plain content hash. This is also the - /// chunk's network address, so it is public; possessing it does NOT prove - /// possession of the bytes (that is what `nonced_hash` is for). + /// `BLAKE3(record_bytes)` — the plain content hash. For a content-addressed + /// chunk this EQUALS `key` (the address IS the content hash), and the round-1 + /// verifier enforces `bytes_hash == key` so credit for `key` cannot be earned + /// by proving possession of unrelated bytes. It is public; possessing it does + /// NOT prove possession of the bytes (that is what `nonced_root` is for). It is + /// also the BLAKE3/Bao root the round-2 slice verifies against. pub bytes_hash: [u8; 32], - /// `compute_audit_digest(nonce, peer_id, key, record_bytes)` — the - /// freshness hash. Only a holder of the actual bytes can produce it for a - /// fresh nonce, so a spot-check on it proves real possession. - pub nonced_hash: [u8; 32], + /// Length of the chunk's content, in bytes. Lets the auditor draw a random + /// 1 KiB block index in range and size the Bao slice for the final (short) + /// block. This field is NOT bound by the signed commitment (the Merkle leaf + /// hashes only `key ‖ bytes_hash`), so the round-2 verifier must not trust it + /// blind: `verify_block_slice` authenticates it against the Bao slice's own + /// length header (validated against the address), rejecting any claim that + /// disagrees with the true content length. Without that check a deflated + /// `content_len` would collapse the challenge to block 0 and forge possession + /// of a large chunk from a ~1 KiB prefix slice. + pub content_len: u32, + /// Root of the responder's fresh **nonced block tree** for this chunk (see + /// [`crate::replication::slice`]): a Merkle root over the chunk's 1 KiB + /// blocks whose leaves each bind the fresh nonce, peer, key and block bytes. + /// Building it requires every byte of the chunk under the fresh nonce, so it + /// commits real possession; round 2 opens one random block against it. + pub nonced_root: [u8; 32], } /// A responder's single-contiguous-subtree proof (ADR-0002 "The proof"). @@ -108,7 +123,8 @@ fn tree_depth(key_count: u32) -> Option { /// leaves. Pure function of geometry — identical on auditor and responder. /// /// `span = 2^(total_depth - depth)`; the node covers `[slot*span, (slot+1)*span)` -/// clamped to `0..key_count`. +/// clamped to `0..key_count`. Test-only geometry helper. +#[cfg(test)] #[must_use] fn real_leaves_under(depth: u32, slot: u64, key_count: u32, total_depth: u32) -> u32 { let levels_below = total_depth - depth; @@ -144,27 +160,44 @@ fn sqrt_floor(key_count: u32) -> u32 { u32::try_from(ceil.max(1)).unwrap_or(u32::MAX) } -/// Read bit `index` of the nonce (bit 0 = MSB of byte 0), `index` 0-based. +/// A stable `u64` drawn from the first 8 bytes of the nonce, used to pick the +/// subtree slot uniformly. The nonce is fresh CSPRNG per audit, so modulo bias +/// over the small slot count (≤ ~1000) is negligible. +#[must_use] +fn nonce_u64(nonce: &[u8; 32]) -> u64 { + let mut b = [0u8; 8]; + b.copy_from_slice(&nonce[..8]); + u64::from_le_bytes(b) +} + +/// The largest real-leaf count any selected subtree can have for `key_count`. /// -/// `1 → left child, 0 → right child` (ADR). With a 256-bit nonce and a tree -/// depth ≤ 20 we never run out of bits. +/// A full fixed-depth block of `next_power_of_two(ceil(sqrt(key_count)))` leaves. +/// Bounds a single round-1 request's read/hash work (≤ 1024 leaves at the +/// `MAX_COMMITMENT_KEY_COUNT` cap). Used both to size selection and as a +/// pre-read defense-in-depth ceiling in [`subtree_plan`]. #[must_use] -fn nonce_bit(nonce: &[u8; 32], index: u32) -> bool { - let byte = (index / 8) as usize; - let bit = 7 - (index % 8); - // byte < 32 because index < 256 for any reachable depth; guard anyway. - nonce.get(byte).is_some_and(|b| (b >> bit) & 1 == 1) +pub fn max_subtree_leaves(key_count: u32) -> u32 { + if key_count <= SMALL_TREE_FULL_AUDIT_FLOOR { + return key_count; + } + sqrt_floor(key_count).next_power_of_two() } /// Deterministically select one contiguous subtree from `(nonce, key_count)`. /// -/// Walks the nonce bits from the root, descending into the child the bit picks, -/// and **stops at the smallest branch whose real-leaf count is still ≥ -/// `ceil(sqrt(key_count))`**. Because an all-padding child has zero real leaves -/// (< the floor), the walk never descends into one — so the selection always -/// covers ≥ `sqrt` real leaves and is never empty (ADR dead-block fix). +/// Picks one **fixed-depth block** of `span = next_power_of_two(ceil(sqrt(N)))` +/// leaves, choosing the block uniformly by `nonce_u64 % slot_count`. Every block +/// (including the short tail) is selectable with roughly equal probability, so +/// the whole tree is covered over many audits, and a single request always reads +/// at most `span` (≤ ~sqrt(N), ≤ 1024 at the cap) leaves — regardless of the +/// nonce. This replaces the old nonce-bit descent, whose stop-at-parent rule let +/// an adversarial nonce select a subtree far larger than `sqrt(N)` (up to the +/// whole tree at `2^k + 1`), an unbounded round-1 work amplification. /// -/// For `key_count <= SMALL_TREE_FULL_AUDIT_FLOOR` the whole tree is selected. +/// Unpredictability is preserved: the responder cannot know the selected slot +/// until it receives the fresh nonce. For `key_count <= SMALL_TREE_FULL_AUDIT_FLOOR` +/// the whole tree is selected. /// /// Returns `None` only for an out-of-protocol `key_count` (caller rejects). #[must_use] @@ -181,36 +214,28 @@ pub fn select_subtree_path(nonce: &[u8; 32], key_count: u32) -> Option= sqrt(N), aligned to a tree level so the block IS a + // single subtree node at `depth = total_depth - log2(span)`. + let span = sqrt_floor(key_count).next_power_of_two(); + let span_log2 = span.trailing_zeros(); + // span <= next_power_of_two(key_count) == 2^total_depth, so span_log2 <= + // total_depth and depth is non-negative. + let depth = total_depth.saturating_sub(span_log2); + let slot_count = key_count.div_ceil(span); + let slot = u32::try_from(nonce_u64(nonce) % u64::from(slot_count)).unwrap_or(0); + + let span64 = u64::from(span); + let leaf_start = u32::try_from( + u64::from(slot) + .saturating_mul(span64) .min(u64::from(key_count)), ) .unwrap_or(key_count); + let leaf_end = leaf_start.saturating_add(span).min(key_count); Some(SubtreePath { depth, - slot: u32::try_from(slot).unwrap_or(u32::MAX), + slot, leaf_start, leaf_end, }) @@ -221,13 +246,14 @@ pub fn select_subtree_path(nonce: &[u8; 32], key_count: u32) -> Option, levels: u32) -> [u8; 32] { level.first().copied().unwrap_or([0u8; 32]) } -/// Build the per-leaf nonced freshness hash for a subtree leaf (responder -/// side), reusing the existing audit digest. -#[must_use] -pub fn nonced_leaf_hash( - nonce: &[u8; 32], - challenged_peer_id: &[u8; 32], - key: &XorName, - record_bytes: &[u8], -) -> [u8; 32] { - compute_audit_digest(nonce, challenged_peer_id, key, record_bytes) -} - /// Why a responder could not build a subtree proof for a challenge. #[derive(Debug, Clone, PartialEq, Eq)] pub enum BuildProofError { @@ -505,6 +519,14 @@ pub fn subtree_plan( let key_count = tree.key_count(); let path = select_subtree_path(nonce, key_count).ok_or(BuildProofError::BadKeyCount)?; + // Defense-in-depth: the fixed-depth selection already bounds this to one + // block of `max_subtree_leaves` leaves, but reject BEFORE any leaf read if a + // future selection change ever produced an oversized subtree, so one request + // can never force unbounded round-1 read/hash work. + if path.real_leaf_count() > max_subtree_leaves(key_count) { + return Err(BuildProofError::BadKeyCount); + } + let mut leaf_keys = Vec::with_capacity(path.real_leaf_count() as usize); for idx in path.leaf_start..path.leaf_end { let key = tree @@ -513,17 +535,16 @@ pub fn subtree_plan( leaf_keys.push(key); } - // Sibling cut-hashes, root-first. At descent step `d` (0-based from the - // root), the chosen child is on the side the nonce bit picks; the sibling - // is the other child at level `total_depth - (d + 1)` (counting up from - // leaves). On an odd-length level the missing sibling self-pairs, i.e. the - // sibling hash is the chosen node itself. + // Sibling cut-hashes, root-first. The fixed-depth slot selection no longer + // follows a per-level nonce walk, so derive the on-path node at each level + // from `path.slot`'s prefix bits: the top `d + 1` bits give the node index at + // level `d + 1` below the root. The sibling is the other child at level + // `total_depth - (d + 1)` (counting up from leaves); on an odd level the + // missing sibling self-pairs (the chosen node is its own sibling). let total_depth = u32::try_from(tree.levels_count().saturating_sub(1)).unwrap_or(0); let mut sibling_cut_hashes = Vec::with_capacity(path.depth as usize); - let mut slot = 0u64; for d in 0..path.depth { - let go_left = nonce_bit(nonce, d); - let child = slot * 2 + u64::from(!go_left); + let child = u64::from(path.slot) >> (path.depth - (d + 1)); let sibling = child ^ 1; let level_from_leaves = (total_depth - (d + 1)) as usize; let chosen_hash = tree.node_at(level_from_leaves, child); @@ -532,7 +553,6 @@ pub fn subtree_plan( .or(chosen_hash) .ok_or(BuildProofError::BadKeyCount)?; sibling_cut_hashes.push(sib_hash); - slot = child; } Ok(SubtreePlan { @@ -542,6 +562,10 @@ pub fn subtree_plan( } /// Build one subtree leaf from its key and the chunk bytes the responder holds. +/// +/// The plain `bytes_hash` is the chunk's content address; the `nonced_root` is +/// the fresh per-audit nonced block-tree root over the same bytes, committing +/// possession at round-1 time (see [`crate::replication::slice`]). #[must_use] pub fn subtree_leaf( nonce: &[u8; 32], @@ -552,7 +576,13 @@ pub fn subtree_leaf( SubtreeLeaf { key: *key, bytes_hash: *blake3::hash(bytes).as_bytes(), - nonced_hash: nonced_leaf_hash(nonce, challenged_peer_id, key, bytes), + content_len: u32::try_from(bytes.len()).unwrap_or(u32::MAX), + nonced_root: crate::replication::slice::nonced_block_root( + nonce, + challenged_peer_id, + key, + bytes, + ), } } @@ -609,28 +639,83 @@ mod tests { // ---- select_subtree_path: dead-block regression ----------------------- #[test] - fn selection_never_empty_across_many_sizes_and_nonces() { + fn selection_never_empty_and_within_ceiling_across_sizes_and_nonces() { for n in [ 5u32, 6, 7, 9, 13, 17, 33, 65, 100, 129, 333, 1000, 1024, 1025, ] { - let floor = sqrt_floor(n); + let ceiling = max_subtree_leaves(n); for seed in 0u8..=255 { let path = select_subtree_path(&nonce_of(seed), n).unwrap(); - assert!( - path.real_leaf_count() >= floor.min(n), - "n={n} seed={seed}: real={} < floor={floor}", - path.real_leaf_count() - ); + // Never empty (ADR dead-block fix). assert!( path.real_leaf_count() >= 1, "n={n} seed={seed}: empty selection" ); + // Bounded (DoS fix): one request reads at most one fixed-depth + // block, regardless of the (possibly adversarial) nonce. + assert!( + path.real_leaf_count() <= ceiling, + "n={n} seed={seed}: real={} > ceiling={ceiling}", + path.real_leaf_count() + ); assert!(path.leaf_end <= n); assert!(path.leaf_start < path.leaf_end); } } } + /// The denial-of-service the fixed-depth selection closes: under the OLD + /// nonce-bit descent, `key_count = 2^19 + 1` with a nonce steering to the + /// sparse side selected the WHOLE tree (~2 TiB of reads). Now every nonce + /// selects at most one `max_subtree_leaves`-sized block, and the tail slot is + /// a single leaf — not + /// the root. + #[test] + fn adversarial_nonce_cannot_amplify_round1_work() { + for &n in &[513u32, 1025, 524_289, 1_000_000, MAX_COMMITMENT_KEY_COUNT] { + let ceiling = max_subtree_leaves(n); + // A generous cap: sqrt-scale, never the whole tree for a large N. + assert!( + u64::from(ceiling) <= 1024.max(u64::from(n)), + "n={n}: ceiling {ceiling} not sqrt-scale" + ); + for nonce in [[0x00u8; 32], [0xFFu8; 32], [0xAAu8; 32], [0x55u8; 32]] { + let path = select_subtree_path(&nonce, n).unwrap(); + assert!( + path.real_leaf_count() <= ceiling && path.real_leaf_count() >= 1, + "n={n}: real={} outside 1..={ceiling}", + path.real_leaf_count() + ); + if n >= 512 { + assert!( + u64::from(path.real_leaf_count()) < u64::from(n), + "n={n}: a single request must not cover the whole tree" + ); + } + } + } + } + + /// Every slot is selectable and the slots partition `0..key_count` with no + /// gaps, so the tail leaf is never permanently unauditable (coverage is + /// uniform over many audits). + #[test] + fn slots_partition_all_leaves_with_no_gaps() { + for &n in &[5u32, 17, 100, 1025, 524_289] { + let span = max_subtree_leaves(n); + let slot_count = n.div_ceil(span); + let mut covered = 0u32; + for slot in 0..slot_count { + let start = slot.saturating_mul(span); + let end = start.saturating_add(span).min(n); + assert_eq!(start, covered, "n={n} slot={slot}: gap before this slot"); + assert!(end > start, "n={n} slot={slot}: empty slot"); + covered = end; + } + assert_eq!(covered, n, "n={n}: slots do not cover all leaves"); + } + } + #[test] fn small_trees_select_whole_tree() { for n in 1..=SMALL_TREE_FULL_AUDIT_FLOOR { @@ -980,25 +1065,27 @@ mod tests { } #[test] - fn fabricated_nonced_hash_caught_by_spotcheck_probability() { + fn fabricated_nonced_root_caught_by_spotcheck_probability() { // Simulate the realness check: a responder fabricates a fraction x of - // nonced hashes. The auditor spot-checks k leaves; probability all k - // land on honest leaves is (1-x)^k. Here we just assert the auditor - // *would* catch a fabricated leaf when it samples that position. + // nonced roots. The auditor opens k leaves; probability all k land on + // honest leaves is (1-x)^k. Here we just assert the auditor *would* catch + // a fabricated leaf: its committed root differs from the honest root + // recomputed from the real chunk bytes under the audit nonce. let peer = [1u8; 32]; let entries = entries_for(400); let nonce = nonce_of(9); let (mut proof, _commitment) = build_proof(&entries, &nonce, &peer); - // Fabricate the nonced hash on the first subtree leaf (wrong bytes). - proof.leaves[0].nonced_hash[0] ^= 0xFF; - // The realness check the caller runs: recompute from the real chunk - // bytes (the same fixture the honest tree was built from). - let leaf = &proof.leaves[0]; + // Fabricate the nonced root on the first subtree leaf. + if let Some(first) = proof.leaves.first_mut() { + first.nonced_root[0] ^= 0xFF; + } + let leaf = proof.leaves.first().expect("proof has leaves"); let real_bytes = chunk_bytes(&leaf.key); - let expected = nonced_leaf_hash(&nonce, &peer, &leaf.key, &real_bytes); + let expected = + crate::replication::slice::nonced_block_root(&nonce, &peer, &leaf.key, &real_bytes); assert_ne!( - leaf.nonced_hash, expected, - "fabricated nonced hash must differ from real" + leaf.nonced_root, expected, + "fabricated nonced root must differ from real" ); } diff --git a/tests/e2e/replication.rs b/tests/e2e/replication.rs index 14d3b5a7..782d2caa 100644 --- a/tests/e2e/replication.rs +++ b/tests/e2e/replication.rs @@ -10,7 +10,8 @@ use super::TestHarness; use ant_node::client::compute_address; use ant_node::replication::commitment_state::{BuiltCommitment, ResponderCommitmentState}; use ant_node::replication::config::{ - storage_admission_width, K_BUCKET_SIZE, REPLICATION_PROTOCOL_ID, + storage_admission_width, K_BUCKET_SIZE, POSSESSION_AUDIT_PROTOCOL_ID, REPLICATION_PROTOCOL_ID, + SUBTREE_AUDIT_PROTOCOL_ID, }; use ant_node::replication::protocol::{ compute_audit_digest, AuditChallenge, AuditResponse, FetchRequest, FetchResponse, @@ -47,6 +48,23 @@ const DUMMY_PAYMENT_PROOF_LEN: usize = 64; /// Dummy proof byte used when a test only needs to reach pre-payment gates. const DUMMY_PAYMENT_PROOF_BYTE: u8 = 0x01; +/// The protocol id a body must be sent on for the receive guard to accept it. +/// +/// Mirrors `body_matches_protocol` in `replication::mod`: subtree-audit bodies +/// ride [`SUBTREE_AUDIT_PROTOCOL_ID`], the digest-based possession pair rides +/// [`POSSESSION_AUDIT_PROTOCOL_ID`], everything else stays on the core id. +/// Deriving it from the body — rather than hardcoding the core id — means a +/// future family move cannot silently turn these tests into request timeouts. +fn protocol_id_for(body: &ReplicationMessageBody) -> &'static str { + if body.is_subtree_audit() { + SUBTREE_AUDIT_PROTOCOL_ID + } else if body.is_possession_audit() { + POSSESSION_AUDIT_PROTOCOL_ID + } else { + REPLICATION_PROTOCOL_ID + } +} + /// Send a replication request via saorsa-core's request-response mechanism /// and decode the response. /// @@ -59,9 +77,10 @@ async fn send_replication_request( msg: ReplicationMessage, timeout: Duration, ) -> ReplicationMessage { + let protocol = protocol_id_for(&msg.body); let encoded = msg.encode().expect("encode replication request"); let response = sender - .send_request(target, REPLICATION_PROTOCOL_ID, encoded, timeout) + .send_request(target, protocol, encoded, timeout) .await .expect("send_request"); ReplicationMessage::decode(&response.data).expect("decode replication response") diff --git a/tests/e2e/testnet.rs b/tests/e2e/testnet.rs index 551cbbbe..a281f5ea 100644 --- a/tests/e2e/testnet.rs +++ b/tests/e2e/testnet.rs @@ -292,6 +292,14 @@ impl TestNetworkConfig { node_count: SMALL_NODE_COUNT, bootstrap_count: DEFAULT_BOOTSTRAP_COUNT, stabilization_timeout: Duration::from_secs(SMALL_STABILIZATION_TIMEOUT_SECS), + // The heavy subtree round-1 responder cooldown defaults to 30 min for + // production rate-limiting; tests audit the same holder repeatedly in + // quick succession, so drop it to near-zero so audits are not + // rate-dropped (a dropped round-1 would surface as a timeout). + replication_config: Some(ReplicationConfig { + subtree_round1_responder_cooldown: Duration::from_millis(1), + ..ReplicationConfig::default() + }), ..Default::default() } } diff --git a/tests/poc_audit_handler_live.rs b/tests/poc_audit_handler_live.rs index 11c857d8..af813c31 100644 --- a/tests/poc_audit_handler_live.rs +++ b/tests/poc_audit_handler_live.rs @@ -25,13 +25,14 @@ use std::sync::Arc; use ant_node::replication::commitment_state::{BuiltCommitment, ResponderCommitmentState}; -use ant_node::replication::config::MAX_BYTE_CHALLENGE_KEYS; +use ant_node::replication::config::MAX_SLICE_OPENINGS; use ant_node::replication::protocol::{ - SubtreeAuditChallenge, SubtreeAuditResponse, SubtreeByteChallenge, SubtreeByteItem, - SubtreeByteResponse, + SubtreeAuditChallenge, SubtreeAuditResponse, SubtreeSliceChallenge, SubtreeSliceItem, + SubtreeSliceOpening, SubtreeSliceResponse, }; +use ant_node::replication::slice::{nonced_block_root, verify_block_slice, verify_nonced_block}; use ant_node::replication::storage_commitment_audit::{ - handle_subtree_byte_challenge, handle_subtree_challenge, + handle_subtree_challenge, handle_subtree_challenge_measured, handle_subtree_slice_challenge, }; use ant_node::replication::subtree::{verify_subtree_proof, StructureVerdict}; use ant_node::storage::{LmdbStorage, LmdbStorageConfig}; @@ -335,48 +336,191 @@ async fn committed_key_with_missing_bytes_is_rejected() { } // --------------------------------------------------------------------------- -// 6. Round 2 (byte challenge): honest serve + oversize-request rejection +// 5b. Partial round-1 work is charged even when the response is a rejection // --------------------------------------------------------------------------- -/// Round-2 happy path: a byte challenge pinned to the responder's retained -/// commitment, for keys it committed to and still stores, returns `Items` with -/// the ORIGINAL bytes (`Present`) for every requested key. +/// A successful proof reports exactly the chunk content it read and hashed. +/// Anchors the rejection case below: it fixes what the measurement means. +#[tokio::test] +async fn round1_proof_reports_the_content_it_read() { + let (storage, _t) = test_storage().await; + let indices: Vec = (1..=64u8).collect(); + let r = Responder::new(&storage, &indices).await; + let challenge = challenge_for(&r, r.current_hash(), [0x11u8; 32]); + + let work = + handle_subtree_challenge_measured(&challenge, &storage, &r.peer_id, false, Some(&r.state)) + .await; + + let SubtreeAuditResponse::Proof { ref proof, .. } = work.response else { + panic!("expected Proof, got {:?}", work.response); + }; + let from_proof: i64 = proof + .leaves + .iter() + .map(|leaf| i64::from(leaf.content_len)) + .sum(); + assert_eq!( + work.content_bytes, from_proof, + "reported work must equal the content the proof covers" + ); + assert!(work.content_bytes > 0, "a real proof reads real bytes"); +} + +/// Regression (dirvine, PR #181): round 1 reads and hashes leaf by leaf, so a +/// commitment holding ONE unreadable key still costs a full run of reads and +/// keyed-BLAKE3 passes over every leaf before it — and then rejects. +/// +/// The responder used to charge its work budget only on the `Proof` arm, so +/// that run was free. An attacker who found such a commitment could replay +/// subtrees over it indefinitely at no cost, and the per-peer cooldown does not +/// help because it is escapable by rotating identity — the responder-wide budget +/// is the only bound that applies to this, so it has to see the work. /// -/// FLIPS IF: the responder stops serving original bytes for committed keys — -/// the auditor would then see byte-verification failures for honest nodes. +/// FLIPS IF: the accounting goes back to charging only successful proofs. #[tokio::test] -async fn byte_challenge_serves_original_bytes_for_committed_keys() { +async fn rejected_round1_still_reports_the_work_it_spent() { let (storage, _t) = test_storage().await; - let r = Responder::new(&storage, &[1, 2, 3, 4]).await; + let indices: Vec = (1..=64u8).collect(); + let r = Responder::new(&storage, &indices).await; let pin = r.current_hash(); - let keys = vec![Responder::address(1), Responder::address(2)]; - let challenge = SubtreeByteChallenge { + // Find a nonce whose selected subtree spans more than one leaf, then delete + // the bytes behind its LAST leaf only. Everything before it is readable, so + // the responder does real work and then hits the hole. + let challenge = challenge_for(&r, pin, [0x11u8; 32]); + let baseline = + handle_subtree_challenge_measured(&challenge, &storage, &r.peer_id, false, Some(&r.state)) + .await; + let SubtreeAuditResponse::Proof { ref proof, .. } = baseline.response else { + panic!("expected a baseline Proof, got {:?}", baseline.response); + }; + assert!( + proof.leaves.len() > 1, + "need a multi-leaf subtree for partial work to exist; got {}", + proof.leaves.len() + ); + let last_leaf = proof.leaves.last().expect("non-empty"); + let (last, deleted_len) = (last_leaf.key, i64::from(last_leaf.content_len)); + let baseline_bytes = baseline.content_bytes; + storage.delete(&last).await.expect("delete last leaf chunk"); + + let work = + handle_subtree_challenge_measured(&challenge, &storage, &r.peer_id, false, Some(&r.state)) + .await; + + match work.response { + SubtreeAuditResponse::Rejected { ref reason, .. } => { + assert!( + reason.contains("missing bytes for committed key"), + "expected the missing-bytes rejection, got: {reason}" + ); + } + ref other => panic!("expected Rejected(missing bytes), got {other:?}"), + } + assert!( + work.content_bytes > 0, + "the leaves read before the hole must be charged, not written off" + ); + // Everything except the deleted leaf was read, so the charge is the + // baseline minus that one leaf. + assert_eq!( + work.content_bytes, + baseline_bytes - deleted_len, + "charge must cover exactly the leaves actually read" + ); +} + +// --------------------------------------------------------------------------- +// 6. Round 2 (slice challenge): honest open + oversize-request rejection +// --------------------------------------------------------------------------- + +/// Round-2 happy path: a slice challenge pinned to the responder's retained +/// commitment, for keys it committed to and still stores, returns `Items` with a +/// `Present` verified opening for every requested key. The opening is fully +/// verified here — the Bao slice decodes against the chunk address to the real +/// block, and the nonced opening folds to the honest nonced root — so this FAILS +/// if the responder produces a malformed or non-possessing proof. +/// +/// FLIPS IF: the responder stops opening valid slices for committed keys — the +/// auditor would then see verification failures for honest nodes. +#[tokio::test] +async fn slice_challenge_opens_valid_blocks_for_committed_keys() { + let (storage, _t) = test_storage().await; + let r = Responder::new(&storage, &[1, 2, 3, 4]).await; + let pin = r.current_hash(); + let nonce = [0x22u8; 32]; + + let openings = vec![ + SubtreeSliceOpening { + key: Responder::address(1), + block_index: 0, + }, + SubtreeSliceOpening { + key: Responder::address(2), + block_index: 0, + }, + ]; + let challenge = SubtreeSliceChallenge { challenge_id: 43, - nonce: [0x22u8; 32], + nonce, challenged_peer_id: r.peer_id_bytes, expected_commitment_hash: pin, - keys: keys.clone(), + openings: openings.clone(), }; let resp = - handle_subtree_byte_challenge(&challenge, &storage, &r.peer_id, false, Some(&r.state)) + handle_subtree_slice_challenge(&challenge, &storage, &r.peer_id, false, Some(&r.state)) .await; match resp { - SubtreeByteResponse::Items { + SubtreeSliceResponse::Items { challenge_id, items, } => { assert_eq!(challenge_id, 43); - assert_eq!(items.len(), keys.len(), "one item per requested key"); - for (item, (i, key)) in items.iter().zip([1u8, 2].into_iter().zip(keys)) { + assert_eq!( + items.len(), + openings.len(), + "one item per requested opening" + ); + for (item, (i, opening)) in items.iter().zip([1u8, 2].into_iter().zip(openings)) { match item { - SubtreeByteItem::Present { key: k, bytes } => { - assert_eq!(*k, key); - assert_eq!(*bytes, chunk_content(i), "must serve the ORIGINAL bytes"); + SubtreeSliceItem::Present { + key: k, + block_index, + bao_slice, + nonced_siblings, + } => { + assert_eq!(*k, opening.key); + assert_eq!(*block_index, 0); + let content = chunk_content(i); + let bytes_hash = *blake3::hash(&content).as_bytes(); + // Chain 1: the Bao slice decodes against the address to + // the real block (a 1024-byte chunk is a single block). + let block = + verify_block_slice(bao_slice, &bytes_hash, content.len() as u64, 0) + .expect("bao slice must verify against the chunk address"); + assert_eq!(block, content, "opened block must be the ORIGINAL bytes"); + // Chain 2: the nonced opening folds to the honest nonced + // root over the real content under this audit's nonce. + let root = + nonced_block_root(&nonce, &r.peer_id_bytes, &opening.key, &content); + assert!( + verify_nonced_block( + &nonce, + &r.peer_id_bytes, + &opening.key, + 0, + &block, + nonced_siblings, + &root, + ant_node::replication::slice::block_count(content.len() as u64), + ), + "nonced opening must fold to the honest nonced root" + ); } - other @ SubtreeByteItem::Absent { .. } => { + other @ SubtreeSliceItem::Absent { .. } => { panic!("expected Present for stored committed key, got {other:?}") } } @@ -386,43 +530,291 @@ async fn byte_challenge_serves_original_bytes_for_committed_keys() { } } -/// A byte challenge requesting more than `MAX_BYTE_CHALLENGE_KEYS` keys is -/// rejected up front: an honest auditor batches its sample to that cap so the -/// worst-case response (all chunks at max size) fits the replication wire cap; -/// anything larger is a forged/over-size request the responder must not try to -/// serve (the response could not encode, and reading the chunks first would be -/// disk-read amplification). +/// Coalescing at the live responder boundary: a challenge carrying DUPLICATE and +/// interleaved openings for the same `(key, block_index)` returns one `Present` +/// item per DISTINCT opening, not one per raw opening. This is the contract that +/// lets `serve_committed_key_openings` (via `ChunkOpener`) read and hash each +/// committed chunk once instead of re-reading it per repeated opening, so a +/// forged auditor cannot amplify responder disk/CPU work by repeating openings +/// while staying under the total-openings cap. /// -/// FLIPS IF: the per-challenge key cap is removed from the responder. +/// FLIPS IF: the responder stops deduplicating openings (item count would grow +/// with the raw request) or drops/duplicates a distinct identity. #[tokio::test] -async fn oversize_byte_challenge_is_rejected() { +async fn slice_challenge_coalesces_duplicate_and_interleaved_openings() { let (storage, _t) = test_storage().await; let r = Responder::new(&storage, &[1, 2, 3, 4]).await; let pin = r.current_hash(); + let nonce = [0x55u8; 32]; + + let k1 = Responder::address(1); + let k2 = Responder::address(2); + // Five raw openings, interleaved, over only TWO distinct (key, block) pairs. + let openings = vec![ + SubtreeSliceOpening { + key: k1, + block_index: 0, + }, + SubtreeSliceOpening { + key: k2, + block_index: 0, + }, + SubtreeSliceOpening { + key: k1, + block_index: 0, + }, + SubtreeSliceOpening { + key: k2, + block_index: 0, + }, + SubtreeSliceOpening { + key: k1, + block_index: 0, + }, + ]; + assert!(openings.len() <= MAX_SLICE_OPENINGS); + let challenge = SubtreeSliceChallenge { + challenge_id: 46, + nonce, + challenged_peer_id: r.peer_id_bytes, + expected_commitment_hash: pin, + openings, + }; + + let resp = + handle_subtree_slice_challenge(&challenge, &storage, &r.peer_id, false, Some(&r.state)) + .await; - let keys: Vec<[u8; 32]> = (0..=MAX_BYTE_CHALLENGE_KEYS) - .map(|i| [u8::try_from(i % 251).unwrap_or(0); 32]) + match resp { + SubtreeSliceResponse::Items { + challenge_id, + items, + } => { + assert_eq!(challenge_id, 46); + // Coalesced: one item per DISTINCT (key, block_index), not per raw opening. + assert_eq!( + items.len(), + 2, + "duplicate openings must not multiply response items" + ); + let mut seen: Vec<([u8; 32], u32)> = Vec::new(); + for item in &items { + match item { + SubtreeSliceItem::Present { + key, block_index, .. + } => { + assert!( + !seen.contains(&(*key, *block_index)), + "each (key, block) identity must appear at most once" + ); + seen.push((*key, *block_index)); + } + other @ SubtreeSliceItem::Absent { .. } => { + panic!("expected Present for a stored committed key, got {other:?}") + } + } + } + seen.sort_unstable(); + let mut want = vec![(k1, 0u32), (k2, 0u32)]; + want.sort_unstable(); + assert_eq!( + seen, want, + "both distinct openings must be served exactly once, order-independent" + ); + } + other => panic!("expected Items, got {other:?}"), + } +} + +/// Multi-block coverage: open a DEEP block of a genuinely large (multi-block) +/// chunk end-to-end through the live handler. The single-block happy-path test +/// above cannot exercise the Bao parent-hash chain or the multi-level nonced +/// tree; this one does — it stores a ~100 KB chunk (98 blocks), opens block 50, +/// and verifies both chains against that exact block. +/// +/// FLIPS IF: the Bao slice or nonced opening for a non-trivial tree is malformed +/// (e.g. wrong offset, wrong sibling ordering). +#[tokio::test] +async fn slice_challenge_opens_a_deep_block_of_a_large_chunk() { + use ant_node::replication::commitment::commitment_hash; + use ant_node::replication::slice::block_count; + + let (storage, _t) = test_storage().await; + + // One large chunk: ~100 KB => 98 one-KiB blocks. + let content: Vec = (0..100_000u32) + .map(|n| (n.wrapping_mul(2_654_435_761) >> 13) as u8) .collect(); - assert!(keys.len() > MAX_BYTE_CHALLENGE_KEYS); - let challenge = SubtreeByteChallenge { + let addr = LmdbStorage::compute_address(&content); + storage.put(&addr, &content).await.expect("put chunk"); + let bytes_hash = *blake3::hash(&content).as_bytes(); + + let (pk, sk) = keypair(); + let peer_id_bytes = *blake3::hash(&pk.to_bytes()).as_bytes(); + let peer_id = PeerId::from_bytes(peer_id_bytes); + let built = BuiltCommitment::build( + vec![(addr, bytes_hash)], + &peer_id_bytes, + &sk, + &pk.to_bytes(), + ) + .expect("build"); + let pin = commitment_hash(built.commitment()).unwrap(); + let state = Arc::new(ResponderCommitmentState::new()); + state.rotate(built); + + let count = block_count(content.len() as u64); + assert!( + count > 64, + "test needs a genuinely multi-block chunk, got {count} blocks" + ); + let block_index = 50u32; + let nonce = [0x5Au8; 32]; + let challenge = SubtreeSliceChallenge { + challenge_id: 77, + nonce, + challenged_peer_id: peer_id_bytes, + expected_commitment_hash: pin, + openings: vec![SubtreeSliceOpening { + key: addr, + block_index, + }], + }; + + let resp = + handle_subtree_slice_challenge(&challenge, &storage, &peer_id, false, Some(&state)).await; + + let SubtreeSliceResponse::Items { items, .. } = resp else { + panic!("expected Items, got {resp:?}"); + }; + let Some(SubtreeSliceItem::Present { + block_index: bi, + bao_slice, + nonced_siblings, + .. + }) = items.first() + else { + panic!("expected a single Present opening, got {items:?}"); + }; + assert_eq!(*bi, block_index); + + // Chain 1: the Bao slice decodes against the address to exactly block 50. + let block = verify_block_slice(bao_slice, &bytes_hash, content.len() as u64, block_index) + .expect("deep-block bao slice must verify"); + let start = block_index as usize * 1024; + assert_eq!( + block, + content[start..start + 1024], + "opened block must be block 50's bytes" + ); + + // Chain 2: the nonced opening folds up the multi-level tree to the honest root. + let root = nonced_block_root(&nonce, &peer_id_bytes, &addr, &content); + assert!( + verify_nonced_block( + &nonce, + &peer_id_bytes, + &addr, + block_index, + &block, + nonced_siblings, + &root, + ant_node::replication::slice::block_count(content.len() as u64) + ), + "deep-block nonced opening must fold to the honest root" + ); + // And the proof is genuinely small — a slice, not the whole 100 KB chunk. + assert!( + bao_slice.len() < content.len() / 4, + "the slice ({} bytes) must be far smaller than the chunk ({} bytes)", + bao_slice.len(), + content.len() + ); +} + +/// A slice challenge requesting more than `MAX_SLICE_OPENINGS` openings is +/// rejected up front: an honest auditor opens at most that many blocks; anything +/// larger is a forged/over-size request the responder must not try to serve +/// (each opening forces a full chunk read to build its proof — disk-read +/// amplification). +/// +/// FLIPS IF: the per-challenge openings cap is removed from the responder. +#[tokio::test] +async fn oversize_slice_challenge_is_rejected() { + let (storage, _t) = test_storage().await; + let r = Responder::new(&storage, &[1, 2, 3, 4]).await; + let pin = r.current_hash(); + + let openings: Vec = (0..=MAX_SLICE_OPENINGS) + .map(|i| SubtreeSliceOpening { + key: [u8::try_from(i % 251).unwrap_or(0); 32], + block_index: 0, + }) + .collect(); + assert!(openings.len() > MAX_SLICE_OPENINGS); + let challenge = SubtreeSliceChallenge { challenge_id: 44, nonce: [0x33u8; 32], challenged_peer_id: r.peer_id_bytes, expected_commitment_hash: pin, - keys, + openings, }; let resp = - handle_subtree_byte_challenge(&challenge, &storage, &r.peer_id, false, Some(&r.state)) + handle_subtree_slice_challenge(&challenge, &storage, &r.peer_id, false, Some(&r.state)) .await; match resp { - SubtreeByteResponse::Rejected { reason, .. } => { + SubtreeSliceResponse::Rejected { reason, .. } => { assert!( reason.contains("max"), - "expected per-challenge key-cap rejection, got: {reason}" + "expected per-challenge openings-cap rejection, got: {reason}" ); } other => panic!("expected Rejected(oversize), got {other:?}"), } } + +/// A slice challenge spanning more DISTINCT keys than the honest spot-check sample +/// is rejected even when it stays under the total-openings cap. Coalescing saves +/// work only when keys repeat, so a forged auditor could otherwise force a full +/// chunk read per distinct key (up to ~40 MiB) without exceeding `MAX_SLICE_OPENINGS`. +/// +/// FLIPS IF: the distinct-key cap is removed from the responder. +#[tokio::test] +async fn slice_challenge_with_too_many_distinct_keys_is_rejected() { + let (storage, _t) = test_storage().await; + let r = Responder::new(&storage, &[1, 2, 3, 4]).await; + let pin = r.current_hash(); + + // 6 distinct keys × 1 opening = 6 openings: under MAX_SLICE_OPENINGS (10) but + // over the honest BYTE_SPOTCHECK_MAX (5) distinct-key sample. + let openings: Vec = (0..6u8) + .map(|i| SubtreeSliceOpening { + key: [i + 1; 32], + block_index: 0, + }) + .collect(); + assert!(openings.len() <= MAX_SLICE_OPENINGS); + let challenge = SubtreeSliceChallenge { + challenge_id: 45, + nonce: [0x44u8; 32], + challenged_peer_id: r.peer_id_bytes, + expected_commitment_hash: pin, + openings, + }; + + let resp = + handle_subtree_slice_challenge(&challenge, &storage, &r.peer_id, false, Some(&r.state)) + .await; + + match resp { + SubtreeSliceResponse::Rejected { reason, .. } => { + assert!( + reason.contains("distinct keys"), + "expected distinct-key rejection, got: {reason}" + ); + } + other => panic!("expected Rejected(distinct keys), got {other:?}"), + } +} diff --git a/tests/poc_commitment_audit_attacks.rs b/tests/poc_commitment_audit_attacks.rs index bc67bc77..903af243 100644 --- a/tests/poc_commitment_audit_attacks.rs +++ b/tests/poc_commitment_audit_attacks.rs @@ -13,17 +13,17 @@ //! The production auditor's `verify_subtree_response` (in //! `src/replication/storage_commitment_audit.rs`) is private, so this file //! reproduces the exact ordered gates it runs — pin, peer-id binding, -//! signature, structural [`verify_subtree_proof`], then the **round-2 byte -//! challenge**: the auditor demands the ORIGINAL chunk bytes for a -//! nonce-selected sample of the just-proven leaves FROM THE RESPONDER and -//! verifies the served content against each leaf's committed `bytes_hash` -//! (content address) and `nonced_hash` (freshness). Possession is -//! non-delegable: the auditor needs to hold NONE of the responder's chunks, -//! and a committed key the responder cannot serve is a deterministic, -//! confirmed failure (`DigestMismatch` in production — never inconclusive, -//! never graced). The helper [`auditor_accepts`] runs these gates in the same -//! order with the same failure semantics, so a reviewer can see each attack -//! is caught at the same gate the network code would catch it. +//! signature, structural [`verify_subtree_proof`], then the **round-2 slice +//! challenge** (V2-685): the auditor opens one 1 KiB block of a fresh-random +//! sample of the just-proven leaves FROM THE RESPONDER and verifies each opened +//! block against the leaf's committed `bytes_hash` (a Bao verified slice against +//! the chunk address) and `nonced_root` (a nonced block-tree opening proving +//! round-1 possession). Possession is non-delegable: the auditor needs to hold +//! NONE of the responder's chunks, and a committed key the responder cannot +//! serve is a deterministic, confirmed failure (`DigestMismatch` in production — +//! never inconclusive, never graced). The helper [`auditor_accepts`] runs these +//! gates in the same order with the same failure semantics, so a reviewer can +//! see each attack is caught at the same gate the network code would catch it. //! //! ## What changed from the old per-key audit (and why) //! @@ -68,9 +68,13 @@ use ant_node::replication::commitment::{ }; use ant_node::replication::commitment_state::{BuiltCommitment, ResponderCommitmentState}; use ant_node::replication::config::AUDIT_SPOTCHECK_COUNT; +use ant_node::replication::slice::{ + extract_block_slice, nonced_block_root, nonced_block_siblings, verify_block_slice, + verify_nonced_block, +}; use ant_node::replication::subtree::{ - build_subtree_proof, nonced_leaf_hash, select_spotcheck_indices, select_subtree_path, - verify_subtree_proof, StructureVerdict, SubtreeProof, + build_subtree_proof, select_spotcheck_indices, select_subtree_path, verify_subtree_proof, + StructureVerdict, SubtreeLeaf, SubtreeProof, }; use rand::Rng; use saorsa_pqc::api::sig::{ml_dsa_65, MlDsaPublicKey, MlDsaSecretKey}; @@ -83,12 +87,12 @@ fn keypair() -> (MlDsaPublicKey, MlDsaSecretKey) { ml_dsa_65().generate_keypair().unwrap() } -/// Deterministic chunk bytes for key index `i`. The committed tree is built -/// from `BLAKE3(content(i))`, so an honest proof — which hashes the same bytes — -/// reconstructs the committed root and passes the real-bytes spot-check. +/// Deterministic chunk bytes for fixture index `i`. Keys are CONTENT-ADDRESSED +/// (`key(i) == BLAKE3(content(i)) == content_hash(i)`), so a committed leaf is +/// `(key, key)` exactly as production, and an honest proof — which hashes the +/// same bytes — reconstructs the committed root and passes the real-bytes check. fn content(i: u32) -> Vec { - let mut v = key(i).to_vec(); - v.extend_from_slice(b"subtree-audit-chunk-body"); + let mut v = b"subtree-audit-chunk-body".to_vec(); v.extend_from_slice(&i.to_le_bytes()); v } @@ -97,12 +101,12 @@ fn content_hash(i: u32) -> [u8; 32] { *blake3::hash(&content(i)).as_bytes() } -/// Big-endian key so numeric order matches the MerkleTree sort order; this lets -/// us reason about leaf positions when we tamper with them. +/// The content-addressed key for index `i`: `key(i) == content_hash(i)`, so +/// `bytes_hash == key` on every committed leaf. Leaf positions in the committed +/// tree therefore follow the hash order, not `i`; the attack tests below read the +/// actual key at each position via `key_at` rather than assuming `key(i)`. fn key(i: u32) -> [u8; 32] { - let mut k = [0u8; 32]; - k[..4].copy_from_slice(&i.to_be_bytes()); - k + content_hash(i) } /// A responder identity (real ML-DSA keypair) plus its retention state. Peer @@ -145,7 +149,7 @@ impl Responder { } /// Bytes source for an HONEST responder: it really holds every chunk it -/// committed to, so it can always produce a correct `nonced_hash`. +/// committed to, so it can always open a real slice + nonced block opening. fn honest_bytes(k: &[u8; 32]) -> Option> { for i in 0..4096u32 { if &key(i) == k { @@ -158,13 +162,15 @@ fn honest_bytes(k: &[u8; 32]) -> Option> { /// The auditor's full ordered verification, mirroring the production /// `verify_subtree_response` gates. Returns `Ok(byte_checked_count)` on accept. /// -/// `responder_serves(k)` models round 2 (`SubtreeByteChallenge`): what the -/// RESPONDER returns when the auditor demands the original bytes of sampled -/// leaf `k`. `Some(bytes)` is a `SubtreeByteItem::Present`; `None` is an -/// explicit `Absent` or an omitted key — a committed key the responder will -/// not serve, which production `verify_byte_response` counts as a confirmed -/// `DigestMismatch`. The auditor verifies the SERVED content, so it needs to -/// hold none of the responder's chunks and no inconclusive lane exists. +/// `responder_serves(k)` models round 2 (`SubtreeSliceChallenge`): the chunk +/// content the RESPONDER can produce for sampled leaf `k`, from which it builds a +/// Bao verified slice + nonced block opening exactly as +/// `handle_subtree_slice_challenge` does. `Some(bytes)` is a +/// `SubtreeSliceItem::Present`; `None` is an explicit `Absent` or an omitted key +/// — a committed key the responder will not serve, which production +/// `verify_slice_response` counts as a confirmed `DigestMismatch`. The auditor +/// verifies the served block against the committed address and nonced root, so it +/// needs to hold none of the responder's chunks and no inconclusive lane exists. fn auditor_accepts( challenged_peer_id: &[u8; 32], expected_commitment_hash: &[u8; 32], @@ -194,13 +200,21 @@ fn auditor_accepts( return Err(AuditError::StructureInvalid(why)); } - // -- Gate: round-2 byte challenge (responder-served possession) ---------- + // -- Gate: content-address binding (possession-forgery guard) ----------- + // Mirrors production `evaluate_subtree_structure`: a leaf whose `bytes_hash` + // is decoupled from its credited `key` is rejected at round 1 (content- + // addressed chunks always have `bytes_hash == key`). + if proof.leaves.iter().any(|l| l.bytes_hash != l.key) { + return Err(AuditError::StructureInvalid("leaf bytes_hash != key")); + } + + // -- Gate: round-2 slice challenge (responder-served possession) ---------- // Mirrors `verify_subtree_response` round 2: the sample is chosen with FRESH // randomness over the RECEIVED proof leaves (NOT nonce-derived), AFTER round - // 1, so the responder cannot predict which leaves will be opened (§1 - // cut-and-choose soundness). EVERY sampled leaf must verify from the bytes - // the responder serves. There is no skip and no inconclusive lane: a - // committed key the responder cannot serve is a provable lie. + // 1, so the responder cannot predict which leaves — or which block within — + // will be opened (§1 cut-and-choose soundness). EVERY sampled leaf must + // verify from the slice the responder serves. There is no skip and no + // inconclusive lane: a committed key the responder cannot serve is a lie. let spot = random_sample_indices( proof.leaves.len(), AUDIT_SPOTCHECK_COUNT.clamp(3, 5) as usize, @@ -221,9 +235,40 @@ fn auditor_accepts( // maps this to `DigestMismatch`), NOT a skip. return Err(AuditError::CommittedKeyUnserved); }; - let plain = *blake3::hash(&bytes).as_bytes(); - let nonced = nonced_leaf_hash(nonce, &commitment.sender_peer_id, &leaf.key, &bytes); - if leaf.bytes_hash != plain || leaf.nonced_hash != nonced { + // The fixtures use short (single-block) chunks, so the only block is 0; + // production draws a fresh random index. The responder builds its slice + // and nonced opening from the content it serves. + let block_index = 0u32; + let bao_slice = extract_block_slice(&bytes, block_index).unwrap(); + let nonced_siblings = nonced_block_siblings( + nonce, + &commitment.sender_peer_id, + &leaf.key, + &bytes, + block_index, + ) + .unwrap_or_default(); + + // Auditor chain 1: authenticate the block against the committed address. + let Some(block) = verify_block_slice( + &bao_slice, + &leaf.bytes_hash, + u64::from(leaf.content_len), + block_index, + ) else { + return Err(AuditError::RealBytesMismatch); + }; + // Auditor chain 2: fold the nonced opening to the committed nonced_root. + if !verify_nonced_block( + nonce, + &commitment.sender_peer_id, + &leaf.key, + block_index, + &block, + &nonced_siblings, + &leaf.nonced_root, + ant_node::replication::slice::block_count(u64::from(leaf.content_len)), + ) { return Err(AuditError::RealBytesMismatch); } checked += 1; @@ -255,8 +300,9 @@ enum AuditError { CommitmentHashMismatch, SignatureInvalid, StructureInvalid(&'static str), - /// Round 2: the responder served content that does not hash to the - /// committed address / freshness hash (production: `DigestMismatch`). + /// Round 2: the responder's opened block failed a chain — the Bao slice did + /// not decode against the committed address, or the nonced opening did not + /// fold to the committed `nonced_root` (production: `DigestMismatch`). RealBytesMismatch, /// Round 2: the responder would not serve a committed, sampled key /// (production: `DigestMismatch` — a deterministic, confirmed failure). @@ -310,15 +356,15 @@ fn honest_responder_passes_audit() { /// leaf's `bytes_hash` (that value IS the chunk's network address, which is /// public), but it DROPPED the actual bytes. It fabricates a proof: correct /// `key` and correct `bytes_hash` for every selected leaf (so the structural -/// root rebuild passes), but it cannot compute the `nonced_hash`, which requires -/// the real bytes under a fresh nonce. It fills in a forged `nonced_hash`. +/// root rebuild passes), but it cannot compute the `nonced_root`, which requires +/// the real bytes under a fresh nonce. It fills in a forged `nonced_root`. /// /// The structural gate PASSES (addresses alone rebuild the root), proving that /// structure is NOT sufficient — exactly the storage-binding hole. Round 2 is what -/// catches it: the auditor demands the original bytes FROM THE RELAY, and the -/// relay has nothing to serve. Refusing/omitting a sampled committed key is a -/// confirmed failure, and serving fabricated bytes cannot hash to the -/// committed content address (a preimage break) — both lanes are asserted. +/// catches it: the auditor opens a block FROM THE RELAY, and the relay has nothing +/// to serve. Refusing/omitting a sampled committed key is a confirmed failure, and +/// serving a fabricated block cannot decode against the committed content address +/// (a preimage break) — both lanes are asserted. #[test] fn relay_holding_only_addresses_caught_by_real_bytes_check() { let nonce = [0x77; 32]; @@ -328,18 +374,21 @@ fn relay_holding_only_addresses_caught_by_real_bytes_check() { // The lazy node fabricates the proof from PUBLIC data only: it knows each // leaf key and its bytes_hash (== address), but NOT the bytes, so it forges - // every nonced_hash. + // every nonced_root. let path = select_subtree_path(&nonce, built.commitment().key_count).unwrap(); let mut leaves = Vec::new(); for idx in path.leaf_start..path.leaf_end { let k = built.tree().key_at(idx as usize).unwrap(); - // bytes_hash is public (== the chunk address); the responder fakes the - // possession hash because it lacks the bytes. - let forged_nonced = *blake3::hash(b"i-do-not-have-the-bytes").as_bytes(); - leaves.push(ant_node::replication::subtree::SubtreeLeaf { + let c = honest_bytes(&k).unwrap(); + // For a content-addressed chunk bytes_hash == key (both public), so the + // relay can present a structurally valid leaf; it fakes the possession + // commitment because it lacks the bytes. + let forged_nonced_root = *blake3::hash(b"i-do-not-have-the-bytes").as_bytes(); + leaves.push(SubtreeLeaf { key: k, - bytes_hash: content_hash(idx), - nonced_hash: forged_nonced, + bytes_hash: k, + content_len: u32::try_from(c.len()).unwrap(), + nonced_root: forged_nonced_root, }); } // Real sibling cut-hashes from the committed tree (public, derivable). @@ -394,17 +443,17 @@ fn relay_holding_only_addresses_caught_by_real_bytes_check() { /// the relay attack, and the one the §1 review found: a relay holds only public /// addresses, but it knows the round-1 nonce, so under the OLD nonce-derived /// sampling it could compute EXACTLY which 3..=5 leaves round 2 would open, -/// fetch only those few chunks from neighbours, fill in correct `nonced_hash` -/// for them, and fabricate `nonced_hash` for every other leaf — passing the +/// fetch only those few chunks from neighbours, commit a correct `nonced_root` +/// for them, and fabricate `nonced_root` for every other leaf — passing the /// audit while holding almost nothing. /// /// With the fix, the auditor draws the sample with fresh randomness AFTER the /// proof is committed, so the relay's bet on the nonce-derived indices is -/// uncorrelated with what actually gets opened. We model the relay holding the -/// nonce-derived prediction set and nothing else: the random sample lands on a -/// leaf the relay did NOT fetch with overwhelming probability, and the audit -/// fails. Repeated across many nonces to make the probabilistic catch a -/// near-certainty in aggregate. +/// uncorrelated with what actually gets opened. We model the relay committing a +/// CORRECT `nonced_root` (and holding the bytes) only for the nonce-derived +/// prediction set and forging the rest: the random sample lands on a leaf the +/// relay did NOT prepare with overwhelming probability, and the audit fails. +/// Repeated across many nonces to make the probabilistic catch a near-certainty. #[test] fn predict_and_fetch_relay_is_caught_by_fresh_random_sample() { let r = Responder::new(); @@ -418,17 +467,34 @@ fn predict_and_fetch_relay_is_caught_by_fresh_random_sample() { let mut nonce = [0u8; 32]; nonce[..4].copy_from_slice(&t.to_le_bytes()); - // The relay builds a structurally-valid proof from PUBLIC data, forging - // every leaf's nonced_hash (it holds no bytes). let plan = ant_node::replication::subtree::subtree_plan(built.tree(), &nonce).unwrap(); let path = select_subtree_path(&nonce, n).unwrap(); + // The relay PREDICTS the old nonce-derived sample: those are the only + // leaves it fetches and can commit a correct nonced_root for. + let predicted_local: std::collections::HashSet = + select_spotcheck_indices(&nonce, &path, AUDIT_SPOTCHECK_COUNT.clamp(3, 5)) + .into_iter() + .collect(); + + // Build a structurally-valid proof: correct `nonced_root` for predicted + // leaves (it holds those bytes), forged for every other leaf. let mut leaves = Vec::new(); - for idx in path.leaf_start..path.leaf_end { + let mut predicted_keys = std::collections::HashSet::new(); + for (local, idx) in (path.leaf_start..path.leaf_end).enumerate() { let k = built.tree().key_at(idx as usize).unwrap(); - leaves.push(ant_node::replication::subtree::SubtreeLeaf { + let c = honest_bytes(&k).unwrap(); + let is_predicted = predicted_local.contains(&u32::try_from(local).unwrap()); + let nonced_root = if is_predicted { + predicted_keys.insert(k); + nonced_block_root(&nonce, &r.peer_id_bytes, &k, &c) + } else { + *blake3::hash(b"forged").as_bytes() + }; + leaves.push(SubtreeLeaf { key: k, - bytes_hash: content_hash(idx), - nonced_hash: *blake3::hash(b"forged").as_bytes(), + bytes_hash: k, + content_len: u32::try_from(c.len()).unwrap(), + nonced_root, }); } let forged = SubtreeProof { @@ -436,14 +502,6 @@ fn predict_and_fetch_relay_is_caught_by_fresh_random_sample() { sibling_cut_hashes: plan.sibling_cut_hashes, }; - // The relay PREDICTS the old nonce-derived sample and fetches exactly - // those chunks (correct bytes for them only). - let predicted: std::collections::HashSet<[u8; 32]> = - select_spotcheck_indices(&nonce, &path, AUDIT_SPOTCHECK_COUNT.clamp(3, 5)) - .into_iter() - .filter_map(|i| forged.leaves.get(i as usize).map(|l| l.key)) - .collect(); - // Responder serves real bytes ONLY for the predicted set; everything // else it cannot serve (it holds no other bytes). let res = auditor_accepts( @@ -453,9 +511,7 @@ fn predict_and_fetch_relay_is_caught_by_fresh_random_sample() { built.commitment(), &forged, |k| { - // The relay can only serve bytes for the chunks it fetched (the - // predicted set); for those it returns the real content. - if predicted.contains(k) { + if predicted_keys.contains(k) { (0..n).find(|&i| &key(i) == k).map(content) } else { None @@ -483,7 +539,7 @@ fn predict_and_fetch_relay_is_caught_by_fresh_random_sample() { /// the auditor now picks the sample with FRESH randomness after the proof is in /// hand (§1), the attacker cannot aim its forgeries away from the sample. We /// model the worst case for the attacker — every leaf's freshness forged — so -/// any random sample is fatal; round 2 re-derives the freshness hash from the +/// any random sample is fatal; round 2 re-derives the nonced opening from the /// served bytes and exposes it. #[test] fn fabricated_fraction_is_caught_when_a_forged_leaf_is_sampled() { @@ -492,14 +548,14 @@ fn fabricated_fraction_is_caught_when_a_forged_leaf_is_sampled() { let pin = r.commit_to_range(400); let (mut proof, commitment) = honest_proof_and_commitment(&r, &nonce); - // Forge every leaf's nonced hash. Under fresh-random sampling the auditor + // Forge every leaf's nonced root. Under fresh-random sampling the auditor // is guaranteed to open a forged leaf, so the audit must fail. for leaf in &mut proof.leaves { - leaf.nonced_hash[0] ^= 0xFF; + leaf.nonced_root[0] ^= 0xFF; } - // Even if the responder serves the REAL bytes in round 2, the freshness - // hash recomputed from that served content exposes the forgery. + // Even if the responder serves the REAL bytes in round 2, the nonced opening + // recomputed from that served content cannot fold to the forged root. let res = auditor_accepts( &r.peer_id_bytes, &pin, @@ -511,7 +567,7 @@ fn fabricated_fraction_is_caught_when_a_forged_leaf_is_sampled() { assert_eq!( res, Err(AuditError::RealBytesMismatch), - "a forged leaf landing under the byte challenge must fail, got {res:?}" + "a forged leaf landing under the slice challenge must fail, got {res:?}" ); } @@ -990,26 +1046,26 @@ fn signature_round_trips_correctly() { assert!(!verify_commitment_signature(&c2)); } -/// The per-leaf possession hash binds nonce, peer, key, and bytes — the -/// foundation of the real-bytes spot-check. Changing any input changes it, so a -/// responder cannot reuse a possession hash across nonces/peers/keys/chunks. +/// The per-leaf nonced block-tree root binds nonce, peer, key, and bytes — the +/// foundation of the possession opening. Changing any input changes it, so a +/// responder cannot reuse a nonced root across nonces/peers/keys/chunks. #[test] -fn nonced_leaf_hash_binds_all_inputs() { - let base = nonced_leaf_hash(&[1; 32], &[2; 32], &key(3), b"chunk"); +fn nonced_block_root_binds_all_inputs() { + let base = nonced_block_root(&[1; 32], &[2; 32], &key(3), b"chunk"); assert_ne!( base, - nonced_leaf_hash(&[9; 32], &[2; 32], &key(3), b"chunk") + nonced_block_root(&[9; 32], &[2; 32], &key(3), b"chunk") ); assert_ne!( base, - nonced_leaf_hash(&[1; 32], &[9; 32], &key(3), b"chunk") + nonced_block_root(&[1; 32], &[9; 32], &key(3), b"chunk") ); assert_ne!( base, - nonced_leaf_hash(&[1; 32], &[2; 32], &key(9), b"chunk") + nonced_block_root(&[1; 32], &[2; 32], &key(9), b"chunk") ); assert_ne!( base, - nonced_leaf_hash(&[1; 32], &[2; 32], &key(3), b"other") + nonced_block_root(&[1; 32], &[2; 32], &key(3), b"other") ); }